Wednesday, February 10, 2016

Android 6 semi-adopted storage

How to split your SD card between adopted internal storage and portable external storage.

(see how-to instructions below)

The pain


Fed up of waiting on Motorola for the Marshmallow upgrade on my XT1072, and being short of internal storage, and being very fed up of having to move-to-sd apps after upgrade (along with occasional tricks such as deleting all data for drive, google+, chrome to scavenge extra memory when clear-cache wasn't enough) I decided to upgrade to CyanogenMod 13 to get Marshmallow that way.

I was really looking forward to being able to use my SD card as internal memory and have no more problems about storage.

What really happens is that with Marshmallow you cannot move-to-sd without adopting your SD card as internal. And then the only apps that will move-to-sd (now called Change) are those that could already move-to-sd.

And one of those apps that won't move-to-sd (even after the adoption of the SD card as internal storage) is Google Music.

Only now there is no SD card for Google Music to store the music on. So it stores it on the *internal* memory.

Whaaaat! My music collection is way bigger than my app collection, how does storing my music internally instead of some apps (all of which could move to SD anyway) help anything?

After messing abuot with Links2SD, Apps2SD, root shells and mount points, reading about volume manager and changing API's I decided that even loopback file fat32 systems in the adopted storage probably wouldn't work.

So I looked at re-partitioning an adopted storage card to shrink the adopted partition to make room for a fat32 partition.

In searching how to access the encrypted partition outside of Android (so that I could resize the file system within it) I came across these notes http://nelenkov.blogspot.co.uk/2015/06/decrypting-android-m-adopted-storage.html I came across the sm command which can create a mixed or public or private volume.

Mixed turned out to be exactly what I was looking for!



How to split your card


I found how to partition my SD card to give 8G as internal storage to which all apps that can be moved will be moved and leave ~20GB as portable storage to hold music, etc.

First, you need adb working, and your SD card inserted and formatted as portable.

$ adb shell sm list-disks adoptable
disk:179_64

disk:168_64 is my SD card that can be made adoptable, I want 75% as external SD:

$ adb shell sm partition disk:179_64 mixed 75


After erasing the entire SD card, it gives me 75% as portable storage and the rest as adopted internal storage.

However I advise a reboot after setting the new music storage location this as Google Music may get the wrong idea about much space is available.

Also try combing that with:

$ adb shell pm set-install-location 2

to have apps installed on the storage by default where possible. It is very effective. (Location 1 means internal, and 0 means auto-choose, but I don't know on what criteria).

Monday, February 1, 2016

Using flock in bash without invoking a subshell

flock -c can call external commands but not bash functions. Consequently users mess about with file descriptors, and often making a mess of it.

Inspired by Ivan's post http://blog.famzah.net/2013/07/31/using-flock-in-bash-without-invoking-a-subshell I've written a flock wrapper for bash, that uses flock underneath but allows flock -c to work for bash functions.

It can be called just like the regular flock command, with the benefit that the -c invocation is supported for bash functions; so you can use it like this:

flock -o /tmp/process my_thing "$@"

and I strongly recommend the -o option so that the file descriptor used for the lock is not passed to any sub-processes, which could be problematic if long live sub-processes (e.g. re-spawned daemons) keep it open.

It will pass through and invoke the regular flock if the command isn't a bash function, or if you aren't trying to execute a function.

Sadly, it doesn't recognize bash built-in's.

But the good news is, you can use flock -o ...file... on a shell function inside your shell script without having to worry about file descriptors.

# Helper function (in order to preserve $@ in the caller)
# If this isn't used to call a shell function then return 0
# otherwise return $? as the argumment number which represents
# the command/function to be called
_is_standard_flock() {
  local args=$#
  # find the first argument that doesn't begin with a -
  while test $# != 0
  do case "$1" in
     -*) shift ; continue ;;
     *) break ;;
     esac
  done

  # if it is numeric and there are not additional arguments 
  test $# = 1 -a -n "$1" -a -z "${1//[0-9]}" && return 0
  shift
  # (skipping -c if present)
  # or the following argument is also not a shell function then use the original flock
  if test "$1" = "-c"
  then declare -F "$2" >/dev/null || return 0
  else declare -F "$1" >/dev/null || return 0
  fi

  # we can't have shifted many args if this is a legitimate use of flock
  # so we will be in range of the exit code
  return $(( args - $# + 1))
}

# Help function to determine if -o or --close was given in the flock arguments
_wants_close() {
  test "${*/#--close/}" != "$*" && return # will also match bogus arguments like --closed
  # remove any -- options
  set -- "${@/#--*/}"
  # look for options with o
  test "${*/#-*o/}" != "$*" && return
  return 1
}

flock() {
  if _is_standard_flock "$@"
  then : # do outside the if-clause so bash can optimise exec where possible
  else # save the exit code (offset) as $1
       set -- $? "$@"
       # ${!$1} is the lock file 
       # ${@:$(($1 + 1))} might be -c
       test "${@:$(($1 + 1)):1}" = "-c" && set -- "${@:1:$1}" "${@:$(($1 + 2))}"
       if _wants_close "${*:2:$(( $1 - 1))}"
       then { set -- "$1" "$_" "${@:2}" ; command flock "${@:3:$(( $1 - 2))}" $2 && eval '"${@:$(( $1 + 2))}"' "$2>&-" ; set -- $? $2 ; command flock -u $2 ; return $1 ; } {_}<"${!1}"
       else { set -- "$1" "$_" "${@:2}" ; command flock "${@:3:$(( $1 - 2))}" $2 &&       "${@:$(( $1 + 2))}"          ; set -- $? $2 ; command flock -u $2 ; return $1 ; } {_}<"${!1}"
       fi
  fi
  command flock "$@"
}

x

Wednesday, January 27, 2016

Per-PC font sizes, etc

My home network has NFS homedirs, and so it doesn't matter which computer or laptop family members log into. They get all their files.

Now, one of the PC's has a 40 inch monitor, and users prefer a text-scaling-factor of 2 when using that PC.

Changing their personal settings means changing it back again; fortunately a system-wide default of 2 can be set on that PC, and apply to all users that don't override the system value.

Edit the text-scaling-factor in the schema:
sudo nano /usr/share/glib-2.0/schemas/org.gnome.desktop.interface.gschema.xml

and rebuild the schema:
sudo glib-compile-schemas /usr/share/glib-2.0/schemas

[from http://askubuntu.com/a/470253]

And then to reset the users custom value (if any):

gsettings reset org.gnome.desktop.interface text-scaling-factor
[from http://askubuntu.com/questions/60044/how-do-i-change-the-font-dpi-settings]

But doing the same for org.cinnamon.desktop as I do for org.gnome.desktop

text-scaling-factor
scaling-factor
cursor-size

Also default-zoom-level to larger and default-use-tighter-layouts to true, in /usr/share/glib-2.0/schemas/org.nemo.gschema.xml

Bash: setting and testing $?

You want to set $? in bash

?() {
  return ${1:-$?}
}

and then:

$ \? 2
$ echo $?
2 

You also want to test $?:

$ \? 2
$ $? && echo yes $? || echo no $?
no 2 

which is simpler than
$ test $? = 0 && echo yes $? || echo no $?
no 1 
which also replaces $? with the result of the test.

But how about this extended form that allows you to run a command while preserving (or forcing) the return code?

?() {
  set -- $? "$@"
  if test "$#" -le 2 -a -z "${2//[0-9]}"
  then return ${2:-$1}
  else "${@:2}"
       return $1
  fi
}

e.g.:

$ tar -xzf "$tar"
$ \? rm -fr "$tar"

which leaves $? set to the result of the tar extraction.

Of course a command which is purely numeric with no arguments is mistaken for an exit code.

Note the use of: set -- $? "$@" as function arguments are which is the only lexical scoped variables in bash.

Timing bash commands

Try this, it runs a simple command with arguments and puts the times $real $user $sys and preserves the exit code. It also does not fork subshells or trample on any variables except real user sys, and does not otherwise interfere with the running of the script

timer () {
  { time { "$@" ; } 2>${_} {_}>&- ; } {_}>&2 2>"/tmp/$$.$BASHPID.${#FUNCNAME[@]}"
  set -- $?
  read -d "" _ real _ user _ sys _ < "/tmp/$$.$BASHPID.${#FUNCNAME[@]}"
  rm -f "/tmp/$$.$BASHPID.${#FUNCNAME[@]}"
  return $1
}

e.g.

  timer find /bin /sbin /usr rm /tmp/ | grep '.sh$'
  echo $real $user $sys

note: it only times the simple command, not the entire pipeline
This version allows you to specify as $1 the name of the variables that should receive the 3 times:

timer () {
  { time { "${@:4}" ; } 2>${_} {_}>&- ; } {_}>&2 2>"/tmp/$$.$BASHPID.${#FUNCNAME[@]}"
  set -- $? "$@"
  read -d "" _ "$2" _ "$3" _ "$4" _ < "/tmp/$$.$BASHPID.${#FUNCNAME[@]}"
  rm -f "/tmp/$$.$BASHPID.${#FUNCNAME[@]}"
  return $1
}

e.g.

  timer r u s find /bin /sbin /usr rm /tmp/ | grep '.sh$'
  echo $r $u $s

and may be useful if it ends up being called recursively, to avoid trampling on times; but then r u s etc should be declared local in their use.

Note:
"/tmp/$$.$BASHPID.${#FUNCNAME[@]}"
is a way of specifying a temporary file name that will not be trampled on until after this function exits.

Shared here: http://unix.stackexchange.com/a/257964/139357

Filtering stderr


This helper function will take $1 as a simple command to be used on stderr, on the rest of the command. The filtered output is emitted on stderr.

e.g. stderr "sed -e s/^/tar: /" tar -xvzf -

The function is short, but obscure, making use of a few tricks

stderr() {
  { set -- $_ "$@" ; } {_}>&1

  { eval '"${@:3}"' "$1>&-" ; } 2>&1 >&${1} | eval '$2' ">&2" "$1>&-" 


  set -- $1 ${PIPESTATUS[0]} "${@:2}"
  eval "exec $1>&-"
  return $2
}

An explanation is here:

stderr() {
  { set -- $_ "$@" ; } {_}>&1

The first line uses the temporary variable _ (underscore), which cannot generally be relied upon, but is safe enough in this context. This variable is used to avoid this helper leaving any imprint. Trampling on variables or declaring any local variables could affect destroy transparency and potentially affect the rest of the script.

So _ becomes a copy of stdout; and then inside the { ... } we update the function arguments so that this copy of stdout is now argument 1.

The function arguments are the only lexically scoped variables in bash. We can set them here in this function knowing that they will not have any other affect anywhere else.

So $1 now refers to a copy of stdout, $2 is now the filter to be applied to stderr, and "$@:3" ($3 and onwards) is the command to be filtered.

We want to run the command with the $1 copy of standard out closed, in case the command spawns other processes that might inherit this copy and leave it open. Its a private copy, and as bash doesn't support close-on-exec we must close it.

We want to do this: "${@:3}" $1>&- but bash can't take a parameter variable on the left hand side of a redirector (not even as ${!1}) so we must use eval. We put the command in single quotes to prevent it being interpolated at all prior to eval, but the redirector is in double quotes so that the interpolated string is passed to eval; thus: eval '"${@:3}"' "$1>&-"

We want to run this command with stdout passed to the spare copy we made in $1  because we will redirect stderr to stdout to be fed into the filter. This redirection specification is: 2>&1 >&${1} (variables are allowed on the right hand side of a redirector).

However we can't append these redirectors to the previous one which already closed $1, so we use a brace scope { ... ; }  in which $1 is closed.

This gives us so far: { eval '"${@:3}"' "$1>&-" ; } 2>&1 >&${1} which has stdout going to our copy of stdout, and stderr going to actual stdout ready to pipe to the next stage.

The next stage also wants to close $1 for the same reason as before, and is:
eval '$2' ">&2" "$1>&-"

So the whole invocation is:
{ eval '"${@:3}"' "$1>&-" ; } 2>&1 >&${1} | eval '$2' ">&2" "$1>&-"

We now want to close $1  for the rest of the script without losing the exit code. As we have finished calling other commands we could save $? in a local variable, but I use the function arguments again to save as $2. Note that PIPESTATUS[0] holds the result of the first stage of the pipeline.

  set -- $1 ${PIPESTATUS[0]} "${@:2}"
  eval "exec $1>&-"
  return $2
}

And there it is.

Monday, January 18, 2016

Check status of entiire bash pipeline

My useful answer here: http://stackoverflow.com/questions/1221833/bash-pipe-output-and-capture-exit-status/34814471#34814471

pipestatus() {
  local S=(${PIPESTATUS[*]})

  if test -n "$*"
  then test "$*" = "${S[*]}"
  else ! [[ "${S[@]}" =~ [^0\ ] ]]
  fi
}

Note that S is not set to ("${PIPESTATUS[@]}"); this is so that we can re-create an array if PIPESTATUS is passed as a string, like this:

PIPE_STATUS="${PIPESTATUS[*]}" pipestatus

because an array cannot be passed in that fashion. Why would anyone want to do that? Probably not directly, but other helper commands just and also may want to preserve PIPESTATUS as best as possible to permit an: also pipestatus combination.

Usage examples:

1. get_bad_things must succeed, but it should produce no output; but we want to see output that it does produce

get_bad_things | grep '^'
pipeinfo 0 1 || return

2: all pipeline must succeed

thing | something -q | thingy
pipeinfo || return