Run time detection: Does the terminal emulator support unicode?

You can test the terminal by setting the cursor position to column 1 and outputing a multibyte unicode character. If the cursor moves by more than 1 position then the terminal does not support unicode.

On in this case we emit a 3 byte sequence which is a zero width space, so if the cursor moves at all, the terminal cannot process unicode

IFS=$';\x1B[' read -p $'\r\xE2\x80\x8B\x1B[6n\r   \r' -d R -rst 1 _ _ _ X _ </dev/tty 2>/dev/tty && test "$X" = 1

Here we output \r to get to position 1 and then emit a 3 byte sequence which is a zero width space, and then emit ESC [ 6n which asks the cursor position, followed by \r \r to overwrite any junk that will have appeared if the terminal handled each byte as a separate character.

Then we read the cusor position with a 1 second timeout and check whether the X position is position 1 which it will be if the terminal can process unicode.

A better function is:

is-tty-unicode() {
  local X

  test -c /dev/tty &&
  if test -t 0
  then IFS=$';\x1B[' read -p $'\r\xE2\x80\x8B\x1B[6n\r   \r' -d R -rst 1 _ _ _ X _ 2>&1
  fi <>/dev/tty && test "$X" = 1
}




BASH inline functions or macros

Sometimes you want to write bash macros instead of bash functions.

You want to write some bash which can declare local variables in the caller context.

Maybe it parses some data and creates a bunch of hashes with a known prefix (that's as much namespace management as bash gives you) and populates a named list with the names of those hashes.

And then you need to use that twice so you want to factor it into a function. Only you can't these aren't meant to be global variables.

So you have a few solutions, all using source in one way or another

You could put it into a separate file and source it whenever you need it.

You could define it in a string or here document and source it when needed (but you loose bash syntax support in the editor).

You can call the function using command substitution and have it execute declare -p on the variables you want to export.

e.g. source <( my-function args )

Or to enjoy all the side effects directly, you can export the function body using declare -f and strip the first line (which is the function declaration) my masking it with a comment, and then execute the rest directly, using source.

First iteration

Hint: this one is reliable.

function-body() {

echo -n '#'

declare -f "$1"

}

And to invoke it:

source <( function-body my-function ) arg1 arg2 arg3

Example

my-function()

{

echo "my-function $*";

echo a=$1;

a=$1

}

test-my-function() {

local a

source <( function-body my-function ) "$@"

echo "test says a=$a"

}

$ a=nothing ; test-my-function something ; echo "shell says a=$a"

my-function something

a=something

test says a=something

shell says a=nothing

Maybe  source <( function-body my-function ) "$@"  is a bit verbose when calling a function inline every time.

Bugs

The first iteration is fine which makes sense as eval is not used:

$ v='$thing' ; a=nothing ; test-my-function '$v $v' ; echo "shell says a=$a"

my-function $v $v

a=$v $v

test says a=$v $v

shell says a=nothing

Second iteration

Hint: this one is not reliable.

Let's create a function to invoke the function inline. But that's not going to work for the same reason, so let's invoke something that looks like a function. How about this? $inline my-function args

The printf '\x23' is an easy way to avoid # taking effect in the wrong level of the eval context.

declare -- inline="eval eval \"\$_inline\" <<<"

declare -- _inline="source <( read && printf '\x23' && declare -f \$REPLY )"

Example

my-function()

{

echo "my-function $*";

echo a=$1;

a=$1

}

test-my-function() {

local a

$inline my-function args "$@"

echo "test says a=$a"

}

$ a=nothing ; test-my-function something ; echo "shell says a=$a"

my-function something

a=something

test says a=something

shell says a=nothing

Note the double-eval mechanism so that <<< can be used to read the function name on stdin while the remaining arguments are applied to the body as $@ for the lifetime of the body (thanks, bash)

Bugs

The second iteration surprisingly appears to work fine despite the double eval:

$ v='$thing' ; a=nothing ; test-my-function '$v $v' ; echo "shell says a=$a"

my-function $v $v

a=$v $v

test says a=$v $v

shell says a=nothing

but bafflingly, if thing is defined then suddenly a double evaluation is exposed:

$ thing=xxx; v='$thing' ; a=nothing ; test-my-function '$v $v' ; echo "shell says a=$a"

my-function xxx xxx

a=xxx

test says a=xxx

shell says a=nothing

how was '$v' preserved when thing wasn't defined?

Third Iteration

Hint: this one is not reliable.

It's a shame that a fork has to be incurred each time for the command substitution to generate the code to be sourced from the function body.

Maybe, rather than $inline my-function ... we could do $my-function except that variable naming has stricter rules than function naming.

But still, that's a minor inconvenience

We can extract the function body using the current technique and assign that to a variable to be sourced, and declare another variable as syntactic sugar to source it.

It gets a little more awkward trying to find a file descriptor or device name from which to source the function without blatantly destroying stdin (which to be fair, we may be doing in the other examples -- it needs testing), so skipping that for now:

make-macro() {

declare -g "_$1" "$1"

{ read -r ; IFS="" read -d '' -r _$1 ; } < <( declare -f "$2" )

printf -v "$1" 'eval source /dev/stdin <<<"$_my_function"'

}

Example

my-function()

{

echo "my-function $*";

echo a=$1;

a=$1

}

test-my-function() {

local a

$my_function "$@"

echo "test says a=$a"

}

$ make-macro my_function my-function

$ a=nothing ; test-my-function something ; echo "shell says a=$a"

my-function something

a=something

tast says a=something

shell says a=nothing

Bugs

Third iteration has arguments subject to one extra evaluation:

$ v='$thing' ; a=nothing ; test-my-function '$v $v' ; echo "shell says a=$a"

my-function $thing $thing

a=$thing

test says a=$thing

shell says a=nothing

To prevent argument evaluation, source must be invoked first, and any eval done within what is sourced.

Bugs

We aren't being careful to make sure we don't destroy stdin. Maybe the function needs stdin. 

Writing hygenic macros is hard and using eval right is harder. The arguments are subject to a varying amount of eval evaluation.

eval had been required so that <<< could be used to paste a here-string and there is no way around that using source from a string with a command defined in a variable, as <<< is not part of a command list

we could work around this by sourcing from an actual file instead of a here doc.

Fourth Iteration

Hint: this one is reliable.

We can avoid here-strings and here-docs and the associated redirections without the need for a file to source, by declaring a temporary file as a here doc, like this:

exec {_source_}<<<'source /dev/stdin <<<"${!1}" "${@:2}"'

Now we can refer to that here-string as /dev/fd/${_source_} and read it as often as we like

source /dev/fd/${_source_} f '$v $v'

my-function $v $v

a=$v $v

So the fuller solution based on the third iteration is

make-macro() {

declare -g _macro_

test -z "$_macro_" && exec {_macro_}<<<'source /dev/stdin <<<"${!1}" "${@:2}"'

declare -g "_$1" "$1"

{ read -r ; IFS="" read -d '' -r _$1 ; } < <( declare -f "$2" )

printf -v "$1" 'source /dev/fd/%d %s' "${_macro_}" "_$1"

}

Example

my-function()

{

echo "my-function $*";

echo a=$1;

a=$1

}

test-my-function() {

local a

$my_function "$@"

echo "test says a=$a"

}

make-macro my_function my-function

thing=xxx ; v='$thing' ; a=nothing ; test-my-function '$v $v' ; echo "shell says a=$a"

my-function $v $v

a=$v $v

test says a=$v $v

shell says a=nothing

And that's enough nastyness for one day

dd is a pretentious fraud, don't trust it except for disks, files, and tape

Especially busybox dd.

[Note, specifying ibs and obs separately seems to work as expected (except with busybox dd). Using bs where ibs and obs have the same value is not a good idea]

 I used to feel guilty writing to the tape drive with cat: cat /tmp/backup.tar.gz > /dev/rct0

What dd was good at, (I had somehow learned by osmosis) was reading and writing full block sizes where the device driver can't do it so well.

What nonsense.

dd doesn't to "impedance matching" of block sizes, it has barely any regard for them. 

This all of a sudden mattered when I was using a shell to set efi variables using the efivarfs file system:

printf "\x07\x00\x00\x00%s" "$var-value" > "my-var-${owner-UUID}"

That works fine, but this was a problem:

some-process | cat /dev/fd/2 2<<<$'\x07\x00\x00\x00' - > "my-var-${owner-UUID}"

why? Because the data written to the variable pseudo-file had to be written all-in-one-go, and as this example shows, it might not happen, the output is written in two parts:

( echo hello ; sleep 1 ; echo goodbye ) | cat /dev/fd/2 2<<<$'\x07\x00\x00\x00' - | cat | cat

I supposed that this would be ideal for dd, I set the block size to 4096 which is the maximum write to the efi-var psuedo files anyway, giving:

( echo hello ; sleep 1 ; echo goodbye ) | \
cat /dev/fd/2 2<<<$'\x07\x00\x00\x00' - | dd bs=4096 of="my-var-${owner-UUID}"

but it didn't do the trick, strace showed multiple writes from dd, as we can also see here

( echo hello ; sleep 1 ; echo goodbye ) | cat /dev/fd/2 2<<<$'\x07\x00\x00\x00' - | dd bs=4096 | cat | cat

it turns out that dd doesn't care if it does a partial read from a pipe, it takes the block size as a maximum hint, rather than a requirement.

So... dd is ostensibly good at impedance matching of block sizes, lets set an input block size of 1 and an output block size of 4096 and let it accumulate the input blocks for output:

( echo hello ; sleep 1 ; echo goodbye ) | cat /dev/fd/2 2<<<$'\x07\x00\x00\x00' - | dd bs=1 obs=4096 | cat | cat

exactly the same result! dd just doesn't care!

GNU has the life-saving non-posix iflag=fullblock which actually reads a full input block (unless eof).

Without which, what is the point of dd? Well yes, we know it has incidental features such as very limited character set conversion, along with skip and seek, and unlike head, tail, etc, it won't read more input than it intends to use (which makes it very convenient for reading some of stdin from a shell script).

But it's main purpose is unmet, and with solid risk in some circles. Want to generate a random password for openssl?

password=$(dd if=/dev/random bs=32 count=1 | base64)

do you see the danger yet? What if dd blocks for want of randomness, dd will return a partial block!

simulate thus, try to read 12 random bytes and see how many we might get:

( printf "a" ; sleep 1 ; printf "bc" ; sleep 1 ; printf "def" ; sleep 1 ; printf "ghij" ; sleep 1 ; printf "klmno" ) | dd bs=12 count=1 | wc -c

we get one character instead of 12 (or instead of 256 or however many you expected)

The fix is to swap bs and count, so we keep reading 1 byte until we have enough

( printf "a" ; sleep 1 ; printf "bc" ; sleep 1 ; printf "def" ; sleep 1 ; printf "ghij" ; sleep 1 ; printf "klmno" ) | dd bs=1 count=12 | wc -c

dd, you are a pretentious ass, without the GNU extension you cannot read lots of small blocks and write one big block, and even on a non-terminating stream with more data becoming available, you will read less than block size multiplied by count, and think yourself smart!

Except with disks, files, and tape, where the driver managers block size, and partial reads won't happen, you can't be trusted to get your most basic task right.

Many interesting remarks at:

• https://unix.stackexchange.com/questions/121865/create-random-data-with-dd-and-get-partial-read-warning-is-the-data-after-the
• https://unix.stackexchange.com/questions/17295/when-is-dd-suitable-for-copying-data-or-when-are-read-and-write-partial
• https://superuser.com/questions/520601/why-does-dd-only-copy-128-bytes-from-dev-random-when-i-request-more

Working with ttystudio and ttyrec

The best of all the tty to animated gif converters, is ttystudio because it outputs small gifs.

Sadly it is slow, and uses up way to much memory -- even running out of memory for even a small session.

Ironically it boasts of not using imagemagick while requiring node! No wonder it runs out of memory. Sadly the frams also seem to be based on clock time instead of ttyrec frames.

Even worse, it only works for recording live sessions.

This shell script causes ttystudio to convert a ttyrec file.

Use: ./ttyrec2studio ttyrecfilename
to make ttyrecfilename.gif


#! /bin/bash
ttystudio() {
  local ttyrec="$1"
  local fifo=$(mktemp -u)
  mkfifo "$fifo"

  coproc ttypipe { SHELL=$(realpath "$0") QUITPIPE="$fifo" TTYPLAYFILE="$ttyrec" command ttystudio "$ttyrec.gif" >/dev/tty 2>&1 ; }

  # when we've finished with fifo, ttyplay has finished
  cat "$fifo"
  # send ^Q
  printf $'\x11' >/proc/$$/fd/${ttypipe[1]}
  # close pipe to coproc
  eval "exec ${ttypipe[1]}<&-"
  # Drain pipe from coproc
  #eval "exec ${ttypipe[0]}<&-"
  eval "cat /proc/$$/fd/${ttypipe[0]}"
  wait $! # is coproc
  echo Done all $?
}

emit() {
  rm -f "$QUITPIPE"
  echo "Playing $1" >&3
  ttyplay "$1"
  echo "Finished playing $1 [$?]" >&3
} 3>"$QUITPIPE"

main() {
  if test -z "$QUITPIPE"
  then ttystudio "$@"
  else emit "$TTYPLAYFILE"
  fi
}

main "$@"

It works by tricking ttysudio to re-invoke itself as the shell for what would have been the interactive subshell. As arguments can't be passed to the subshell, it smuggles them as environment variables.

When invoked as the subshell it simply uses ttyplay to re-play the ttyrec file.

readline with bash's read -e

Want to use bash's read -e for full editing and readline support, but without bash completion or exposing the bash history?

bind -f /dev/stdin <<<"set disable-completion on"

HISTSIZE=0 HISTIGNORE="*" HISTFILE=/dev/null read -e ...

synchronous pipe based task monitoring

I want a process to monitor another and know when it quits.

An obvious way if I control the launch is to tie them together with a pipe and they can detect when the pipe closes.

Maybe the other process would launch many further processes, all inheriting the pipe-fd, which I want to avoid.

So clearly the launcher needs to hold the pipe-fd but not share it (close-on-exec, don't fork too much), and then wait in the usual way for the launched process to quit, and then close the pipe.

Here's a bash incantation lifetime_fd which runs a simple list of arguments and can share an fd with another process through process substitution.

# just run a simple list, without affecting $?

just() {
  set -- $? "$@"

  "${@:2}"
  return $1
}

lifetime_fd() {
  set -- $_ "$@" ; eval "$1>&-" '"${@:2}"' ; just eval exec "$1>&-"
}

So if you want to run command fooly barly bazly and link it to the lame read && echo done then this will do the trick

lifetime_fd fooly barly bazly {_}> >( read && echo done )

So to attach a pipe descriptor to a sub-process, it is clear to use the trailing invocation {_}> >( sub-process ) which will attach stdin of the sub-process to a file descriptor to be stored in $_ which is managed in lifetime_fd

The variable $_ is used to avoid messing with any other variables. $_ is constantly adjusted and should do no harm if we abuse it; but as it is constantly adjusted, the first thing we do in lifetime_fd is to save it.

We don't use a local variable in case of a name clash that affects something else, so we store it as $1

We then run "$@" (or "${@:2}" as it now would be) but with the fd closed, so that it is not inherited.

We then close the fd while preserving the exit code.

You can invoke it in a pipeline like this:

get_report_request | lifetime_fd get_report {_}> >( monitor ) | send_report

An illustrative example of monitor (which reads until eof), might be:

monitor() {
  while read -t 1 || test $? = 142 # 142 is timeout code
  do echo -n '*'
  done
}

which displays a star every second until stdin closes; by continually waiting up to 1 second to fail to read anything from stdin (until it closes, having a different exit code), and displays a star.

Of course it might read other data too.... if you can send it...

Defining bash functions in a Makefile

As I answered here; a makefile recipe may depend on bash functions rather than bash scripts.

Because of bash's export -f function-name feature to export a bash function definition into the environment where it will be picked up by sub-child invocations of bash, we can define a bash variable in the environment in the same form in the Makefile.

If we define bash functions in the Makefile using the same format bash does, and export them into the environment, and set SHELL=bash then we can use these bash functions in the make recipe.

Let's see how bash formats functions exported in the environment:

$ # define the function
$ something() { echo do something here ; }
$ export -f something

$ # the Makefile
$ cat > Makefile <<END
SHELL=bash
all: ; something
END
$

Try it out

$ make
something
do something here
$

How does this look in the environment?

$ env | grep something
BASH_FUNC_something%%=() { echo do something here

The pattern is (currently):
  BASH_FUNC_function-name%%=() { function-body }

So we do this in the Makefile:

SHELL=bash
define BASH_FUNC_something-else%%
() {
  echo something else
}
endef
export BASH_FUNC_something-else%%

all: ; something-else

and try it:

$ make
something-else
something else

It's a bit ugly in the Makefile but presents no ugliness for make -n

Parsing paths & deleting empty items from a bash list

This spell will split a path into directories:

IFS=/ read -d "" -r -a path <<<"$path"

but because of the action of <<<, the final item will have a newline appended -- but that can be removed thus:

path[-1]="${path[-1]%$'\n'}"

If the path began with a /, then path[0] will be empty, so as the leading / is important, we'll recover that

test -z "${path[0]}" && path[0]="/"

This spell will remove empty items from the list (and also renumber the indexes):

IFS= eval 'path=(${path[@]})'

(We could avoid the eval by saving IFS, but eval is safe enough here not to bother).

Now it is simple enough to iterate over "${path[@]}" and perform a chdir on each stage.

Using flock in bash without invoking a subshell

flock -c can call external commands but not bash functions. Consequently users mess about with file descriptors, and often making a mess of it.

Inspired by Ivan's post http://blog.famzah.net/2013/07/31/using-flock-in-bash-without-invoking-a-subshell I've written a flock wrapper for bash, that uses flock underneath but allows flock -c to work for bash functions.

It can be called just like the regular flock command, with the benefit that the -c invocation is supported for bash functions; so you can use it like this:

flock -o /tmp/process my_thing "$@"

and I strongly recommend the -o option so that the file descriptor used for the lock is not passed to any sub-processes, which could be problematic if long live sub-processes (e.g. re-spawned daemons) keep it open.

It will pass through and invoke the regular flock if the command isn't a bash function, or if you aren't trying to execute a function.

Sadly, it doesn't recognize bash built-in's.

But the good news is, you can use flock -o ...file... on a shell function inside your shell script without having to worry about file descriptors.

# Helper function (in order to preserve $@ in the caller)
# If this isn't used to call a shell function then return 0
# otherwise return $? as the argumment number which represents
# the command/function to be called
_is_standard_flock() {
  local args=$#
  # find the first argument that doesn't begin with a -
  while test $# != 0
  do case "$1" in
     -*) shift ; continue ;;
     *) break ;;
     esac
  done

  # if it is numeric and there are not additional arguments 
  test $# = 1 -a -n "$1" -a -z "${1//[0-9]}" && return 0
  shift
  # (skipping -c if present)
  # or the following argument is also not a shell function then use the original flock
  if test "$1" = "-c"
  then declare -F "$2" >/dev/null || return 0
  else declare -F "$1" >/dev/null || return 0
  fi

  # we can't have shifted many args if this is a legitimate use of flock
  # so we will be in range of the exit code
  return $(( args - $# + 1))
}

# Help function to determine if -o or --close was given in the flock arguments
_wants_close() {
  test "${*/#--close/}" != "$*" && return # will also match bogus arguments like --closed
  # remove any -- options
  set -- "${@/#--*/}"
  # look for options with o
  test "${*/#-*o/}" != "$*" && return
  return 1
}

flock() {
  if _is_standard_flock "$@"
  then : # do outside the if-clause so bash can optimise exec where possible
  else # save the exit code (offset) as $1
       set -- $? "$@"
       # ${!$1} is the lock file 
       # ${@:$(($1 + 1))} might be -c
       test "${@:$(($1 + 1)):1}" = "-c" && set -- "${@:1:$1}" "${@:$(($1 + 2))}"
       if _wants_close "${*:2:$(( $1 - 1))}"
       then { set -- "$1" "$_" "${@:2}" ; command flock "${@:3:$(( $1 - 2))}" $2 && eval '"${@:$(( $1 + 2))}"' "$2>&-" ; set -- $? $2 ; command flock -u $2 ; return $1 ; } {_}<"${!1}"
       else { set -- "$1" "$_" "${@:2}" ; command flock "${@:3:$(( $1 - 2))}" $2 &&       "${@:$(( $1 + 2))}"          ; set -- $? $2 ; command flock -u $2 ; return $1 ; } {_}<"${!1}"
       fi
  fi
  command flock "$@"
}

x

Bash: setting and testing $?

You want to set $? in bash

?() {
  return ${1:-$?}
}

and then:

$ \? 2
$ echo $?
2 

You also want to test $?:

$ \? 2
$ \? && echo yes $? || echo no $?
no 2 

which is simpler than

$ test $? = 0 && echo yes $? || echo no $?
no 1 

which also replaces $? with the result of the test.

But how about this extended form that allows you to run a command while preserving (or forcing) the return code?

?() {
  set -- $? "$@"
  if test "$#" -le 2 -a -z "${2//[0-9]}"
  then return ${2:-$1}
  else "${@:2}"
       return $1
  fi
}

e.g.:

$ tar -xzf "$tar"
$ \? rm -fr "$tar"

which leaves $? set to the result of the tar extraction.

Of course a command which is purely numeric with no arguments is mistaken for an exit code.

Note the use of: set -- $? "$@" as function arguments are which is the only lexical scoped variables in bash.

Timing bash commands

Try this, it runs a simple command with arguments and puts the times $real $user $sys and preserves the exit code. It also does not fork subshells or trample on any variables except real user sys, and does not otherwise interfere with the running of the script

timer () {
  { time { "$@" ; } 2>${_} {_}>&- ; } {_}>&2 2>"/tmp/$$.$BASHPID.${#FUNCNAME[@]}"
  set -- $?
  read -d "" _ real _ user _ sys _ < "/tmp/$$.$BASHPID.${#FUNCNAME[@]}"
  rm -f "/tmp/$$.$BASHPID.${#FUNCNAME[@]}"
  return $1
}

e.g.

  timer find /bin /sbin /usr rm /tmp/
  echo $real $user $sys

note: it only times a simple command, not any part of a pipeline (all parts of which are run in a sub-shell).

This version allows you to specify as $1 $2 $3 the name of the variables that should receive the 3 times:

timer () {
  { time { "${@:4}" ; } 2>${_} {_}>&- ; } {_}>&2 2>"/tmp/$$.$BASHPID.${#FUNCNAME[@]}"
  set -- $? "$@"
  read -d "" _ "$2" _ "$3" _ "$4" _ < "/tmp/$$.$BASHPID.${#FUNCNAME[@]}"
  rm -f "/tmp/$$.$BASHPID.${#FUNCNAME[@]}"
  return $1
}

e.g.

  timer r u s find /bin /sbin /usr rm /tmp/
  echo $r $u $s

and may be useful if it ends up being called recursively, to avoid trampling on times; but then r u s etc should be declared local in their use.

Note:

"/tmp/$$.$BASHPID.${#FUNCNAME[@]}"

is a way of specifying a temporary file name that will not be trampled on until after this function exits.

Shared here: http://unix.stackexchange.com/a/257964/139357

Filtering stderr

This helper function will take $1 as a simple command to be used on stderr, on the rest of the command. The filtered output is emitted on stderr.

e.g. stderr "sed -e s/^/tar: /" tar -xvzf -

The function is short, but obscure, making use of a few tricks

stderr() {
  { set -- $_ "$@" ; } {_}>&1

  { eval '"${@:3}"' "$1>&-" ; } 2>&1 >&${1} | eval '$2' ">&2" "$1>&-" 


  set -- $1 ${PIPESTATUS[0]} "${@:2}"
  eval "exec $1>&-"
  return $2
}

An explanation is here:

stderr() {
  { set -- $_ "$@" ; } {_}>&1

The first line uses the temporary variable _ (underscore), which cannot generally be relied upon, but is safe enough in this context. This variable is used to avoid this helper leaving any imprint. Trampling on variables or declaring any local variables could affect destroy transparency and potentially affect the rest of the script.

So _ becomes a copy of stdout; and then inside the { ... } we update the function arguments so that this copy of stdout is now argument 1.

The function arguments are the only lexically scoped variables in bash. We can set them here in this function knowing that they will not have any other affect anywhere else.

So $1 now refers to a copy of stdout, $2 is now the filter to be applied to stderr, and "${@:3}" ($3 and onwards) is the command to be filtered.

We want to run the command with the $1 copy of standard out closed, in case the command spawns other processes that might inherit this copy and leave it open. Its a private copy, and as bash doesn't support close-on-exec we must close it.

We want to do this: "${@:3}" $1>&- but bash can't take a parameter variable on the left hand side of a redirector (not even as ${!1}) so we must use eval. We put the command in single quotes to prevent it being interpolated at all prior to eval, but the redirector is in double quotes so that the interpolated string is passed to eval; thus: eval '"${@:3}"' "$1>&-"

We want to run this command with stdout passed to the spare copy we made in $1  because we will redirect stderr to stdout to be fed into the filter. This redirection specification is: 2>&1 >&${1} (variables are allowed on the right hand side of a redirector).

However we can't append these redirectors to the previous one which already closed $1, so we use a brace scope { ... ; }  in which $1 is closed.

This gives us so far: { eval '"${@:3}"' "$1>&-" ; } 2>&1 >&${1} which has stdout going to our copy of stdout, and stderr going to actual stdout ready to pipe to the next stage.

The next stage also wants to close $1 for the same reason as before, and is:

eval '$2' ">&2" "$1>&-"

So the whole invocation is:

{ eval '"${@:3}"' "$1>&-" ; } 2>&1 >&${1} | eval '$2' ">&2" "$1>&-"

We now want to close $1  for the rest of the script without losing the exit code. As we have finished calling other commands we could save $? in a local variable, but I use the function arguments again to save as $2. Note that PIPESTATUS[0] holds the result of the first stage of the pipeline.

  set -- $1 ${PIPESTATUS[0]} "${@:2}"
  eval "exec $1>&-"
  return $2
}

And there it is.

Check status of entiire bash pipeline

My useful answer here: http://stackoverflow.com/questions/1221833/bash-pipe-output-and-capture-exit-status/34814471#34814471

pipestatus() {
  local S=(${PIPESTATUS[*]})

  if test -n "$*"
  then test "$*" = "${S[*]}"
  else ! [[ "${S[@]}" =~ [^0\ ] ]]
  fi
}

Note that S is not set to ("${PIPESTATUS[@]}"); this is so that we can re-create an array if PIPESTATUS is passed as a string, like this:

PIPE_STATUS="${PIPESTATUS[*]}" pipestatus

because an array cannot be passed in that fashion. Why would anyone want to do that? Probably not directly, but other helper commands just and also may want to preserve PIPESTATUS as best as possible to permit an: also pipestatus combination.

Usage examples:

1. get_bad_things must succeed, but it should produce no output; but we want to see output that it does produce

get_bad_things | grep '^'
pipeinfo 0 1 || return

2: all pipeline must succeed

thing | something -q | thingy
pipeinfo || return

Switch hdmi audio when TV is off using cec-client

I have a HDMI TV used as a computer monitor.

When the TV is turned on, the audio plays through the TV, so that the TV remote volume control works.

When  the TV is turned off, I want the audio to play through analog audio out so that I can still hear it.

I use the PulseEIGHT CEC controller and cec-client program to talk to the TV.

I have cec-client run as root under SOCAT to broadcast cec-data by UDP.

I have another SOCAT process running as the logged in user to pick up the data, monitor it for TV on/off status and move the playing audio streams.

It works, but is very rough.

In rc.local: /usr/local/bin/cec -s &

In gnome/cinnamon startup programs: /usr/local/bin/cec

And then the script:

#! /bin/bash

MYADDR=8 # because it is. I should probably read this from the cec-client output 
 
# 0 and 1 and the audio device indexes associated with the device names
# later I will normalise these too, but "pacmd list" will show yours.
ext() {
  pacmd set-default-sink alsa_output.pci-0000_00_1f.3.analog-stereo
  for input in $( pacmd list-sink-inputs | sed -e 's/index: //;t;d' )
  do pacmd move-sink-input $input 1
  done
}

hdmi() {
  pacmd set-default-sink alsa_output.pci-0000_01_00.1.hdmi-stereo-extra1
  for input in $( pacmd list-sink-inputs | sed -e 's/index: //;t;d' )
  do pacmd move-sink-input $input 0
  done
}

cec_client() {
  socat -u UDP4-RECV:4224,reuseaddr - | while read type _ _ dir rest
  do # echo "$type    $dir    :$rest"
     case "$rest" in
       "0f:36") echo "POWER OFF" ; ext ;;
       "(0): power status changed"*"to 'on'") echo "POWER ON" ; hdmi ;;
#       *"($MYADDR) as inactive source"*) echo INACTIVE ; ext ;;
#       *"($MYADDR) the active source"*) echo ACTIVATED ; hdmi ;;
#       *active*) echo "$type    $dir    $rest" ;;
# mmkeys-mate2mpris2
     esac
  done
}

cec_server() {
  read hostname _ < <( hostname )
  hex_hostname=$( echo -n "${hostname%.*}" | od -tx1 | sed -e '1!d;s/0* //' )

  <<< "tx 80 47 $hex_hostname" exec -a CEC socat -u -L/tmp/cec \
      EXEC:'cec-client -t p -p 2' UDP-DATAGRAM:127.0.0.1:4224,broadcast &

  wait
}

main() {
  if test "$1" = "-s"
  then cec_server
  else cec_client
  fi
}

main "$@"

Packed binary in bash

Using bash, and other accessible commands, I wanted to output some based 64 encode packed binary string representations of 64 bit decimal numbers, in reverse byte order.

I used bc to convert the arbitrarily long decimal number into hex

bc <<< "obase=16; $N"

e.g.

$ N=123456789
$ bc <<< "obase=16; $N"
75BCD15

I then split this into hex digit pairs to represent one byte each, in reverse order, and prefixing with \x to produce a printf format string.  I do this by appending a space to the line, and then move two (or one, if two not available) digits that precede the space to instead append to the end of the string, (with \x prefix). And then finally, remove the space which is now a leading space.

sed -re 's/$/ /;:start;s/([a-fA-F0-9]{1,2}) (.*)/ \2\\x\1/;Tdone;bstart;:done;s/^ *//;'

e.g. the following pattern space is iterated

75BCD15

75BCD15␣

75BCD␣\15

75B␣\15\CD

7␣\15\CD\5B

␣\15\CD\5B\7

\15\CD\5B\7

Another method of separation might have been to pad with 0 to an even length, and then split off pairs of digits from the front. I then use all of that as the argument to printf which interpolates the characters, and then pipe to base64.

printf $( bc <<< "obase=16; $N" |
  sed -re 's/$/ /;:start;s/([a-fA-F0-9]{1,2}) (.*)/ \2\\x\1/;Tdone;bstart;:done;s/^ *//;'
) | base64

e.g.

$ N=123456789
$ printf $( bc <<< "obase=16; $N" |
  sed -re 's/$/ /;:start;s/([a-fA-F0-9]{1,2}) (.*)/ \2\\x\1/;Tdone;bstart;:done;s/^ *//;'
) | base64
Fc1bBw==

An alternate method for fixed with fields that don't need byte order fixings:
Convert to hex using printf:

printf -v hex '%08x' $dec
printf '%b' "\x${hex:0:2}" "\x${hex:2:2}" "\x${hex:4:2}" "\x${hex:6:2}"

To convert a string of space separated hex to characters:

input=" $input"
printf -v input '%b' ${input// /\\x}

.

Finding xterm Terminal Window Size for serial console resize

I use a networked serial console displaying in an xterm (TERM=xterm), and to my frustration it can't cope when I resize the xterm, instead either garbling the output or just using the original portion.

I don't know why this should be.
# shopt | grep checkwinsize
checkwinsize on

Various combinations of # kill -SIGWINCH $$ and # kill -s WINCH $$ had no effect at all, and # stty size proved to be 0 0, and thus not very useful.

Edit: It turns out that bash does not directly query the terminal size directly as a result of SIGWINCH or anything else. Changing the size of the tty with stty cols $COLUMNS rows $LINES is what causes the tty driver to send SIGWINCH to the foreground application which then queries the tty driver. Resizing an xterm is what causes xterm to modify the tty, thus sending the SIGWINCH and caused bash to take notice.

With a little help from the source to xterm's resize command and also this python interpretation I came up with this 1-liner for bash, to read the xterm width and height into LINES and COLUMNS and set these in the tty driver ready for vi or other programs to pick up:

IFS=$';\x1B[' read -p $'\x1B7\x1B[r\x1B[999;999H\x1B[6n\x1B8' -d R -rst 1 _ _ LINES COLUMNS _ </dev/tty \
&& stty cols "$COLUMNS" rows "$LINES"

It's worth looking at how this works.

$'...' is a bash quoted string that allows character entities to be expressed in hexadecimal, thus $'\x1B' is the character named ESC.

The string emitted as a prompt instructs the xterm to save the current cursor position, move the cursor to 999,999 which should be beyond the terminal bounds and so instead move to the bottom right corner; it then instructs the xterm to report the current cursor position, and then restore the previous cursor position.

The current cursor position is returned as a string (without newline) in this form: \x1B[lines;columnsR and so this can be read into two variables LINES and COLUMNS that bash uses. But how to do that?

The bash read function will emit the prompt, and then read the response. As the response is terminated in R instead of a newline, -d R is passed to read.

Other useless characters are \x1B [and ; so we put these into IFS causing read to use these to split the input. This gives us 2 empty strings, which we read into the bash underscore variable which gets overwritten every line anyway, so our read variable specification is _ _ LINES COLUMNS _ causing LINES and COLUMNS to take the 3rd and 4th values, with a final _ to take any potential junk that otherwise would have been appended to COLUMNS. Raw mode and silent mode are advised obviously, hence -r -s and a timeout of 1 second is set in case an xterm isn't in use and there will be no response.

So why not define this as a function...

resize() {
  IFS=$';\x1B[' read -p $'\x1B7\x1B[r\x1B[999;999H\x1B[6n\x1B8' \
                     -d R -rst 1 _ _ LINES COLUMNS _ 2>&1 &&
  stty cols "$COLUMNS" rows "$LINES"
} </dev/tty >/dev/tty 2>/dev/null

The stdio redirections are to ensure that stderr is thrown away (in case set -x debug is active, which could mess this up), to ensure that the tty is being accessed whatever stdin and stdout are, and to ensure that the read command has access to stderr (for the prompt) to the tty.

Depending when/how you invoke it you may wish to not loose the previous value of $?, saved here without the use of local variables (which through dynamic scoping might affect the called function).

just() {
  set -- $? "$@"
  "${@:2}" 
  return $1
}

so now you can invoke: just resize without affecting $?, perhaps from something ghastly like:

export PS1="$PS1"'$(just resize)'

or as part of PROMPT_COMMAND although ideally it would run before executing a command.

It now remains to learn why (although a serial terminal cannot be expected to send a SIGWINCH) bash was not responding to SIGWINCH. (Note: Explained here, the signal was not to signify to basg that it should query the terminal using the methods described here, but to tell bash that the STTY rows and columns had already been changed and that it should read those. As we set those using STTY in the resize function, the signal is then sent to bash).

update: a colleague points out the a malicious terminal (or spoofed user input) could emit a bad cursor position response, so $COLUMNS and $LINES must be quoted where used to avoid some stty argument injection, or perhaps worse to other programs which might make careless use of COLUMNS or LINES without validating them as sane integers

MessageFormat {0} for bash

Here is an sample string formatter written in bash, that works along the lines of the java string format class, see http://docs.oracle.com/javase/7/docs/api/java/text/MessageFormat.html.

#! /bin/bash
# Sam Liddicott sam@liddicott.com
# e.g. formatecho "Hello {1}, date of {0} is {0,datetime}, using '{0,datetime}'\n" 1401444493 "$USER"

cutstr() {
  printf -v "$1" "%s" "${4%%$3*}"
  printf -v "$2" "%s" "${4#"${!1}"}"
}

formatstr() {
  local target="$1"
  shift
  local format="$1"
  shift

  local result
  local start
  local arg
  local func

  # slice up to ' or { and process
  while test -n "$format"
  do cutstr start format "['{]" "$format"
     result="$result$start"
     case "$format" in
          {*) cutstr start format "}" "$format"
              format="${format:1}"
              cutstr arg start , "${start:1}"
              arg=$(( arg + 1 ))
              if test "${start:0:1}" = ","
              then result="$result"$(${start:1} "${!arg}")
              else result="$result${!arg}"
              fi
            ;;
        "'"*) format="${format:1}"
              cutstr start format "'" "$format"
              # empty string means ' but bash 4.2 errors stop me defaulting 
              # to ' or $'\x29' so I copy the ' from format
              result="$result${start:-${format:0:1}}"
              format="${format:1}"
            ;;
     esac
  done

  printf -v "$target" "$result"
}

formatecho() {
  local _message
  formatstr _message "$@"
  printf "%s" "$_message"
}

datetime() {
  date -d @"$@"
}

formatecho "$@"

MAKEDEV and double virtualization

MAKEDEV can run in a couple of seconds - even on a virtual machine.

But if your virtual machine is hosting a qemu guest, then in that guest a fork/exec can take 0.2 of a second, and MAKEDEV generic-i386 can take a couple of hours. (This is because the kqemu kernel module is not available, http://www.linuxquestions.org/questions/linux-virtualization-and-cloud-90/qemu-running-on-ubuntu-vmware-guest-cannot-find-dev-kvm-936253/).

The first hacky-hack to cut down on the number of fork/exec is to stop calling sed quite so often, cue this patch (requires MAKEDEV to run under bash).

--- /sbin/MAKEDEV 2009-07-30 08:39:09.000000000 -0700
+++ /MAKEDEV 2014-02-21 06:31:43.000000000 -0800
@@ -1,4 +1,4 @@
-#! /bin/sh -
+#! /bin/bash -
 # $Id$
 
 #---#---#---#---#---#---#---#---#---#---#---#---#---#---#---#---#---#---#---#
@@ -116,7 +116,9 @@
 
 devicename () { # translate device names to something safe
  # A-Z is not full alphabet in all locales (e.g. in et_EE)
- echo "$*" | LC_ALL=C sed -e 's/[^A-Za-z0-9_]/_/g' 
+ #echo "$*" | LC_ALL=C sed -e 's/[^A-Za-z0-9_]/_/g' 
+ echo "${*//[^ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789]/_}"
+  
 }
 
 makedev () { # usage: makedev name [bcu] major minor owner group mode
@@ -231,12 +233,14 @@
  exec 3<$procfs/devices
  while read major device extra <&3
  do
-  device=`echo $device | sed 's#/.*##'`
+  #device=`echo $device | sed 's#/.*##'`
+  device="${device%%/*}"
   case "$major" in
    Character|Block|'')
     ;;
    *)
-    safedevname=`devicename $device`
+    safedevname="${device//[^ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz01234566
+789]/_}"
     eval "major_$safedevname=$major"
     devices="$devices $device"
     ;;
@@ -247,7 +251,8 @@
 
 Major () {
  device=$2
- devname=`devicename $1`
+ devname="${1//[^ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz01234566
+789]/_}"
  if [ "$opt_d" ]
  then
   echo -1 # don't care
@@ -2149,7 +2154,8 @@
    exec 3<$procfs/devices
    while read major device extra <&3
    do
-    device=`echo $device | sed 's#/.*##'`
+    # device=`echo $device | sed 's#/.*##'`
+    device="${device%%/*}"
     case "$major" in
      Character|Block|'')
       ;;

The next hack is to stop the 5 or 6 fork/exec when MAKEDEV deletes a device node, creates a (temporary device node), chmods it, chowns it, and renames it.

How do we do that? We run MAKEDEV -n so that it does none of these and then feed the output to a perl script which will do them.

Sadly perl has no native mknod call and there are no libraries in this environment, and so I use perl's blessed syscall function with a hard-wired syscall 133 for mknod (as it is, on my kernel):

time /MAKEDEV -n generic-i386 | perl -ne '
print; 
umask(0);
($c, $f, $node, $major, $minor, $own, $perm) = split; 
if ($node eq "->") { 
  symlink($major, $f) || die "sym: $!";
} else {
  # system("mknod",$f,$node,$major,$minor) && die "mknod($f,$node,$major,$minor): $? $!"; 
  # chmod(oct($perm), $f) || die "chmod: $!"; 
  $n=0;
  $n=0010000 if ($node eq "f");
  $n=0020000 if ($node eq "c");
  $n=0060000 if ($node eq "b");
  $n=0140000 if ($node eq "s");
  if (syscall(133, $f, oct($perm) | $n, (($minor & 0xff) | (($major & 0xfff) << 8)
          | (( ($minor & ~0xff)) << 12)
          | (( ($major & ~0xfff)) << 32))) == -1) { die "syscall: $!"; }
  ($user,$group)=split(/:/,$own);
  $user=getpwnam($user);
  $group=getgrnam($group);
  chown($user,$group,$f) || die "chown: $own $!"; 
}
'

Leaving in the system("mknod,...) took around 24 minutes, but moving straight to syscall(mknod,...) takes1 minute 8 seconds.

I did toy with having bash use printf or something to pack a binary tar archive (or even getting perl to pack a tar archive) to pipe to tar -x, but... this will have to do for now.

I suspect this depends on a bug in MAKEDEV which seems to still create sub-directories needed even in -n mode.

It would have been better to pipe the output to a c program which would parse it, I will do that, another day...

using sed to split a stream into 2 streams

An expensive file listing operation needs to invoke an action on the listed files.

xargs is normally the candidate for that, but what when there are multiple file types with varied actions?

Normally I would pipe into a bash scriptlet like this

... | while read "$file" ; do if [ $(expr "$file" :  "$pattern" ) = "0" ] ; then ... ; else ...

but it lacks the bulk appeal of xargs which can reduce the number of command invocations by thousands of times for a large file list.

So here I make use of sed, and bash's >( ... ) construct to open a subshell and substitute a magic filename that refers a file descriptor that writes to the input of the subshell. (The substituted filename is typically something like /dev/fd/63). The newline can be entered on a terminal session with ^V ^J. It is also essential that there are no spaces between the ' and >( and also between the ) and ', otherwise the sed script will be presented to sed as multiple arguments instead of one argument.

... | sed -e '/\.ko$/{w'>( xargs strip --strip-debug )'
;d}' | xargs strip

This allows kernel objects to be stripped of debug only but other objects to be stripped entirely.

An alternative would be to use tee and a separate grep

... | tee >( grep '\.ko$' | xargs strip --strip-debug ) | grep -v '\.ko$' | xargs strip

bash signal handling

A signal trap will not be executed while bash is implicitly waiting for an external command to complete.

I mean:

#! /bin/bash

trap "echo signal" 10

( sleep 1 ; kill -s 10 $$ ) &

sleep 5

We would expect the word signal to be emitted after 1 second, but the signal handler will not run until the sleep 5 is complete.

However, the built-in bash command wait does not have that problem, and so we can run the task in the background and immediately wait as in this example where the signal runs after 1 second.

#! /bin/bash

trap "echo signal" 10

( sleep 1 ; kill -s 10 $$ ) &

<&0 sleep 5 & wait

The <&0 is required to cover the case that the background command needs to read stdin which would otherwise not be connected for a background command.

The script also quits after 1 second even though the sleep is still running, so this variant detects if the background is still running and waits again:

#! /bin/bash

trap "echo signal" 10

( sleep 1 ; kill -s 10 $$ ) &

sleep 5 & wait

kill -s 0 $! &>- && wait

Of course this is insufficient for 2 reasons; the extra wait should be in a loop, but worse, the exit code is lost.