bottles to make me well

Thank you, kind parents, for giving me bottles to make me well.

Mr Darling in "Peter Pan" by J. M. Barrie

Every parent wishes to raise their child to be capable, independent, and yet still remain appreciative enough to visit once in a while in those declining autumn years, so that every longing look from the window shall not be in vain, and most of those that are will be sufficiently trimmed with happy memories, and with happy anticipations of many more to come.

Are any of us wise or foresightful enough to choose each experience that would develop and improve our natural situation and abilities, while eschewing every experience that would do damage to the same? Few of us are wise enough to even think upon wishing that we were so.

There is time enough for early morning study on every other blessed early morning yet to rise, it is the mornings in bed that are in short supply. Does this homework assignment really matter among so many?

It is not the tomorrows that multiply all those good intentions, for they are all born of the now which says wait and not yet, and so it is the now, in the moment, that the unyielding twig must be unbent for the sake of the tree. There will be other shining moments to improve, says the youth to the parent attempting to incline him towards improving on any of them.

The debate between can't and won't is a narrow distraction and often chosen by a schoolmaster as a more familiar and a less exerting strategy and almost as effective as combining all their wit with honeyed words, which makes only a very small helm by which to steer reluctant pupils. The schoolmaster has been beaten back by a never-ending sea of youths, wave upon wave, year upon year, dashing his castles in the sand, and leaving a pristine shore as they depart to whence they came.

The parent is a specialist craftsman, with a lifetime commitment, and with no end in view so narrow as mere matriculation, which is only one possible stepping stone of many on a carefully consulted route.

When the young lad won't go in and make so many friends for life from a group of strangers his age, what is a father to do? Where reason and determined entreaties may fail, force is futile and will only cultivate immunity against all the more gentle influences in the future.

Complete capitulation can't be countenanced -- who wants the present problem to persist permanently?

Do you know how to make a donkey turn around on a narrow staircase? It's very easy when you know the trick, which I am at liberty to reveal to you at some future time in the form of a song. The refrain is based upon the idea that a donkey will do what a donkey wants to do, and when the donkey wants to do it; and it goes like this: you got to make him want to.

It is a strategy that has often succeeded. Children will often want to do what they should when they realise that everything else is more boring. 

I was once losing influence due to constant brow-beating. My over-used arguments were losing their force. The lad and I were getting fed up, and I was blowed if I was going to carry on like that all night. I needed something to strengthen the relationship, to give a distraction to clear the mind, to give time for everyone to think, while not removing the choice or obligation.

Inspiration spoke, and the solution was to go for a walk up a steep hill in the summer evening, away from the present pressing problem. Time to explore, time to talk - and if about nothing else, then about what we explore, about what we might find next, whether there will be a chip shop, and which road leads back. If the troublesome topic became too touchy, we temporarily talk of trivial things.

It was an activity that would require exertion, rewarded by a quite predictable though somewhat novel experience, and new sights of minor interest.

It was also sure to be just as steep next time, and slightly less interesting.

There were worse ways to spend the evening, and time for one to conclude (who knows?) as dread dissipates in the daylight outdoors, that going in and meeting new people might actually be better than this.

The obligation to overcome the obstacle remains. Any paths that ultimately lead back to this fearsome obstacle would be a blessing, for any steps which did not would need to be retraced.

The parent's duty is to create an environment that reflects this truth: The way out is the way through. 

The boy became the master. Such situations cannot dominate him. Having paid the price, he knows their secret, and one secret more: 

By paying the price, man may know the secret, and overcome.

BASH inline functions or macros

Sometimes you want to write bash macros instead of bash functions.

You want to write some bash which can declare local variables in the caller context.

Maybe it parses some data and creates a bunch of hashes with a known prefix (that's as much namespace management as bash gives you) and populates a named list with the names of those hashes.

And then you need to use that twice so you want to factor it into a function. Only you can't these aren't meant to be global variables.

So you have a few solutions, all using source in one way or another

You could put it into a separate file and source it whenever you need it.

You could define it in a string or here document and source it when needed (but you loose bash syntax support in the editor).

You can call the function using command substitution and have it execute declare -p on the variables you want to export.

e.g. source <( my-function args )

Or to enjoy all the side effects directly, you can export the function body using declare -f and strip the first line (which is the function declaration) my masking it with a comment, and then execute the rest directly, using source.

First iteration

Hint: this one is reliable.

function-body() {

echo -n '#'

declare -f "$1"

}

And to invoke it:

source <( function-body my-function ) arg1 arg2 arg3

Example

my-function()

{

echo "my-function $*";

echo a=$1;

a=$1

}

test-my-function() {

local a

source <( function-body my-function ) "$@"

echo "test says a=$a"

}

$ a=nothing ; test-my-function something ; echo "shell says a=$a"

my-function something

a=something

test says a=something

shell says a=nothing

Maybe  source <( function-body my-function ) "$@"  is a bit verbose when calling a function inline every time.

Bugs

The first iteration is fine which makes sense as eval is not used:

$ v='$thing' ; a=nothing ; test-my-function '$v $v' ; echo "shell says a=$a"

my-function $v $v

a=$v $v

test says a=$v $v

shell says a=nothing

Second iteration

Hint: this one is not reliable.

Let's create a function to invoke the function inline. But that's not going to work for the same reason, so let's invoke something that looks like a function. How about this? $inline my-function args

The printf '\x23' is an easy way to avoid # taking effect in the wrong level of the eval context.

declare -- inline="eval eval \"\$_inline\" <<<"

declare -- _inline="source <( read && printf '\x23' && declare -f \$REPLY )"

Example

my-function()

{

echo "my-function $*";

echo a=$1;

a=$1

}

test-my-function() {

local a

$inline my-function args "$@"

echo "test says a=$a"

}

$ a=nothing ; test-my-function something ; echo "shell says a=$a"

my-function something

a=something

test says a=something

shell says a=nothing

Note the double-eval mechanism so that <<< can be used to read the function name on stdin while the remaining arguments are applied to the body as $@ for the lifetime of the body (thanks, bash)

Bugs

The second iteration surprisingly appears to work fine despite the double eval:

$ v='$thing' ; a=nothing ; test-my-function '$v $v' ; echo "shell says a=$a"

my-function $v $v

a=$v $v

test says a=$v $v

shell says a=nothing

but bafflingly, if thing is defined then suddenly a double evaluation is exposed:

$ thing=xxx; v='$thing' ; a=nothing ; test-my-function '$v $v' ; echo "shell says a=$a"

my-function xxx xxx

a=xxx

test says a=xxx

shell says a=nothing

how was '$v' preserved when thing wasn't defined?

Third Iteration

Hint: this one is not reliable.

It's a shame that a fork has to be incurred each time for the command substitution to generate the code to be sourced from the function body.

Maybe, rather than $inline my-function ... we could do $my-function except that variable naming has stricter rules than function naming.

But still, that's a minor inconvenience

We can extract the function body using the current technique and assign that to a variable to be sourced, and declare another variable as syntactic sugar to source it.

It gets a little more awkward trying to find a file descriptor or device name from which to source the function without blatantly destroying stdin (which to be fair, we may be doing in the other examples -- it needs testing), so skipping that for now:

make-macro() {

declare -g "_$1" "$1"

{ read -r ; IFS="" read -d '' -r _$1 ; } < <( declare -f "$2" )

printf -v "$1" 'eval source /dev/stdin <<<"$_my_function"'

}

Example

my-function()

{

echo "my-function $*";

echo a=$1;

a=$1

}

test-my-function() {

local a

$my_function "$@"

echo "test says a=$a"

}

$ make-macro my_function my-function

$ a=nothing ; test-my-function something ; echo "shell says a=$a"

my-function something

a=something

tast says a=something

shell says a=nothing

Bugs

Third iteration has arguments subject to one extra evaluation:

$ v='$thing' ; a=nothing ; test-my-function '$v $v' ; echo "shell says a=$a"

my-function $thing $thing

a=$thing

test says a=$thing

shell says a=nothing

To prevent argument evaluation, source must be invoked first, and any eval done within what is sourced.

Bugs

We aren't being careful to make sure we don't destroy stdin. Maybe the function needs stdin. 

Writing hygenic macros is hard and using eval right is harder. The arguments are subject to a varying amount of eval evaluation.

eval had been required so that <<< could be used to paste a here-string and there is no way around that using source from a string with a command defined in a variable, as <<< is not part of a command list

we could work around this by sourcing from an actual file instead of a here doc.

Fourth Iteration

Hint: this one is reliable.

We can avoid here-strings and here-docs and the associated redirections without the need for a file to source, by declaring a temporary file as a here doc, like this:

exec {_source_}<<<'source /dev/stdin <<<"${!1}" "${@:2}"'

Now we can refer to that here-string as /dev/fd/${_source_} and read it as often as we like

source /dev/fd/${_source_} f '$v $v'

my-function $v $v

a=$v $v

So the fuller solution based on the third iteration is

make-macro() {

declare -g _macro_

test -z "$_macro_" && exec {_macro_}<<<'source /dev/stdin <<<"${!1}" "${@:2}"'

declare -g "_$1" "$1"

{ read -r ; IFS="" read -d '' -r _$1 ; } < <( declare -f "$2" )

printf -v "$1" 'source /dev/fd/%d %s' "${_macro_}" "_$1"

}

Example

my-function()

{

echo "my-function $*";

echo a=$1;

a=$1

}

test-my-function() {

local a

$my_function "$@"

echo "test says a=$a"

}

make-macro my_function my-function

thing=xxx ; v='$thing' ; a=nothing ; test-my-function '$v $v' ; echo "shell says a=$a"

my-function $v $v

a=$v $v

test says a=$v $v

shell says a=nothing

And that's enough nastyness for one day

Jovano Jovanke

 Johnathon, Oh Johnny is my answer to Jovano Jovanke.

The original folk song lycrics are sung beautifully by Biljana Krstic

Macedonian	English translation[6]
Јовано, Jованке Jовано, Jованке Крај Вардарот седиш, мори Бело платно белиш, Бело платно белиш, душо Се нагоре гледаш. (X2) Jовано, Jованке, јас те тебе чeкам, мори Дома да ми дојдеш, А ти не доаѓаш, душо Срце мое, Jовано. (X2) Jовано, Jованке, Твојата мајка, мори Тебе не те пушта, Крај мене да дојдеш, душо Срце мое, Jовано. (X2)	Jovana, Jovanka, you sit by the river, bleaching your white linen, bleaching your white linen, my dear, looking upward. (X2)   Jovana, Jovanka, I'm waiting for you to come to my home, and you don't come, my dear, my heart, Jovana. (X2) Jovana, Jovanka, your mother doesn't let you come to me, my dear, my heart, Jovana. (X2)

A more up-to-date rendering of this timeless piece is provided for the youth to reflect their own faults instead of blaming their parents.

Johnathon, my Johnny

Johnathon, oh Johnny
You sit in your bedroom, playing on your computer
instead of studying, my dear
sitting with the curtains shut

Johnathon, my Johnny
I'm waiting here for you
To come to my home,
But you don't come, my dear,
My dear, my Johnny.

Johnathon, oh Johnny
Your mother, won't let you,
your school work isn't done
Oh dear, oh Johnny.

© Sam Liddicott 2021

And if you got this far, don't miss Biljana singing the original.

dd is a pretentious fraud, don't trust it except for disks, files, and tape

Especially busybox dd.

[Note, specifying ibs and obs separately seems to work as expected (except with busybox dd). Using bs where ibs and obs have the same value is not a good idea]

 I used to feel guilty writing to the tape drive with cat: cat /tmp/backup.tar.gz > /dev/rct0

What dd was good at, (I had somehow learned by osmosis) was reading and writing full block sizes where the device driver can't do it so well.

What nonsense.

dd doesn't to "impedance matching" of block sizes, it has barely any regard for them. 

This all of a sudden mattered when I was using a shell to set efi variables using the efivarfs file system:

printf "\x07\x00\x00\x00%s" "$var-value" > "my-var-${owner-UUID}"

That works fine, but this was a problem:

some-process | cat /dev/fd/2 2<<<$'\x07\x00\x00\x00' - > "my-var-${owner-UUID}"

why? Because the data written to the variable pseudo-file had to be written all-in-one-go, and as this example shows, it might not happen, the output is written in two parts:

( echo hello ; sleep 1 ; echo goodbye ) | cat /dev/fd/2 2<<<$'\x07\x00\x00\x00' - | cat | cat

I supposed that this would be ideal for dd, I set the block size to 4096 which is the maximum write to the efi-var psuedo files anyway, giving:

( echo hello ; sleep 1 ; echo goodbye ) | \
cat /dev/fd/2 2<<<$'\x07\x00\x00\x00' - | dd bs=4096 of="my-var-${owner-UUID}"

but it didn't do the trick, strace showed multiple writes from dd, as we can also see here

( echo hello ; sleep 1 ; echo goodbye ) | cat /dev/fd/2 2<<<$'\x07\x00\x00\x00' - | dd bs=4096 | cat | cat

it turns out that dd doesn't care if it does a partial read from a pipe, it takes the block size as a maximum hint, rather than a requirement.

So... dd is ostensibly good at impedance matching of block sizes, lets set an input block size of 1 and an output block size of 4096 and let it accumulate the input blocks for output:

( echo hello ; sleep 1 ; echo goodbye ) | cat /dev/fd/2 2<<<$'\x07\x00\x00\x00' - | dd bs=1 obs=4096 | cat | cat

exactly the same result! dd just doesn't care!

GNU has the life-saving non-posix iflag=fullblock which actually reads a full input block (unless eof).

Without which, what is the point of dd? Well yes, we know it has incidental features such as very limited character set conversion, along with skip and seek, and unlike head, tail, etc, it won't read more input than it intends to use (which makes it very convenient for reading some of stdin from a shell script).

But it's main purpose is unmet, and with solid risk in some circles. Want to generate a random password for openssl?

password=$(dd if=/dev/random bs=32 count=1 | base64)

do you see the danger yet? What if dd blocks for want of randomness, dd will return a partial block!

simulate thus, try to read 12 random bytes and see how many we might get:

( printf "a" ; sleep 1 ; printf "bc" ; sleep 1 ; printf "def" ; sleep 1 ; printf "ghij" ; sleep 1 ; printf "klmno" ) | dd bs=12 count=1 | wc -c

we get one character instead of 12 (or instead of 256 or however many you expected)

The fix is to swap bs and count, so we keep reading 1 byte until we have enough

( printf "a" ; sleep 1 ; printf "bc" ; sleep 1 ; printf "def" ; sleep 1 ; printf "ghij" ; sleep 1 ; printf "klmno" ) | dd bs=1 count=12 | wc -c

dd, you are a pretentious ass, without the GNU extension you cannot read lots of small blocks and write one big block, and even on a non-terminating stream with more data becoming available, you will read less than block size multiplied by count, and think yourself smart!

Except with disks, files, and tape, where the driver managers block size, and partial reads won't happen, you can't be trusted to get your most basic task right.

Many interesting remarks at:

• https://unix.stackexchange.com/questions/121865/create-random-data-with-dd-and-get-partial-read-warning-is-the-data-after-the
• https://unix.stackexchange.com/questions/17295/when-is-dd-suitable-for-copying-data-or-when-are-read-and-write-partial
• https://superuser.com/questions/520601/why-does-dd-only-copy-128-bytes-from-dev-random-when-i-request-more

Working with ttystudio and ttyrec

The best of all the tty to animated gif converters, is ttystudio because it outputs small gifs.

Sadly it is slow, and uses up way to much memory -- even running out of memory for even a small session.

Ironically it boasts of not using imagemagick while requiring node! No wonder it runs out of memory. Sadly the frams also seem to be based on clock time instead of ttyrec frames.

Even worse, it only works for recording live sessions.

This shell script causes ttystudio to convert a ttyrec file.

Use: ./ttyrec2studio ttyrecfilename
to make ttyrecfilename.gif


#! /bin/bash
ttystudio() {
  local ttyrec="$1"
  local fifo=$(mktemp -u)
  mkfifo "$fifo"

  coproc ttypipe { SHELL=$(realpath "$0") QUITPIPE="$fifo" TTYPLAYFILE="$ttyrec" command ttystudio "$ttyrec.gif" >/dev/tty 2>&1 ; }

  # when we've finished with fifo, ttyplay has finished
  cat "$fifo"
  # send ^Q
  printf $'\x11' >/proc/$$/fd/${ttypipe[1]}
  # close pipe to coproc
  eval "exec ${ttypipe[1]}<&-"
  # Drain pipe from coproc
  #eval "exec ${ttypipe[0]}<&-"
  eval "cat /proc/$$/fd/${ttypipe[0]}"
  wait $! # is coproc
  echo Done all $?
}

emit() {
  rm -f "$QUITPIPE"
  echo "Playing $1" >&3
  ttyplay "$1"
  echo "Finished playing $1 [$?]" >&3
} 3>"$QUITPIPE"

main() {
  if test -z "$QUITPIPE"
  then ttystudio "$@"
  else emit "$TTYPLAYFILE"
  fi
}

main "$@"

It works by tricking ttysudio to re-invoke itself as the shell for what would have been the interactive subshell. As arguments can't be passed to the subshell, it smuggles them as environment variables.

When invoked as the subshell it simply uses ttyplay to re-play the ttyrec file.

readline with bash's read -e

Want to use bash's read -e for full editing and readline support, but without bash completion or exposing the bash history?

bind -f /dev/stdin <<<"set disable-completion on"

HISTSIZE=0 HISTIGNORE="*" HISTFILE=/dev/null read -e ...

synchronous pipe based task monitoring

I want a process to monitor another and know when it quits.

An obvious way if I control the launch is to tie them together with a pipe and they can detect when the pipe closes.

Maybe the other process would launch many further processes, all inheriting the pipe-fd, which I want to avoid.

So clearly the launcher needs to hold the pipe-fd but not share it (close-on-exec, don't fork too much), and then wait in the usual way for the launched process to quit, and then close the pipe.

Here's a bash incantation lifetime_fd which runs a simple list of arguments and can share an fd with another process through process substitution.

# just run a simple list, without affecting $?

just() {
  set -- $? "$@"

  "${@:2}"
  return $1
}

lifetime_fd() {
  set -- $_ "$@" ; eval "$1>&-" '"${@:2}"' ; just eval exec "$1>&-"
}

So if you want to run command fooly barly bazly and link it to the lame read && echo done then this will do the trick

lifetime_fd fooly barly bazly {_}> >( read && echo done )

So to attach a pipe descriptor to a sub-process, it is clear to use the trailing invocation {_}> >( sub-process ) which will attach stdin of the sub-process to a file descriptor to be stored in $_ which is managed in lifetime_fd

The variable $_ is used to avoid messing with any other variables. $_ is constantly adjusted and should do no harm if we abuse it; but as it is constantly adjusted, the first thing we do in lifetime_fd is to save it.

We don't use a local variable in case of a name clash that affects something else, so we store it as $1

We then run "$@" (or "${@:2}" as it now would be) but with the fd closed, so that it is not inherited.

We then close the fd while preserving the exit code.

You can invoke it in a pipeline like this:

get_report_request | lifetime_fd get_report {_}> >( monitor ) | send_report

An illustrative example of monitor (which reads until eof), might be:

monitor() {
  while read -t 1 || test $? = 142 # 142 is timeout code
  do echo -n '*'
  done
}

which displays a star every second until stdin closes; by continually waiting up to 1 second to fail to read anything from stdin (until it closes, having a different exit code), and displays a star.

Of course it might read other data too.... if you can send it...