Wednesday, 24 November 2021

BASH inline functions or macros

Sometimes you want to write bash macros instead of bash functions.

You want to write some bash which can declare local variables in the caller context.

Maybe it parses some data and creates a bunch of hashes with a known prefix (that's as much namespace management as bash gives you) and populates a named list with the names of those hashes.

And then you need to use that twice so you want to factor it into a function. Only you can't these aren't meant to be global variables.

So you have a few solutions, all using source in one way or another

You could put it into a separate file and source it whenever you need it.

You could define it in a string or here document and source it when needed (but you loose bash syntax support in the editor).

You can call the function using command substitution and have it execute declare -p on the variables you want to export.

e.g. source <( my-function args )

Or to enjoy all the side effects directly, you can export the function body using declare -f and strip the first line (which is the function declaration) my masking it with a comment, and then execute the rest directly, using source.

First iteration

Hint: this one is reliable.
function-body() {
echo -n '#'
declare -f "$1"
}

And to invoke it:

source <( function-body my-function ) arg1 arg2 arg3

Example

my-function()
{
echo "my-function $*";
echo a=$1;
a=$1
}
test-my-function() {
local a
source <( function-body my-function ) "$@"
echo "test says a=$a"
}

$ a=nothing ; test-my-function something ; echo "shell says a=$a"
my-function something
a=something
test says a=something
shell says a=nothing

Maybe  source <( function-body my-function ) "$@"  is a bit verbose when calling a function inline every time.

Bugs

The first iteration is fine which makes sense as eval is not used:
$ v='$thing' ; a=nothing ; test-my-function '$v $v' ; echo "shell says a=$a"
my-function $v $v
a=$v $v
test says a=$v $v
shell says a=nothing

Second iteration

Hint: this one is not reliable.
Let's create a function to invoke the function inline. But that's not going to work for the same reason, so let's invoke something that looks like a function. How about this? $inline my-function args

The printf '\x23' is an easy way to avoid # taking effect in the wrong level of the eval context.

declare -- inline="eval eval \"\$_inline\" <<<"
declare -- _inline="source <( read && printf '\x23' && declare -f \$REPLY )"

Example

my-function()
{
echo "my-function $*";
echo a=$1;
a=$1
}
test-my-function() {
local a
$inline my-function args "$@"
echo "test says a=$a"
}
$ a=nothing ; test-my-function something ; echo "shell says a=$a"
my-function something
a=something
test says a=something
shell says a=nothing

Note the double-eval mechanism so that <<< can be used to read the function name on stdin while the remaining arguments are applied to the body as $@ for the lifetime of the body (thanks, bash)

Bugs

The second iteration surprisingly appears to work fine despite the double eval:
$ v='$thing' ; a=nothing ; test-my-function '$v $v' ; echo "shell says a=$a"
my-function $v $v
a=$v $v
test says a=$v $v
shell says a=nothing

but bafflingly, if thing is defined then suddenly a double evaluation is exposed:
$ thing=xxx; v='$thing' ; a=nothing ; test-my-function '$v $v' ; echo "shell says a=$a"
my-function xxx xxx
a=xxx
test says a=xxx
shell says a=nothing
how was '$v' preserved when thing wasn't defined?

Third Iteration

Hint: this one is not reliable.
It's a shame that a fork has to be incurred each time for the command substitution to generate the code to be sourced from the function body.

Maybe, rather than $inline my-function ... we could do $my-function except that variable naming has stricter rules than function naming.

But still, that's a minor inconvenience

We can extract the function body using the current technique and assign that to a variable to be sourced, and declare another variable as syntactic sugar to source it.

It gets a little more awkward trying to find a file descriptor or device name from which to source the function without blatantly destroying stdin (which to be fair, we may be doing in the other examples -- it needs testing), so skipping that for now:

make-macro() {
declare -g "_$1" "$1"
{ read -r ; IFS="" read -d '' -r _$1 ; } < <( declare -f "$2" )
printf -v "$1" 'eval source /dev/stdin <<<"$_my_function"'
}

Example

my-function()
{
echo "my-function $*";
echo a=$1;
a=$1
}
test-my-function() {
local a
$my_function "$@"
echo "test says a=$a"
}

$ make-macro my_function my-function
$ a=nothing ; test-my-function something ; echo "shell says a=$a"
my-function something
a=something
tast says a=something
shell says a=nothing

Bugs

Third iteration has arguments subject to one extra evaluation:
$ v='$thing' ; a=nothing ; test-my-function '$v $v' ; echo "shell says a=$a"
my-function $thing $thing
a=$thing
test says a=$thing
shell says a=nothing

To prevent argument evaluation, source must be invoked first, and any eval done within what is sourced.

Bugs

We aren't being careful to make sure we don't destroy stdin. Maybe the function needs stdin. 

Writing hygenic macros is hard and using eval right is harder. The arguments are subject to a varying amount of eval evaluation.

eval had been required so that <<< could be used to paste a here-string and there is no way around that using source from a string with a command defined in a variable, as <<< is not part of a command list

we could work around this by sourcing from an actual file instead of a here doc.

Fourth Iteration

Hint: this one is reliable.
We can avoid here-strings and here-docs and the associated redirections without the need for a file to source, by declaring a temporary file as a here doc, like this:

exec {_source_}<<<'source /dev/stdin <<<"${!1}" "${@:2}"'

Now we can refer to that here-string as /dev/fd/${_source_} and read it as often as we like
source /dev/fd/${_source_} f '$v $v'
my-function $v $v
a=$v $v

So the fuller solution based on the third iteration is
make-macro() {
declare -g _macro_
test -z "$_macro_" && exec {_macro_}<<<'source /dev/stdin <<<"${!1}" "${@:2}"'
declare -g "_$1" "$1"
{ read -r ; IFS="" read -d '' -r _$1 ; } < <( declare -f "$2" )
printf -v "$1" 'source /dev/fd/%d %s' "${_macro_}" "_$1"
}

Example

my-function()
{
echo "my-function $*";
echo a=$1;
a=$1
}
test-my-function() {
local a
$my_function "$@"
echo "test says a=$a"
}

make-macro my_function my-function
thing=xxx ; v='$thing' ; a=nothing ; test-my-function '$v $v' ; echo "shell says a=$a"
my-function $v $v
a=$v $v
test says a=$v $v
shell says a=nothing

And that's enough nastyness for one day

Friday, 26 February 2021

Jovano Jovanke

 Johnathon, Oh Johnny is my answer to Jovano Jovanke.

The original folk song lycrics are sung beautifully by Biljana Krstic

MacedonianEnglish translation[6]

Јовано, Jованке
Jовано, Jованке
Крај Вардарот седиш, мори
Бело платно белиш,
Бело платно белиш, душо
Се нагоре гледаш. (X2)

Jовано, Jованке,
јас те тебе чeкам, мори
Дома да ми дојдеш,
А ти не доаѓаш, душо
Срце мое, Jовано. (X2)

Jовано, Jованке,
Твојата мајка, мори
Тебе не те пушта,
Крај мене да дојдеш, душо
Срце мое, Jовано. (X2)


Jovana, Jovanka,
you sit by the river,
bleaching your white linen,
bleaching your white linen, my dear,
looking upward. (X2)
 
Jovana, Jovanka,
I'm waiting for you
to come to my home,
and you don't come, my dear,
my heart, Jovana. (X2)

Jovana, Jovanka,
your mother
doesn't let you
come to me, my dear,
my heart, Jovana. (X2)

A more up-to-date rendering of this timeless piece is provided for the youth to reflect their own faults instead of blaming their parents.

Johnathon, my Johnny

Johnathon, oh Johnny
You sit in your bedroom, playing on your computer
instead of studying, my dear
sitting with the curtains shut

Johnathon, my Johnny
I'm waiting here for you
To come to my home,
But you don't come, my dear,
My dear, my Johnny.

Johnathon, oh Johnny
Your mother, won't let you,
your school work isn't done
Oh dear, oh Johnny.

© Sam Liddicott 2021
And if you got this far, don't miss Biljana singing the original.


Friday, 23 October 2020

dd is a pretentious fraud, don't trust it except for disks, files, and tape

Especially busybox dd.

[Note, specifying ibs and obs separately seems to work as expected (except with busybox dd). Using bs where ibs and obs have the same value is not a good idea]

 I used to feel guilty writing to the tape drive with cat: cat /tmp/backup.tar.gz > /dev/rct0

What dd was good at, (I had somehow learned by osmosis) was reading and writing full block sizes where the device driver can't do it so well.

What nonsense.

dd doesn't to "impedance matching" of block sizes, it has barely any regard for them. 

This all of a sudden mattered when I was using a shell to set efi variables using the efivarfs file system:

printf "\x07\x00\x00\x00%s" "$var-value" > "my-var-${owner-UUID}"

That works fine, but this was a problem:

some-process | cat /dev/fd/2 2<<<$'\x07\x00\x00\x00' - > "my-var-${owner-UUID}"

why? Because the data written to the variable pseudo-file had to be written all-in-one-go, and as this example shows, it might not happen, the output is written in two parts:

( echo hello ; sleep 1 ; echo goodbye ) | cat /dev/fd/2 2<<<$'\x07\x00\x00\x00' - | cat | cat

I supposed that this would be ideal for dd, I set the block size to 4096 which is the maximum write to the efi-var psuedo files anyway, giving:

( echo hello ; sleep 1 ; echo goodbye ) | \
cat /dev/fd/2 2<<<$'\x07\x00\x00\x00' - | dd bs=4096 of="my-var-${owner-UUID}"

but it didn't do the trick, strace showed multiple writes from dd, as we can also see here

( echo hello ; sleep 1 ; echo goodbye ) | cat /dev/fd/2 2<<<$'\x07\x00\x00\x00' - | dd bs=4096 | cat | cat

it turns out that dd doesn't care if it does a partial read from a pipe, it takes the block size as a maximum hint, rather than a requirement.

So... dd is ostensibly good at impedance matching of block sizes, lets set an input block size of 1 and an output block size of 4096 and let it accumulate the input blocks for output:

( echo hello ; sleep 1 ; echo goodbye ) | cat /dev/fd/2 2<<<$'\x07\x00\x00\x00' - | dd bs=1 obs=4096 | cat | cat

exactly the same result! dd just doesn't care!

GNU has the life-saving non-posix iflag=fullblock which actually reads a full input block (unless eof).

Without which, what is the point of dd? Well yes, we know it has incidental features such as very limited character set conversion, along with skip and seek, and unlike head, tail, etc, it won't read more input than it intends to use (which makes it very convenient for reading some of stdin from a shell script).

But it's main purpose is unmet, and with solid risk in some circles. Want to generate a random password for openssl?

password=$(dd if=/dev/random bs=32 count=1 | base64)

do you see the danger yet? What if dd blocks for want of randomness, dd will return a partial block!

simulate thus, try to read 12 random bytes and see how many we might get:

( printf "a" ; sleep 1 ; printf "bc" ; sleep 1 ; printf "def" ; sleep 1 ; printf "ghij" ; sleep 1 ; printf "klmno" ) | dd bs=12 count=1 | wc -c

we get one character instead of 12 (or instead of 256 or however many you expected)

The fix is to swap bs and count, so we keep reading 1 byte until we have enough

( printf "a" ; sleep 1 ; printf "bc" ; sleep 1 ; printf "def" ; sleep 1 ; printf "ghij" ; sleep 1 ; printf "klmno" ) | dd bs=1 count=12 | wc -c

dd, you are a pretentious ass, without the GNU extension you cannot read lots of small blocks and write one big block, and even on a non-terminating stream with more data becoming available, you will read less than block size multiplied by count, and think yourself smart!

Except with disks, files, and tape, where the driver managers block size, and partial reads won't happen, you can't be trusted to get your most basic task right.

Many interesting remarks at:

  • https://unix.stackexchange.com/questions/121865/create-random-data-with-dd-and-get-partial-read-warning-is-the-data-after-the
  • https://unix.stackexchange.com/questions/17295/when-is-dd-suitable-for-copying-data-or-when-are-read-and-write-partial
  • https://superuser.com/questions/520601/why-does-dd-only-copy-128-bytes-from-dev-random-when-i-request-more

Saturday, 21 March 2020

Working with ttystudio and ttyrec

The best of all the tty to animated gif converters, is ttystudio because it outputs small gifs.

Sadly it is slow, and uses up way to much memory -- even running out of memory for even a small session.

Ironically it boasts of not using imagemagick while requiring node! No wonder it runs out of memory. Sadly the frams also seem to be based on clock time instead of ttyrec frames.

Even worse, it only works for recording live sessions.

This shell script causes ttystudio to convert a ttyrec file.

Use: ./ttyrec2studio ttyrecfilename
to make ttyrecfilename.gif


#! /bin/bash
ttystudio() {
  local ttyrec="$1"
  local fifo=$(mktemp -u)
  mkfifo "$fifo"

  coproc ttypipe { SHELL=$(realpath "$0") QUITPIPE="$fifo" TTYPLAYFILE="$ttyrec" command ttystudio "$ttyrec.gif" >/dev/tty 2>&1 ; }

  # when we've finished with fifo, ttyplay has finished
  cat "$fifo"
  # send ^Q
  printf $'\x11' >/proc/$$/fd/${ttypipe[1]}
  # close pipe to coproc
  eval "exec ${ttypipe[1]}<&-"
  # Drain pipe from coproc
  #eval "exec ${ttypipe[0]}<&-"
  eval "cat /proc/$$/fd/${ttypipe[0]}"
  wait $! # is coproc
  echo Done all $?
}

emit() {
  rm -f "$QUITPIPE"
  echo "Playing $1" >&3
  ttyplay "$1"
  echo "Finished playing $1 [$?]" >&3
} 3>"$QUITPIPE"

main() {
  if test -z "$QUITPIPE"
  then ttystudio "$@"
  else emit "$TTYPLAYFILE"
  fi
}

main "$@"

It works by tricking ttysudio to re-invoke itself as the shell for what would have been the interactive subshell. As arguments can't be passed to the subshell, it smuggles them as environment variables.

When invoked as the subshell it simply uses ttyplay to re-play the ttyrec file.

Thursday, 18 July 2019

readline with bash's read -e

Want to use bash's read -e for full editing and readline support, but without bash completion or exposing the bash history?

bind -f /dev/stdin <<<"set disable-completion on"
HISTSIZE=0 HISTIGNORE="*" HISTFILE=/dev/null read -e ...

Thursday, 6 June 2019

synchronous pipe based task monitoring

I want a process to monitor another and know when it quits.

An obvious way if I control the launch is to tie them together with a pipe and they can detect when the pipe closes.

Maybe the other process would launch many further processes, all inheriting the pipe-fd, which I want to avoid.

So clearly the launcher needs to hold the pipe-fd but not share it (close-on-exec, don't fork too much), and then wait in the usual way for the launched process to quit, and then close the pipe.

Here's a bash incantation lifetime_fd which runs a simple list of arguments and can share an fd with another process through process substitution.

# just run a simple list, without affecting $?
just() {
set -- $? "$@"
  "${@:2}"
  return $1
}

lifetime_fd() {
  set -- $_ "$@" ; eval "$1>&-" '"${@:2}"' ; just eval exec "$1>&-"
}

So if you want to run command fooly barly bazly and link it to the lame read && echo done then this will do the trick

lifetime_fd fooly barly bazly {_}> >( read && echo done )

So to attach a pipe descriptor to a sub-process, it is clear to use the trailing invocation {_}> >( sub-process ) which will attach stdin of the sub-process to a file descriptor to be stored in $_ which is managed in lifetime_fd

The variable $_ is used to avoid messing with any other variables. $_ is constantly adjusted and should do no harm if we abuse it; but as it is constantly adjusted, the first thing we do in lifetime_fd is to save it.

We don't use a local variable in case of a name clash that affects something else, so we store it as $1

We then run "$@" (or "${@:2}" as it now would be) but with the fd closed, so that it is not inherited.

We then close the fd while preserving the exit code.

You can invoke it in a pipeline like this:

get_report_request | lifetime_fd get_report {_}> >( monitor ) | send_report

An illustrative example of monitor (which reads until eof), might be:

monitor() {
  while read -t 1 || test $? = 142 # 142 is timeout code
  do echo -n '*'
  done
}

which displays a star every second until stdin closes; by continually waiting up to 1 second to fail to read anything from stdin (until it closes, having a different exit code), and displays a star.

Of course it might read other data too.... if you can send it...

Monday, 25 February 2019

The Old Summer House


The Old Summer House

Sam Liddicott
The Fat Controller didn't seem at all bothered when he heard that Diesel had lost all the jobi wood into the sea. Diesel didn't seem too worried either, though a trifle scared at nearly falling into the sea himself.
Instead of salvaging what he could, the Fat Controller declared "More jobi wood must be found", "the search and rescue will be delayed".
My suspicious were not aroused, this waste of resources was typical on the Island of Sodor railway.
But my suspicions were aroused when Rocky the large breakdown crane happened to mention one morning that he was surprised just how deep the water was where Diesel lost the jobi wood. Rocky had come to the island to help put Gordon back on the rails, and wasn't involved in the jobi incident, and it was Thomas that had rescued Diesel. So what did Rocky know about it?
Later that year, after the search and rescue centre was complete, the Fat Controller invited everyone to a summer garden party, and to formally open his new summer house and gazebo.
I was there of course, as were most of the engines. Goodness knows how much it cost to lay the siding so that they could attend. It's hardly putting the shareholders money to good use.
I examined the summer house and gazebo. They were made-to-measure, and done very nicely, but the wood was rather cheaper than I expected, and not properly finished. "It must be painted with preservative" said the fat controller, "but there was not time to do it before the party". I carved my initials behind one the posts at the back. They would remember that "I woz ere".
The works manager was very pleased as the fat controller had promised him the old summer house and gazebo, if he would come and collect it. I went with him to help load the disassembled pieces from the front of the Fat Controllers driveway, and onto the back of a lorry, and then helped to assemble it.
"It's not very large!" said the works manager after we had finished assembling it a few days later, "but it will make a good shed for my model railway, though it is in worse condition than I expected."
As we sat looking at it with out drinks, I got to thinking about the poor use of shareholders money in laying a siding to the Fat Controllers house, and I decided to pay another visit.
As I walked up the driveway, I saw Harvey on the new siding by the Fat Controllers summer house, tearing up the track behind him.
The Fat Controller's wife came out showed me into the garden where the Fat Controller sat under the gazebo with the company auditor enjoying some drinks together, and I was cordially invited to join them.
"Our Fat Controller has rather carelessly cut himself carving some wood!" explained the auditor, pointing towards the bandaged.
The newness of the wound was evident by the bright red mark leaking through, whose colour I thought matched a bloodstain on the floor at the back of the post just where I could just see my initials carved, if I lent back in my chair.
After some talk about assets and amortisations which I could not understand, the Fat Controller excused himself, and so the auditor and I left.
"That's rather a nice summer house," I remarked, and a lovely colour.
"Yes," says the auditor, "it required the treatment of a special preservative which I understand affects it in a remarkable way, and imparts a uniquely heavy feel to the wood. It should last a long time"
And then, after a thought he added as we reached a lorry at the end of the driveway, "the Fat Controller has given me his old summer house, and I understand that you have experience assembling this sort of thing and may be able to help".
One of the posts had my initials carved on it. I noticed this as the auditor slapped the tarpaulin and said "You can fit a lot of bad boys on this one"
I noped out of there.