Friday, 23 October 2020

dd is a pretentious fraud, don't trust it except for disks, files, and tape

Especially busybox dd.

[Note, specifying ibs and obs separately seems to work as expected (except with busybox dd). Using bs where ibs and obs have the same value is not a good idea]

 I used to feel guilty writing to the tape drive with cat: cat /tmp/backup.tar.gz > /dev/rct0

What dd was good at, (I had somehow learned by osmosis) was reading and writing full block sizes where the device driver can't do it so well.

What nonsense.

dd doesn't to "impedance matching" of block sizes, it has barely any regard for them. 

This all of a sudden mattered when I was using a shell to set efi variables using the efivarfs file system:

printf "\x07\x00\x00\x00%s" "$var-value" > "my-var-${owner-UUID}"

That works fine, but this was a problem:

some-process | cat /dev/fd/2 2<<<$'\x07\x00\x00\x00' - > "my-var-${owner-UUID}"

why? Because the data written to the variable pseudo-file had to be written all-in-one-go, and as this example shows, it might not happen, the output is written in two parts:

( echo hello ; sleep 1 ; echo goodbye ) | cat /dev/fd/2 2<<<$'\x07\x00\x00\x00' - | cat | cat

I supposed that this would be ideal for dd, I set the block size to 4096 which is the maximum write to the efi-var psuedo files anyway, giving:

( echo hello ; sleep 1 ; echo goodbye ) | \
cat /dev/fd/2 2<<<$'\x07\x00\x00\x00' - | dd bs=4096 of="my-var-${owner-UUID}"

but it didn't do the trick, strace showed multiple writes from dd, as we can also see here

( echo hello ; sleep 1 ; echo goodbye ) | cat /dev/fd/2 2<<<$'\x07\x00\x00\x00' - | dd bs=4096 | cat | cat

it turns out that dd doesn't care if it does a partial read from a pipe, it takes the block size as a maximum hint, rather than a requirement.

So... dd is ostensibly good at impedance matching of block sizes, lets set an input block size of 1 and an output block size of 4096 and let it accumulate the input blocks for output:

( echo hello ; sleep 1 ; echo goodbye ) | cat /dev/fd/2 2<<<$'\x07\x00\x00\x00' - | dd bs=1 obs=4096 | cat | cat

exactly the same result! dd just doesn't care!

GNU has the life-saving non-posix iflag=fullblock which actually reads a full input block (unless eof).

Without which, what is the point of dd? Well yes, we know it has incidental features such as very limited character set conversion, along with skip and seek, and unlike head, tail, etc, it won't read more input than it intends to use (which makes it very convenient for reading some of stdin from a shell script).

But it's main purpose is unmet, and with solid risk in some circles. Want to generate a random password for openssl?

password=$(dd if=/dev/random bs=32 count=1 | base64)

do you see the danger yet? What if dd blocks for want of randomness, dd will return a partial block!

simulate thus, try to read 12 random bytes and see how many we might get:

( printf "a" ; sleep 1 ; printf "bc" ; sleep 1 ; printf "def" ; sleep 1 ; printf "ghij" ; sleep 1 ; printf "klmno" ) | dd bs=12 count=1 | wc -c

we get one character instead of 12 (or instead of 256 or however many you expected)

The fix is to swap bs and count, so we keep reading 1 byte until we have enough

( printf "a" ; sleep 1 ; printf "bc" ; sleep 1 ; printf "def" ; sleep 1 ; printf "ghij" ; sleep 1 ; printf "klmno" ) | dd bs=1 count=12 | wc -c

dd, you are a pretentious ass, without the GNU extension you cannot read lots of small blocks and write one big block, and even on a non-terminating stream with more data becoming available, you will read less than block size multiplied by count, and think yourself smart!

Except with disks, files, and tape, where the driver managers block size, and partial reads won't happen, you can't be trusted to get your most basic task right.

Many interesting remarks at:

  • https://unix.stackexchange.com/questions/121865/create-random-data-with-dd-and-get-partial-read-warning-is-the-data-after-the
  • https://unix.stackexchange.com/questions/17295/when-is-dd-suitable-for-copying-data-or-when-are-read-and-write-partial
  • https://superuser.com/questions/520601/why-does-dd-only-copy-128-bytes-from-dev-random-when-i-request-more

Saturday, 21 March 2020

Working with ttystudio and ttyrec

The best of all the tty to animated gif converters, is ttystudio because it outputs small gifs.

Sadly it is slow, and uses up way to much memory -- even running out of memory for even a small session.

Ironically it boasts of not using imagemagick while requiring node! No wonder it runs out of memory. Sadly the frams also seem to be based on clock time instead of ttyrec frames.

Even worse, it only works for recording live sessions.

This shell script causes ttystudio to convert a ttyrec file.

Use: ./ttyrec2studio ttyrecfilename
to make ttyrecfilename.gif


#! /bin/bash
ttystudio() {
  local ttyrec="$1"
  local fifo=$(mktemp -u)
  mkfifo "$fifo"

  coproc ttypipe { SHELL=$(realpath "$0") QUITPIPE="$fifo" TTYPLAYFILE="$ttyrec" command ttystudio "$ttyrec.gif" >/dev/tty 2>&1 ; }

  # when we've finished with fifo, ttyplay has finished
  cat "$fifo"
  # send ^Q
  printf $'\x11' >/proc/$$/fd/${ttypipe[1]}
  # close pipe to coproc
  eval "exec ${ttypipe[1]}<&-"
  # Drain pipe from coproc
  #eval "exec ${ttypipe[0]}<&-"
  eval "cat /proc/$$/fd/${ttypipe[0]}"
  wait $! # is coproc
  echo Done all $?
}

emit() {
  rm -f "$QUITPIPE"
  echo "Playing $1" >&3
  ttyplay "$1"
  echo "Finished playing $1 [$?]" >&3
} 3>"$QUITPIPE"

main() {
  if test -z "$QUITPIPE"
  then ttystudio "$@"
  else emit "$TTYPLAYFILE"
  fi
}

main "$@"

It works by tricking ttysudio to re-invoke itself as the shell for what would have been the interactive subshell. As arguments can't be passed to the subshell, it smuggles them as environment variables.

When invoked as the subshell it simply uses ttyplay to re-play the ttyrec file.

Thursday, 18 July 2019

readline with bash's read -e

Want to use bash's read -e for full editing and readline support, but without bash completion or exposing the bash history?

bind -f /dev/stdin <<<"set disable-completion on"
HISTSIZE=0 HISTIGNORE="*" HISTFILE=/dev/null read -e ...

Thursday, 6 June 2019

synchronous pipe based task monitoring

I want a process to monitor another and know when it quits.

An obvious way if I control the launch is to tie them together with a pipe and they can detect when the pipe closes.

Maybe the other process would launch many further processes, all inheriting the pipe-fd, which I want to avoid.

So clearly the launcher needs to hold the pipe-fd but not share it (close-on-exec, don't fork too much), and then wait in the usual way for the launched process to quit, and then close the pipe.

Here's a bash incantation lifetime_fd which runs a simple list of arguments and can share an fd with another process through process substitution.

# just run a simple list, without affecting $?
just() {
set -- $? "$@"
  "${@:2}"
  return $1
}

lifetime_fd() {
  set -- $_ "$@" ; eval "$1>&-" '"${@:2}"' ; just eval exec "$1>&-"
}

So if you want to run command fooly barly bazly and link it to the lame read && echo done then this will do the trick

lifetime_fd fooly barly bazly {_}> >( read && echo done )

So to attach a pipe descriptor to a sub-process, it is clear to use the trailing invocation {_}> >( sub-process ) which will attach stdin of the sub-process to a file descriptor to be stored in $_ which is managed in lifetime_fd

The variable $_ is used to avoid messing with any other variables. $_ is constantly adjusted and should do no harm if we abuse it; but as it is constantly adjusted, the first thing we do in lifetime_fd is to save it.

We don't use a local variable in case of a name clash that affects something else, so we store it as $1

We then run "$@" (or "${@:2}" as it now would be) but with the fd closed, so that it is not inherited.

We then close the fd while preserving the exit code.

You can invoke it in a pipeline like this:

get_report_request | lifetime_fd get_report {_}> >( monitor ) | send_report

An illustrative example of monitor (which reads until eof), might be:

monitor() {
  while read -t 1 || test $? = 142 # 142 is timeout code
  do echo -n '*'
  done
}

which displays a star every second until stdin closes; by continually waiting up to 1 second to fail to read anything from stdin (until it closes, having a different exit code), and displays a star.

Of course it might read other data too.... if you can send it...

Monday, 25 February 2019

The Old Summer House


The Old Summer House

Sam Liddicott
The Fat Controller didn't seem at all bothered when he heard that Diesel had lost all the jobi wood into the sea. Diesel didn't seem too worried either, though a trifle scared at nearly falling into the sea himself.
Instead of salvaging what he could, the Fat Controller declared "More jobi wood must be found", "the search and rescue will be delayed".
My suspicious were not aroused, this waste of resources was typical on the Island of Sodor railway.
But my suspicions were aroused when Rocky the large breakdown crane happened to mention one morning that he was surprised just how deep the water was where Diesel lost the jobi wood. Rocky had come to the island to help put Gordon back on the rails, and wasn't involved in the jobi incident, and it was Thomas that had rescued Diesel. So what did Rocky know about it?
Later that year, after the search and rescue centre was complete, the Fat Controller invited everyone to a summer garden party, and to formally open his new summer house and gazebo.
I was there of course, as were most of the engines. Goodness knows how much it cost to lay the siding so that they could attend. It's hardly putting the shareholders money to good use.
I examined the summer house and gazebo. They were made-to-measure, and done very nicely, but the wood was rather cheaper than I expected, and not properly finished. "It must be painted with preservative" said the fat controller, "but there was not time to do it before the party". I carved my initials behind one the posts at the back. They would remember that "I woz ere".
The works manager was very pleased as the fat controller had promised him the old summer house and gazebo, if he would come and collect it. I went with him to help load the disassembled pieces from the front of the Fat Controllers driveway, and onto the back of a lorry, and then helped to assemble it.
"It's not very large!" said the works manager after we had finished assembling it a few days later, "but it will make a good shed for my model railway, though it is in worse condition than I expected."
As we sat looking at it with out drinks, I got to thinking about the poor use of shareholders money in laying a siding to the Fat Controllers house, and I decided to pay another visit.
As I walked up the driveway, I saw Harvey on the new siding by the Fat Controllers summer house, tearing up the track behind him.
The Fat Controller's wife came out showed me into the garden where the Fat Controller sat under the gazebo with the company auditor enjoying some drinks together, and I was cordially invited to join them.
"Our Fat Controller has rather carelessly cut himself carving some wood!" explained the auditor, pointing towards the bandaged.
The newness of the wound was evident by the bright red mark leaking through, whose colour I thought matched a bloodstain on the floor at the back of the post just where I could just see my initials carved, if I lent back in my chair.
After some talk about assets and amortisations which I could not understand, the Fat Controller excused himself, and so the auditor and I left.
"That's rather a nice summer house," I remarked, and a lovely colour.
"Yes," says the auditor, "it required the treatment of a special preservative which I understand affects it in a remarkable way, and imparts a uniquely heavy feel to the wood. It should last a long time"
And then, after a thought he added as we reached a lorry at the end of the driveway, "the Fat Controller has given me his old summer house, and I understand that you have experience assembling this sort of thing and may be able to help".
One of the posts had my initials carved on it. I noticed this as the auditor slapped the tarpaulin and said "You can fit a lot of bad boys on this one"
I noped out of there.

Tuesday, 11 September 2018

I Speak in Kingsley

I walked up and down
through the town, and round and round
and wondered what to say and why to say it

Then I purposed with an air
of sureness, and took care,
That when I did it, saying said it, I would say it.

Unfortunately it didn't work out like that
because I stubbed my toe
And said some other things instead,
which didn't really go,
I was quite embarrassed and my mother, she was too
And I really couldn't think of what I ought to do-
to break the silence that I started, so:

                                                        I screamed:
                      "Oh look, ohh look behind you!"
                      So they did, and when they had,
I took off running down the street
the pit pat padding of my feet
was all I heard besides my panting
'till I heard a train man chanting
"All aboard for Kingsley Station, time to go, the train is waiting"

So I climbed in and took a seat, nonchalantly trying not to attract attention;
which I can do quite well
(unless I stub my toe)
and the train started to go.

So I waited, looking out of the window as we went,
my foot going clackety clack,
my mind keeping time with the train,
and it started to rain

I was finally starting to calm down when we came to the next station,
and who do you think was waiting
with my mother
to get on the train?

I considered my solutions,
and came to my conclusion,
and operating smoothly I got into the toilet compartment,
waited till we started, pulled the cord and the alarm went -
ringing
"Stop the train" the cry went up,
the brakeman working frantically, a scraping screaming sound,

and stop we did! Stopped dead on the tracks
Nobody knew why so after a while we went again,
with a chuff chuff sound, and I carefully jumped out
landing softly on the ground, but as I stood and rose to my height,
past me went a window, and there, pressed to the glass,
were the faces of my mother and her three friends

Sam Liddicott
March 1992














I Wander

I wondered as I wandered
what was what,
and who was where,
And I thought a little more
as I wandered up the stair
and 'cos I wasn't looking
I slipped on three pairs of roller blades and two yin and a yang,
I bumped my pore head     and I trod on my hand
I didn't much like it, I shan't say what I said
But that should explain why I'm sitting in bed.

I should be out working
it's true that I'm not,
when I trod on my hand
I'm not sure what I got,
But it got all twisted
and pulled here and there,
which all should explain
why not to play on the stair

Sam Liddicott
December 1991 and March 1992