Filtering stderr

This helper function will take $1 as a simple command to be used on stderr, on the rest of the command. The filtered output is emitted on stderr.

e.g. stderr "sed -e s/^/tar: /" tar -xvzf -

The function is short, but obscure, making use of a few tricks

stderr() {
  { set -- $_ "$@" ; } {_}>&1

  { eval '"${@:3}"' "$1>&-" ; } 2>&1 >&${1} | eval '$2' ">&2" "$1>&-" 


  set -- $1 ${PIPESTATUS[0]} "${@:2}"
  eval "exec $1>&-"
  return $2
}

An explanation is here:

stderr() {
  { set -- $_ "$@" ; } {_}>&1

The first line uses the temporary variable _ (underscore), which cannot generally be relied upon, but is safe enough in this context. This variable is used to avoid this helper leaving any imprint. Trampling on variables or declaring any local variables could affect destroy transparency and potentially affect the rest of the script.

So _ becomes a copy of stdout; and then inside the { ... } we update the function arguments so that this copy of stdout is now argument 1.

The function arguments are the only lexically scoped variables in bash. We can set them here in this function knowing that they will not have any other affect anywhere else.

So $1 now refers to a copy of stdout, $2 is now the filter to be applied to stderr, and "${@:3}" ($3 and onwards) is the command to be filtered.

We want to run the command with the $1 copy of standard out closed, in case the command spawns other processes that might inherit this copy and leave it open. Its a private copy, and as bash doesn't support close-on-exec we must close it.

We want to do this: "${@:3}" $1>&- but bash can't take a parameter variable on the left hand side of a redirector (not even as ${!1}) so we must use eval. We put the command in single quotes to prevent it being interpolated at all prior to eval, but the redirector is in double quotes so that the interpolated string is passed to eval; thus: eval '"${@:3}"' "$1>&-"

We want to run this command with stdout passed to the spare copy we made in $1  because we will redirect stderr to stdout to be fed into the filter. This redirection specification is: 2>&1 >&${1} (variables are allowed on the right hand side of a redirector).

However we can't append these redirectors to the previous one which already closed $1, so we use a brace scope { ... ; }  in which $1 is closed.

This gives us so far: { eval '"${@:3}"' "$1>&-" ; } 2>&1 >&${1} which has stdout going to our copy of stdout, and stderr going to actual stdout ready to pipe to the next stage.

The next stage also wants to close $1 for the same reason as before, and is:

eval '$2' ">&2" "$1>&-"

So the whole invocation is:

{ eval '"${@:3}"' "$1>&-" ; } 2>&1 >&${1} | eval '$2' ">&2" "$1>&-"

We now want to close $1  for the rest of the script without losing the exit code. As we have finished calling other commands we could save $? in a local variable, but I use the function arguments again to save as $2. Note that PIPESTATUS[0] holds the result of the first stage of the pipeline.

  set -- $1 ${PIPESTATUS[0]} "${@:2}"
  eval "exec $1>&-"
  return $2
}

And there it is.

Check status of entiire bash pipeline

My useful answer here: http://stackoverflow.com/questions/1221833/bash-pipe-output-and-capture-exit-status/34814471#34814471

pipestatus() {
  local S=(${PIPESTATUS[*]})

  if test -n "$*"
  then test "$*" = "${S[*]}"
  else ! [[ "${S[@]}" =~ [^0\ ] ]]
  fi
}

Note that S is not set to ("${PIPESTATUS[@]}"); this is so that we can re-create an array if PIPESTATUS is passed as a string, like this:

PIPE_STATUS="${PIPESTATUS[*]}" pipestatus

because an array cannot be passed in that fashion. Why would anyone want to do that? Probably not directly, but other helper commands just and also may want to preserve PIPESTATUS as best as possible to permit an: also pipestatus combination.

Usage examples:

1. get_bad_things must succeed, but it should produce no output; but we want to see output that it does produce

get_bad_things | grep '^'
pipeinfo 0 1 || return

2: all pipeline must succeed

thing | something -q | thingy
pipeinfo || return

Switch hdmi audio when TV is off using cec-client

I have a HDMI TV used as a computer monitor.

When the TV is turned on, the audio plays through the TV, so that the TV remote volume control works.

When  the TV is turned off, I want the audio to play through analog audio out so that I can still hear it.

I use the PulseEIGHT CEC controller and cec-client program to talk to the TV.

I have cec-client run as root under SOCAT to broadcast cec-data by UDP.

I have another SOCAT process running as the logged in user to pick up the data, monitor it for TV on/off status and move the playing audio streams.

It works, but is very rough.

In rc.local: /usr/local/bin/cec -s &

In gnome/cinnamon startup programs: /usr/local/bin/cec

And then the script:

#! /bin/bash

MYADDR=8 # because it is. I should probably read this from the cec-client output 
 
# 0 and 1 and the audio device indexes associated with the device names
# later I will normalise these too, but "pacmd list" will show yours.
ext() {
  pacmd set-default-sink alsa_output.pci-0000_00_1f.3.analog-stereo
  for input in $( pacmd list-sink-inputs | sed -e 's/index: //;t;d' )
  do pacmd move-sink-input $input 1
  done
}

hdmi() {
  pacmd set-default-sink alsa_output.pci-0000_01_00.1.hdmi-stereo-extra1
  for input in $( pacmd list-sink-inputs | sed -e 's/index: //;t;d' )
  do pacmd move-sink-input $input 0
  done
}

cec_client() {
  socat -u UDP4-RECV:4224,reuseaddr - | while read type _ _ dir rest
  do # echo "$type    $dir    :$rest"
     case "$rest" in
       "0f:36") echo "POWER OFF" ; ext ;;
       "(0): power status changed"*"to 'on'") echo "POWER ON" ; hdmi ;;
#       *"($MYADDR) as inactive source"*) echo INACTIVE ; ext ;;
#       *"($MYADDR) the active source"*) echo ACTIVATED ; hdmi ;;
#       *active*) echo "$type    $dir    $rest" ;;
# mmkeys-mate2mpris2
     esac
  done
}

cec_server() {
  read hostname _ < <( hostname )
  hex_hostname=$( echo -n "${hostname%.*}" | od -tx1 | sed -e '1!d;s/0* //' )

  <<< "tx 80 47 $hex_hostname" exec -a CEC socat -u -L/tmp/cec \
      EXEC:'cec-client -t p -p 2' UDP-DATAGRAM:127.0.0.1:4224,broadcast &

  wait
}

main() {
  if test "$1" = "-s"
  then cec_server
  else cec_client
  fi
}

main "$@"

LinuxMint 17.3 on dell XPS 8900

I'm having trouble installing Linux Mint 17.3 on my Dell XPS 8900

I re-sized the windows partition from within Windows, but the Ubiquity installer chokes on the partition table.

sda is my hard disk, and sdb is some intel SSD cache to go with the hard disk.

To avoid Ubiquity choking, I hide them, thus: (tips from http://unix.stackexchange.com/questions/102648/how-to-tell-linux-kernel-3-0-to-completely-ignore-a-failing-disk)

mint ~ # echo 1 > /sys/block/sda/device/delete
mint ~ # echo 1 > /sys/block/sdb/device/delete

and then install on a spare USB hard disk.

That worked, and I can boot off the USB hard disk by using F12 on power-up to get a boot menu.

I will later try to DD that installation to the spare partition, and somehow merge the UEFI partition.

I don't seem to need the tips here: http://askubuntu.com/questions/700617/problems-installing-ubuntu-14-04-3-on-dell-xps-8900

This may be useful: http://forums.linuxmint.com/viewtopic.php?f=90&t=211740&p=1104716

--

dmraid -ay

fixes the problem and supports Intel Matrix Raid which is the combination I have. http://www.linuxquestions.org/questions/ubuntu-63/intel-matrix-storage-manager-with-ubuntu-804108/

But when I came to install, having selected the new partition made under windows, it is confirmed as being /dev/mapper/blah_blah_5 (striped).

striped? So I daren't click yes. I would expect "sporadically mirrored" but not "striped".

One solution was to use "dmraid -ay" but install onto a separate internal SATA disk, and install the bootloader to the same partition I install onto, but the bootloader would not install on that disk, failing due to not being able to find the real mount point of /cdrom (!!) (and I don't want to install it onto SDA).

My working solution was to get a new internal SATA disk and connect it as an external USB disk and install on that, first using the "echo 1 > /sys/block/sda/device/delete" trick above on sda and sdb.

Then I move that disk to internal, and the UEFI boot entry still seems to work.

I will be repeating that trick later today when a spare SATA SSD arrives.

(I did try adjusting grub.cfg on /dev/sda1 or creating my own boot entries according to https://wiki.gentoo.org/wiki/Efibootmgr but could never get those to work).

--

Convert pdf to monochrome

From StackExchange

gs \
 -sOutputFile="BW-$1" \
 -sDEVICE=pdfwrite \
 -sColorConversionStrategy=Gray \
 -dProcessColorModel=/DeviceGray \
 -dCompatibilityLevel=1.4 \
 -dNOPAUSE \
 -dBATCH "$1"

Writes sample.pdf to BW-sample.pdf

Marriage of alloca and asprintf as asprintfa

It all comes down to not being able to return variable length arrays in C.

[Note: I realise that I am pushing obscenity to the limit, I don't really use this]

I think that asprintf is the way sprintf should work, but not everyone has it, and so folk either cart around an asprintf implementation, or enjoy getting the tedium of snprintf(NULL, 0, ...) or the worse error-prone tedium of adding up buffer sizes.

A single line  asprintf and corresponding  free() make a much simpler 1-2 lines of use.

Alloca

But while I'm not being portable it would be nice to be able to have asprintf use alloca as it's allocator...

alloca returns space from the current function stack frame (not scope) and is automatically freed when the function returns.

If alloca can't allocate enough memory, then who knows what might happen, but on the other hand If asprintf or malloc can't allocate enough memory, how does your program handle it?

So the plan is to alloca a buffer of the right size, and then sprintf into that.

This 1 line macro has the same syntax as the official asprintf.

#define asprintfa(PTR, ...) sprintf( (*(PTR)) = alloca(1 + snprintf(NULL, 0, __VA_ARGS__)), __VA_ARGS__)

and must be used with two lines:

char* x;
asprintfa(&x, "a %s b %d c\n", "<>", -1);

But I'd prefer a single line use:

char* x = asprintfa("a %s b %d c\n", "<>", -1);

So the plan is to alloca a buffer of the right size, and the right size is measured with 1 + nprintf(NULL, 0, ...). So we define a helpful macro:

#define measure_printf(...) snprintf(NULL, 0, __VA_ARGS__)

Once the size is measured, we can allocated with alloca and call sprintf(alloc, ...) to populate our alloc'd buffer. usual warning that macro arguments might get evaluated twice, but I don't see any other way around that without using a function call to stash the values, and require the heap or static buffers to hold them.

#define asprintfa(...) sprintf(alloca(1 + measure_printf(__VA_ARGS__)), __VA_ARGS__);

However, that will return whatever sprintf returns, which is the size, not the buffer. We need a temporary variable to hold the alloca pointer, and maybe return via a special GNU macro, like this:

#define asprintfa(...) ({ char* x=alloca(1 + measure_printf(__VA_ARGS__)); sprintf(x, __VA_ARGS__); x })

but that also requires us to come up with a variable name that will never be one of the __VA_ARGS__. And relying on alloca is bad enough without insisting on GNU extensions.

Maybe a tuple, like this:

#define asprintfa(...) ( char* x=alloca(measure_printf(__VA_ARGS__)), sprintf(x, __VA_ARGS__), x ) 

It's not a GNU extension but x still might clash with one of the __VA_ARGS__ and now we are declaring a variable in the middle of a scope.

It is simpler to define a wrapper to snprintf (or vsprintf ) that returns the buffer instead of the size. (Although perhaps a simple assembler push/pop would have saved the value nicely).

static char* a_sprintf(char* v, char* format, ...)
{
    if (! v) return NULL; // However alloca is undefined if it fails...
    va_list args;
    va_start(args, format);
    vsprintf(v, format, args);
    va_end(args);
    return v;
}

Which gives this winning combination which returns the string instead of the length (more useful):

#define measure_printf(...) snprintf(NULL, 0, __VA_ARGS__)
#define asprintfa(...) a_sprintf(alloca(measure_printf(__VA_ARGS__)), __VA_ARGS__);
/* sprintf wrapper that returns the buffer address. 
   Does not check size, intended to be used on a 
   properly allocated buffer as part of aasprintf */
char* a_sprintf(char* v, char* format, ...)
{
    if (! v) return NULL; // However alloca is undefined if it fails...
    va_list args;
    va_start(args, format);
    vsprintf(v, format, args);
    va_end(args);
    return v;
}

Here is another version that can be called like this:

char* x = asprintfa("a %s b %d c\n", "<>", -1);

it returns the pointer from the macro directly, but uses a stinky temp var along with GNU statement macro extensions again:

#define asprintfa(...) ({\
  char* __buf__;\
  sprintf(__buf__ = alloca(1 + snprintf(NULL, 0, __VA_ARGS__), __VAR_ARGS__);\
  __buf__; })

Variable Length Arrays

With this, you type:

char asprintfa(x, "a %s b %d c\n", "<>", -1);

and it expands to:

char x[1 + snprintf(NULL, 0, "a %s b %d c\n", "<>", -1)];
sprintf[x, sprintf("a %s b %d c\n", "<>", -1);

The code is:

#define asprintfa(name, ...) name[1 + snprintf(NULL, 0, __VA_ARGS__)]; \
sprintf(name, __VA_ARGS__);

Memory Allocation Failures

It doesn't warn you or fail sensibly if you try to print too much, but on the other hand it's 1 line to use and no need to worry about freeing.

Which is more likely?
With snprintf that you either:

• get the snprintf buffer arithmetic wrong
• or don't cope with allocating too much memory
• or forget to free the memory

or with asprintfa
that you

• allocates too much memory

There's certainly a lot less to get wrong, although admitedly you can't get it all right either - but there's asprintf for that.

sprintf was designed to get wrong, and snprint was designed to also get the arithmetic wrong.

And for completeness, asprintf (now tested):

int asprintf(char **v, const char* format, ...)
{ 
    int len = 0;
    if (! v) return -1;
 
    va_list args;
    va_start(args, format);
    len = vsnprintf(NULL, 0, format, args);
    va_end(args);
    va_start(args, format);
    *v = malloc(len + 1); 
    if (! *v) return -1;
       len = vsnprintf(*v, len + 1, format, args);
    va_end(args);
    return len;
}

[ Edit: Fixed loads of missing 1 + to the result of snprintf while measuring. snprintf returns just the number of characters written except the NULL character, but the length provided should also have space for the NULL.]

Packed binary in bash

Using bash, and other accessible commands, I wanted to output some based 64 encode packed binary string representations of 64 bit decimal numbers, in reverse byte order.

I used bc to convert the arbitrarily long decimal number into hex

bc <<< "obase=16; $N"

e.g.

$ N=123456789
$ bc <<< "obase=16; $N"
75BCD15

I then split this into hex digit pairs to represent one byte each, in reverse order, and prefixing with \x to produce a printf format string.  I do this by appending a space to the line, and then move two (or one, if two not available) digits that precede the space to instead append to the end of the string, (with \x prefix). And then finally, remove the space which is now a leading space.

sed -re 's/$/ /;:start;s/([a-fA-F0-9]{1,2}) (.*)/ \2\\x\1/;Tdone;bstart;:done;s/^ *//;'

e.g. the following pattern space is iterated

75BCD15

75BCD15␣

75BCD␣\15

75B␣\15\CD

7␣\15\CD\5B

␣\15\CD\5B\7

\15\CD\5B\7

Another method of separation might have been to pad with 0 to an even length, and then split off pairs of digits from the front. I then use all of that as the argument to printf which interpolates the characters, and then pipe to base64.

printf $( bc <<< "obase=16; $N" |
  sed -re 's/$/ /;:start;s/([a-fA-F0-9]{1,2}) (.*)/ \2\\x\1/;Tdone;bstart;:done;s/^ *//;'
) | base64

e.g.

$ N=123456789
$ printf $( bc <<< "obase=16; $N" |
  sed -re 's/$/ /;:start;s/([a-fA-F0-9]{1,2}) (.*)/ \2\\x\1/;Tdone;bstart;:done;s/^ *//;'
) | base64
Fc1bBw==

An alternate method for fixed with fields that don't need byte order fixings:
Convert to hex using printf:

printf -v hex '%08x' $dec
printf '%b' "\x${hex:0:2}" "\x${hex:2:2}" "\x${hex:4:2}" "\x${hex:6:2}"

To convert a string of space separated hex to characters:

input=" $input"
printf -v input '%b' ${input// /\\x}

.