Sunday, April 20, 2014

MAKEDEV and double virtualization

MAKEDEV can run in a couple of seconds - even on a virtual machine.

But if your virtual machine is hosting a qemu guest, then in that guest a fork/exec can take 0.2 of a second, and MAKEDEV generic-i386 can take a couple of hours. (This is because the kqemu kernel module is not available, http://www.linuxquestions.org/questions/linux-virtualization-and-cloud-90/qemu-running-on-ubuntu-vmware-guest-cannot-find-dev-kvm-936253/).

The first hacky-hack to cut down on the number of fork/exec is to stop calling sed quite so often, cue this patch (requires MAKEDEV to run under bash).

--- /sbin/MAKEDEV 2009-07-30 08:39:09.000000000 -0700
+++ /sbin/MAKEDEV 2014-02-21 06:31:43.000000000 -0800
@@ -1,4 +1,4 @@
-#! /bin/sh -
+#! /bin/bash -
 # $Id$
 
 #---#---#---#---#---#---#---#---#---#---#---#---#---#---#---#---#---#---#---#
@@ -116,7 +116,9 @@
 
 devicename () { # translate device names to something safe
  # A-Z is not full alphabet in all locales (e.g. in et_EE)
- echo "$*" | LC_ALL=C sed -e 's/[^A-Za-z0-9_]/_/g' 
+ #echo "$*" | LC_ALL=C sed -e 's/[^A-Za-z0-9_]/_/g' 
+ echo "${*//[^ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789]/_}"
+  
 }
 
 makedev () { # usage: makedev name [bcu] major minor owner group mode
@@ -231,12 +233,14 @@
  exec 3<$procfs/devices
  while read major device extra <&3
  do
-  device=`echo $device | sed 's#/.*##'`
+  #device=`echo $device | sed 's#/.*##'`
+  device="${device%%/*}"
   case "$major" in
    Character|Block|'')
     ;;
    *)
-    safedevname=`devicename $device`
+    safedevname="${device//[^ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz01234566
+789]/_}"
     eval "major_$safedevname=$major"
     devices="$devices $device"
     ;;
@@ -247,7 +251,8 @@
 
 Major () {
  device=$2
- devname=`devicename $1`
+ devname="${1//[^ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz01234566
+789]/_}"
  if [ "$opt_d" ]
  then
   echo -1 # don't care
@@ -2149,7 +2154,8 @@
    exec 3<$procfs/devices
    while read major device extra <&3
    do
-    device=`echo $device | sed 's#/.*##'`
+    # device=`echo $device | sed 's#/.*##'`
+    device="${device%%/*}"
     case "$major" in
sh-4.1# diff -u /sbin/MAKEDEV /MAKEDEV 
--- /sbin/MAKEDEV 2009-07-30 08:39:09.000000000 -0700
+++ /MAKEDEV 2014-02-21 06:31:43.000000000 -0800
@@ -1,4 +1,4 @@
-#! /bin/sh -
+#! /bin/bash -
 # $Id$
 
 #---#---#---#---#---#---#---#---#---#---#---#---#---#---#---#---#---#---#---#
@@ -116,7 +116,9 @@
 
 devicename () { # translate device names to something safe
  # A-Z is not full alphabet in all locales (e.g. in et_EE)
- echo "$*" | LC_ALL=C sed -e 's/[^A-Za-z0-9_]/_/g' 
+ #echo "$*" | LC_ALL=C sed -e 's/[^A-Za-z0-9_]/_/g' 
+ echo "${*//[^ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789]/_}"
+  
 }
 
 makedev () { # usage: makedev name [bcu] major minor owner group mode
@@ -231,12 +233,14 @@
  exec 3<$procfs/devices
  while read major device extra <&3
  do
-  device=`echo $device | sed 's#/.*##'`
+  #device=`echo $device | sed 's#/.*##'`
+  device="${device%%/*}"
   case "$major" in
    Character|Block|'')
     ;;
    *)
-    safedevname=`devicename $device`
+    safedevname="${device//[^ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz01234566
+789]/_}"
     eval "major_$safedevname=$major"
     devices="$devices $device"
     ;;
@@ -247,7 +251,8 @@
 
 Major () {
  device=$2
- devname=`devicename $1`
+ devname="${1//[^ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz01234566
+789]/_}"
  if [ "$opt_d" ]
  then
   echo -1 # don't care
@@ -2149,7 +2154,8 @@
    exec 3<$procfs/devices
    while read major device extra <&3
    do
-    device=`echo $device | sed 's#/.*##'`
+    # device=`echo $device | sed 's#/.*##'`
+    device="${device%%/*}"
     case "$major" in
      Character|Block|'')
       ;;


The next hack is to stop the 5 or 6 fork/exec when MAKEDEV deletes a device node, creates a (temporary device node), chmods it, chowns it, and renames it.

How do we do that? We run MAKEDEV -n so that it does none of these and then feed the output to a perl script which will do them.

Sadly perl has no native mknod call and there are no libraries in this environment, and so I use perl's blessed syscall function with a hard-wired syscall 133 for mknod (as it is, on my kernel):

time /MAKEDEV -n generic-i386 | perl -ne '
print; 
umask(0);
($c, $f, $node, $major, $minor, $own, $perm) = split; 
if ($node eq "->") { 
  symlink($major, $f) || die "sym: $!";
} else {
  # system("mknod",$f,$node,$major,$minor) && die "mknod($f,$node,$major,$minor): $? $!"; 
  # chmod(oct($perm), $f) || die "chmod: $!"; 
  $n=0;
  $n=0010000 if ($node eq "f");
  $n=0020000 if ($node eq "c");
  $n=0060000 if ($node eq "b");
  $n=0140000 if ($node eq "s");
  if (syscall(133, $f, oct($perm) | $n, (($minor & 0xff) | (($major & 0xfff) << 8)
          | (( ($minor & ~0xff)) << 12)
          | (( ($major & ~0xfff)) << 32))) == -1) { die "syscall: $!"; }
  ($user,$group)=split(/:/,$own);
  $user=getpwnam($user);
  $group=getgrnam($group);
  chown($user,$group,$f) || die "chown: $own $!"; 
}
'

Leaving in the system("mknod,...) took around 24 minutes, but moving straight to syscall(mknod,...) takes1 minute 8 seconds.

I did toy with having bash use printf or something to pack a binary tar archive (or even getting perl to pack a tar archive) to pipe to tar -x, but... this will have to do for now.

I suspect this depends on a bug in MAKEDEV which seems to still create sub-directories needed even in -n mode.

It would have been better to pipe the output to a c program which would parse it, I will do that, another day...

Wednesday, March 19, 2014

NFS/LDAP only at home

I have this pre-init script in /etc/init/autofs.conf

start on runlevel [2345] and net-device-up IFACE=wlan0
...
pre-start script
    modprobe -q autofs4 || true
    if iwconfig | grep 00:18:84:26:7E:1B 
    then sed -e 's/ #ldap/ ldap/' /etc/nsswitch.conf > /var/tmp/nsswitch.conf && mount /var/tmp/nsswitch.conf /etc/nsswitch.conf -o bind
    else ! true
    fi
end script

and these lines in my /etc/nsswitch.conf

passwd:         compat #ldap
group:          compat #ldap
shadow:         compat #ldap

When I'm on my home wireless network (detected by the access-point mac address, autofs will run to mount home directories, and ldap will be enabled and used to validate user accounts.

Otherwise it is a standalone laptop and will revert to built in accounts and home directories.

Tuesday, February 25, 2014

A Christmastime Computer Ghost Story, or, a loop in time

The poor coder was running out of time for his deadline, and right before the Christmas holidays

The new system was not working. PAM had been updated to a newer version which choked on the old style pam.d files which used the old @ style include directives.

This meant that CRON was failing with permission denied because it's config file could not be processed.

Examination of the source version control system showed that old versions of the system used the same new version of pam and yet did not have that problem at all.

The pam.d config files were not supplied as part of the pam source, and so had come from somewhere else - and came from the base file system on which the system was based.

An older version of the system was fired up, and sure enough, it had a different set of pam.d files to cope with the newer pam release that was also in use on the older system.

It seemed simpler to just transfer the pam.d files from the old system to the new system - and indeed that fixed the problem.

These were then added as a fixup to the pam project, so that the new pam.d files would be installed whenever the new pam project was used. Otherwise, the original files would still be used.

The build system was tested and it did indeed deposit the new pam.d files in the correct place.

A new build was fired up and just before delivery for QA was found to fail in exactly the same way as before.

Things get spooky...

Analysis showed that the old pam files were installed after all; and this was because the old pam files had been packed in the install image, and this was because the fixup to the pam project contained the old pam files.

What?

Didn't the coder fix that to have new pam files the day before? Version control showed that he did, but version control showed that the fixup archive contained the old files, known not to work; and only those files were committed.

The previous system was fired up again, and to his surprise it was using the old pam files too and yet was NOT failing - because it was using the old pam libraries, and not the new pam libraries!

The command history showed that it's pam files had been packaged up using tar ... | base64, for cut-n-paste from window to window, so there was sadly not left behind any temporary files of evidence of what was actually packaged. But it was clearly the machine from which the files were taken.

The new system on which the replacement pam files had been tested had been re-installed to test the new build, and so it had the newly installed old broken pam files and not the new ones that had been proved to work.

There had been a fix and it had worked but all that was left was the same old pam.d files that did not work. And nowhere to get a fix from.

The coder then remembered that the previous installation is always preserved on a backup partition, and so quickly mounted this on the new system and found the new, different, pam.d files - which worked.

The coder packaged these up to a fixup to the pam project, just like 2 days before.

And kept a back up just to be sure.

The coder was glad to have his fix in time for the deadline, and have his Christmas holidays, but if the newer fixed files had not come from the working older system (which did not use new pam after all), where had they come from... and where did they go to... But he did not let these questions prevent him from starting a new build for QA, and going off to enjoy his long anticipated holidays.

Monday, January 27, 2014

A Plea for Social Equality

True democratic wealth is time: everybody has 24 hours of it a day.

The significant social inequality of the day is that some peoples time is not exchangeable for much value, while other peoples time is very much in demand and exchangeable at a high rate.

I simplify somewhat, for a person can sometimes exert control over their exchange rate by means of how much effort they put in; but even at maximum effort there is still a large disparity between the exchangeable values of the time of different people, or in different locations.

In the corrupt imperial west there is a notion that 8 hours of effort per day should be exchangeable for necessities of life (and some discretionary comforts) but the opportunity to make this exchange is not granted equally to all people (and nor is the standard of life).

In the name of social conscience and humanity, this inequality is often reasonably addressed by moving a surfeit of value from those who found favourable exchange, to those who who found a poor rate of exchange.

Yet paradoxically, the extreme inequality of those who found no exchange is addressed by an attempt to provide the value of life necessities with no exchange required.

This does not introduce equality but instead introduces further inequality for in transferring the surfeit of value to those who find no exchange, it leaves them with a surfeit of time. It reduces them to the status of beggar and provides no way to contribute.

It would be more equitable to provide opportunity of exchange to those who found no exchange, making all equal contributors to society from the universal and collective wealth of time, and condemning none to the status of beggar.

TLDR: Or in other words, why in the UK when taxes are going up, and public works are going down, why, are we paying people to do nothing?

It is no socialist triumph to lose the working class and gain another idle class (however unwilling).

Let the time of each be valued equally - sufficient for life, and let all contribute.

And here's hoping for a shorter working day for all, enough time for dancing and singing.

Friday, November 29, 2013

Makefile mash

The dumbest thing so far seen in a makefile:

ECHO      = `which echo`

Apart from the apparent fact that the writer assumes that the external command which is on the path but echo might not be (hint: they don't want to use the built-in echo for whatever reason), the main error is that everywhere $(ECHO) is used it will be replaced with `which echo`

The author probably intends

 ECHO      := $(shell which echo)

which will define ECHO to be something like /bin/echo which is what they intended.

Note the := otherwise which echo would be evaluated each time $(ECHO) is used.


Friday, October 4, 2013

TalkTalk moble data configuration

Today I learned that TalkTalk mobile users can txt 40127 with: TALK and get the configuration sent back.

Monday, September 16, 2013

Why are setjmp /volatile hacks STILL needed?

We all know the old gotcha of setjmp (but I'm going to recount it anyway) which is that when longjmp is executed (and setjmp appears to complete again, but this time with a non-zero result) some (or all) of the original register set will be restored.

Any local variables which were (possibly temporarily) stored in the local registers reserved for such use could appear to revert back to their previous values until they are re-freshed from the stack frame.

And so all pre-existing local variables that would be accessed after the longjmp should be declared volatile so that they will not be fetched from the registers.

Well... why is there no special #pragma or other option attached to the setjmp function indicating a register clobber-list to the compiler that all the local registers could have been modified and it should not depend on cached values any more?

Why do we need such a pragma? Why can't we #define setjmp to be something like:

#define setjmp(env) (setjmp(env) + __asm__ volatile ("mov %%eax,0" : : : "memory" "ebx" "ecx" "etc..." ));

I'm sure that example is insufficient, but I also think that some working method could be contrived, so I'll work on it...

and then...


OK, having tried; I realise that first problems are

  • not stopping the extra return from setjmp from using stale values in restored registers
  • that the correct values from the registers will not have been copied back into the stack frame, but will somewhere be pushed on the stack by a called function where we can't retrieve them

But the worst problem is that the optimiser can remove operations that ought to have been noticeable:

#include 
#include 

int getint() {
  static int i=3;

  i+=2;

  return i;
}

int main(void) {
  static jmp_buf j;
  volatile int x;
  int y;
  x = getint();
  y = getint();
  if (setjmp(j) == 1) {
    asm volatile("nop" : : : "memory", "ecx", "edx");
    printf("%d %d\n", x, y);
    return 0;
  }
  y++;
  x++;
//  printf("%d %d\n", x, y); // Second Printf
  longjmp(j, 1);
}

Unless the second printf is uncommented, the line for y++ is not emitted even at optimisation -O1
The reason is that y is not used after that point, even though the longjmp (acting somewhat like a goto) might have the effect of returning to a point where the y is used.

If I replace the longjmp with a goto then y++ is emitted in the code.

The first lesson: Use volatile in the way everyone says you should; all variables to be accessed in the second return of setjmp should be volatile.

The second lession, some contrivance of:
blah: if (setjmp(j) == 1) {
...
  if (never_true()) goto blah; else longjmp(j, 1);

generates the right code too by recognizing the effect of the goto rather than the longjmp however this cannot be sensibly managed as there can easily be multiple setjmp in a function.

So I still search for some means to cause a block of code to commit all temporary registers back to the stack frame before calling another function, and a means to cause all temporary registers to be flushed on second return from setjmp (although this may be covered by the first case).

further reading


I read http://stackoverflow.com/a/7734313 that:
The returns_twice attribute tells the compiler that a function may return more than one time. The compiler will ensure that all registers are dead before calling such a function and will emit a warning about the variables that may be clobbered after the second return from the function. Examples of such functions are setjmp and vfork. The longjmp-like counterpart of such function, if any, might need to be marked with the noreturn attribute.
So in fact gcc at least does evict any registers before calling setjmp but that doesn't prevent local variables being cached later on and not comitted to the stack frame before longjmp is called.

So I really just need a way to mark that functions might call longjmp and that temporary registers should be evicted before calling; but I know deep down that this is rubbish as longjmp might be called even from a signal handler.