Deleting Duplicate Files in Backups

rsync-backup (and other variants) can reduce backup size by referring to previous backups; although this can make the backup process slower and requires some fore-thought.

This script allows post-backup size reduction by detecting duplicate files in backups and making them share the same hardlink target, saving disk space.

We keep a placeholder link in MD5 folder. Each file whose MD5 file already exists in the MD5 folder is forced as a link to that MD5 file. Otherwise we link the file to create that MD5 file.

cd BACKUP-PATH
mkdir -p MD5
find . -path ./MD5 -prune -o -type f | xargs md5sum  | while read md5 file
do if test -f "MD5/$md5"
   then ln -f "MD5/$md5" "$file"
   else ln "$file" "MD5/$md5"
   fi
   echo "$md5 $file"
done

Because we keep an MD5 directory we don't need the md5 list to be sorted and we can re-run the script later on a smaller disk subset without needing to refer to the full MD5 list.

We can also easily examine the MD5 directory to see how many copies of a specific file exist and how much disk space is saved.

Any files in MD5 directory with only 1 link have been deleted from the normal file system tree and could also be deleted... but serve as a backup-backup!

Media Buttons on Mint 14 Nadia

Sadly keyboard multi-media control buttons still have no effect on Mint 14 when logged in using the Gnome2 fork Mate desktop.

Gladly, Matteo Italia has written a short python script to convert the DBUS events into MPRIS2 events.

http://pastebin.com/N0Jv2W5Y

Start the script on login and media keys work! Thanks Matteo

#!/usr/bin/env python

'''
Created on 30.05.2012

@author: Matteo Italia <mi_1@mitalia.net>
'''

import dbus
import dbus.mainloop.glib
import gobject

app_name = 'mmkeys-mate2mpris2'
Version=0.1

MediaKeysObjName = 'org.mate.SettingsDaemon'
MediaKeysObjectPath = '/org/mate/SettingsDaemon/MediaKeys'
MediaKeysInterface = 'org.mate.SettingsDaemon.MediaKeys'

MPRIS2Prefix = 'org.mpris.MediaPlayer2'

ActionMappings = {
        'Play': 'PlayPause',
        'Pause': 'Pause',
        'Stop': 'Stop',
        'Next': 'Next',
        'Previous': 'Previous'}


def onMediaKeyPress(app_name, action):
    sb = dbus.SessionBus()
    # Get the compatible players
    players = [n for n in sb.list_names() if n.startswith(MPRIS2Prefix + ".") ]

    # Send them the command
    for n in players:
        # TODO: it doesn't make sense to perform the action on *all* the players!
        # find a sensible criterion to choose the "best one"
        sb.get_object(n, '/org/mpris/MediaPlayer2').__getattr__(ActionMappings[action])()

if __name__ == '__main__':

    # DBUS boilerplate
    dbus.mainloop.glib.DBusGMainLoop(set_as_default=True)
    sb = dbus.SessionBus()

    # Get the media keys notificator object
    mediaKeysObj = sb.get_object(MediaKeysObjName, MediaKeysObjectPath)

    # Register to receive media keys notifications
    mediaKeysObj.GrabMediaPlayerKeys(app_name, 0, dbus_interface=MediaKeysInterface)
    mediaKeysObj.connect_to_signal('MediaPlayerKeyPressed', onMediaKeyPress)

    # Start the main loop
    mainLoop = gobject.MainLoop()
    mainLoop.run()



Discontinuous Existence

I had to laugh when I read about the discontinuous state of variables with fluid-let (on account of re-usable continuations).

It related nicely to the discontinuous existence of the planet earth as revealed in the extended trilogy, that wholly remarkable book, the Hitchhikers Guide to the Galaxy.

You can read about the discontinuities introduced with fluid-let and continuations here: http://www.gnu.org/software/mit-scheme/documentation/mit-scheme-ref/Dynamic-Binding.html but you should remember those largely friendly letters on the front of the other book: Don't Panic

I quote the key phrases here:

The extent of a dynamic binding is defined to be the time period during which the variable contains the new value. Normally this time period begins when the body is entered and ends when it is exited; on a sequential machine it is normally a contiguous time period. However, because Scheme has first-class continuations, it is possible to leave the body and then reenter it, as many times as desired. In this situation, the extent becomes non-contiguous. 

When the body is exited by invoking a continuation, the new value is saved, and the variable is set to the old value. Then, if the body is reentered by invoking a continuation, the old value is saved, and the variable is set to the new value. In addition, side effects to the variable that occur both inside and outside of body are preserved, even if continuations are used to jump in and out of body repeatedly.

I'm going to have fun with fluid-let but I just wonder about the hackiness that went into implementing that, and is (I suspect) related to dynamic-unwind.

nih, and alternative to talloc

libnih could contain an alternative to libtalloc

Latin1 to utf-8 without iconv

I recently helped someone convert latin1 text to utf-8 on a minimal system with no access to iconv.

A bash script had a latin1 field and needed to encode it to utf-8.

Fortunately, latin-1 only has 256 characters and only the top 128 are special, and (not that it makes any difference) most of those are the same.

The minimal system had busybox od command, so I decided to convert the variable to a numeric octal stream, like this:

$ read FIELD

Hello everybody I am the thing

Which can be converted to octal like this

$ echo "$FIELD" | od -b

0000000 110 145 154 154 157 040 145 166 145 162 171 142 157 144 171 040

0000020 111 040 141 155 040 164 150 145 040 164 150 151 156 147 012

0000037

and then strip to just the octal character values preceded by a space

echo "$FIELD" | od -b | sed -e 's/[^ ]*//;s/ *$//' 

 110 145 154 154 157 040 145 166 145 162 171 142 157 144 171 040

 111 040 141 155 040 164 150 145 040 164 150 151 156 147 012

and then join lines together

$ echo "$FIELD" | od -b | sed -e 's/[^ ]*//;s/ *$//' | tr -d $'\012'

 110 145 154 154 157 040 145 166 145 162 171 142 157 144 171 040 111 040 141 155 040 164 150 145 040 164 150 151 156 147 012

and then convert each space to $_lu_ which is a nice variable prefix

$ echo "$FIELD" | od -b | sed -e 's/[^ ]*//;s/ *$//' | tr -d $'\012' | sed -e 's/ /$_lu_/g'

$_lu_110$_lu_145$_lu_154$_lu_154$_lu_157$_lu_040$_lu_145$_lu_166$_lu_145$_lu_162$_lu_171$_lu_142$_lu_157$_lu_144$_lu_171$_lu_040$_lu_111$_lu_040$_lu_141$_lu_155$_lu_040$_lu_164$_lu_150$_lu_145$_lu_040$_lu_164$_lu_150$_lu_151$_lu_156$_lu_147$_lu_012




Now if all those variables were defined to hold the utf-8 values, we could convert the field, like this:

$ FIELD=$(eval echo \"$(echo -n "$FIELD" | od -b | sed -e 's/[^ ]*//;s/ *$//' | tr -d $'\012' | sed -e 's/ /$_lu_/g' )\")

as a bash function:

latin1_to_utf8() {
  eval echo -n \"$( <<<"$1" od -b | sed -e 's/[^ ]*//;s/ *$//' | tr -d $'\012' | sed -e 's/ /$_lu_/g' )\"
}

Here is how we define those variables; this code must be run on a fully-featured box with access to iconv.

for i in `seq 1 255`

do

  echo "_lu_$(printf "%03o" $i)"=\$\'$( printf $( printf '\\x%x' $i ) | iconv -f latin1 -t utf-8 | od -b | sed -e 's/[^ ]*//;s/ *$//;s/ /\\/g' )\'

done

and the text it outputs

...
...
_lu_176=$'\176'
_lu_177=$'\177'
_lu_200=$'\302\200'
_lu_201=$'\302\201'
_lu_202=$'\302\202'
...
...

Will be pasted into the script that runs on the reduced environment

Recover deleted photos

In March 2007, some missionaries called around with a sad tale - they'd accidentally deleted all the photos on their memory card, could I get them back?

I quickly knocked up this short perl script to do it.


#! /usr/bin/perl
# Quick hack by Sam Liddicott: <sam@liddicott.com> www.liddicott.com/~sam
# Try and look for jpeg images in a stacked file.
# Reads 512 blocks and when it finds one that starts with oxd8ff it starts a new file
# Invoke on a raw image of the card (maybe taken using rawread or rawrite) like this:
# perl recover.pl < IMAGE
# or
# perl recover.pl < /dev/sde

our $count=0;
our $file="/dev/null"; # use nul for dos/windows
our $buffer;

open STDOUT,'>',$file;

while (read(STDIN,$buffer,512)) {
  if ($buffer=~/^\377\330\377[^\333]/) {
    close(STDOUT);
    $count++;
    $file=sprintf("img_%03d.jpg",$count);
    print STDERR "$file\n";
    open STDOUT,'>',$file;
  }
  print STDOUT $buffer;
}

It did recover his photos.

Of course that \377\330\377 sequence was taken from /etc/magic - where else!?

The New Park

The New Park

(C) Sam Liddicott 2004

As we got near the park, my mother started to walk in the other direction, taking me with her.

This wasn't right; she had promised me that I could go to the park in the afternoon if I behaved in the morning, and I had behaved. I hadn't played with the scissors and cut the curtains, or got my shoes muddy when I played outside. I read my books quietly and put them away again, and I had eaten all my bread at lunchtime without complaining.

I had behaved, but just as we arrived at the park she started taking me somewhere else. I protested with a yell.

"Be quiet!" she said abruptly, which surprised me because my mother had been in a good mood when we set out. I was hoping we could stay longer at the park because it was sunny, and mum could sit on the bench and read her book.

"I want to go to the park, you said I could..." - it was worth a try, but she didn't seem to be in the mood. Why had she changed so suddenly?

I looked behind me towards the park as my mother dragged me along. There were some boys standing by the the red bus shelter in a group as they often did. I'd seen them before so I think they live nearby. One of them is called Harry. Maybe they were waiting to catch a bus to see their grandma, like we do with when dad has to stay at work late.

My mother dragged me on, still hurrying. I twisted my head as far as I could to see behind me. There was broken glass about the pavement. That was not unusual, sometimes the park had glass in it from broken bottles. I saw a broken window once. My big cousin Jamie kicked his leather football through it. He was not supposed to use his leather football in the garden. His dad had told him again and again.

Just as my mother dragged me round the corner I noticed that the side of bus shelter was broken. That must be where the glass had come from. It looked as if someone had spilt red paint on it.

“Where are we going, mum?” I asked.

“We're going to a different park today,” she said.

That was good news, and I began to walk a little faster. I wondered if the new park would have the same sort of swings as the old one; but best of all, on the way, I saw an ambulance. It had its lights flashing and it made the noise that makes the cars get out of the way, but they turned those off when it went round the corner.

The new park was bigger than the old one, but it was quite far away so we came home on the bus.

On the way home I asked my mother where she thought the ambulance went. She said she expected it had gone to help some people who had hurt themselves.

Mum said we can go to the new park again sometimes, but not every day.