Latin1 to utf-8 without iconv

I recently helped someone convert latin1 text to utf-8 on a minimal system with no access to iconv.

A bash script had a latin1 field and needed to encode it to utf-8.

Fortunately, latin-1 only has 256 characters and only the top 128 are special, and (not that it makes any difference) most of those are the same.

The minimal system had busybox od command, so I decided to convert the variable to a numeric octal stream, like this:

$ read FIELD

Hello everybody I am the thing

Which can be converted to octal like this

$ echo "$FIELD" | od -b

0000000 110 145 154 154 157 040 145 166 145 162 171 142 157 144 171 040

0000020 111 040 141 155 040 164 150 145 040 164 150 151 156 147 012

0000037

and then strip to just the octal character values preceded by a space

echo "$FIELD" | od -b | sed -e 's/[^ ]*//;s/ *$//' 

 110 145 154 154 157 040 145 166 145 162 171 142 157 144 171 040

 111 040 141 155 040 164 150 145 040 164 150 151 156 147 012

and then join lines together

$ echo "$FIELD" | od -b | sed -e 's/[^ ]*//;s/ *$//' | tr -d $'\012'

 110 145 154 154 157 040 145 166 145 162 171 142 157 144 171 040 111 040 141 155 040 164 150 145 040 164 150 151 156 147 012

and then convert each space to $_lu_ which is a nice variable prefix

$ echo "$FIELD" | od -b | sed -e 's/[^ ]*//;s/ *$//' | tr -d $'\012' | sed -e 's/ /$_lu_/g'

$_lu_110$_lu_145$_lu_154$_lu_154$_lu_157$_lu_040$_lu_145$_lu_166$_lu_145$_lu_162$_lu_171$_lu_142$_lu_157$_lu_144$_lu_171$_lu_040$_lu_111$_lu_040$_lu_141$_lu_155$_lu_040$_lu_164$_lu_150$_lu_145$_lu_040$_lu_164$_lu_150$_lu_151$_lu_156$_lu_147$_lu_012




Now if all those variables were defined to hold the utf-8 values, we could convert the field, like this:

$ FIELD=$(eval echo \"$(echo -n "$FIELD" | od -b | sed -e 's/[^ ]*//;s/ *$//' | tr -d $'\012' | sed -e 's/ /$_lu_/g' )\")

as a bash function:

latin1_to_utf8() {
  eval echo -n \"$( <<<"$1" od -b | sed -e 's/[^ ]*//;s/ *$//' | tr -d $'\012' | sed -e 's/ /$_lu_/g' )\"
}

Here is how we define those variables; this code must be run on a fully-featured box with access to iconv.

for i in `seq 1 255`

do

  echo "_lu_$(printf "%03o" $i)"=\$\'$( printf $( printf '\\x%x' $i ) | iconv -f latin1 -t utf-8 | od -b | sed -e 's/[^ ]*//;s/ *$//;s/ /\\/g' )\'

done

and the text it outputs

...
...
_lu_176=$'\176'
_lu_177=$'\177'
_lu_200=$'\302\200'
_lu_201=$'\302\201'
_lu_202=$'\302\202'
...
...

Will be pasted into the script that runs on the reduced environment

Recover deleted photos

In March 2007, some missionaries called around with a sad tale - they'd accidentally deleted all the photos on their memory card, could I get them back?

I quickly knocked up this short perl script to do it.


#! /usr/bin/perl
# Quick hack by Sam Liddicott: <sam@liddicott.com> www.liddicott.com/~sam
# Try and look for jpeg images in a stacked file.
# Reads 512 blocks and when it finds one that starts with oxd8ff it starts a new file
# Invoke on a raw image of the card (maybe taken using rawread or rawrite) like this:
# perl recover.pl < IMAGE
# or
# perl recover.pl < /dev/sde

our $count=0;
our $file="/dev/null"; # use nul for dos/windows
our $buffer;

open STDOUT,'>',$file;

while (read(STDIN,$buffer,512)) {
  if ($buffer=~/^\377\330\377[^\333]/) {
    close(STDOUT);
    $count++;
    $file=sprintf("img_%03d.jpg",$count);
    print STDERR "$file\n";
    open STDOUT,'>',$file;
  }
  print STDOUT $buffer;
}

It did recover his photos.

Of course that \377\330\377 sequence was taken from /etc/magic - where else!?

The New Park

The New Park

(C) Sam Liddicott 2004

As we got near the park, my mother started to walk in the other direction, taking me with her.

This wasn't right; she had promised me that I could go to the park in the afternoon if I behaved in the morning, and I had behaved. I hadn't played with the scissors and cut the curtains, or got my shoes muddy when I played outside. I read my books quietly and put them away again, and I had eaten all my bread at lunchtime without complaining.

I had behaved, but just as we arrived at the park she started taking me somewhere else. I protested with a yell.

"Be quiet!" she said abruptly, which surprised me because my mother had been in a good mood when we set out. I was hoping we could stay longer at the park because it was sunny, and mum could sit on the bench and read her book.

"I want to go to the park, you said I could..." - it was worth a try, but she didn't seem to be in the mood. Why had she changed so suddenly?

I looked behind me towards the park as my mother dragged me along. There were some boys standing by the the red bus shelter in a group as they often did. I'd seen them before so I think they live nearby. One of them is called Harry. Maybe they were waiting to catch a bus to see their grandma, like we do with when dad has to stay at work late.

My mother dragged me on, still hurrying. I twisted my head as far as I could to see behind me. There was broken glass about the pavement. That was not unusual, sometimes the park had glass in it from broken bottles. I saw a broken window once. My big cousin Jamie kicked his leather football through it. He was not supposed to use his leather football in the garden. His dad had told him again and again.

Just as my mother dragged me round the corner I noticed that the side of bus shelter was broken. That must be where the glass had come from. It looked as if someone had spilt red paint on it.

“Where are we going, mum?” I asked.

“We're going to a different park today,” she said.

That was good news, and I began to walk a little faster. I wondered if the new park would have the same sort of swings as the old one; but best of all, on the way, I saw an ambulance. It had its lights flashing and it made the noise that makes the cars get out of the way, but they turned those off when it went round the corner.

The new park was bigger than the old one, but it was quite far away so we came home on the bus.

On the way home I asked my mother where she thought the ambulance went. She said she expected it had gone to help some people who had hurt themselves.

Mum said we can go to the new park again sometimes, but not every day.

Overcast

Overcast

(C) Sam Liddicott 2004

Of course, he'd heard of boats. Instead of a steering wheel they had two sticks – paddle things – that you waved around in the water to tell the boat which way to go. Sort of like a steering wheel cut in half with the two halves dipping in the water. Like steering wheel – but in the water. That didn't sound too hard, he'd used steering wheels before.

The water stank of ducks, rotting leaves, rank damp bread and the sort of things that gets left behind by ducks that eat too much rank damp bread thats been floating around in rotting leaves. And it was wet – raining that is. He had no hat and so tried to make do with putting his hands over his head and wriggling down further into the bushes by the boat house. Looking up in the dusk light, he noticed with resigned pessimism that he was right below the eaves, which, without guttering, were dripping a regular beat upon him. There had been a guttering. He looked at it beneath his feet, wondering if it would be of any use for anything.

There were no boats in the water. This escaped him for quite some time, and he cursed quietly as he realised. First, panic that he was in the wrong place; that maybe this wasn't the boat house on the lake by the south entrance as the map had shown.  Then he fetched the crumpled map out of his pocket and squinted at in in the halflight, parting the bushes for more light to be able to see clearly.  This was the place, and the boat house was marked on the map, and there was the island in the middle of the lake.  Boat house – what a funny name, he thought to himself. It was probably called a boat house because it was right by the place they kept the boats.

His legs were beginning to ache from too much crouching in the low bushes, and shuffling his feet only served to make him lose his balance and sit down in the mud.  He noticed (as he turned around) a litter bin close by. There was something moving in the bin, he could tell by the noise. Probably a rat. Then he remembered about rats in the water. It would be hard to tell if rats had weed in this water, it was so dirty anyway.

These hardships recalled his purpose to mind.  Soon he'd be on the island, a new home of his own with no-one, no-one, to bother him; and a big bonfire to dry him out, warm him up and cook his dinner. A tin of scotch broth! Later he'd trap rabbits and things, but that took time, of course. The idea of a fire was comforting. He felt in his pockets for his cigarette lighter - just a flame to wam his hands while he waited - but it was missing; his mother had been through his pockets again.

Then a clap of thunder broke out and the rain increased to a torrent. So much for “overcast” he thought as he trudged back home. At least the rain would wash the mud off his trousers.

Mi Charango

I wrote this in 2004 as part of my coursework

Mi Charango

© 2004 Sam Liddicott

Jorge awoke as the distant sound of the festival music leaked through his broken shutters to accompany the sunlight already spilling over the strings of the charango hanging by his bed. He did not look at it, it mocked him, but he picked it up with a care and tenderness that contradicted his countenance, and began to play.

In playing, he found no comfort. No familiar melodies could move his heart, he felt no joy from the lively dance tunes and became all the more melancholy with each love song. Finally, he found himself playing something unfamiliar, but though each phrase was new, his fingers drew from the strings a gentle melody whose notes fell with a comforting reassurance and familiarity that soothed his soul. He had heard it somewhere before.

A clatter of feet on the stairs erupted through the door of his room bringing him the greetings of a small crowd.

"And how is our maestro today?" – this was not a question.

"Today your music will honour us all."

"Why have you not called at my shop? I have brought for you a suit here that will show everyone from miles around what fine clothes we folk wear!"

"And when you are dressed you must call in to see me. I have a special table laid for you – outside; I will prepare for you a meal that will truly inspire your playing. And my sister, – she will serve you"

"And here is a fine hat for you. When people look at the man who plays the best I want them to see he knows where to go for an excellent hat!"

Jorge said nothing. The little crowd paused. The foremost looked at Jorge and dimly perceiving his mood decided that some encouragement was needed.

"Can't you see he is resting?", he chastised the small crowd, and pushed and hustled them down the stairs, but did not immediately descend himself:

"Don't be concerned, my friend," he entreated gently, “remember, even though you compete against the villages across the lake, you are surrounded by friends!”

Jorge just shrugged, and wearing his new suit, went to join his band of supporters for breakfast.

A special table had been placed in the street for all to see, and covered in a bright white cloth, which already his attendees had stained with wine. To keep the ascending sun from his head, Jorge put on his new hat, the action receiving rapturous acclaim from its giver.

The young lady serving the table paid particular attention to Jorge, though not to take his order, for a special dish was already being prepared in his honour.

Jorge looked at his supporters seated around the table, who, while waiting, contended to give the best account of the forthcoming victory. He wondered when they had first become friends.

His mind slipped back seven years ago:

He was hiding in the framework under the platform. The music from the performers did not drown out the cries of his tormentors as they sought him out, having already spent their money on the first day of the festival. Knowing that he would be found near the music their search was bringing them closer to his hiding place, and although the framework raised him from the ground and thus from casual inspection he knew he would be found before long.

“Jorge?”

“Shhh! He's thinking! Can't you see he's thinking?”

“I was only going to ask what it was he was playing when we came to rouse him!”

“Yes, Jorge, what were you playing this morning?”

“It's his surprise piece of course! He will play it today and win the competition and show those, those, ... across the lake that real music belongs here!”

“I only wanted to know its name?”

“Yes, Jorge, what was is its name?”

Jorge looked across the table. That tune he had been playing when they disturbed him certainly had the power captivate all who heard it. It was music worthy of a great master. And then he remembered.

It was music from a great master, from the man whose charango lay on the table beside him. Memories so unexpectedly freed raced back into to his mind:

With surprise, the face that showed itself beneath the platform belonged not to his tormentors, but to an old man. Recognizing the situation he immediately withdrew his head and began to scold the young boys for playing so noisily and sent them play away from the music.

The same face returned and begged him to come out, but Jorge refused, "I can hear the music better here", he said.

"You like music?" the old man asked.

Jorge nodded.

"Can you play?" enquired the man.

Jorge shook his head. "I could", he said, "if.. if..."

"Would you like to play this?", asked the face, holding down a charango.

Jorge opened his eyes and looked at the same charango, now in his hands. His fingers alternately strummed and plucked the strings as his mind recalled his first lessons.

"You have a great gift for music," said the man as Jorge, without being taught directly, began to select chords to play against the simple melodies taught him by this friendly stranger.

"If I had a son,” said the old man, “I would teach him the song of our family. But I am old and have no children. Shall I play it to you?"

Jorge, listened enthralled. He had not known such power in music. The melody and rhythm wrapped him in sunlight and danced him over the hills. In a moment he was bathed in the cool stream of the hillside, and in another moment he was soaring in the sky, his loneliness gone, giving way to the joy known only to songbirds.

"I shall play this tomorrow," said the old man, "I have no son, so I shall play for my countrymen, and so win the competition."

Jorge's breakfast was served carefully and ostentatiously by someone's beautiful sister but he didn't notice.

By the third day Jorge could play the piece "Mi Charango" with tolerable skill and his new friend and mentor proposed that he play this as a beginner solo musician before the final competition at the festival. He would lend his instrument to Jorge, and afterwards himself compete to take the seven year prize, and be the pride of his countrymen.

Jorge basked in the memory of the joy he had then felt, more vivid and more brief than life itself. He stepped out onto the platform.

“Jorge, what are you doing?”

“I told you, he's thinking. Let him think!”

He stepped out onto the platform. Beneath the crowd of hats, children running among the crowd, mothers looking after babies, men talking to each other and paying no attention to their wives or babies, children chattering and laughing. And laughing – at him!

“Jorge wants to play, look at him.“

“He doesn't even know how!”

Defiantly, Jorge raised his hand to play, but somehow – either he held it wrong, or started on the wrong strings – he played something that bore just enough similarity to “Mi Charango” to be recognizable.

The laughter increased. Men stopped talking, and women turned away from their children to watch the spectacle. Jorge made another attempt.

"Get off!" he hissed his father, "you have never played music in your life before!" and the crowd laughed even harder.

The old man stepped forward to offer encouragement, but blinded by his tears and fright Jorge pushed past him and ran down the steps and through the taunts of his enemies.

"Where have you been Jorge?" they cried, "We've been looking for you!"

"Give me that!", they said, pointing at the charango, "let us play!"

Running on, Jorge made his way through the village and into the corn where he lay, sobbing. After about an hour he began to play to himself. First "Mi Charango" as he had meant to play it, and then picking out the various melodies that came into his mind, as his mother had sung them years ago. He played on into the dusk, and it was only when the evening coolness turned to chill that he remembered competition and who the charango really belonged to.

With every step home Jorge felt a rod beat his back. Finding his father drunk and asleep he hid the charango under his bed. The next day nothing was said, and so for years he secretly practised the charango after his work, and when his father died he practised it openly.

Jorge looked at the charango in his hands. It wasn't his. It belonged to a man across the lake. An old man who had taught him to play.

Jorge sprang to his feet, and smiled at around the table.

“He's ready!”

“I knew he'd be all right!”

“You all worried for nothing!”

"When he wins, I shall win enough money to by a team of mules."

With the calmness and serenity of seven years Jorge waited by the platform until his turn, heedless of all questions and encouragement. His turn was to be the last performance before the festival broke up for another seven years. This time the crowd was hushed. His fame was known. He stepped towards the front of the platform, found what he sought, and began to play.

As he finished, no-one moved, all ears straining for the final imperceptible echoes from the hills. Jorge broke the silence. "I play for my new home" and ignoring the enquiring calls from the audience, he descended the steps, and passed by his newly generous friends and stopped to embrace a tired old man. That evening he returned across the lake to his new home, as son to a new mother and new father – the only friend he'd ever known.

The Handkerchiefs

I wrote this in 2004

The Handkerchiefs

© 2004 Sam Liddicott

Dear June,
                   you see, I know your name,
your handkerchief was spelt the same,
the one I kept, when in the park,
you dropped it on a summer walk.

  You haven't dropped one since, all year!
  The truth most likely is, I fear
  You quite forgot to buy some more
  To drop for me upon the floor

So please accept this gift from me
Some silken handkerchiefs times three
And should you now be quite sure who
To drop them for - I'm wearing blue

Jon, The Brainworker

Jon, The Brainworker

© 2004 Sam Liddicott

Jon opened one eye. On the opposite wall there were the shadows of the clutter on the window sill. The pile of plates cast a shadow like some kind of tree with the past weeks knives and forks looking like so many spindly branches as if more growth was too much effort for such a shadow as this. As the sunlight passed through a half full glass of dusty drink it cast a bright pattern on the wall with a shimmering beauty caused through the efforts of heavy traffic outside.

Jon noticed none of this. He saw nothing unexpected, nothing to require further examination of his surroundings. His eye closed.

He rolled over and briefly considered the chances of finding something clean to wear. A moments thought concluded that it would not be worth the effort of getting out of bed. It seemed to be late evening. If he spend long enough thinking, the laundrette would be closed – that was some consolation.

The shadows of the plates and drinks in the reddening light seemed to wake some recollection in his empty mind. Food! If the laundrette was closed, so was the shop. John stirred. If only he could see the clock at the other side of the room. He smirked when he remembered the sound it made when it hit the wall – when had that been?

He rolled out of bed onto his knees and struggled up. He gathered nearby clothes onto his bed-sheet and checked the pockets for enough change for a wash and dry. He would do without the powder if necessary, but not the dry. Nice, warm clothes, warm sheets to snuggle down in. He pulled the corners of the sheet into a bundle and shuffled down the steps.

The laundry and the shop had been closed for an hour, but on the way back an old lady gave him a pound.

Jon didn't put the sheet back on the bed, it didn't seem worth it.