Thursday, 26 January 2017

Trampolines on MIPS for GCC nested functions under VxWorks

Taking the address of a GCC nested function generates a trampoline  on most platforms so that the stack frame address can be recovered during the callback.

Nested functions are a GCC extension to C (available for Pascal), with an alternative of code blocks under clang.

Trampolines are described in Lexical Closures for C++ (Thomas M. Breuel, USENIX C++ Conference Proceedings, October 17-21, 1988), and this trampoline is a piece of generated code that can recover a stack frame pointer or any other useful value by loading a literal before making a jump to the callback address, so that auto variables of the containing function can be accessed in their stack frame.

It means that your callback can recover any number of auto-variables without the need for a cookie pointer or an associated struct.

The MIPS trampoline structure is explained here by Ian Lance Taylor in 2006.

On Systems with a unified cache, data-writes to generate the code automatically become visible to the instruction cache, but on other systems (MIPS, ARM) the data cache and instruction cache may be independent, so after generating the trampoline, the data cache must be flushed and the instruction cache invalidated for that region, to be sure that the CPU will read the newly generated instructions.

This Arm Community post explains the difficulty quite well, although do not expect the specification for the __clear_cache builtin function to have the same calling signature as the _flush_cache function whose calling is generated by a set of definable macros.

The _flush_cache function is system specific, you need whatever flushes the data cache and invalidates the instruction cache properly across all CPU's for your target system.

If it were easy, the compiler would have done it; but it doesn't know how to clear the caches for whatever system you might be targeting.

The proper way to do this on a MIPS system is not clear. There may be privileged instructions required, maybe user instructions such as SYNCI will work, maybe the OS kernel exports a system call or function to deal with this.

Unless this is specificed, the GCC nested function trampolines on MIPS under VxWorks tend to fail at link time, like this:

(.text+0x146370): undefined reference to `_flush_cache'
vxWorks.bin: In function `zig':
(.text+0x146370): relocation truncated to fit: R_MIPS_26 against `_flush_cache'

because the compiler has no real idea how to flush the caches on your MIPS system, and this function is not provided.

I can redefine the name of the missing function, so it calls the cache flush function on my system with this compiler option:
-mflush-func=really_flush_cache

(.text+0x146370): undefined reference to `really_flush_cache'
vxWorks.bin: In function `zig':
(.text+0x146370): relocation truncated to fit: R_MIPS_26 against `really_flush_cache'

If only I knew what the name of the cache flush function was, and that it had a matching calling signature.

Flush function signature

The documentation is very poor on the arguments to be provided to the flush cache function, and this seems to vary from platform to platform.

In the documentation, of special relevance are CLEAR_INSN_CACHE and TARGET_TRAMPOLINE_INIT which in one case is given this definition:

#define CLEAR_INSN_CACHE(beg, end) mips_sync_icache (beg, end - beg)

By adding debugging to inspect the arguments of a flush-cache function wrapper, I see that this matches the arguments passed when compiling under my VxWorks tool chain; but given that those macros can be re-defined it may not match GCC building for a different system so I must acknowledge that my flush function is not portable.

Possibly a portable implementation is given here, and I note that other JIT systems (e.g. SLJT used by PCRE-SLJT) and code generators (libffcall) attempt to provide their own portable implementations but which may not work on some multi-core systems.

Cache Operations

Linux

#include <asm/cachectl.h>
int cacheflush(char *addr, int nbytes, int cache);

VxWorks

VxWorks provides cacheInvalidate, cacheFlush and cacheClear (invalidate and flush). As cacheClear does both, we could just have the compiler option:

-mflush-func=cacheClear
although why VxWorks didn't fill that in for us already, I don't know.

But we only want to flush the data cache and invalidate the instruction cache for this range, so we might prefer this wrapper:

#include <cacheLib.h>

void _flush_cache(char* beg, size_t bytes) {
    cacheFlush(DATA_CACHE, beg, bytes);
    cacheInvalidate(INSTRUCTION_CACHE, beg, bytes);
}

It Lives!

Does it work? Well, the trampoline works; but is it treating the cache properly or will it randomly fail? I don't know.

Note: I don't need to provide the _flush_cache function here if I pass -mflush-func=cacheClear to use the VxWorks function directly.

#include <cacheLib.h>

void _flush_cache(char* beg, size_t bytes) {
    cacheFlush(DATA_CACHE, beg, bytes);
    cacheInvalidate(INSTRUCTION_CACHE, beg, bytes);
}

void zag(void(*zog)(void)) {
    printf("ZAG\n");
    zog();
}

void zig() {
  const char* where = NULL;

  void zog() {
    printf("ZOG from %s\n", where);
  }

  where = __FUNCTION__;

  printf("ZIG\n");
  zag(zog);
}

Call function zag( ) to see

ZIG
ZAG
ZOG from zig

If you want to see the generated trampoline code, use GCC flags -save-temps -fverbose-asm and then look at the .s file.

Wednesday, 28 December 2016

Augmenting Apple iCloud PhotoStream

As discussed previously, if you want to share in the photos of an Apple user, you have to do it the Apple way.

But once my windows PC has an incrementally downloading folder of photos, I want to upload them to Google Photos where everyone can view them.

I can upload the photos, but Google Photo's uses the file system dates when ordering the photos and ignores any timestamps in the exif tags; so somthing needs coding to extract that information from the tags to touch the files.

Furthermore, photos (and more likely, movies) that aren't tagged are totally without timestamp information.

And yet... iCloud photos viewer on windows has this information, along with comments -- none of which is stored in the images or movies.

The task

My self-appointed task is to find the source of this information for iCloud Photostream, embed it into the images and movies, and update file timestamps with it prior to upload to Google Photos.

Clues

There are some clues here: https://www.braxtonehle.com/posts/replicating-shared-photostream-to-dropbox/ but I'd already got that far by using Process Explorer to examine what files iCloud Photo was accessing.

The App

My original intention was to write a window app to upload to the Windows App Store, as the only one there to help with this seems highly priced at 

But having examined the database schema I see that this could be about 10 lines of bash scripting that works for just me; is it worth the extra work to produce a windows app?

No, but I'll do both anyway; downloading Visual Studio Express Community Edition

The bash script

The bash script is plain sqlite3 query, with the results piping into a loop at updates each file; with the usual bash presumptions that filenames won't contain a newline character, etc.

#! /bin/bash

USERDIR=/mnt/c/Users
WINUSER=flipflap
ICLOUD=AppData/Roaming/Apple\ Computer/MediaStream
DBNAME=local.db

DB="$USERDIR/$WINUSER/$ICLOUD/$DBNAME"

sq() {
  sqlite3 -line "$DB" "$@"
}

albums() {
  sq "select albumName from MSASAlbums"
}

photos() {
  sq "select assetfilepath, caption, downloaded, deleted, datetime(dateCreated + 978307200, 'unixepoch') || 'Z' as datetime, createdbyme from MSASAlbumAssets left join MSASAlbums on MSASAlbumAssets.albumGuid = MSASAlbums.albumGuid where albumName = '$1' order by dateCreated"
}

readRecord() {
  local field value count
  while read -r field _ value && test -n "$field"
  do printf -v "$field" "%s" "$value"
     count=$(( count +1 ))
  done
  test -n "$count"
}

updatePhotos() {
  while readRecord
  do file="${assetfilepath//C://mnt/c}"
     file="${file//\\//}"
     echo update "$datetime $file"
     touch -c -d "$datetime" "$file"
  done
}


if test -z "$1"
then albums
     exit
fi

photos "$@" | updatePhotos

And that works well enough.

Wednesday, 14 December 2016

Two touchpads for one

http://askubuntu.com/questions/783838/disable-touchpad-while-typing-do-not-work

Solution was to add in /etc/modprobe.d/blacklist.conf
blacklist i2c_designware-platform
And reboot the system. After that syndaemon works fine.
or

https://ubuntuforums.org/showthread.php?t=2316240

/usr/share/X11/xorg.conf.d/51-synaptics-quirks.conf
and added this entry:

Code:
# Disable generic Synaptics device, as we're using
# "DLL0704:01 06CB:76AE Touchpad"
# Having multiple touchpad devices running confuses syndaemon
Section "InputClass"
        Identifier "SynPS/2 Synaptics TouchPad"
        MatchProduct "SynPS/2 Synaptics TouchPad"
        MatchIsTouchpad "on"
        MatchOS "Linux"
        MatchDevicePath "/dev/input/event*"
        Option "Ignore" "on"
EndSection



Wednesday, 7 December 2016

What is wrong with Apple?

Apple make easy things very very hard

Sharing photos is easy with google


If I want to share my photos with someone who has a google account, I just name them in the share dialog, and then they can see my photos, via their own google account from any web browser.

Or if they don't have a google account, I can share the "link" to the album so that any one who has the hard-to-guess link can see my photos.

Anyone with a vaguely modern web browser.

That's all that's needed for 21st century communication.

But not with Apple


A family member uses an ipad and wants to share photos with me from her "photo stream".

So they send me an invitation to subscribe to their  photostream.

Luckily I have an icloud account from when I tried to use itunes to buy music (that's another story).

And I click on the link, to be greeted with:
To subscribe to ... photo stream on your iPhone, iPad, iPod touch or Mac, open your invitation in the Mail app and click the Subscribe button in the message.
To subscribe you need to be signed in to iCloud on:
  • an iPhone, iPad, or iPod touch with iOS 6 or later or
  • a Mac with macOS 10.8.2 or later and iPhoto 9.4 or Aperture 3.4 or later
I actually need to possess an apple device to view some photographs that someone has taken using their apple device!

I get my paws on an apple device

I manage to lay my hands on a MacBook Air. I give it all the updates. It's running Lion, the latest release of MacOS that it can run.

I install iCloud.

Do you want to guess if I can subscribe to the photostream?

I can't -- I need iPhoto 9. Free upgrades to iPhoto 9.4 are available if I have iPhoto 9, but I don't. It has iPhoto 8.

I can't buy iPhoto 9 from the Apple app store because it has been discontinued.

Apple won't let me pay them the money that they are extorting from me because... it isn't enough.

I track down iPhoto 9


I find that I can get iPhoto 9 if I buy a CDROM containing iLife 11. (iLife 11 has also been discontinued from the app store).

iLife 11 typically sells for $50 or £50 on Amazon, but I track down a copy for around £20 on ebay.

I'm not confident, and neither is my wife.

And then I notice, iPhoto 9 could possibly not be enough.

The small print in the email says: with macOS 10.8.2 or later
and I only have macOS 10.6.6 (Lion) and this MacBook Air  won't take a newer version.

So now I'm an apple user (with this MacBook Air I got hold of) and I still can't view the photostream. I need to shell out a few hundred pounds to but a newer macbook or ipad.

But iCloud is available for Windows.

iCloud for Windows

I install iCloud for windows.

It doesn't work.

After logging in (authenticating me successfully against the apple authentication servers) it then declares that I am not connected to the internet. c_a_murphy4 comments:
I hope you receive more professional support than I did. Their senior team took more than a week with this issue and then blamed Microsoft (in a rather backhand way by saying there was nothing more they could do from an Apple standpoint). I called up Microsoft, and they in turn blamed Apple. If you ever do find a solution to this problem, please let me know. Apple and Microsoft just aren't coming through with the goods - at least not for me.
Some people seem to suggest that it is only the windows 10 anniversary edition that has this problem. Well I'm not downgrading.

Windows 7

So I try again on an old windows 7 laptop and get exactly the same problem.

I suppose I can blame Apple that now it's not just windows 10 but apparently no version of windows can run the latest version of their software, and it's not there fault.

Taking advice, I downgrade to the previous version of iCloud for windows.

This time  get the Error occurred during authentication error.

So I try the previous version of iCloud for windows.

Surely one point Apple have managed to produce working software for windows? And it works.

But no photostream

Now iCloud works but isn't any use. The photostream subscription has to be accepted as describe in the sucks-be-to-you message I had when I originally tried to subscribe from an unclean non-apple system:
a Mac with macOS 10.8.2 or later and iPhoto 9.4 or Aperture 3.4 or later
So I lookup virtual mac machines in the cloud. I could pay a few dollars an hour and perhaps get it working. Or pay $20 a month with the first day free and then cancel right away.

Via a friend, I get long distance access to a mac, with all the right software, and the photostream subscription is accepted.

Back to windows

Now the photostream subscription is accepted I can begin to download it on the windows 7 PC and then use google photo uploader to sync that folder to a photo album.

Back to google

And so now I can share the google photo album with other non-apple family members.

But this is normal for Apple

They break your stuff on purpose

If a you sacrifice at the wrong altar and have a non-Apple vendor repair your phone, Apple prevent your phone from working.

If you jailbreak your phone so that you can install software that you didn't pay Apple for, they will brick your phone on the next firmware update.

I showed here how an Apple user with a MacBook Air is prevented from subscribing to the photostream of another Apple user, because they didn't sacrifice recently enough at the Apple altar.

But Apple made a mistake 

Their new much-cursed top-of-the-range laptop that is missing a load of ports -- no HDMI, no USB2, no sdcard, etc

Now some independent developers funded by kickstarter have put together The HyperDrive.
It’s a $100 dongle that slips neatly into both USB Type-C slots to give you a whole lot more connectivity. Not only do you get your two USB Type-C ports back, but you gain a couple of USB 3.1 ports; a microSD and SD slot; and an HDMI video port.
Did you notice? "slips neatly into both USB Type-C slots"

Because of a stupid Apple oversight in providing standard USB-C ports users have overcome Apple! How are they going to stop that?

Of course the most annoying thing is that there will have been room inside the case to fit this dongle.

Afraid of perfection

Did you ever know a company so afraid of perfection that they spent so much time and effort deliberately lousing things up for their customers?

Or a bunch of captive customers so suffering from Stockholm Syndrome that they keep paying over much money for such rubbish?

Stockholm syndrome, or capture-bonding, is a psychological phenomenon first described in 1973 in which hostages express empathy and sympathy and have positive feelings toward their captors, sometimes to the point of defending and identifying with the captors. These feelings are generally considered irrational in light of the danger or risk endured by the victims, who essentially mistake a lack of abuse from their captors for an act of kindness.
And we are not seeing that much lack of abuse.
The FBI's Hostage Barricade Database System shows that roughly eight percent of victims show evidence of Stockholm syndrome.
And yet Apple market share is quite a bit more than 8%, around 12%.

I wonder, what is the excuse of the other 4%?

Friday, 2 December 2016

The End

The time is past
  the bell has rung
The deeds to do
  that have been done
    are gathered up
      into the store
The rest: undone,
  for ever more

(C) Sam Liddicott 2016

Wednesday, 30 November 2016

Calibri and Cambria on Linux

I'm no font fanatic. I can't tell the difference between Calibri and Carlito but I can tell the a busted document layout.

Fortunately, Carlito has the same front metrics as Calibiri (and Caladea has the same metrics as Cambria).

The font-substitution table in LibreOffice then allows me to work on documents using these fonts while getting a layout the same as my colleagues on Microsoft Office.

Quoting from the Debian Wiki:

LibreOffice font substitution


To install them, issue these commands as root in a shell:
# apt-get update
# apt-get install fonts-crosextra-carlito fonts-crosextra-caladea


In LibreOffice, you exchange Calibri and Cambria with Carlito and Caladea this way:
  • Open the "Extras" menu
  • Go to "Options"
  • Choose "LibreOffice"
  • Choose "Fonts"
  • Define a substitution for each of the two fonts (Calibri -> Carlito, Cambria -> Caladea).
  • Remember to check "Always" in the substitution lines.

Once the program is restarted, documents sent from MS Office look almost the same on your screen and printouts.