Thursday 19 October 2017

Why we won't have a third referendum

Why we won't have a third referendum on the terms of leaving the EU

The 1975 referendum was on single market membership, not EU membership.

There is no authority for EU membership and the associated demonstrated loss of national sovereignty that subsequently occurred, the possibility of which was denied by Edward Heath.

A referendum would be needed to get that authority, and that authority was not obtained in either the 1975 referendum or the 2016 referendum.

Prime Minister Cameron, doing his best (we assume) was unable to find terms on which to get voters to accept membership.

As voters never accepted EU membership and associated loss of sovereignty, our EU membership is illegitimate, and so the default position is therefore to not be a member.

It becomes ludicrous then to seek for national agreement of the terms under which we won't be a member; it is membership that is illegitimate, the non-membership does not need legitimising in the least.

How could a possible rejection of the specific initial terms of leaving be construed as implicit legitimisation of EU membership, in the face of the explicit rejection of membership?

Should we remain by default simply because a few mardy ministers or civil servants will persistently to a bad job and refuse to facilitate a "good deal" in order to continually fail to get national approval?

No, we should assume the legitimate position of non-membership, and give them that to work from.

This is why we shan't have a third referendum on the terms of leaving.

We might have a third referendum on whether and on what terms to re-join the EU, in a few dozen years or so.

Monday 2 October 2017

Vodafone Broadband

Using your own VDSL router

Thanks to aquazi who reports that you can use your own broadband router if you add to the end of your PPOE username.

Others report that Vodafone will no longer give out the PPOE username and password to you, however I just had online support chat and was given the POE username and password complete with at the end!

Someone pointed that Vodafone currently supports Vodafone supplied hardware only, and doesn’t allow the use of third party routers or modems which is not something I was aware of and seems like a pretty severe limitation to my unlimited broadband.

Avoiding Content Control

Although content control may be disabled, many HTTPS MITM warnings are given for image sharing sites like imgur.

Thanks to its5am who reports that changing to not use Vodafones DNS solves the problem of Vodafones content control performing man-in-the-middle attacks on your HTTPS traffic. (And presumably also returning bogus DNS resolutions).

Wednesday 21 June 2017

Better update your photo with powershell for windows Outlook/Lync/Skype-for-business/etc

Yesterdays post required the Remote Server Administration Toolskit installed.

Here I show how to do it without that, just using powershell. Thanks to maxx-ode who gives the answer on this Microsoft Technet Forum post .

This code will set picture.jpg as your thumbnail picture.

$root = [System.DirectoryServices.ActiveDirectory.Domain]::GetCurrentDomain().GetDirectoryEntry()
$search = [System.DirectoryServices.DirectorySearcher]$root
$search.Filter = "(&(objectclass=user)(objectcategory=person) (samAccountName=$env:username))"
$result = $search.FindOne()
$user = $result.GetDirectoryEntry()
$user.put("thumbnailPhoto", [byte[]](Get-Content "picture.jpg" -Encoding byte))

or you want to delete the photo:

$user.putex(1, "thumbnailPhoto", "")

but if you are re-using $user you may want to precede with: $user.refreshcache()

If you get an error like this:

Exception calling "setinfo" with "0" argument(s): "A constraint violation occurred."
At line:1 char:1
+ $user.setinfo()
+ ~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : CatchFromBaseAdapterMethodInvokeTI

then it could be that your picture is too big, keep it less that 100K, and ideally much smaller than that.
A 100x100 picture should be around 4K which is more ideal.

Tuesday 20 June 2017

Update your photo with powershell for windows Outlook/Lync/Skype-for-business/etc

NOTE: See my following post which does not required the Remote Server Administration Tools to be installed.

Thanks to the EZ IT guy who writes explaining that by installing the Remote Server Administration Tools for Windows 10 (or Windows 7 if you need it) you can then read your windows domain thumbnail photo picture.jpg like this:

$record = Get-ADUser $ENV:USERNAME -Properties thumbnailPhoto
$record.thumbnailPhoto | Set-Content "picture.jpg" -Encoding byte

And, provided that your photo is less that 100K you can also upload it:

Set-ADUser $ENV:USERNAME -Replace @{thumbnailPhoto=([byte[]](Get-Content "picture.jpg" -Encoding byte))}

All this requires that you be logged into your domain. Also thanks to the WOSHUB for help simplifying it.

If only there were a way that didn't require installing Remote Server Administration Tools.

Also some talk here from heyscriptingguy.

Friday 16 June 2017

Putting self-tests in a C header file

You can compile a header file to C using gcc -x c like this:

gcc -x c -o feature-test feature.h

Without the -x c options it would have built a pre-compiled header instead of a program you can run.

But of course the compile will ultimately fail as you don't have a main( ) function, and when you think about it you know that you won't normally want one.

Except when compiling the top level header file directly.

The pre-processor symbol __INCLUDE_LEVEL__ is conveniently 0 for the top level file, which lets us do:

int test1( ) {
int main(int argc, char **argv) {
  return test1() && test2() ... ;

So if the top level header file is build directly, then it gets a main and some test functions too.

Thursday 18 May 2017

Macro detection overload - Special Access Macros

This article talks about how one macro can call another macro if it exists, but if not, it can fallback to a default macro. This can be useful for behaviour vectoring or interception, debugging macros, etc.

Preferring Special Access Macros, but falling back to default macros

It would be nice if a C pre-processor macro can tell if another macro is defined.

It can... in a way. If you know in advance the name of the other macro you are looking it, you can do some comparison tricks as explained by Paul Fultz II in his C Preprocessor tricks, tips, and idioms.

But what if you don't know in advance what it will be called?

What if you are composing a combination of C macros and functions, where the developer can override a special case of the default behaviour for reasons of efficiency?

Maybe you have a cross-bar / plug-board, with data from a variety of sources, to undergo a variety of processing and be sent to a variety of destinations. Maybe you have a general route from a to b but sometimes want to drop in a specific optimisation.

Maybe you want to call a macro process(data, handler) where handler is the name of a macro or function used to emit the slowly generically processed data. Maybe process itself is actually an argument passed to you.

#define work(process, data, handler) process(data, handler)
work(mangle, input, archive);
work(mangle, input, deliver);
A conceptual mangle might be defined (using GCC / clang statement expressions):
#define mangle(handler, data) ({
  for(int x, int i=0; ITERATATE_DATA(&x, i, data); i++) {
but perhaps it's not efficient for mangle to call archive for each processed input element, and we want to have a specialised function to mangle the input for archive instead.

After all there is a big difference between dest[index++]=c; and http_post_value(dest, c); when being called on every piece of data!

Of course the archive handler is simply a macro name and can't be passed as a parameter to a function (this isn't C++ templates you know...).

But that's OK, we write a specific function to mangle the input for the archive output, by calling the archive macro inside the function:
void mangle_archive(void* data) {
  void* buffer = bulk_mangle_to_buffer(data);
That's nice.

But we'd like work(mangle, input, archive) to expand to call function mangle_archive(data) but we'd still like work(mangle, input, deliver) to expand to call the generic macro mangle(data, deliver) because it's not worth writing a more efficient handler to that. (But we might write one later, and if we did, we would want it to just work).

Can we do this? The answer is yes!


Now the difficulty is that (unlike in Paul's example) that we don't know in advance all the different handlers or processes that the user might have, so we can't go declaring in advance a load of macros like COMPARE_foo in advance. (Furthermore this is an ugly requirement to put on the author.)

However, in our case we can require that the special macro body (with which to override the default behaviour) should be entirely enclosed in parenthesis.

So while mangle_archive may well ultimately be a function, we will also define a macro wrapper thus:
#define mangle_archive(...) (mangle_archive(__VA_ARGS__))
Or something like that. As long as it is a macro whose body is in parenthesis and whose name is formed by joining mangle with archive. How does this help?

We have mangle and archive as input parameters, and if we can construct the name mangle_archive from mangle and archive, and then attempt to invoke it (as a macro, with the right number of parameters) it would result in (mangle_archive(...)) which (as the more enlightened will excitedly note) is an expression in parenthesis.

As anybody who has been reading Fultz2Heathcote and Gustedt will know, a well formed expression contained within parenthesis can be made to disappear entirely if prefixed by something such as the name of a macro which was defined thus:
#define VANISH(...)
e.g., with the right amount of coaxing:
VANISH (mangle_archive(__VA_ARGS__))
is quite easily persuaded to be nothing at all. And we can detect nothing at all. (See Jen's piece Detect empty macro arguments).

Clearly, if mangle_archive were not defined, we would have:
VANISH mangle_archive(__VA_ARGS__) which clearly won't be persuaded to vanish, and if mangle_archive were defined to something (not entirely) enclosed in parenthesis, we might have:
VANISH something (not entirely) enclosed in parenthesis
which clearly won't vanish either.

So to detect if the developer has defined a valid overriding macro, we simply attempt to invoke the macro with a VANISHing prefix and then use standard mechanisms already discovered to see if anything is left. (Using EVAL, naturally).

The actual trick is that rather than
#define VANISH(...)
we leave a comma (thanks Jens)
And put this at the start of an argument list to be followed by
We then grab the second argument with
and whether that is the concatenation of the process and handler, or just the process depends on whether or not an extra comma was inserted which would offset the first argument to become the second argument.

The resultant call to the overriding macro is called with the same arguments as would be supplied to the default macro, including the name of the handler. Of course, the handler is implicit now, being defined in the overriding function body, so the overriding macro can dispose of this (perhaps after a static assert) before calling the overriding function.
The simplified full solution is here.

MAGIC_DISPATCH takes at least two or three arguments, which are:

  1. default process name
  2. output handler, which when joined to the default process name becomes a potential overriding process name
  3. other arguments to pass to whichever of the processes is selected, should include at least the handler as the default macro will need to know where to send the data.
#define MAGIC_MAKE_NAME_(a, b) a##_##b
#define MAGIC_MAKE_NAME(a, b) MAGIC_MAKE_NAME_(a, b)
#define MAGIC_EMPTY()
#define MAGIC_DEFER(id) id MAGIC_EMPTY()
#define MAGIC_EVAL(...) __VA_ARGS__


#define work(process, data, handler) MAGIC_DISPATCH(process, handler, handler, data)
#define mangle_archive(handler, ...) (STATIC_ASSERT(#handler == "archive"), do_mangle_archive(__VA_ARGS__))

work(mangle, data, archive); // This will be overridden with mangle_archive
work(mangle, data, deliver); // This will call the default mangle
giving the output:
(STATIC_ASSERT("archive" == "archive"), do_mangle_archive(data));
mangle(deliver, data);
A last minute entry:
#define mangle_deliver(handler, data) (crunch_data(data))
work(mangle, data, deliver); // this will be overridden with mangle_deliver
If this looks too theoretical, consider that handler could be a list whose first member is the macro to call and whose other members could be trailing (or leading) arguments that are inserted into the process invocation. Such a handler would be decomposed by the invoker work to form a correct argument set for MAGIC_DISPATCH.

In such decomposition, these macros may prove useful.
#define MAGIC_LIST(...) (__VA_ARGS__)
#define MAGIC_UNLIST(...) __VA_ARGS__ 
though a more powerful level of EVAL may be required.

Much thanks to Paul Fultz II (C Preprocessor tricks, tips, and idioms), Jonathan Heathcote ( C Pre-Processor Magic), and particularly Jens Gustedt (P99 Macros for Emulation of C11), for documenting their work, which I have read, confused over, and slept on for many a night.

Tuesday 25 April 2017

Too many arguments; or: keyword arguments in C

Sometimes you extend a generic helper function and it starts to take too many optional arguments, which can be left as NULL if they don't matter.

Maybe your function formats and signs a message that could have quite a variety of fields.

int sign(struct key *key, const char *this, const char *that)
    x_sign(key, this);

int result = sign(key, "This", "That");

and then you find yourself wanting to add const char* other.

Yeah... and there will be a lot of caller code to fixup (unless you want to use the other macro trick for default arguments, which I'll write up later). 

Or will there?

struct sign_args {
    const char* this;
    const char* that;
    const char* other;

int sign(struct key *key, struct sign_args *args)
#define sign(key, ...) sign(key, &(struct sign_args){ __VA_ARGS__ })
and it is called the same way:

sign(key, "This", "That", "Other");

but you also get keyword arguments for free!

sign(key, .other = "Other"); // leaves this & that NULL

And this expands out to a valid compound literal expression. The temporary struct (an lvalue) is created (probably on the stack) and disposed of automatically at the end of the expression.

sign(key, &(struct sign_args){ .other = "Other" });

Another advantage is that now as your arguments are passed as a pointer, it is very easy to tunnel them all through the usual void* callback systems. In the case below, it briefly obtains an interactively unlocked key to sign with:

int batch_sign(struct key *key, struct sign_args *args);
int sign(struct key *key, struct sign_args *args) 
#define sign(key, ...) sign(key, &(struct sign_args){ __VA_ARGS__})
  x_with_unlocked_key(key, batch_sign, args);

Is it efficient? Every time the args are passed, only a pointer is passed, so that can be efficient; but the struct does have to be created once and may be a little more work than passing a struct as an argument or passing the arguments.

The main points to note are that

  • the original arguments should keep their same order at the start of the args struct
  • as the macro has the same name as your function, be sure to define it AFTER your function
  • the arguments are passed by pointer and so shared as they are tunnelled.
    This can make it easy to pass back a temporary value from an auto-variable. 
Of course you don't have to pass a pointer to the struct, you could pass the struct, by removing the & from the macro:

int sign(struct key *key, struct sign_args *args)
#define sign(key, ...) sign(key, (struct sign_args){ __VA_ARGS__ })
it is called the same way:

sign(key, "This", "That", "Other");

but expands out slightly differently, passing the struct instead of a pointer to it, e.g.:

sign(key, .other = "Other"); // leaves this & that NULL

which this expands out to:

sign(key, (struct sign_args){ .other = "Other" });

Of course there is still nothing to prevent you from passing &args via a callback, or making and passing a copy.

Thursday 26 January 2017

Trampolines on MIPS for GCC nested functions under VxWorks

Taking the address of a GCC nested function generates a trampoline  on most platforms so that the stack frame address can be recovered during the callback.

Nested functions are a GCC extension to C (available for Pascal), with an alternative of code blocks under clang.

Trampolines are described in Lexical Closures for C++ (Thomas M. Breuel, USENIX C++ Conference Proceedings, October 17-21, 1988), and this trampoline is a piece of generated code that can recover a stack frame pointer or any other useful value by loading a literal before making a jump to the callback address, so that auto variables of the containing function can be accessed in their stack frame.

It means that your callback can recover any number of auto-variables without the need for a cookie pointer or an associated struct.

The MIPS trampoline structure is explained here by Ian Lance Taylor in 2006.

On Systems with a unified cache, data-writes to generate the code automatically become visible to the instruction cache, but on other systems (MIPS, ARM) the data cache and instruction cache may be independent, so after generating the trampoline, the data cache must be flushed and the instruction cache invalidated for that region, to be sure that the CPU will read the newly generated instructions.

This Arm Community post explains the difficulty quite well, although do not expect the specification for the __clear_cache builtin function to have the same calling signature as the _flush_cache function whose calling is generated by a set of definable macros.

The _flush_cache function is system specific, you need whatever flushes the data cache and invalidates the instruction cache properly across all CPU's for your target system.

If it were easy, the compiler would have done it; but it doesn't know how to clear the caches for whatever system you might be targeting.

The proper way to do this on a MIPS system is not clear. There may be privileged instructions required, maybe user instructions such as SYNCI will work.

Since GCC 3.1 GCC has support for the -msynci option
GCC now supports an -msynci option, which specifies that synci is enough to flush the instruction cache, without help from the operating system. GCC uses this information to optimize automatically-generated cache flush operations, such as those used for nested functions in C. There is also a --with-synci configure-time option, which makes -msynci the default.
Maybe the OS kernel exports a system call or function to deal with this, so unless this is specified, the GCC nested function trampolines on MIPS under VxWorks tend to fail at link time, like this:

(.text+0x146370): undefined reference to `_flush_cache'
vxWorks.bin: In function `zig':
(.text+0x146370): relocation truncated to fit: R_MIPS_26 against `_flush_cache'

because the compiler has no real idea how to flush the caches on your MIPS system, and this function is not provided.

I can redefine the name of the missing function, so it calls the cache flush function on my system with this compiler option:

(.text+0x146370): undefined reference to `really_flush_cache'
vxWorks.bin: In function `zig':
(.text+0x146370): relocation truncated to fit: R_MIPS_26 against `really_flush_cache'

If only I knew what the name of the cache flush function was, and that it had a matching calling signature.

Flush function signature

The documentation is very poor on the arguments to be provided to the flush cache function, and this seems to vary from platform to platform.

In the documentation, of special relevance are CLEAR_INSN_CACHE and TARGET_TRAMPOLINE_INIT which in one case is given this definition:

#define CLEAR_INSN_CACHE(beg, end) mips_sync_icache (beg, end - beg)

By adding debugging to inspect the arguments of a flush-cache function wrapper, I see that this matches the arguments passed when compiling under my VxWorks tool chain; but given that those macros can be re-defined it may not match GCC building for a different system so I must acknowledge that my flush function is not portable.

Possibly a portable implementation is given here, and I note that other JIT systems (e.g. SLJT used by PCRE-SLJT) and code generators (libffcall) attempt to provide their own portable implementations but which may not work on some multi-core systems.

Cache Operations


#include <asm/cachectl.h>
int cacheflush(char *addr, int nbytes, int cache);


VxWorks provides cacheInvalidate, cacheFlush and cacheClear (invalidate and flush); and particularly cacheTextUpdate which seems just what we want:

If cacheTextUpdate works, we could just have the compiler option:


Or we might prefer a wrapper that allows us to vary it, or add debugging statements:

#include <cacheLib.h>

void _flush_cache(char* beg, size_t bytes) {
    cacheFlush(DATA_CACHE, beg, bytes);
    cacheInvalidate(INSTRUCTION_CACHE, beg, bytes);


#include <cacheLib.h>
void _flush_cache(char* beg, size_t bytes) {
    cacheTextUpdate(beg, bytes);

A clue, search for cacheTextUpdate in this initial attempt to add MIPS vxWorks support: 

It Lives!

Does it work? Well, the trampoline works; but is it treating the cache properly or will it randomly fail? I don't know.

#include <cacheLib.h>

void _flush_cache(char* beg, size_t bytes) {
    cacheTextUpdate(beg, bytes);

void zag(void(*zog)(void)) {

void zig() {
  const char* where = NULL;

  void zog() {
    printf("ZOG from %s\n", where);

  where = __FUNCTION__;


Call function zag( ) to see

ZOG from zig

If you want to see the generated trampoline code, use GCC flags -save-temps -fverbose-asm and then look at the .s file.

An end note for smug x86 users

If you do the sub-word write using a regular store, you are now invoking the _one_ non-coherent part of the x86 memory pipeline: the store buffer. Normal stores can (and will) be forwarded to subsequent loads from the store buffer, and they are not strongly ordered wrt cache coherency while they are buffered.
...if you do a regular store to a partial word, with no serializing instructions between that and a subsequent load of the whole word, the value of the store can be bypassed from the store buffer, and the load from the other part of the word can be carried out _before_ the store has actually gotten that cacheline exclusively!
So when you do
  movb reg,(byteptr)
  movl (byteptr),reg
you may actually get old data in the upper 24 bits, along with new data in the lower 8.
I think.
Anyway, be careful. The cacheline itself will always be coherent, but the store buffer is not going to be part of the coherency rules, and without serialization (or locked ops), you _are_ going to invoke the store buffer!