Thursday 7 December 2023

Even More on Attachable Scopes

 When I wrote More on Attachable Scopes I forgot to refer to my StackOverflow Post convert gcc forward-declared nested function to clang blocks where most of the thinking was done.

An archive of my question and first unsuccessful answer and second successful answer is given here.

convert gcc forward-declared nested function to clang blocks

I know this is impossible because I've spent hours on it and also read these below but I am determined to approximate a clang block forward definition even if it takes linker tricks or inline assembly:

I did my homework

I'm using a macro library written for gcc (which is 1,000 - 10,000 times faster than jansson at outputting large quantities of json) which encourages horrors such as this test case:

JSON_with_BUFFERED_WRITER((1001, result),
    JSON_OBJ(
        JSON_STRING("logGroupName", "\005group©"),
        JSON_STRING("logStreamName", "stream")
    )
) (out) {
    strncpy(out, data, length); // warning not checking target bounds
    out+=length; 
    *out=0; 
    return 0; 
};

Yes, that block at the end is actually partially sucked into the macro expansion to produce a function definition something like this below:

int __WRITER__1849_emit(const char *data, int length, typeof(result) out) {
  strncpy(out, data, length);
  out += length;
  *out = 0;
  return 0;
};

((out) appearing after the macro expansion actually becomes an argument to a macro name which is the result of the macro expansion, and which consumes out and then emits it as the last item of the generated parameter list, how sick is that? I'm very proud of that!)

Under gcc, this function (by means of an early forward definition in the macro expansion auto int __WRITER__1849_emit(const char *data, int length, typeof(result) out); is assigned to a struct instance somewhere in the macro expansion and invoked as a callback as the macro generated code is executed.

Most of the library has been converted to also work under clang by means of clang blocks, generally with that final function prefixed with a slightly late assignment of the block to the struct field ready for use.

__WRITER__1849.emit = ^int(const char *data, int length, typeof(result) out) {...

and that works fine where no use is made until after that assignment but in the case of JSON_with_BUFFERED_WRITER the processing is happening inside the macro arguments expansion before the assignment can take place.

So it is clear how important forward declarations are. The values are all known at link time anyway, but I need the value to be inserted into the struct instance static initialization before the generated code from the macro executes.

I know could beat-up the usage and have a 2-stage declare-and-then-use but it isn't that kind of quitting that has me waking up wanting to write more code (probably of the type that should never be written, but this isn't reddit so don't judge me).

Although that final function can get sucked into the tail of the macro expansion, and even be assigned to something by means of the macro tail expanding to an l-value, I don't have the option to have anything invoked after that assignment. C has no setters, I can't work out how to have anything go out of scope to run a destructor (within which I could do something) after the assignment, so that's a dead end.

I've tried persuading clang with attempts at alias attributes:

static __attribute__((alias("__WRITER__1849_emit__"))) int (^__WRITER__1849_emit)(const char *data, int length, typeof(result) cookie);
...
__attribute__((external)) static int (^__WRITER__1849_emit__)(const char *data, int length, typeof(result) cookie) =
^int(const char *data, int length, typeof(result) out) {
  strncpy(out, data, length);
  out += length;
  *out = 0;
  return 0;
};

but it doesn't work, and inspecting the .s file, the alias attribute has had no effect.

It's not possible to add the reverse __attribute__((alias("__WRITER__1849_emit")) to the static block definition as it was the "forward" declaration, perhaps because it is statically defined.

This hints at use of extern but I haven't yet divined the required trick to have the subsequent static definition be linked and matched with the extern definition:

extern __attribute__((alias("__WRITER__1849_emit__"))) int (^__WRITER__1849_emit)(const char *data, int length, typeof(result) cookie);
...
static int (^__WRITER__1849_emit__)(const char *data, int length, typeof(result) cookie) =
^int(const char *data, int length, typeof(result) out) {
  strncpy(out, data, length);
  out += length;
  *out = 0;
  return 0;
};

This merely gives undefined reference to '__WRITER__1849_emit' and nothing in the .s file to indicate that any aliasing or linking between the symbols will occur, and the only definition in the .s file of the extern "forward declaration" hack is:

    .addrsig_sym __WRITER__1849_emit

So now I'm looking at a way to declare/emit a global symbol from the static block definition in the function and/or maybe some inline assembly, which can map up to the extern "forward declaration"

Any tips? Special linker scripts?

It looks like I need to emit something like this somehow, to go with the extern method:

        .globl  __WRITER__1849_emit
.set __WRITER__1849_emit, __WRITER__1849_emit__

maybe like this:

__asm__("        .globl  __WRITER__1849_emit\n"
        ".set __WRITER__1849_emit, __WRITER__1849_emit__\n");

Well it almost works; a little warning: warning: relocation against '__WRITER__1849_emit__' in read-only section '.text' but the real problem is that it is emitted before __WRITER__1849_emit__\ is defined (a static const initializer) giving rise to: undefined reference to '__WRITER__1849_emit__' during link, logged against the use of __WRITER__1849_emit the ostensible extern "forward declaration"

I need to qualify my symbols as clang does, this worked in combination with extern! and without the need for .globl

extern int (^__WRITER__1849_emit)(const char *data, int length, typeof(result) cookie);
__asm__(".set __WRITER__1849_emit, selftest_json.__WRITER__1849_emit__\n");

So I just need to normalise the generation of the prefix selftest_json. which might be hard as that is the outer function name which is not available as a character sequence at macro time, but it becomes hopeful!

I'd rather make this work and then not use the solution because it is unclean, than not be able to make it work. I will bend clang to my will!

I'm using Ubuntu clang version 15.0.7

Maybe I could substitute a global function pointer which receives a symbol name as an argument to use with dlopen/dlsym to find the real address - nope the block is "data" and dlsym doesn't find it.

First Answer

This original answer is not complete, a complete answer to this question is below

In clang, there is no declaration of a block prior to its definition, but there is a workaround.

We can make an extern declaration as described here and then satisfy that extern declaration anywhere in the source file with an asm .equiv like this:

extern int (^BLOCK_DECLARATION)(const char *data, int length, typeof(result) cookie);
...
__asm__(".equiv BLOCK_DECLARATION, FUNCTION_NAME.BLOCK_DEFINITION\n");
static int (^BLOCK_DEFINITION)(const char *data, int length, typeof(result) cookie) =
^int(const char *data, int length, typeof(result) out)  {
  ...
  return 0;
};

The asm statement won't take effect until link time at which point the code and data records for the block definition are already emitted.

The thing to note is that the asm code needs the name of the function (given as FUNCTION_NAME in this example) in order to prepend to the BLOCK_DEFINITION name in order to produce the asm: .equiv BLOCK_DECLARATION, FUNCTION_NAME.BLOCK_DEFINITION

This is because clang generates static variable symbols as the function name, a dot, and then the symbol name. I can't find any way to override this.

The function name is not available as a token at pre-processor time, and I've not managed to find any asm or alias tricks to set a symbol whose name we do know to be defined at a similar location to what is the next static variable definition (and they could possibly move about anyway).

A custom section __attribute__((section ("BLOCK_DEFINITION"))) might protect against the movement but don't know any way to take useful advantage of knowing the name of the section that contains only one item.

I don't find any linker flags for gnu-ld or llvm-linker that can usefully and generally map one symbol to another.

So with this method you definitely can do block forward definitions if you don't mind repeating the current function name somewhere in your code, or defining it as the first line in your function or something like that.

The caution is that there is no type checking going on, if your definition is not compatible with the declaration, then bad things might happen. But if your code is generated by a macro you probably have full control over this.

Second Answer

This answer is complete and tested with a sample program provided

The macro composition is required to be:

  1. declare block
  2. use block
  3. define block

because the function/block definition attaches to the macro expansion but is not part of the parameters.

However, with clang we can't use the block until it is defined, but the definition must come last. Sadly there is no universal forward-declaration for blocks, although my previous answer made a good attempt, it failed at not having pre-processor access to the containing function name which was necessary to .equiv the fake extern forward declaration to the actual definition later.

However after re-reading some work of @jens-gustedt and re-reading some previous discussion on his blog I have a better solution that is more complete and also more standards compliant (although clang objective-C style blocks are themselves quite a diversion).

Fortunately, in standard C, we can re-arrange the order of execution of statements from order in source code: 1 → 2 → 3 → 4 to order of execution: 1 → 2 → 4 → 3 which is what we want, as that allows us to make the block assignment at 4 but have the assignment take place before it is used at 3.

The for statement is what causes this re-ordering, with a little hackery

for(1;2;3) **4**

will execute each of 1 2 4 3 once in that order, which is managed something like this (skipping position 2 which we aren't too bothered about as we break at 3)

... for(struct S s = { ... };;({ doing stuff with s.func() ; break; })) s.func = ...

clearly this way s.func will be assigned before use, even though the assignment code appears after the using code.

Technically such constructs block the action of break in the trailing scope, that is not a problem here is intended to be the trailing scope of a block/function body anyway.

A working minimal example that can be de-constructed and re-used is:

/* Demo of forward declaration method for clang style static objective-C blocks

   With gcc a forward declaration of a nested function is sufficient to assign
   and link against; that is: you can assign the function address after the
   forward declaration but before the definition.
   
   With clang objective-C style static blocks this is not possible, you can't
   take the address of the block until after the definition.
   
   By cunning use of the for(;;); construct we can re-order execution so that
   the definition can occur after the use in the source file, but take effect
   before use at runtime.
   
   Author: sam@liddicott.com
   for: https://stackoverflow.com/questions/76771957/convert-gcc-forward-declared-nested-function-to-clang-blocks

   0. Save this program as: trailing-block.c
   1. Make sure you have blocks runtime, e.g.
      sudo apt-get install libblocksruntime-dev
   2. build with: clang -fblocks -lBlocksRuntime trailing-block.c -o trailing-block
   3. run with: ./trailing-block
   4. expect output:
Are you ready? 0x7ffea2f5cea0
Message: 0x7ffea2f5ce70 Hello world

   Thus showing that a block invocation and a task passed as a macro parameter
   would be executed after the definition of the block whose body follows the
   the macro invocation and expansion site
*/
#include <stdio.h>
#include <string.h>

#define THING(text, task) \
struct S; \
for(struct S { \
      int size; \
      const char* message; \
      int count; \
      int (^emit)(const char *data, int length, struct S* s); \
    } s = { \
      .size=strlen(text), \
      .message=text \
    }; \
    ; \
    ({ task; (void)s.emit(s.message, s.size, &s); break; }) \
    ) s.emit = ^int(const char *data, int length, struct S* s) \

int main(int argc, char** argv) {
  THING("Hello world", printf("Are you ready? %p\n", &s)) {
    printf("Message: %p %.*s\n", &s, length, data);
    s->count++;
    return 0;
  };
}

I have to acknowledge @jens-gustedt as an inspiration and precise implementer of clearly and well thought ideas, compared with whom I am just a dirty hacker of the lowest order. What he does with standard C will amaze anybody P99 By using new tools from C99 we implement default arguments for functions, scope bound resource management, transparent allocation and initialization, ...defer This is a reference implementation of a defer feature for C. This is a tool that is designed with golang's defer/panic/recover in mind, adapting it to the specificities of C. It aims to provide four properties to the handling of circumstantial cleanup or error handling code, and where and how he draws the line for going beyond standard C is a revelation, e.g. schnell locally replaceable code snippets can be used to easily specify and prototype compiler and language enhancements for the C language that work by local source-to-source transformationModular C The change to the C language is minimal since we only add one feature, composed identifiers, to the core language

Follow his blog and Check out his Modern C book.

Additional Comments

I also learned that for(;;); is a good mandatory NOP to follow a label where a statement is absolutely required, e.g. if (0) { label_1: ... } where the user doesn't want to provide an action for execution when break is detected in the trailing scope captured by a the trailing for of a preceding macro. (yes, break propagation becomes desirable)

If you don't have libBlocksRuntime.a or libBlocksRuntime.so, you can just insert this line near the top of the example, it is sufficient for our minimal non-capturing blocks. void * _NSConcreteGlobalBlock[32] = { 0 }; I'm as surprised as you are. Looking at the generated assembly and the libBlocksRuntime source, it seems safe enough.