Thursday, 7 December 2023

Even More on Attachable Scopes

 When I wrote More on Attachable Scopes I forgot to refer to my StackOverflow Post convert gcc forward-declared nested function to clang blocks where most of the thinking was done.

An archive of my question and first unsuccessful answer and second successful answer is given here.

convert gcc forward-declared nested function to clang blocks

I know this is impossible because I've spent hours on it and also read these below but I am determined to approximate a clang block forward definition even if it takes linker tricks or inline assembly:

I did my homework

I'm using a macro library written for gcc (which is 1,000 - 10,000 times faster than jansson at outputting large quantities of json) which encourages horrors such as this test case:

JSON_with_BUFFERED_WRITER((1001, result),
    JSON_OBJ(
        JSON_STRING("logGroupName", "\005group©"),
        JSON_STRING("logStreamName", "stream")
    )
) (out) {
    strncpy(out, data, length); // warning not checking target bounds
    out+=length; 
    *out=0; 
    return 0; 
};

Yes, that block at the end is actually partially sucked into the macro expansion to produce a function definition something like this below:

int __WRITER__1849_emit(const char *data, int length, typeof(result) out) {
  strncpy(out, data, length);
  out += length;
  *out = 0;
  return 0;
};

((out) appearing after the macro expansion actually becomes an argument to a macro name which is the result of the macro expansion, and which consumes out and then emits it as the last item of the generated parameter list, how sick is that? I'm very proud of that!)

Under gcc, this function (by means of an early forward definition in the macro expansion auto int __WRITER__1849_emit(const char *data, int length, typeof(result) out); is assigned to a struct instance somewhere in the macro expansion and invoked as a callback as the macro generated code is executed.

Most of the library has been converted to also work under clang by means of clang blocks, generally with that final function prefixed with a slightly late assignment of the block to the struct field ready for use.

__WRITER__1849.emit = ^int(const char *data, int length, typeof(result) out) {...

and that works fine where no use is made until after that assignment but in the case of JSON_with_BUFFERED_WRITER the processing is happening inside the macro arguments expansion before the assignment can take place.

So it is clear how important forward declarations are. The values are all known at link time anyway, but I need the value to be inserted into the struct instance static initialization before the generated code from the macro executes.

I know could beat-up the usage and have a 2-stage declare-and-then-use but it isn't that kind of quitting that has me waking up wanting to write more code (probably of the type that should never be written, but this isn't reddit so don't judge me).

Although that final function can get sucked into the tail of the macro expansion, and even be assigned to something by means of the macro tail expanding to an l-value, I don't have the option to have anything invoked after that assignment. C has no setters, I can't work out how to have anything go out of scope to run a destructor (within which I could do something) after the assignment, so that's a dead end.

I've tried persuading clang with attempts at alias attributes:

static __attribute__((alias("__WRITER__1849_emit__"))) int (^__WRITER__1849_emit)(const char *data, int length, typeof(result) cookie);
...
__attribute__((external)) static int (^__WRITER__1849_emit__)(const char *data, int length, typeof(result) cookie) =
^int(const char *data, int length, typeof(result) out) {
  strncpy(out, data, length);
  out += length;
  *out = 0;
  return 0;
};

but it doesn't work, and inspecting the .s file, the alias attribute has had no effect.

It's not possible to add the reverse __attribute__((alias("__WRITER__1849_emit")) to the static block definition as it was the "forward" declaration, perhaps because it is statically defined.

This hints at use of extern but I haven't yet divined the required trick to have the subsequent static definition be linked and matched with the extern definition:

extern __attribute__((alias("__WRITER__1849_emit__"))) int (^__WRITER__1849_emit)(const char *data, int length, typeof(result) cookie);
...
static int (^__WRITER__1849_emit__)(const char *data, int length, typeof(result) cookie) =
^int(const char *data, int length, typeof(result) out) {
  strncpy(out, data, length);
  out += length;
  *out = 0;
  return 0;
};

This merely gives undefined reference to '__WRITER__1849_emit' and nothing in the .s file to indicate that any aliasing or linking between the symbols will occur, and the only definition in the .s file of the extern "forward declaration" hack is:

    .addrsig_sym __WRITER__1849_emit

So now I'm looking at a way to declare/emit a global symbol from the static block definition in the function and/or maybe some inline assembly, which can map up to the extern "forward declaration"

Any tips? Special linker scripts?

It looks like I need to emit something like this somehow, to go with the extern method:

        .globl  __WRITER__1849_emit
.set __WRITER__1849_emit, __WRITER__1849_emit__

maybe like this:

__asm__("        .globl  __WRITER__1849_emit\n"
        ".set __WRITER__1849_emit, __WRITER__1849_emit__\n");

Well it almost works; a little warning: warning: relocation against '__WRITER__1849_emit__' in read-only section '.text' but the real problem is that it is emitted before __WRITER__1849_emit__\ is defined (a static const initializer) giving rise to: undefined reference to '__WRITER__1849_emit__' during link, logged against the use of __WRITER__1849_emit the ostensible extern "forward declaration"

I need to qualify my symbols as clang does, this worked in combination with extern! and without the need for .globl

extern int (^__WRITER__1849_emit)(const char *data, int length, typeof(result) cookie);
__asm__(".set __WRITER__1849_emit, selftest_json.__WRITER__1849_emit__\n");

So I just need to normalise the generation of the prefix selftest_json. which might be hard as that is the outer function name which is not available as a character sequence at macro time, but it becomes hopeful!

I'd rather make this work and then not use the solution because it is unclean, than not be able to make it work. I will bend clang to my will!

I'm using Ubuntu clang version 15.0.7

Maybe I could substitute a global function pointer which receives a symbol name as an argument to use with dlopen/dlsym to find the real address - nope the block is "data" and dlsym doesn't find it.

First Answer

This original answer is not complete, a complete answer to this question is below

In clang, there is no declaration of a block prior to its definition, but there is a workaround.

We can make an extern declaration as described here and then satisfy that extern declaration anywhere in the source file with an asm .equiv like this:

extern int (^BLOCK_DECLARATION)(const char *data, int length, typeof(result) cookie);
...
__asm__(".equiv BLOCK_DECLARATION, FUNCTION_NAME.BLOCK_DEFINITION\n");
static int (^BLOCK_DEFINITION)(const char *data, int length, typeof(result) cookie) =
^int(const char *data, int length, typeof(result) out)  {
  ...
  return 0;
};

The asm statement won't take effect until link time at which point the code and data records for the block definition are already emitted.

The thing to note is that the asm code needs the name of the function (given as FUNCTION_NAME in this example) in order to prepend to the BLOCK_DEFINITION name in order to produce the asm: .equiv BLOCK_DECLARATION, FUNCTION_NAME.BLOCK_DEFINITION

This is because clang generates static variable symbols as the function name, a dot, and then the symbol name. I can't find any way to override this.

The function name is not available as a token at pre-processor time, and I've not managed to find any asm or alias tricks to set a symbol whose name we do know to be defined at a similar location to what is the next static variable definition (and they could possibly move about anyway).

A custom section __attribute__((section ("BLOCK_DEFINITION"))) might protect against the movement but don't know any way to take useful advantage of knowing the name of the section that contains only one item.

I don't find any linker flags for gnu-ld or llvm-linker that can usefully and generally map one symbol to another.

So with this method you definitely can do block forward definitions if you don't mind repeating the current function name somewhere in your code, or defining it as the first line in your function or something like that.

The caution is that there is no type checking going on, if your definition is not compatible with the declaration, then bad things might happen. But if your code is generated by a macro you probably have full control over this.

Second Answer

This answer is complete and tested with a sample program provided

The macro composition is required to be:

  1. declare block
  2. use block
  3. define block

because the function/block definition attaches to the macro expansion but is not part of the parameters.

However, with clang we can't use the block until it is defined, but the definition must come last. Sadly there is no universal forward-declaration for blocks, although my previous answer made a good attempt, it failed at not having pre-processor access to the containing function name which was necessary to .equiv the fake extern forward declaration to the actual definition later.

However after re-reading some work of @jens-gustedt and re-reading some previous discussion on his blog I have a better solution that is more complete and also more standards compliant (although clang objective-C style blocks are themselves quite a diversion).

Fortunately, in standard C, we can re-arrange the order of execution of statements from order in source code: 1 → 2 → 3 → 4 to order of execution: 1 → 2 → 4 → 3 which is what we want, as that allows us to make the block assignment at 4 but have the assignment take place before it is used at 3.

The for statement is what causes this re-ordering, with a little hackery

for(1;2;3) **4**

will execute each of 1 2 4 3 once in that order, which is managed something like this (skipping position 2 which we aren't too bothered about as we break at 3)

... for(struct S s = { ... };;({ doing stuff with s.func() ; break; })) s.func = ...

clearly this way s.func will be assigned before use, even though the assignment code appears after the using code.

Technically such constructs block the action of break in the trailing scope, that is not a problem here is intended to be the trailing scope of a block/function body anyway.

A working minimal example that can be de-constructed and re-used is:

/* Demo of forward declaration method for clang style static objective-C blocks

   With gcc a forward declaration of a nested function is sufficient to assign
   and link against; that is: you can assign the function address after the
   forward declaration but before the definition.
   
   With clang objective-C style static blocks this is not possible, you can't
   take the address of the block until after the definition.
   
   By cunning use of the for(;;); construct we can re-order execution so that
   the definition can occur after the use in the source file, but take effect
   before use at runtime.
   
   Author: sam@liddicott.com
   for: https://stackoverflow.com/questions/76771957/convert-gcc-forward-declared-nested-function-to-clang-blocks

   0. Save this program as: trailing-block.c
   1. Make sure you have blocks runtime, e.g.
      sudo apt-get install libblocksruntime-dev
   2. build with: clang -fblocks -lBlocksRuntime trailing-block.c -o trailing-block
   3. run with: ./trailing-block
   4. expect output:
Are you ready? 0x7ffea2f5cea0
Message: 0x7ffea2f5ce70 Hello world

   Thus showing that a block invocation and a task passed as a macro parameter
   would be executed after the definition of the block whose body follows the
   the macro invocation and expansion site
*/
#include <stdio.h>
#include <string.h>

#define THING(text, task) \
struct S; \
for(struct S { \
      int size; \
      const char* message; \
      int count; \
      int (^emit)(const char *data, int length, struct S* s); \
    } s = { \
      .size=strlen(text), \
      .message=text \
    }; \
    ; \
    ({ task; (void)s.emit(s.message, s.size, &s); break; }) \
    ) s.emit = ^int(const char *data, int length, struct S* s) \

int main(int argc, char** argv) {
  THING("Hello world", printf("Are you ready? %p\n", &s)) {
    printf("Message: %p %.*s\n", &s, length, data);
    s->count++;
    return 0;
  };
}

I have to acknowledge @jens-gustedt as an inspiration and precise implementer of clearly and well thought ideas, compared with whom I am just a dirty hacker of the lowest order. What he does with standard C will amaze anybody P99 By using new tools from C99 we implement default arguments for functions, scope bound resource management, transparent allocation and initialization, ..., defer This is a reference implementation of a defer feature for C. This is a tool that is designed with golang's defer/panic/recover in mind, adapting it to the specificities of C. It aims to provide four properties to the handling of circumstantial cleanup or error handling code, and where and how he draws the line for going beyond standard C is a revelation, e.g. schnell locally replaceable code snippets can be used to easily specify and prototype compiler and language enhancements for the C language that work by local source-to-source transformation, Modular C The change to the C language is minimal since we only add one feature, composed identifiers, to the core language

Follow his blog and Check out his Modern C book.

Additional Comments

I also learned that for(;;); is a good mandatory NOP to follow a label where a statement is absolutely required, e.g. if (0) { label_1: ... } where the user doesn't want to provide an action for execution when break is detected in the trailing scope captured by a the trailing for of a preceding macro. (yes, break propagation becomes desirable)

If you don't have libBlocksRuntime.a or libBlocksRuntime.so, you can just insert this line near the top of the example, it is sufficient for our minimal non-capturing blocks. void * _NSConcreteGlobalBlock[32] = { 0 }; I'm as surprised as you are. Looking at the generated assembly and the libBlocksRuntime source, it seems safe enough.



Thursday, 5 October 2023

Extra monitor won't sync under kernel 6.2 but OK on 5.19

I have a Dell XPS 17 9700 with dock driving 3 monitors:

  Monitor-1: 1366x768 (old viewsonic LCD)
  Monitor-2: 5120x1440 (big and wiiide)
  Monitor-3: 1920x1200 (laptop LCD)

Under kernel 5.19 all is well, but under kernel 6.2 (under Mint Vera or Victoria) the Monitor-1 viewsonic won't sync properly:


I think the whole image is there, but just squashed up with trailing junk.

I still have my 5.19 kernel available since before upgrade so I can still work, but I'm not sure where to look to find what to blame for this. Do I need to manually specify a modeline?

Ubuntu 22.04.03 booted from USB has the same problem:


Ubuntu 23.10 with kernel 6.5 below works just fine:



And so fixed on Linuxmint 21.2 with:
sudo apt install linux-oem-22.04d

https://ubuntuhandbook.org/index.php/2023/08/linux-kernel-6-5-released-how-to-install-it-in-ubuntu/


Monday, 31 July 2023

More on attachable scopes

I previously called these braceless scopes, and ended up writing more in discussion with Jens that I did on my blog - but now I come to needed it, and further back in time than I expected I find this gem carbuncle below.

(By attachable-scope I mean a braced-scope that get's attached to the expansion of previous macro, perhaps as the body of a function definition, or a for loop, or even an else clause).

Hah! Your example there was the clue to propagating breaks.

Thanks to your tip on the gcc equivalence using BREAKOUT and BREAKIN I can contrive a bunch of macros that expand to this:

1
2
3
4
5
6
7
8
9
main() {
  int a;
  for (a=1; a<=2; a++) {
    printf("Main context a=%d\n", a);
 
    if (0) { break_line_6: break ; } else for (int o = 0; o >=0; ) if (o == 1) { printf("Detected break\n"); goto break_line_6; } else
      for (int i=0; !o && (o=1), i==0; o=-1, i=-1 ) { printf("Inner context\n"); break; }
  }
}

it does not require gcc-isms – sadly the C style is lacking but it functions as desired

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
#define GENSYM_CONCAT(name, salt) name ## salt
#define GENSYM2(name, salt) GENSYM_CONCAT(name, salt)
#define GENSYM(name) GENSYM2(name, __LINE__)
 
#define _BRACELESS_SCOPE(__label__, _declaration_) \
    if (0) { __label__: break ; } else for (int _o = 0; _o >=0; ) if (_o == 1) goto __label__; else \
      for (_declaration_; !_o && (_o=1); _o=-1) \
 
#define BRACELESS_SCOPE(_declaration_) _BRACELESS_SCOPE(GENSYM(break_line_) , _declaration_)
 
main() {
  int a;
  for (a=1; a<=2; a++) {
    printf("Main context a=%d\n", a);
 
    BRACELESS_SCOPE(char* message="hello") { printf("Inner context: %s\n", message); break; }
  }
}

The main deficiency is that a single macro cannot cannot ultimately expand to more than one of these constructs, nor can it be used twice on the same line, or the generated labels will conflict. __COUNTER__ where supported could be passed instead of __LINE__. I guess this is something you already addressed in P99 if I look again…

It's just what I needed as I convert a narrating block logger (that annotates progress through a block scope) from gcc-isms to clang-isms.


It originally expanded to a nested function forward definition, a setup block, a nested function invocation, a clean-up block, followed by a the first half of a nested function definition which sticks onto a following scope that becomes the body.


clang won't take nested functions (even without arguments) and clang's objective-C style blocks is no help as it can't access the variables in the calling scope as gcc does.


Knowing that I've passed this way before I look at this and this, before finally recovering the carbuncle above which is to be pressed into service.

Saturday, 8 July 2023

Fixing Linuxmint Audio on HP Envy x360

 Install Linuxmint 21.1 with secure boot turned off and wifi turned off

On first boot turn on secure boot

Join wifi, get all updates

sudo apt install linux-oem-22.04c [https://wiki.ubuntu.com/Kernel/OEMKernel]

git clone https://github.com/xoocoon/hp-15-ew0xxx-snd-fix # see https://h30434.www3.hp.com/t5/Notebook-Audio/No-sound-from-internal-speakers-using-Linux/m-p/8678403/highlight/true#M121500

cat /sys/class/sound/hwC0D0/subsystem_id

0x103c8a28

edit setup_snd-hda-codec-realtek.sh to remove this line
+  SND_PCI_QUIRK(0x103c, 0x8a29, "HP Envy x360 15-ew0xxx", ALC287_FIXUP_CS35L41_I2C_2),
and add this line
+   SND_PCI_QUIRK(0x103c, 0x8a28, "HP ENVY x360 2-in-1 Laptop", ALC287_FIXUP_CS35L41_I2C_2),

bash setup_snd-hda-scodec-cs35l41.sh
bash setup_snd-hda-codec-realtek.sh


Thursday, 25 May 2023

Run time detection: Does the terminal emulator support unicode?

You can test the terminal by setting the cursor position to column 1 and outputing a multibyte unicode character. If the cursor moves by more than 1 position then the terminal does not support unicode.

On in this case we emit a 3 byte sequence which is a zero width space, so if the cursor moves at all, the terminal cannot process unicode

IFS=$';\x1B[' read -p $'\r\xE2\x80\x8B\x1B[6n\r   \r' -d R -rst 1 _ _ _ X _ </dev/tty 2>/dev/tty && test "$X" = 1

Here we output \r to get to position 1 and then emit a 3 byte sequence which is a zero width space, and then emit ESC [ 6n which asks the cursor position, followed by \r \r to overwrite any junk that will have appeared if the terminal handled each byte as a separate character.

Then we read the cusor position with a 1 second timeout and check whether the X position is position 1 which it will be if the terminal can process unicode.

A better function is:

is-tty-unicode() {
  local X

  test -c /dev/tty &&
  if test -t 0
  then IFS=$';\x1B[' read -p $'\r\xE2\x80\x8B\x1B[6n\r   \r' -d R -rst 1 _ _ _ X _ 2>&1
  fi <>/dev/tty && test "$X" = 1
}

Thursday, 27 October 2022

Is Rust sacrificing 3U safety for stability?

My original post is too malign or pejorative or something, apparently imputing dark motives when I mean only to highlight the ostensible appearance. So I try again.

[This rant is a work in Progress and will receive regular updates]

TL;DR: This is a general call for prioritisation of breaking changes over preservation of unexpected undetectable unsafe or undefined behaviour in Rust, and for full transparency with respect to such faults. (And maybe there is and I haven't seen it).

Will Rust sacrifice 3U safety for stability?

3U safety refers to safety from unexpected undetectable unsafe behaviours (or, if you prefer, unexpected undetectable undefined behaviours).

The utility of C and other ancient software development tools harks back to those ages when they could be compared to that dog walking on its hind legs: The wonder isn't that it does it well, but that it does it at all. And in comparison to what preceded them, they did it well.

But it's no longer enough.

The instability of such tools has been a major hindrance to the development and maintenance of secure software.

Youthfully optimistic C developers blithely consider C to be a portable macro assembler. The experience that comes with time teaches them that it isn't. 

C compilers have been secretly throwing away increasing chunks of code for decades, and not just in simple cases like throwing away sub-statement code by remembering that a variable is already in a register, or throwing away whole statements like function calls that zero a sensitive buffer (because it wasn't read-from again), any new compiler edition might throw away whole blocks of code that you inserted for security checks, to fix bugs -- and not tell you. 

This is not an attack on C compiler authors or maintainers but is to illustrate (due to insufficient congruence of priorities) the shifting sands on which much software has been, and is being, built.

Will Rust go the same way?

Stability or Speed?

In a much-contested view (especially by compiler authors) D. J. Bernstein controversially wrote in 2015:

...gcc and clang both feel entitled to arbitrarily change the behavior of "undefined" programs. Pretty much every real-world C program is "undefined" according to the C "standard", and new compiler "optimizations" often produce new security holes in the resulting object code, as illustrated by

and many other examples. Crypto code isn't magically immune to this--- one can easily see how today's crypto code audits will be compromised by tomorrow's compiler optimizations, even if the code is slightly too complicated for today's compilers to screw up.

If you want the short version of the dispute, C compilers take advantage of what to developers is an uncontrollably increasing collection of what is termed "undefined behaviour" in order to make speed optimizations because... speed is obviously what you want, and has been one of the main metrics in the compiler wars. (Smallness of output being the other main metric).

It plays out like this: There is an ever-growing bunch of rules that developers must adhere to, and if they don't, the compiler can do what it likes, and it's your fault. When we say do what it likes we mean things like: ignore your code. And compiler writers stick to that like glue, even when it makes it exceedingly difficult to write secure software, and when the introduction of new undefined behaviour makes existing secure software suddenly become insecure when compiled with a new compiler. 

You have the responsibility to look at the compiler changes list and put all the right flags in your makefile to get the old behaviour. Flags that the old compiler won't recognize. How often have you seen a makefile that assembled CFLAGS flags based on the compiler version?

Now the language designers and compiler authors are right because they make the rules, but the fight to produce correct and safe software becomes more and more like a fight, and every edition of the compiler tools behaves more and more like a fully automatic foot-gun - all the better to shoot yourself in the foot with, and the bullets you fire at security problems are optimised away.

Clearly, compiler authors and compiler users have conflicting views on what the compiler should do. And if you don't believe me, read this sorry tale of the assertions that were optimized away.

I don't exaggerate. New tools will produce worse code. And the compiler doesn't warn you. Because in your makefile you forgot to turn on the new warning flag that didn't exist when you wrote the makefile.

Simply recompiling previously working, secure code with a newer version of the compiler can introduce security vulnerabilities. While the new behaviour can be disabled with a flag, existing makefiles do not have that flag set, obviously. And since no warning is produced, it is not obvious to the developer that the previously reasonable behaviour has changed.

I think I want a compiler that warns me when it throws away lines of code that I took the trouble to write and debug, but what do I know? It's really hard. But so is having your code silently yanked away.

Consider what happens when the default code optimisation settings change in the next release of the compiler. As one chap puts it:

Value range propagation now assumes that the this pointer of C++ member functions is non-null. This eliminates common null pointer checks but also breaks some non-conforming code-bases (such as Qt-5, Chromium, KDevelop). As a temporary work-around -fno-delete-null-pointer-checks can be used. Wrong code can be identified by using -fsanitize=undefined.

http://blog.fefe.de/?ts=a9de792d 

A more even-handed treatment of what compiler authors are trying to achieve is given here: https://gist.github.com/rygorous/e0f055bfb74e3d5f0af20690759de5a7 

It all comes down to a persistent and ever-escalating mismatch of expectations. The scarcely known obligation upon software developers is to be aware of the increasing instances of the increasing definitions of undefined behaviour that exist in existing debugged safe and working, and of the changes to default optimisations, so that with each compiler release they must remove the last year's best practice and replace it with this year's best practice.

And our current state is where:

...the root cause of approximately 70% of security vulnerabilities that Microsoft fixes and assigns a CVE (Common Vulnerabilities and Exposures) are due to memory safety issues. ​

This is despite mitigations including intense code review, training, static analysis, and more.​

While many experienced programmers can write correct systems-level code, it’s clear that no matter the amount of mitigations put in place, it is near impossible to write memory-safe code using traditional systems-level programming languages at scale.

we-need-a-safer-systems-programming-language

Let's expand upon those mitigations. Everyone is compiling C with -Werror and -Wall, but it's not enough. Many look harder by paying good time and money for Coverity, Klocwork, Black Duck, Code Sonar, etc, and trying every C compiler they can get their hands on for maximum warnings, and the information on every change that they can make.

If the compiler authors insist that they are correct (as they may) then the software developers need new tools. For me, and for the author of that previous quote, Rust is that tool.

From C, we have had one long round of compiler-imposed instability in pursuit of speed, and another more recent self-imposed round of instability for safety.

So to answer the question heading this section: 

If you want stability (I mean such as for security bugs to stay fixed) the answer is clear: do not update your compiler tools.

If you want speed, you need the latest compiler tools. Your programs will be very fast, but you will not know what they are doing. Of course, there are ways to find out, but the answer will vary from one compiler edition to the next.

In the question of stability or speed, the C compiler maintainers chose speed for too many decades.

What about Rust?

Stability or Safety?

Don't think that Rust isn't optimising your code. It is. The most popular Rust implementation is based on Clang/LLVM and has various bugs originating in the optimising compiler backend, throwing away code that Rust wants to keep.

But Rust makes certain guarantees of safety, in the same way that C promises to throw away apparently arbitrary multi-line chunks of code. 

You can see the failure list of what are termed Unsound Bugs: Today I see 65 open and 307 closed. If I exclude C bugs from that list then I see 10 open and 86 closed. (I maybe misusing the bug filter system).