Run time detection: Does the terminal emulator support unicode?

You can test the terminal by setting the cursor position to column 1 and outputing a multibyte unicode character. If the cursor moves by more than 1 position then the terminal does not support unicode.

On in this case we emit a 3 byte sequence which is a zero width space, so if the cursor moves at all, the terminal cannot process unicode

IFS=$';\x1B[' read -p $'\r\xE2\x80\x8B\x1B[6n\r   \r' -d R -rst 1 _ _ _ X _ </dev/tty 2>/dev/tty && test "$X" = 1

Here we output \r to get to position 1 and then emit a 3 byte sequence which is a zero width space, and then emit ESC [ 6n which asks the cursor position, followed by \r \r to overwrite any junk that will have appeared if the terminal handled each byte as a separate character.

Then we read the cusor position with a 1 second timeout and check whether the X position is position 1 which it will be if the terminal can process unicode.

A better function is:

is-tty-unicode() {
  local X

  test -c /dev/tty &&
  if test -t 0
  then IFS=$';\x1B[' read -p $'\r\xE2\x80\x8B\x1B[6n\r   \r' -d R -rst 1 _ _ _ X _ 2>&1
  fi <>/dev/tty && test "$X" = 1
}




Is Rust sacrificing 3U safety for stability?

My original post is too malign or pejorative or something, apparently imputing dark motives when I mean only to highlight the ostensible appearance. So I try again.

[This rant is a work in Progress and will receive regular updates]

TL;DR: This is a general call for prioritisation of breaking changes over preservation of unexpected undetectable unsafe or undefined behaviour in Rust, and for full transparency with respect to such faults. (And maybe there is and I haven't seen it).

Will Rust sacrifice 3U safety for stability?

3U safety refers to safety from unexpected undetectable unsafe behaviours (or, if you prefer, unexpected undetectable undefined behaviours).

The utility of C and other ancient software development tools harks back to those ages when they could be compared to that dog walking on its hind legs: The wonder isn't that it does it well, but that it does it at all. And in comparison to what preceded them, they did it well.

But it's no longer enough.

The instability of such tools has been a major hindrance to the development and maintenance of secure software.

Youthfully optimistic C developers blithely consider C to be a portable macro assembler. The experience that comes with time teaches them that it isn't. 

C compilers have been secretly throwing away increasing chunks of code for decades, and not just in simple cases like throwing away sub-statement code by remembering that a variable is already in a register, or throwing away whole statements like function calls that zero a sensitive buffer (because it wasn't read-from again), any new compiler edition might throw away whole blocks of code that you inserted for security checks, to fix bugs -- and not tell you. 

This is not an attack on C compiler authors or maintainers but is to illustrate (due to insufficient congruence of priorities) the shifting sands on which much software has been, and is being, built.

Will Rust go the same way?

Stability or Speed?

In a much-contested view (especially by compiler authors) D. J. Bernstein controversially wrote in 2015:

...gcc and clang both feel entitled to arbitrarily change the behavior of "undefined" programs. Pretty much every real-world C program is "undefined" according to the C "standard", and new compiler "optimizations" often produce new security holes in the resulting object code, as illustrated by

• https://lwn.net/Articles/342330/ 
• https://kb.isc.org/article/AA-01167

and many other examples. Crypto code isn't magically immune to this--- one can easily see how today's crypto code audits will be compromised by tomorrow's compiler optimizations, even if the code is slightly too complicated for today's compilers to screw up.

If you want the short version of the dispute, C compilers take advantage of what to developers is an uncontrollably increasing collection of what is termed "undefined behaviour" in order to make speed optimizations because... speed is obviously what you want, and has been one of the main metrics in the compiler wars. (Smallness of output being the other main metric).

It plays out like this: There is an ever-growing bunch of rules that developers must adhere to, and if they don't, the compiler can do what it likes, and it's your fault. When we say do what it likes we mean things like: ignore your code. And compiler writers stick to that like glue, even when it makes it exceedingly difficult to write secure software, and when the introduction of new undefined behaviour makes existing secure software suddenly become insecure when compiled with a new compiler. 

You have the responsibility to look at the compiler changes list and put all the right flags in your makefile to get the old behaviour. Flags that the old compiler won't recognize. How often have you seen a makefile that assembled CFLAGS flags based on the compiler version?

Now the language designers and compiler authors are right because they make the rules, but the fight to produce correct and safe software becomes more and more like a fight, and every edition of the compiler tools behaves more and more like a fully automatic foot-gun - all the better to shoot yourself in the foot with, and the bullets you fire at security problems are optimised away.

Clearly, compiler authors and compiler users have conflicting views on what the compiler should do. And if you don't believe me, read this sorry tale of the assertions that were optimized away.

I don't exaggerate. New tools will produce worse code. And the compiler doesn't warn you. Because in your makefile you forgot to turn on the new warning flag that didn't exist when you wrote the makefile.

Simply recompiling previously working, secure code with a newer version of the compiler can introduce security vulnerabilities. While the new behaviour can be disabled with a flag, existing makefiles do not have that flag set, obviously. And since no warning is produced, it is not obvious to the developer that the previously reasonable behaviour has changed.

I think I want a compiler that warns me when it throws away lines of code that I took the trouble to write and debug, but what do I know? It's really hard. But so is having your code silently yanked away.

Consider what happens when the default code optimisation settings change in the next release of the compiler. As one chap puts it:

Value range propagation now assumes that the this pointer of C++ member functions is non-null. This eliminates common null pointer checks but also breaks some non-conforming code-bases (such as Qt-5, Chromium, KDevelop). As a temporary work-around -fno-delete-null-pointer-checks can be used. Wrong code can be identified by using -fsanitize=undefined.

http://blog.fefe.de/?ts=a9de792d 

A more even-handed treatment of what compiler authors are trying to achieve is given here: https://gist.github.com/rygorous/e0f055bfb74e3d5f0af20690759de5a7 

It all comes down to a persistent and ever-escalating mismatch of expectations. The scarcely known obligation upon software developers is to be aware of the increasing instances of the increasing definitions of undefined behaviour that exist in existing debugged safe and working, and of the changes to default optimisations, so that with each compiler release they must remove the last year's best practice and replace it with this year's best practice.

And our current state is where:

...the root cause of approximately 70% of security vulnerabilities that Microsoft fixes and assigns a CVE (Common Vulnerabilities and Exposures) are due to memory safety issues. ​

This is despite mitigations including intense code review, training, static analysis, and more.​

While many experienced programmers can write correct systems-level code, it’s clear that no matter the amount of mitigations put in place, it is near impossible to write memory-safe code using traditional systems-level programming languages at scale.

we-need-a-safer-systems-programming-language

Let's expand upon those mitigations. Everyone is compiling C with -Werror and -Wall, but it's not enough. Many look harder by paying good time and money for Coverity, Klocwork, Black Duck, Code Sonar, etc, and trying every C compiler they can get their hands on for maximum warnings, and the information on every change that they can make.

If the compiler authors insist that they are correct (as they may) then the software developers need new tools. For me, and for the author of that previous quote, Rust is that tool.

From C, we have had one long round of compiler-imposed instability in pursuit of speed, and another more recent self-imposed round of instability for safety.

So to answer the question heading this section: 

If you want stability (I mean such as for security bugs to stay fixed) the answer is clear: do not update your compiler tools.

If you want speed, you need the latest compiler tools. Your programs will be very fast, but you will not know what they are doing. Of course, there are ways to find out, but the answer will vary from one compiler edition to the next.

In the question of stability or speed, the C compiler maintainers chose speed for too many decades.

What about Rust?

Stability or Safety?

Don't think that Rust isn't optimising your code. It is. The most popular Rust implementation is based on Clang/LLVM and has various bugs originating in the optimising compiler backend, throwing away code that Rust wants to keep.

But Rust makes certain guarantees of safety, in the same way that C promises to throw away apparently arbitrary multi-line chunks of code. 

You can see the failure list of what are termed Unsound Bugs: Today I see 65 open and 307 closed. If I exclude C bugs from that list then I see 10 open and 86 closed. (I maybe misusing the bug filter system).

Is Rust sacrificing 3U safety for stability?

[This rant is a work in Progress and will receive regular updates]

TL;DR: This is a general call for prioritisation of breaking changes over preservation of unexpected undetectable unsafe or undefined behaviour in Rust, and for full transparency with respect to such faults. (And maybe there is and I haven't seen it).

  
I was reading the discussion at Surprising soundness trouble around `PollFn` (preceding Zulip discussion, subsequent github issue) and I was appalled.

As a meta-observation, and speaking as someone arguing for use of Rust in organisation projects (and as I am new to Rust, and maybe I misunderstood the whole thing) but:

Seeing people apparently argue to preserve all three U in the the 3U (unexpected undetectable undefined) behaviour undermines the glorious promises of Rust safety, and the claims of the supposed impossibilities of writing various kinds of bugs in Rust.

This very much damages the Rust cause, and that is something that also ought also to be considered along with the issue of introducing safe but breaking changes for existing users, because new users are coming to Rust for safety, and there will be more new users than existing users. (And most existing users also came for safety).

Those who want stability over safety will stay where they are.

The over-caution about breaking changes turns these expectations on their head. How do you think users feel: Yeah, we didn't introduce the safety of this breaking change 'cos a very few of you might need to make a patch your work and recompile to fix an actual bug, so we left it unexpectedly undetectably unsafe as a favour to you.

Because that is what this looks like.

So while everyone and their dog is now compiling C with -Werror and -Wall and literally begging to get as many breaking changes as they can, and looking yet harder by paying good time and money for Coverity and Klocwork and Black Duck etc, and trying every C compiler they can get their hands on for maximum warnings, people are arguing that Rust should to cover it all up. 

Because that is what it looks like. 

I cannot comprehend the mindset behind that. I'd love to know what some of you been smoking so that I can make sure I never ingest any of it.

All the "nobody is writing such bugs" claims, are just begging for it to come up 10 years later in the post-mortem of a severe exploit, yet we just had a lengthy post-mortem discussed on Zulip because somebody wrote such bugs, and some poor chap spent a week trying to find the cause. 

And we're more worried about "breaking changes" than actual breakage? The promise of Rust was that it should have been impossible to write that bug.

And I don't think much about the idea of simply mentioning such risk in a note at the bottom of a locked filing cabinet stuck in a disused lavatory with a sign on the door saying 'Beware of the Leopard'.

It's having this sort of secretly-documented unexpected undetectable undefined behaviour regularly foisted on us by new optimisations in the C language compilers (introducing new bugs and UB in old code in the process) that drive us to Rust in the first place. That's the sort of breaking change we don't like. We really want safety. If we didn't we wouldn't be spending millions and billions across the board between us on rewriting and retooling for Rust.

It honestly looks like I'm watching TLA-sponsored exploits being embedded into Rust. I can't account for it in any other way.

I just say that this is what it looks like, and it is very damaging to the image of Rust, precisely because it could be very damaging to compiled code, and given a useful combination of gadgets, also damaging to the systems using them, and those using the systems.

I'm sure this isn't the only case, but I daren't look. I'm trying to make a case for Rust based on its promises of safety, and it is a real conflict to know that I might undercover an apparent conspiracy to not only keep the failures of such guarantees hidden, but even to maintain those failures as failures!

I beg in the name of transparency and accountability, that whatever rules need changing are changed, so that awareness of the failure of Rust safety guarantees is paramount:

• There is a specific public list of any bug or flaw which could accidentally permit unexpected undetectable undefined or unsafe behaviour, along with a collection of instances where it has been discovered.
  There is: see: https://github.com/rust-lang/rust/labels/I-unsound
• Clippy detection be implemented rapidly, even if there is no fix (or no agreed fix), with the clippy message containing a link to the issue, even if it is unfixable.
• Breaking fixes must be adopted within a small fixed timescale if non-breaking fixes aren't adopted

I also suggest that if there is a strong case to retain these unsafe potentialities, it is not overriding enough to block the fix. If there is a strong case, then maybe potentially unsafe code can continue to be combined and compiled and released by those with such needs, by use of a strongly frowned upon compiler flag to deselect the safety of the breaking fixes.

But this practice of presenting the preservation of unexpected undetectable unsafe behaviours as some kind of unqualified benefit to those who are fleeing that sort of behaviour in other languages should stop. We undergo the expense and inconvenience and business risk to get guarantees of safety, and not guarantees of stability, which if we are honest, we know are better had by doing absolutely nothing.

With ever-increasing adoption, if Rust or libs are breaking the safety guarantees, the best time to fix them is yesterday, not maybe someday.

Emulating the past

“It’s no use going back to who I was yesterday, because I was a different person then”

Lewis Carroll

For so long, now, it's very fashionable to go back to how things were, that retro is almost passé.

It's computing that makes it all possible, of course. Computers today are so powerful that they can emulate older computers even faster than full speed.

Even in my lifetime, new computers became so fast that they weren't slow enough to properly run some of the older programs that they were supposedly compatible with. That's always fun telling the user "It's because your computer is so powerful it crashed the program". It's quite something to watch them dilute their own frustration with a mingling of pride and awe.

My history with computers started as a pre-teenage child when I traded a pair of opera glasses (that I had got from a friend whose dad ran a waste processing business) with a school teacher, to get an old electric school bell. I used to wake up at night from realistic dreams of the electric shocks that I inflicted on myself during the day.

Around the same time, my dad brought a bunch of old Bakelite telephones home from where he worked. I'm not sure what he did with them, but when he'd finished, I knew enough to fix them up with my old train transformer and make some kind of an intercom system (incurably suffused with a 50hz hum loud enough to give you a headache).

It went on from there. What pocket money I didn't spend on sweets I'd spend on the Maplin Electronics catalog and look at all the things I could build, and all the things I could build them with, if only I wouldn't spend all my money on sweets.

And that was the problem. Apart from the odd FM transmitter, I didn't build very much because electronics is an expensive hobby - all those parts to buy. I used to get by filing components from old transistor radio circuit boards before I was given leave to use the antique soldering iron. I still have a scar on my wrist from that!

But one day as I was walking home from school along the main road, that we all traveled, I heard some crazy talk, and that made all the difference.

My brother was talking to a friend about a computer program that he had written at computer club, which made a man walk across the screen, getting lower and lower, and leaving some dots behind him. It sounded fantastic, but yet his ideas intrigued to me.

I don't know what I had been doing instead of going to the computer club, but the very next time I did go to the computer club and asked the computer teacher:

Please write me a program that makes a man walk across the screen getting lower and lower and tell me how it does what it does.

Even now I am as impressed by my ability to synthesize that request as I am of my ability to understand the program.

The program is something like this (Try it on the Beeb Emulator at https://bbc.godbolt.org/):

10 CLS 20 FOR Y=0 T0 25 30 FOR X=0 T0 40 40 PRINT TAB(X,Y) "*"; 50 PRINT TAB(X,Y); 60 A$=INKEY$(10) 70 PRINT TAB(X,Y) " "; 80 NEXT X 90 NEXT Y

The teacher explained how it worked but it was pretty obvious. He was a kind and good teacher, so rather than answer my questions all night, he gave me the user programming guide (for those were the days when the users did the programming).

One misspent youth later, including the notable event of being dragged out of the computer room by the invigilator so that I could sit what was left of my french exam, I had written the following computer software:

1. The school email program (BBC Micro, on the eNet network system)
2. Decode morse code (including punctuation) (Sinclair ZX81)
3. Various programming tools (music output, read-data-&-restore) (Sinclair ZX81)
4. Database (Amstrad CPC6128) where I learned that adults will answer personal questions posed by a child if a computer is used.
5. 3-player personal computer implementation of electronic game Detective Shoestring originally by Grandstand, which involved a cardboard separator on the screen, and added a helpful detectives dog as the third player. (Amstrad CPC6128)
6. Much much more, and I also learned the futility of adding "-on-a-computer" to various other specialist tasks of the time. My attempt at scenery design on a BBC micro was doomed.

I think I only paid for 1 computer, my ZX81, and the seller gave me many a sleepless Friday night as he repeatedly forgot to bring it on Saturday each week. Christmas eve excitement was nothing to getting my first computer. My Dad then bought me a portable TV to stop me hogging the one in the front room.

Through kindness and good fortune I was lent/borrowed/given the following: 

• ZX Spectrum 48K
• Amstrad CPC6128
• TRS80 II with dual disk drive
• 80286 with 2MB RAM and Hercules Graphics Card
• 386SX16 with 4MB

After which I was able to buy a 386DX40 motherboard and 4MB RAM with Crystal sound card, and added a second hand 540MB hard disk for £40

Such glorious memories.

And now -- the computers of the past return in the form of an emulator. All of them. Most of them will even run in a web browser.

I've installed some of them, but mostly I just look and stare in a daze.

What do I want to make it do? 

Nothing. I did it all already. 

I can't go back to my childhood, I'm not a child anymore.

I seek not to follow in my footsteps of old, I attained those things I sought.

Frozen Lemons

“When life gives you lemons, make lemonade or freeze them and throw them at the people that are making your life difficult”

- Unknown

Blessed children skip along scattering sunshine and petals wherever they go, delighting to bless each life they touch for good.

Other children have never heard of such a thing. Being occupied with weightier matters, they receive their parents' duty and love as a matter of course, and get on with their own obsessive labours while the day yet remains.

My brother and I had mastered all of the necessary arts for young children and made it our labour to engage these daily for our enjoyment and delight. 

We knew the secrets of overlapping lego brings to bring strength to a wall. 

We could put a water pistol to hours of judicious use until the inevitable blister on our trigger finger burst and ripped open, but we could also use a knife to split open a water pistol to extract the pump mechanism to serve as the essential core of a fire engine yet to be built.

We knew the secrets of a beach ball, both to inflate it and deflate it, despite the anti-deflation mechanism.

We could slide down the stairs in our sleeping bags without injury but we could also climb up the green and yellow "beanstalk" blanket which hung from the balustrade above the bottom of the stairs. 

Life was worth living, and we lived it with rarely a thought for those who made it so worthwhile.

One long summer day ended too soon. I don't know if we had been sent to bed early, but if so, we probably deserved it. We might have accepted this, but it wasn't dark, and one of us saw a water pistol. 

Now one brother will not be left unarmed in a water fight, and will use all his ingenuity to overcome the lack of a second water pistol. And there never was, nor ever will be a water fight that did not escalate beyond all reason.

The water pistol had not been put to very much use before the soap dish was involved as both a carrier and dispenser of water.  As an efficiency improvement in the art war it is only a minor footnote, and interesting only in relation to the subsequent escalation of which it was the direct cause: the inflatable beach ball.

Your imagination is only faulty in one respect. The inflatable beach ball was not entirely inflated with water.

The inflatable beach ball is a dispenser of water to exceed all in those days before the pump-action super-soaker, but one could suffer a lot of hits in the time it takes to fill. To be well used it doesn't need to hold that much more than the water pistol. And if not fully filled it may also serve as a sort of bowl to hold more water.

This was peak escalation - and there were two beach balls.

From the child's point of view, we were having unpermitted quantities of fun, but the parent would think: "What are they doing up there? They are supposed to be going to sleep!"

Despite all these details being true, I'm not aware that any of the bedding actually got wet.

Our mother, on the other hand, observed that it all needed hanging out on the washing line to dry, and as there was no other bedding to hand we would have to go and play outside in our pyjamas and bare feet! 

See how we like that!

We liked that. 

We rode around the "lawn" on our tricycles in our pyjamas and bare feet, our fun a form of defiance at our supposed punishment.

What to do when your lemons are returned to you, frozen? 

If it's summer, make lovely chilled lemonade.

Unless you broke a window

“It is usually best to admit mistakes when they occur, and to seek to restore honor”. Unless you just broke a window and a really large gentleman is threatening to beat you up. In that case, run!

- Uncle Iroh

I remember that as a child, it was always important to own up when you did something wrong. It was a good theory, but the practice was not so good.

I never was a big fan of using the toilet as a young child. It was a lot of faff that required breaking off the concentration of whatever I was doing at the time. It was prime for procrastination. Why go to the toilet now when I can go later? So I procrastinated. Why procrastinate tomorrow when I can procrastinate right now!? I procrastinated with vigour, right away, with exceeding great diligence.

But you know how it is. If you procrastinate this sort of thing once, you have to procrastinate it again five minutes later, and then again soon after that. Such procrastination requires increasing and more-intrusive concentration to sustain. A slight lapse of mental effort can be followed by instant regret, and a young child then needs help to clean up the mess.

Even after months of careful practice, a young lad can misjudge when he is no longer going to be able to sustain the necessary concentration.

On one occasion I remembered that my mum had told I should just tell her, and not try to hide it, so I told her.

In return for my frank forthrightness, I received a smack on the bottom. I wasn't impressed and reminded my mum what she told me. I wasn't convinced by her response though I could see the logic: "Well I would have smacked you twice if you hadn't told me!" 

Even the best parents have their off-days.

But the problem with owning up, is that if you don't do it right away, you also get in trouble for not owning up, with the inquisition: "Why didn't you own up?" and you have to own up to that if you can understand why you didn't.

The answer which I can articulate only now but which I knew instinctively then was: because I wanted to avoid the sort of inquisition you are now putting me to.

Yes, if you don't own up right away you are better off sticking to your story, and just don't do whatever it was next time. 

And then there is the time I actually broke the glass, of a picture, of Jesus, praying.

I broke it with a ball. I didn't commit the common-or-garden sin of playing with a ball in the house.

I simply had the ball in my hand. I was cross at something, something so trivial compared with the self-inflicted mental trauma which followed, that I can't remember what it was.

But whatever it was helped me justify expressing my annoyance by striking the picture on the wall with the ball in my hand. There!

And the glass cracked. And I was probably over the age of eight so I wasn't going to get any kind of free pass.

And no amount of pleas or entreaties, or vain promises to never be naughty again for the rest of my life persuaded Heavenly Father to mend the picture. He could do it if he wanted to. I suppose he didn't want to. Didn't he know what a bargain he was missing?

And so it stayed broken. And there I was! Ketched!

I don't think Heavenly Father had any difficulty fixing glass or getting as much glass as he wanted. 

I think what he wanted was for me to learn to trust my parents.

I think the kindest thing my parents did then was to fix the picture without the inquisition.

The chocolate room

“Reality leaves a lot to the imagination.”

― John Lennon

As a young father I wanted to give my young children a real treat, something wondrous and amazing that they would never forget.

One Saturday lunchtime an idea came to mind and  I immediately announced it.

I would take them to the Cholocate Room.

Doubtless inspired by Charlie and the Chocolate Factory (book or movie, I couldn't be sure) it was full of all kinds of chocolate goodies that they could take.

They key was that they should be blindfolded, and I would take them the secret way and they would grab what they could.

And so it was.

They would be blindfolded, I carried them up and down the stairs, turned around, and again until as I imagined they did not know where they were, and then carried them down to the cellar where, while still blindfolded, they could choose whatever they wanted from the selection of chocolate that was , unbeknownst to them, continually held in front of their blindfolded face whichever way they turned.

It was a huge success. All the excitement of a real chocolate room but only needing a small stock of chocolate. All the benefit of having a secret passage in the house leading to somewhere wonderful without having to have the trouble of actually digging a secret passage, or laying out the wonderful place.

Except... I found that the wandering about with a blindfold did not confuse them as to where they were. They were smart kids, and with a sense of direction as well as a sense of the ambience of the rooms.

So what had happened? Did they imagine that they had actually been to an actual chocolate room? Or were they just imagining that they had to play along, for free chocolate?

We never went to the chocolate room again, despite multiple requests.

I want to, I long to, but if they don't actually believe that I can take them to an actual chocolate room, then what am I doing?

Am I running upstairs and downstairs as an excuse to give them free chocolate? 

If they want free chocolate, then they can find it in the cupboard.

I was looking to give them a sense of wonder and amazement. Is that what they thought they were doing for me?