Hello! I’ve been working on writing a zine about debugging for a while (here’s an early draft of the table of contents).

As part of that I thought it might be fun to read some academic papers about debugging, and last week Greg Wilson sent me some papers about academic research into debugging.

One of those papers (Towards a framework for teaching debugging [paywalled]) had a categorization I really liked of the different kinds of knowledge/skills we need to debug effectively. It comes from another more general paper on troubleshooting: Learning to Troubleshoot: A New Theory-Based Design Architecture.

I thought the categorization was a very useful structure for thinking about how to get better at debugging, so I’ve reframed the five categories in the paper into actions you can take to get better at debugging.

Here they are:

1. learn the codebase

To debug some code, you need to understand the codebase you’re working with. This seems kind of obvious (of course you can’t debug code without understanding how it works!).

This kind of learning happens pretty naturally over time, and actually debugging is also one of the best ways to learn how a new codebase works – seeing how something breaks helps you learn a lot about how it works.

The paper calls this “System Knowledge”.

2. learn the system

The paper mentions that you need to understand the programming language, but I think there’s more to it than that – to fix bugs, often you need to learn a lot about the broader environment than just the language.

For example, if you’re a backend web developer, some “system” knowledge you might need includes:

• how HTTP caching works
• CORS
• how database transactions work

I find that I often have to be a bit more intentional about learning systemic things like this – I need to actually take the time to look them up and read about them.

The paper calls this “Domain Knowledge”.

3. learn your tools

There are lots of debugging tools out there, for example:

• debuggers (gdb etc)
• browser developer tools
• profilers
• strace / ltrace
• tcpdump / wireshark
• core dumps
• and even basic things like error messages (how do you read them properly)

I’ve written a lot about debugging tools on this blog, and definitely learning these tools has made a huge difference to me.

The paper calls this “Procedural Knowledge”.

4. learn strategies

This is the fuzziest category, we all have a lot of strategies and heuristics we pick up along the way for how to debug efficiently. For example:

• writing a unit test
• writing a tiny standalone program to reproduce the bug
• finding a working version of the code and seeing what changed
• printing out a million things
• adding extra logging
• taking a break
• explaining the bug to a friend and then figuring out what’s wrong halfway through
• looking through the github issues to see if anything matches

I’ve been thinking a lot about this category while writing the zine, but I want to keep this post short so I won’t say more about it here.

The paper calls this “Strategic Knowledge”.

5. get experience

The last category is “experience”. The paper has a really funny comment about this:

Their findings did not show a significant difference in the strategies employed by the novices and experts. Experts simply formed more correct hypotheses and were more efficient at finding the fault. The authors suspect that this result is due to the difference in the programming experience between novices and experts.

This really resonated with me – I’ve had SO MANY bugs that were really frustrating and difficult the first time I ran into them, and very straightforward the fifth or tenth or 20th time.

This also feels like one of the most straightforward categories of knowledge to acquire to me – all you need to do is investigate a million bugs, which is our whole life as programmers anyway :). It takes a long time but I feel like it happens pretty naturally.

The paper calls this “Experiential Knowledge”.

that’s all!

I’m going to keep this post short, I just really liked this categorization and wanted to share it.

Hello! The other day we talked about what happened when you press a key in your terminal.

As a followup, I thought it might be fun to implement a program that’s like a tiny ssh server, but without the security. You can find it on github here, and I’ll explain how it works in this blog post.

the goal: “ssh” to a remote computer

Our goal is to be able to login to a remote computer and run commands, like you do with SSH or telnet.

The biggest difference between this program and SSH is that there’s literally no security (not even a password) – anyone who can make a TCP connection to the server can get a shell and run commands.

Obviously this is not a useful program in real life, but our goal is to learn a little more about how terminals works, not to write a useful program.

(I will run a version of it on the public internet for the next week though, you can see how to connect to it at the end of this blog post)

let’s start with the server!

We’re also going to write a client, but the server is the interesting part, so let’s start there. We’re going to write a server that listens on a TCP port (I picked 7777) and creates remote terminals for any client that connects to it to use.

When the server receives a new connection it needs to:

1. create a pseudoterminal for the client to use
2. start a bash shell process for the client to use
3. connect bash to the pseudoterminal
4. continuously copy information back and forth between the TCP connection and the pseudoterminal

I just said the word “pseudoterminal” a lot, so let’s talk about what that means.

what’s a pseudoterminal?

Okay, what the heck is a pseudoterminal?

A pseudoterminal is a lot like a bidirectional pipe or a socket – you have two ends, and they can both send and receive information. You can read more about the information being sent and received in what happens if you press a key in your terminal

Basically the idea is that on one end, we have a TCP connection, and on the other end, we have a bash shell. So we need to hook one part of the pseudoterminal up to the TCP connection and the other end to bash.

The two parts of the pseudoterminal are called:

• the “pseudoterminal master”. This is the end we’re going to hook up to the TCP connection.
• the “slave pseudoterminal device”. We’re going to set our bash shell’s stdout, stderr, and stdin to this.

Once they’re conected, we can communicate with bash over our TCP connection and we’ll have a remote shell!

why do we need this “pseudoterminal” thing anyway?

You might be wondering – Julia, if a pseudoterminal is kind of like a socket, why can’t we just set our bash shell’s stdout / stderr / stdin to the TCP socket?

And you can! We could write a TCP connection handler like this that does exactly that, it’s not a lot of code (server-notty.go).

func handle(conn net.Conn) {
	tty, _ := conn.(*net.TCPConn).File()
	// start bash with tcp connection as stdin/stdout/stderr
	cmd := exec.Command("bash")
	cmd.Stdin = tty
	cmd.Stdout = tty
	cmd.Stderr = tty
	cmd.Start()
}

It even kind of works – if we connect to it with nc localhost 7778, we can run commands and look at their output.

But there are a few problems. I’m not going to list all of them, just two.

problem 1: Ctrl + C doesn’t work

The way Ctrl + C works in a remote login session is

• you press ctrl + c
• That gets translated to 0x03 and sent through the TCP connection
• The terminal receives it
• the Linux kernel on the other end notes “hey, that was a Ctrl + C!”
• Linux sends a SIGINT to the appropriate process (more on what the “appropriate process” is exactly later)

If the “terminal” is just a TCP connection, this doesn’t work, because when you send 0x04 to a TCP connection, Linux won’t magically send SIGINT to any process.

problem 2: top doesn’t work

When I try to run top in this shell, I get the error message top: failed tty get. If we strace it, we see this system call:

ioctl(2, TCGETS, 0x7ffec4e68d60)        = -1 ENOTTY (Inappropriate ioctl for device)

So top is running an ioctl on its output file descriptor (2) to get some information about the terminal. But Linux is like “hey, this isn’t a terminal!” and returns an error.

There are a bunch of other things that go wrong, but hopefully at this point you’re convinced that we actually need to set bash’s stdout/stderr to be a terminal, not some other thing like a socket.

So let’s start looking at the server code and see what creating a pseudoterminal actually looks like.

step 1: create a pseudoterminal

Here’s some Go code to create a pseudoterminal on Linux. This is copied from github.com/creack/pty, but I removed some of the error handling to make the logic a bit easier to follow:

pty, _ := os.OpenFile("/dev/ptmx", os.O_RDWR, 0)
sname := ptsname(p)
unlockpt(p)
tty, _ := os.OpenFile(sname, os.O_RDWR|syscall.O_NOCTTY, 0)

In English, what we’re doing is:

• open /dev/ptmx to get the “pseudoterminal master” Again, that’s the part we’re going to hook up to the TCP connection
• get the filename of the “slave pseudoterminal device”, which is going to be /dev/pts/13 or something.
• “unlock” the pseudoterminal so that we can use it. I have no idea what the point of this is (why is it locked to begin with?) but you have to do it for some reason
• open /dev/pts/13 (or whatever number we got from ptsname) to get the “slave pseudoterminal device”

What do those ptsname and unlockpt functions do? They just make some ioctl system calls to the Linux kernel. All of the communication with the Linux kernel about terminals seems to be through various ioctl system calls.

Here’s the code, it’s pretty short: (again, I just copied it from creack/pty)

func ptsname(f *os.File) string {
	var n uint32
	ioctl(f.Fd(), syscall.TIOCGPTN, uintptr(unsafe.Pointer(&n)))
	return "/dev/pts/" + strconv.Itoa(int(n))
}

func unlockpt(f *os.File) {
	var u int32
	// use TIOCSPTLCK with a pointer to zero to clear the lock
	ioctl(f.Fd(), syscall.TIOCSPTLCK, uintptr(unsafe.Pointer(&u)))
}

step 2: hook the pseudoterminal up to bash

The next thing we have to do is connect the pseudoterminal to bash. Luckily, that’s really easy – here’s the Go code for it! We just need to start a new process and set the stdin, stdout, and stderr to tty.

cmd := exec.Command("bash")
cmd.Stdin = tty
cmd.Stdout = tty
cmd.Stderr = tty
cmd.SysProcAttr = &syscall.SysProcAttr{
  Setsid: true,
}
cmd.Start()

Easy! Though – why do we need this Setsid: true thing, you might ask? Well, I tried commenting out that code to see what went wrong. It turns out that what goes wrong is – Ctrl + C doesn’t work anymore!

Setsid: true creates a new session for the new bash process. But why does that make Ctrl + C work? How does Linux know which process to send SIGINT to when you press Ctrl + C, and what does that have to do with sessions?

how does Linux know which process to send Ctrl + C to?

I found this pretty confusing, so I reached for my favourite book for learning about this kind of thing: the linux programming interface, specifically chapter 34 on process groups and sessions.

That chapter contains a few key facts: (#3, #4, and #5 are direct quotes from the book)

1. Every process has a session id and a process group id (which may or may not be the same as its PID)
2. A session is made up of multiple process groups
3. All of the processes in a session share a single controlling terminal.
4. A terminal may be the controlling terminal of at most one session.
5. At any point in time, one of the process groups in a session is the foreground process group for the terminal, and the others are background process groups.
6. When you press Ctrl+C in a terminal, SIGINT gets sent to all the processes in the foreground process group

What’s a process group? Well, my understanding is that:

• processes in the same pipe x | y | z are in the same process group
• processes you start on the same shell line (x && y && z) are in the same process group
• child processes are by default in the same process group, unless you explicitly decide otherwise

I didn’t know most of this (I had no idea processes had a session ID!) so this was kind of a lot to absorb. I tried to draw a sketchy ASCII art diagram of the situation

(maybe)  terminal --- session --- process group --- process
                               |                 |- process
                               |                 |- process
                               |- process group 
                               |
                               |- process group 

So when we press Ctrl+C in a terminal, here’s what I think happens:

• \x04 gets written to the “pseudoterminal master” of a terminal
• Linux finds the session for that terminal (if it exists)
• Linux find the foreground process group for that session
• Linux sends SIGINT

If we don’t create a new session for our new bash process, our new pseudoterminal actually won’t have any session associated with it, so nothing happens when we press Ctrl+C. But if we do create a new session, then the new pseudoterminal will have the new session associated with it.

how to get a list of all your sessions

As a quick aside, if you want to get a list of all the sessions on your Linux machine, grouped by session, you can run:

$ ps -eo user,pid,pgid,sess,cmd | sort -k3

This includes the PID, process group ID, and session ID. As an example of the output, here are the two processes in the pipeline:

bork       58080   58080   57922 ps -eo user,pid,pgid,sess,cmd
bork       58081   58080   57922 sort -k3

You can see that they share the same process group ID and session ID, but of course they have different PIDs.

That was kind of a lot but that’s all we’re going to say about sessions and process groups in this post. Let’s keep going!

step 3: set the window size

We need to tell the terminal how big to be!

Again, I just copied this from creack/pty. I decided to hardcode the size to 80x24.

Setsize(tty, &Winsize{
		Cols: 80,
		Rows: 24,
	})

Like with getting the terminal’s pts filename and unlocking it, setting the size is just one ioctl system call:

func Setsize(t *os.File, ws *Winsize) {
	ioctl(t.Fd(), syscall.TIOCSWINSZ, uintptr(unsafe.Pointer(ws)))
}

Pretty simple! We could do something smarter and get the real window size, but I’m too lazy.

step 4: copy information between the TCP connection and the pseudoterminal

As a reminder, our rough steps to set up this remote login server were:

1. create a pseudoterminal for the client to use
2. start a bash shell process
3. connect bash to the pseudoterminal
4. continuously copy information back and forth between the TCP connection and the pseudoterminal

We’ve done 1, 2, and 3, now we just need to ferry information between the TCP connection and the pseudoterminal.

There are two io.Copy calls, one to copy the input from the tcp connection, and one to copy the output to the TCP connection. Here’s what the code looks like:

	go func() {
			io.Copy(pty, conn)
	}()
  io.Copy(conn, pty)

The first one is in a goroutine just so they can both run in parallel.

Pretty simple!

step 5: exit when we’re done

I also added a little bit of code to close the TCP connection when the command exits

go func() {
  cmd.Wait()
  conn.Close()
}()

And that’s it for the server! You can see all of the Go code here: server.go.

next: write a client

Next, we have to write a client. This is a lot easier than the server because we don’t need to do quite as much terminal setup. There are just 3 steps:

1. Put the terminal into raw mode
2. copy stdin/stdout to the TCP connection
3. reset the terminal

client step 1: put the terminal into “raw” mode

We need to put the client terminal into “raw” mode so that every time you press a key, it gets sent to the TCP connection immediately. If we don’t do this, everything will only get sent when you press enter.

“Raw mode” isn’t actually a single thing, it’s a bunch of flags that you want to turn off. There’s a good tutorial explaining all the flags we have to turn off called Entering raw mode.

Like everything else with terminals, this requires ioctl system calls. In this case we get the terminal’s current settings, modify them, and save the old settings so that we can restore them later.

I figured out how to do this in Go by going to https://grep.app and typing in syscall.TCSETS to find some other Go code that was doing the same thing.

func MakeRaw(fd uintptr) syscall.Termios {
	// from https://github.com/getlantern/lantern/blob/devel/archive/src/golang.org/x/crypto/ssh/terminal/util.go
	var oldState syscall.Termios
	ioctl(fd, syscall.TCGETS, uintptr(unsafe.Pointer(&oldState)))

	newState := oldState
	newState.Iflag &^= syscall.ISTRIP | syscall.INLCR | syscall.ICRNL | syscall.IGNCR | syscall.IXON | syscall.IXOFF
	newState.Lflag &^= syscall.ECHO | syscall.ICANON | syscall.ISIG
	ioctl(fd, syscall.TCSETS, uintptr(unsafe.Pointer(&newState)))
	return oldState
}

client step 2: copy stdin/stdout to the TCP connection

This is exactly like what we did with the server. It’s very little code:

go func() {
		io.Copy(conn, os.Stdin)
	}()
	io.Copy(os.Stdout, conn)

client step 3: restore the terminal’s state

We can put the terminal back into the mode it started in like this (another ioctl!):

func Restore(fd uintptr, oldState syscall.Termios) {
	ioctl(fd, syscall.TCSETS, uintptr(unsafe.Pointer(&oldState)))
}

we did it!

We have written a tiny remote login server that lets anyone log in! Hooray!

Obviously this has zero security so I’m not going to talk about that aspect.

it’s running on the public internet! you can try it out!

For the next week or so I’m going to run a demo of this on the internet at tetris.jvns.ca. It runs tetris instead of a shell because I wanted to avoid abuse, but if you want to try it with a shell you can run it on your own computer :).

If you want to try it out, you can use netcat as a client instead of the custom Go client program we wrote, because copying information to/from a TCP connection is what netcat does. Here’s how:

stty raw -echo && nc tetris.jvns.ca 7777 && stty sane

This will let you play a terminal tetris game called tint.

You can also use the client.go program and run go run client.go tetris.jvns.ca 7777.

this is not a good protocol

This protocol where we just copy bytes from the TCP connection to the terminal and nothing else is not good because it doesn’t allow us to send over information information like the terminal or the actual window size of the terminal.

I thought about implementing telnet’s protocol so that we could use telnet as a client, but I didn’t feel like figuring out how telnet works so I didn’t. (the server 30% works with telnet as is, but a lot of things are broken, I don’t quite know why, and I didn’t feel like figuring it out)

it’ll mess up your terminal a bit

As a warning: using this server to play tetris will probably mess up your terminal a bit because it sets the window size to 80x24. To fix that I just closed the terminal tab after running that command.

If we wanted to fix this for real, we’d need to restore the window size after we’re done, but then we’d need a slightly more real protocol than “just blindly copy bytes back and forth with TCP” and I didn’t feel like doing that.

Also it sometimes takes a second to disconnect after the program exits for some reason, I’m not sure why that is.

other tiny projects

That’s all! There are a couple of other similar toy implementations of programs I’ve written here:

• toy tls 1.3 implementation
• toy dns resolver

I’ve been confused about what’s going on with terminals for a long time.

But this past week I was using xterm.js to display an interactive terminal in a browser and I finally thought to ask a pretty basic question: when you press a key on your keyboard in a terminal (like Delete, or Escape, or a), which bytes get sent?

As usual we’ll answer that question by doing some experiments and seeing what happens :)

remote terminals are very old technology

First, I want to say that displaying a terminal in the browser with xterm.js might seem like a New Thing, but it’s really not. In the 70s, computers were expensive. So many employees at an institution would share a single computer, and each person could have their own “terminal” to that computer.

For example, here’s a photo of a VT100 terminal from the 70s or 80s. This looks like it could be a computer (it’s kind of big!), but it’s not – it just displays whatever information the actual computer sends it.

Of course, in the 70s they didn’t use websockets for this, but the information being sent back and forth is more or less the same as it was then.

(the terminal in that photo is from the Living Computer Museum in Seattle which I got to visit once and write FizzBuzz in ed on a very old Unix system, so it’s possible that I’ve actually used that machine or one of its siblings! I really hope the Living Computer Museum opens again, it’s very cool to get to play with old computers.)

what information gets sent?

It’s obvious that if you want to connect to a remote computer (with ssh or using xterm.js and a websocket, or anything else), then some information needs to be sent between the client and the server.

Specifically:

• the client needs to send the keystrokes that the user typed in (like ls -l)
• the server needs to tell the client what to display on the screen

Let’s look at a real program that’s running a remote terminal in a browser and see what information gets sent back and forth!

we’ll use goterm to experiment

I found this tiny program on GitHub called goterm that runs a Go server that lets you interact with a terminal in the browser using xterm.js. This program is very insecure but it’s simple and great for learning.

I forked it to make it work with the latest xterm.js, since it was last updated 6 years ago. Then I added some logging statements to print out every time bytes are sent/received over the websocket.

Let’s look at sent and received during a few different terminal interactions!

example: ls

First, let’s run ls. Here’s what I see on the xterm.js terminal:

bork@kiwi:/play$ ls
file
bork@kiwi:/play$

and here’s what gets sent and received: (in my code, I log sent: [bytes] every time the client sends bytes and recv: [bytes] every time it receives bytes from the server)

sent: "l"
recv: "l"
sent: "s"
recv: "s"
sent: "\r"
recv: "\r\n\x1b[?2004l\r"
recv: "file\r\n"
recv: "\x1b[?2004hbork@kiwi:/play$ "

I noticed 3 things in this output:

1. Echoing: The client sends l and then immediately receives an l sent back. I guess the idea here is that the client is really dumb – it doesn’t know that when I type an l, I want an l to be echoed back to the screen. It has to be told explicitly by the server process to display it.
2. The newline: when I press enter, it sends a \r (carriage return) symbol and not a \n (newline)
3. Escape sequences: \x1b is the ASCII escape character, so \x1b[?2004h is telling the terminal to display something or other. I think this is a colour sequence but I’m not sure. We’ll talk a little more about escape sequences later.

Okay, now let’s do something slightly more complicated.

example: Ctrl+C

Next, let’s see what happens when we interrupt a process with Ctrl+C. Here’s what I see in my terminal:

bork@kiwi:/play$ cat
^C
bork@kiwi:/play$

And here’s what the client sends and receives.

sent: "c"
recv: "c"
sent: "a"
recv: "a"
sent: "t"
recv: "t"
sent: "\r"
recv: "\r\n\x1b[?2004l\r"
sent: "\x03"
recv: "^C"
recv: "\r\n"
recv: "\x1b[?2004h"
recv: "bork@kiwi:/play$ "

When I press Ctrl+C, the client sends \x03. If I look up an ASCII table, \x03 is “End of Text”, which seems reasonable. I thought this was really cool because I’ve always been a bit confused about how Ctrl+C works – it’s good to know that it’s just sending an \x03 character.

I believe the reason cat gets interrupted when we press Ctrl+C is that the Linux kernel on the server side receives this \x03 character, recognizes that it means “interrupt”, and then sends a SIGINT to the process that owns the pseudoterminal’s process group. So it’s handled in the kernel and not in userspace.

example: Ctrl+D

Let’s try the exact same thing, except with Ctrl+D. Here’s what I see in my terminal:

bork@kiwi:/play$ cat
bork@kiwi:/play$

And here’s what gets sent and received:

sent: "c"
recv: "c"
sent: "a"
recv: "a"
sent: "t"
recv: "t"
sent: "\r"
recv: "\r\n\x1b[?2004l\r"
sent: "\x04"
recv: "\x1b[?2004h"
recv: "bork@kiwi:/play$ "

It’s very similar to Ctrl+C, except that \x04 gets sent instead of \x03. Cool! \x04 corresponds to ASCII “End of Transmission”.

what about Ctrl + another letter?

Next I got curious about – if I send Ctrl+e, what byte gets sent?

It turns out that it’s literally just the number of that letter in the alphabet, like this:

• Ctrl+a => 1
• Ctrl+b => 2
• Ctrl+c => 3
• Ctrl+d => 4
• …
• Ctrl+z => 26

Also, Ctrl+Shift+b does the exact same thing as Ctrl+b (it writes 0x2).

What about other keys on the keyboard? Here’s what they map to:

• Tab -> 0x9 (same as Ctrl+I, since I is the 9th letter)
• Escape -> \x1b
  
• Backspace -> \x7f
• Home -> \x1b[H
• End: \x1b[F
• Print Screen: \x1b\x5b\x31\x3b\x35\x41
• Insert: \x1b\x5b\x32\x7e
• Delete -> \x1b\x5b\x33\x7e
• My Meta key does nothing at all

What about Alt? From my experimenting (and some Googling), it seems like Alt is literally the same as “Escape”, except that pressing Alt by itself doesn’t send any characters to the terminal and pressing Escape by itself does. So:

• alt + d => \x1bd (and the same for every other letter)
• alt + shift + d => \x1bD (and the same for every other letter)
• etcetera

Let’s look at one more example!

example: nano

Here’s what gets sent and received when I run the text editor nano:

recv: "\r\x1b[Kbork@kiwi:/play$ "
sent: "n" [[]byte{0x6e}]
recv: "n"
sent: "a" [[]byte{0x61}]
recv: "a"
sent: "n" [[]byte{0x6e}]
recv: "n"
sent: "o" [[]byte{0x6f}]
recv: "o"
sent: "\r" [[]byte{0xd}]
recv: "\r\n\x1b[?2004l\r"
recv: "\x1b[?2004h"
recv: "\x1b[?1049h\x1b[22;0;0t\x1b[1;16r\x1b(B\x1b[m\x1b[4l\x1b[?7h\x1b[39;49m\x1b[?1h\x1b=\x1b[?1h\x1b=\x1b[?25l"
recv: "\x1b[39;49m\x1b(B\x1b[m\x1b[H\x1b[2J"
recv: "\x1b(B\x1b[0;7m  GNU nano 6.2 \x1b[44bNew Buffer \x1b[53b \x1b[1;123H\x1b(B\x1b[m\x1b[14;38H\x1b(B\x1b[0;7m[ Welcome to nano.  For basic help, type Ctrl+G. ]\x1b(B\x1b[m\r\x1b[15d\x1b(B\x1b[0;7m^G\x1b(B\x1b[m Help\x1b[15;16H\x1b(B\x1b[0;7m^O\x1b(B\x1b[m Write Out   \x1b(B\x1b[0;7m^W\x1b(B\x1b[m Where Is    \x1b(B\x1b[0;7m^K\x1b(B\x1b[m Cut\x1b[15;61H"

You can see some text from the UI in there like “GNU nano 6.2”, and these \x1b[27m things are escape sequences. Let’s talk about escape sequences a bit!

ANSI escape sequences

These \x1b[ things above that nano is sending the client are called “escape sequences” or “escape codes”. This is because they all start with \x1b, the “escape” character. . They change the cursor’s position, make text bold or underlined, change colours, etc. Wikipedia has some history if you’re interested.

As a simple example: if you run

echo -e '\e[0;31mhi\e[0m there'

in your terminal, it’ll print out “hi there” where “hi” is in red and “there” is in black. This page has some nice examples of escape codes for colors and formatting.

I think there are a few different standards for escape codes, but my understanding is that the most common set of escape codes that people use on Unix come from the VT100 (that old terminal in the picture at the top of the blog post), and hasn’t really changed much in the last 40 years.

Escape codes are why your terminal can get messed up if you cat a bunch of binary to your screen – usually you’ll end up accidentally printing a bunch of random escape codes which will mess up your terminal – there’s bound to be a 0x1b byte in there somewhere if you cat enough binary to your terminal.

can you type in escape sequences manually?

A few sections back, we talked about how the Home key maps to \x1b[H. Those 3 bytes are Escape + [ + H (because Escape is \x1b).

And if I manually type Escape, then [, then H in the xterm.js terminal, I end up at the beginning of the line, exactly the same as if I’d pressed Home.

I noticed that this didn’t work in fish on my computer though – if I typed Escape and then [, it just printed out [ instead of letting me continue the escape sequence. I asked my friend Jesse who has written a bunch of Rust terminal code about this and Jesse told me that a lot of programs implement a timeout for escape codes – if you don’t press another key after some minimum amount of time, it’ll decide that it’s actually not an escape code anymore.

Apparently this is configurable in fish with fish_escape_delay_ms, so I ran set fish_escape_delay_ms 1000 and then I was able to type in escape codes by hand. Cool!

terminal encoding is kind of weird

I want to pause here for a minute here and say that the way the keys you get pressed get mapped to bytes is pretty weird. Like, if we were designing the way keys are encoded from scratch today, we would probably not set it up so that:

• Ctrl + a does the exact same thing as Ctrl + Shift + a
• Alt is the same as Escape
• control sequences (like colours / moving the cursor around) use the same byte as the Escape key, so that you need to rely on timing to determine if it was a control sequence of the user just meant to press Escape

But all of this was designed in the 70s or 80s or something and then needed to stay the same forever for backwards compatibility, so that’s what we get :)

changing window size

Not everything you can do in a terminal happens via sending bytes back and forth. For example, when the terminal gets resized, we have to tell Linux that the window size has changed in a different way.

Here’s what the Go code in goterm to do that looks like:

syscall.Syscall(
    syscall.SYS_IOCTL,
    tty.Fd(),
    syscall.TIOCSWINSZ,
    uintptr(unsafe.Pointer(&resizeMessage)),
)

This is using the ioctl system call. My understanding of ioctl is that it’s a system call for a bunch of random stuff that isn’t covered by other system calls, generally related to IO I guess.

syscall.TIOCSWINSZ is an integer constant which which tells ioctl which particular thing we want it to to in this case (change the window size of a terminal).

this is also how xterm works

In this post we’ve been talking about remote terminals, where the client and the server are on different computers. But actually if you use a terminal emulator like xterm, all of this works the exact same way, it’s just harder to notice because the bytes aren’t being sent over a network connection.

that’s all for now!

There’s definitely a lot more to know about terminals (we could talk more about colours, or raw vs cooked mode, or unicode support, or the Linux pseudoterminal interface) but I’ll stop here because it’s 10pm, this is getting kind of long, and I think my brain cannot handle more new information about terminals today.

Thanks to Jesse Luehrs for answering a billion of my questions about terminals, all the mistakes are mine :)

Hello! I’ve started to run a few more servers recently (nginx playground, mess with dns, dns lookup), so I’ve been thinking about monitoring.

It wasn’t initially totally obvious to me how to monitor these websites, so I wanted to quickly write up what how I did it.

I’m not going to talk about how to monitor Big Serious Mission Critical websites at all, only tiny unimportant websites.

goal: spend approximately 0 time on operations

I want the sites to mostly work, but I also want to spend approximately 0% of my time on the ongoing operations.

I was initially very wary of running servers at all because at my last job I was on a ²⁴⁄₇ oncall rotation for some critical services, and in my mind “being responsible for servers” meant “get woken up at 2am to fix the servers” and “have lots of complicated dashboards”.

So for a while I only made static websites so that I wouldn’t have to think about servers.

But eventually I realized that any server I was going to write was going to be very low stakes, if they occasionally go down for 2 hours it’s no big deal, and I could just set up some very simple monitoring to help keep them running.

not having monitoring sucks

At first I didn’t set up any monitoring for my servers at all. This had the extremely predictable outcome of – sometimes the site broke, and I didn’t find out about it until somebody told me!

step 1: an uptime checker

The first step was to set up an uptime checker. There are tons of these out there, the ones I’m using right now are updown.io and uptime robot. I like updown’s user interface and pricing structure more (it’s per request instead of a monthly fee), but uptime robot has a more generous free tier.

These

1. check that the site is up
2. if it goes down, it emails me

I find that email notifications are a good level for me, I’ll find out pretty quickly if the site goes down but it doesn’t wake me up or anything.

step 2: an end-to-end healthcheck

Next, let’s talk about what “check that the site is up” actually means.

At first I just made one of my healthcheck endpoints a function that returned 200 OK no matter what.

This is kind of useful – it told me that the server was on!

But unsurprisingly I ran into problems because it wasn’t checking that the API was actually working – sometimes the healthcheck succeeded even though the rest of the service had actually gotten into a bad state.

So I updated it to actually make a real API request and make sure it succeeded.

All of my services do very few things (the nginx playground has just 1 endpoint), so it’s pretty easy to set up a healthcheck that actually runs through most of the actions the service is supposed to do.

Here’s what the end-to-end healthcheck handler for the nginx playground looks like. It’s very basic: it just makes another POST request (to itself) and checks if that request succeeds or fails.

func healthHandler(w http.ResponseWriter, r *http.Request) {
	// make a request to localhost:8080 with `healthcheckJSON` as the body
	// if it works, return 200
	// if it doesn't, return 500
	client := http.Client{}
	resp, err := client.Post("http://localhost:8080/", "application/json", strings.NewReader(healthcheckJSON))
	if err != nil {
		log.Println(err)
		w.WriteHeader(http.StatusInternalServerError)
		return
	}
	if resp.StatusCode != http.StatusOK {
		log.Println(resp.StatusCode)
		w.WriteHeader(http.StatusInternalServerError)
		return
	}
	w.WriteHeader(http.StatusOK)
}

healthcheck frequency: hourly

Right now I’m running most of my healthchecks every hour, and some every 30 minutes.

I run them hourly because updown.io’s pricing is per healthcheck, I’m monitoring 18 different URLs, and I wanted to keep my healthcheck budget pretty minimal at $5/year.

Taking an hour to find out that one of these websites has gone down seems ok to me – if there is a problem there’s no guarantee I’ll get to fixing it all that quickly anyway.

If it were free to run them more often I’d probably run them every 5-10 minutes instead.

step 3: automatically restart if the healthcheck fails

Some of my websites are on fly.io, and fly has a pretty standard feature where I can configure a HTTP healthcheck for a service and restart the service if the healthcheck starts failing.

“Restart a lot” is a very useful strategy to paper over bugs that I haven’t gotten around to fixing yet – for a while the nginx playground had a process leak where nginx processes weren’t getting terminated, so the server kept running out of RAM.

With the healthcheck, the result of this was that every day or so, this would happen:

• the server ran out of RAM
• the healthcheck started failing
• it get restarted
• everything was fine again
• repeat the whole saga again some number of hours later

Eventually I got around to actually fixing the process leak, but it was nice to have a workaround in place that could keep things running while I was procrastinating fixing the bug.

These healthchecks to decide whether to restart the service run more often: every 5 minutes or so.

this is not the best way to monitor Big Services

This is probably obvious and I said this already at the beginning, but “write one HTTP healthcheck” is not the best approach for monitoring a large complex service. But I won’t go into that because that’s not what this post is about.

it’s been working well so far!

I originally wrote this post 3 months ago in April, but I waited until now to publish it to make sure that the whole setup was working.

It’s made a pretty big difference – before I was having some very silly downtime problems, and now for the last few months the sites have been up 99.95% of the time!

Hello! About a year ago I got mad about Docker container startup time. This was because I was building an nginx playground where I was starting a new “container” on every HTTP request, and so for it to feel reasonably snappy, nginx needed to start quickly.

Also, I was running this project on a pretty small cloud machine (256MB RAM), a small CPU, so I really wanted to avoid unnecessary overhead.

I’ve been looking for a way to run containers faster since then, but I couldn’t find one until last week when I discovered bubblewrap!! It’s very fast and I think it’s super cool, but I also ran into a bunch of fun problems that I wanted to write down for my future self.

some disclaimers

• I’m not sure if the way I’m using bubblewrap in this post is maybe not how it’s intended to be used
• there are a lot of sharp edges when using bubblewrap in this way, you need to think a lot about Linux namespaces and how containers work
• bubblewrap is a security tool but I am not a security person and I am only doing this for weird tiny projects. you should definitely not take security advice from me.

Okay, all of that said, let’s talk about I’m trying to use bubblewrap to run containers fast and in a relatively secure way :)

Docker containers take ~300ms to start on my machine

I ran a quick benchmark to see how long a Docker container takes to run a simple command (ls). For both Docker and Podman, it’s about 300ms.

$ time docker run --network none -it ubuntu:20.04 ls / > /dev/null
Executed in  378.42 millis 
$ time podman run --network none -it ubuntu:20.04 ls / > /dev/null
Executed in  279.27 millis

Almost all of this time is overhead from docker and podman – just running ls by itself takes about 3ms:

$ time ls / > /dev/null
Executed in    2.96 millis 

I want to stress that, while I’m not sure exactly what the slowest part of Docker and podman startup time is (I spent 5 minutes trying to profile them and gave up), I’m 100% sure it’s something important.

The way we’re going to run containers faster with bubblewrap has a lot of limitations and it’s a lower level interface which is a lot trickier to use.

goal 1: containers that start quickly

I felt like it should be possible to have containers that start essentially instantly or at least in less than 5ms. My thought process:

• creating a new namespace with unshare is basically instant
• containers are basically just a bunch of namespaces
• what’s the problem?

container startup time is (usually) not that important

Most of the time when people are using containers, they’re running some long-running process inside the container like a webserver, so it doesn’t really matter if it takes 300ms to start.

So it makes sense to me that there aren’t a lot of container tools that optimize for startup time. But I still wanted to optimize for startup time :)

goal 2: run the containers as an unprivileged user

Another goal I had was to be able to run my containers as an unprivileged user instead of root.

I was surprised the first time I learned that Docker actually runs containers as root – even though I run docker run ubuntu:20.04 as an unprivileged user (bork), that message is actually sent to a daemon running as root, and the Docker container process itself also runs as root (albeit a root that’s stripped of all its capabilities).

That’s fine for Docker (they have lots of very smart people making sure that they get it right!), but if I’m going to do container stuff without using Docker (for the speed reasons mentioned above), I’d rather not do it as root to keep everything a bit more secure.

podman can run containers as an non-root user

Before we start talking about how to do weird stuff with bubblewrap, I want to quickly talk about a much more normal tool to run containers: podman!

Podman, unlike Docker, can run containers as an unprivileged user!

If I run this from my normal user:

$ podman run -it ubuntu:20.04 ls

it doesn’t secretly run as root behind the scenes! It just starts the container as my normal user, and then uses something called “user namespaces” so that inside the container I appear to be root.

The other cool thing about podman is that it has exactly the same interface as Docker, so you can just take a Docker command and replace docker with podman and it’ll Just Work. I’ve found that sometimes I need to do some extra work to get podman to work in practice, but it’s still pretty nice that it has the same command line interface.

This “run containers as a non-root user” feature is normally called “rootless containers”. (I find that name kind of counterintuitive, but that’s what people call it)

failed attempt 1: write my own tool using runc

I knew that Docker and podman use runc (or maybe crun? I can’t keep track honestly) under the hood, so I thought – well, maybe I can just use runc directly to make my own tool that starts containers faster than Docker does!

I tried to do this 6 months ago and I don’t remember most of the details, but basically I spent 8 hours working on it, got frustrated because I couldn’t get anything to work, and gave up.

One specific detail I remember struggling with was setting up a working /dev for my programs to use.

enter bubblewrap

Okay, that was a very long preamble so let’s get to the point! Last week, I discovered a tool called bubblewrap that was basically exactly the thing I was trying to build with runc in my failed attempt, except that it actually works and has many more features and it’s built by people who know things about security! Hooray!

The interface to bubblewrap is pretty different than the interface to Docker – it’s much lower level. There’s no concept of a container image – instead you map a bunch of directories on your host to directories in the container.

For example, here’s how to run a container with the same root directory as your host operating system, but with only read access to that root directory, and only write access to /tmp.

bwrap \
    --ro-bind / / \
    --bind /tmp /tmp \
    --proc /proc --dev /dev \
    --unshare-pid \
    --unshare-net \
    bash

For example, you could imagine running some untrusted process under bubblewrap this way and then putting all the files you want the process to be able to access in /tmp.

bubblewrap runs containers as an unprivileged (non-root) user

Like podman, bubblewrap runs containers as a non-root user, using user namespaces. It can also run containers as root, but in this post we’re just going to be talking about using it as an unprivileged user.

bubblewrap is fast

Let’s see how long it takes to run ls in a bubblewrap container!

$ time bwrap --ro-bind / / --proc /proc --dev /dev --unshare-pid ls /
Executed in    8.04 millis

That’s a big difference! 8ms is a lot faster than 279ms.

Of course, like we said before, the reason bubblewrap is faster is that it does a lot less. So let’s talk about some things bubblewrap doesn’t do.

some things bubblewrap doesn’t do

Here are some things that Docker/podman do that bubblewrap doesn’t do:

• set up overlayfs mounts for you, so that your changes to the filesystem don’t affect the base image
• set up networking bridges so that you can connect to a webserver inside the container
• probably a bunch more stuff that I’m not thinking of

In general, bubblewrap is a much lower level tool than something like Docker.

Also, bubblewrap seems to have pretty different goals than Docker – the README seems to say that it’s intended as a tool for sandboxing desktop software (I think it comes from flatpak).

running a container image with bubblewrap

I couldn’t find instructions for running a Docker container image with bubblewrap, so here they are. Basically I just use Docker to download the container image and put it into a directory and then run it with bwrap:

There’s also a tool called bwrap-oci which looks cool but I couldn’t get it to compile.

mkdir rootfs
docker export $(docker create frapsoft/fish) | tar -C rootfs -xf -
bwrap \
    --bind $PWD/rootfs / \
    --proc /proc --dev /dev \
    --uid 0 \
    --unshare-pid \
    --unshare-net \
    fish

One important thing to note is that this doesn’t create a temporary overlay filesystem for the container’s file writes, so it’ll let the container edit files in the image.

I wrote a post about overlay filesystems if you want to see how you could do that yourself though.

running “containers” with bubblewrap isn’t the same as with podman

I just gave an example of how to “run a container” with bubblewrap, and you might think “cool, this is just like podman but faster!”. It is not, and it’s actually unlike using podman in even more ways than I expected.

I put “container” in scare quotes because there are two ways to define “container”:

• something that implements OCI runtime specification
• any way of running a process in a way that’s somehow isolated from the host system

bubblewrap is a “container” tool in the second sense. It definitely provides isolation, and it does that using the same features – Linux namespaces – as Docker.

But it’s not a container tool in the first sense. And it’s a lower level tool so you can get into a bunch of weird states and you really need to think about all the weird details of how container work while using it.

For the rest of the post I’m going to talk about some weird things that can happen with bubblewrap that would not happen with podman/Docker.

weird thing 1: processes that don’t exist

Here’s an example of a weird situation I got into with bubblewrap that confused me for a minute:

$ bwrap --ro-bind / / --unshare-all bash
$ ps aux
... some processes
root      390073  0.0  0.0   2848   124 pts/9    S    14:28   0:00 bwrap --ro-bind / / --unshare-all --uid 0 bash
... some other processes
$ kill 390073
bash: kill: (390073) - No such process
$ ps aux | grep 390073
root      390073  0.0  0.0   2848   124 pts/9    S    14:28   0:00 bwrap --ro-bind / / --unshare-all --uid 0 bash

Here’s what happened

• I started a bash shell inside bubblewrap
• I ran ps aux, and saw a process with PID 390073
• I try to kill the process. It fails with the error no such process. What?
• I ran ps aux, and still see the process with PID 390073

What’s going on? Why doesn’t the process 390073 exist, even though ps says it does? Isn’t that impossible?

Well, the problem is that ps doesn’t actually list all the processes in your current PID namespace. Instead, it iterates through all the entries in /proc and prints those out. Usually, what’s in /proc is actually the same as the processes on your system.

But with Linux containers these things can get out of sync. What’s happening in this example is that we have the /proc from the host PID namespace, but those aren’t actually the processes that we have access to in our PID namespace.

Passing --proc /proc to bwrap fixes the issue – ps then actually lists the correct processes.

$ bwrap --ro-bind / / --unshare-all --dev /dev --proc /proc ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
bork           1  0.0  0.0   3644   136 ?        S+   16:21   0:00 bwrap --ro-bind / / --unshare-all --dev /dev --proc /proc ps au
bork           2  0.0  0.0  21324  1552 ?        R+   16:21   0:00 ps aux

Just 2 processes! Everything is normal!

weird thing 2: trying to listen on port 80

Passing --uid 0 to bubblewrap makes the user inside the container root. You might think that this means that the root user has administrative privileges inside the container, but that’s not true!

For example, let’s try to listen on port 80:

$ bwrap --ro-bind / / --unshare-all --uid 0 nc -l 80
nc: Permission denied

What’s going on here is that the new root user actually doesn’t have the capabilities it needs to listen on port 80. (you need special permissions to listen on ports less than 1024, and 80 is less than 1024)

There’s actually a capability specifically for listening on privileged ports called CAP_NET_BIND_SERVICE.

So to fix this all we need to do is to tell bubblewrap to give our user that capability.

$ bwrap --ro-bind / / --unshare-all --uid 0 --cap-add cap_net_bind_service nc -l 80
(no output, success!!!)

This works! Hooray!

finding the right capabilities is pretty annoying

bubblewrap doesn’t give out any capabilities by default, and I find that figuring out all the right capabilities and adding them manually is kind of annoying. Basically my process is

• run the thing
• see what fails
• read man capabilities to figure out what capabilities I’m missing
• add the capability with --cap-add
• repeat until everything is running

But that’s the price I pay for wanting things to be fast I guess :)

weird thing 2b: --dev /dev makes listening on privileged ports not work

One other strange thing is that if I take the exact same command above (which worked!) and add --dev /dev (to set up the /dev/ directory), it causes it to not work again:

$ bwrap --ro-bind / / --dev /dev --unshare-all --uid 0 --cap-add cap_net_bind_service nc -l 80
nc: Permission denied

I think this might be a bug in bubblewrap, but I haven’t mustered the courage to dive into the bubblewrap code and start investigating yet. Or maybe there’s something obvious I’m missing!

weird thing 3: UID mappings

Another slightly weird thing was – I tried to run apt-get update inside a bubblewrap Ubuntu container and everything went very poorly.

Here’s how I ran apt-get update inside the Ubuntu container:

mkdir rootfs
docker export $(docker create ubuntu:20.04) | tar -C rootfs -xf -
bwrap \
    --bind $PWD/rootfs / \
    --proc /proc\
    --uid 0 \
    --unshare-pid \
    apt-get update

And here are the error messages:

E: setgroups 65534 failed - setgroups (1: Operation not permitted)
E: setegid 65534 failed - setegid (22: Invalid argument)
E: seteuid 100 failed - seteuid (22: Invalid argument)
E: setgroups 0 failed - setgroups (1: Operation not permitted)
.... lots more similar errors

At first I thought “ok, this is a capabilities problem, I need to set CAP_SETGID or something to give the container permission to change groups. But I did that and it didn’t help at all!

I think what’s going on here is a problem with UID maps. What are UID maps? Well, every time you run a container using “user namespaces” (which podman is doing), it creates a mapping of UIDs inside the container to UIDs on the host.

Let’s look that the UID maps! Here’s how to do that:

root@kiwi:/# cat /proc/self/uid_map 
         0       1000          1
root@kiwi:/# cat /proc/self/gid_map 
      1000       1000          1

This is saying that user 0 in the container is mapped to user 1000 on in the host, and group 1000 is mapped to group 1000. (My normal user’s UID/GID is 1000, so this makes sense). You can find out about this uid_map file in man user_namespaces.

All other users/groups that aren’t 1000 are mapped to user 65534 by default, according to man user_namespaces.

what’s going on: non-mapped users can’t be used

The only users and groups that have been mapped are 0 and 1000. But man user_namespaces says:

After the uid_map and gid_map files have been written, only the mapped values may be used in system calls that change user and group IDs.

apt is trying to use users 100 and 65534. Those aren’t on the list of mapped users! So they can’t be used!

This works fine in podman, because podman sets up its UID and GID mappings differently:

$ podman run -it ubuntu:20.04 bash
root@793d03a4d773:/# cat /proc/self/uid_map
         0       1000          1
         1     100000      65536
root@793d03a4d773:/# cat /proc/self/gid_map
         0       1000          1
         1     100000      65536

All the users get mapped, not just 1000.

I don’t quite know how to fix this, but I think it’s probably possible in bubblewrap to set up the uid mappings the same way as podman does – there’s an issue about it here that links to a workaround.

But this wasn’t an actual problem I was trying to solve so I didn’t dig further into it.

a quick note on Firecracker

Someone asked “would Firecracker work here?” (I wrote about Firecracker last year).

My experience with Firecracker VMs is that they use kind of a lot of RAM (like 50MB?), which makes sense because they’re VMs. And when I tried Firecracker on a tiny machine (with ~256MB of RAM / a tiny CPU), the startup times were 2-3 seconds.

I’m sure it’s possible to optimize Firecracker to be a bit faster, but at the end of the day I think it’s a VM and it’s not going to be anywhere near as low overhead as a process – there’s a whole operating system to start!

So Firecracker would add a lot more overhead than I want in this case.

bubblewrap works pretty great!

I’ve talked about a bunch of issues, but the things I’ve been trying to do in bubblewrap have been very constrained and it’s actually been pretty simple. For example, I was working on a git project where I really just want to run git inside a container and map a git repository from the host.

That’s very simple to get to work with bubblewrap! There were basically no weird problems! It’s really fast!

So I’m pretty excited about this tool and I might use it for more stuff in the future.