Hello! On Wednesday I was talking to a friend about how it would be cool to have an nginx playground website where you can just paste in an nginx config and test it out. And then I realized it might actually be pretty easy to build, so got excited and started coding and I built it. It’s at https://nginx-playground.wizardzines.com.

Here’s a screenshot:

For the rest of the post I’m mostly going to talk about how I built the project, because there were a few decisions that weren’t obvious to me.

how to use it

You need to enter both an nginx config and a curl or http command to make a HTTP request to that nginx instance

And then you click “Run” in the top right, and it’ll output either:

1. the results of the command you executed (if nginx started successfully), or
2. the nginx error logs (if nginx failed to start)

why a playground?

I find that playgrounds really help me learn – it’s incredibly useful to be able to quickly and safely experiment and try out different options without worrying that Something Terrible is going to happen if you make a mistake.

And nginx in particular is EXTREMELY finicky to configure, so I think it’s extra important to have a playground for nginx to quickly test things out.

Here are 3 playgrounds I’ve made in the past:

• SQL playground, which uses sql.js to let you run arbitrary SQLite queries on some small example data
• CSS examples, which uses codepen to show some examples of surprising CSS behaviour that you can play with
• DNS lookup, which makes DNS queries to any website you want

and a few other great playgrounds that others have made:

• CodePen for CSS/JS/HTML
• regexr for regular expressions
• db-fiddle for SQL
• this nginx location match tester that reimplements nginx’s location matching in typescript

building it quickly by keeping it simple

This site has

1. a static frontend (using vue.js and tailwind, my usual frontend stack)
2. a Go backend with a single API endpoint that just does 1 thing (run an nginx config)

This made pretty easy to build the project quickly (I just needed to write 1 backend endpoint and then a frontend that uses that endpoint!). This is also how the dns lookup tool I made works – I like this approach a lot and I think I’ll do other projects in the same way.

Let’s talk about what that backend code does when the frontend makes a request to it!

what happens when you make a request

Here’s what the Go backend does when you click ‘run’: (also here’s a gist with the code right now)

1. write the config to a temp file
2. create a new network namespace (ip netns add $RANDOM_NAMESPACE_NAME)
3. start go-httpbin on port 777 so that people can use that as a backend in their nginx configs
4. start nginx
5. wait 100ms to make sure nginx started, if it failed then return nginx’s error logs to the client
6. run the command that the user requested (and make sure that the command starts with curl or http)
7. return the command’s output
8. done

the security problem

The whole point of the tool is to let people run arbitrary nginx configurations, and it’s easy to imagine how that could result in the server being compromised. But I wanted run the service for free and not spend a lot of money on hosting. So I wanted to just use 1 shared server for all requests.

This kind of thing often stops me from doing projects like this, but this one felt a little more tractable to me.

the security approach: a little bit of isolation, a little bit of YOLO

Here’s how I decided to approach the security to start, after talking to a friend about it:

1. Host the frontend on a CDN, separately from the backend (so that if the backend gets compromised nobody can serve malware from the frontend)
2. Don’t use a database, just browser local storage (can’t hack the database if there isn’t one!)
3. Put every nginx in its own network namespace, don’t let it connect to the internet
4. Use fly.io’s free tier (so it’s isolated on its own VM, as far as I know it doesn’t have access to anything sensitive, and I can potentially destroy and redeploy the VM every hour if I want to)
5. Ask people to be nice in the FAQ (that’s the “YOLO” part :) )

I think that with these constraints, the only bad things that could happen are:

a. Someone gets access to the test nginx configs of other people using the website at the same time b. Someone replaces the backend API server so that it returns some sort of malicious or offensive output. c. Someone tries to mine bitcoin on the tiny instance it’s running on (1 shared CPU, 256 MB RAM)

I don’t think any of those are thaaaat harmful in the grand scheme of things, though it’s possible I’m missing something. Someone already showed how to read the /etc/passwd file which is fun, but there’s nothing sensitive in there.

I might switch to running each nginx in a container later instead of just running it in a network namespace, but I didn’t do it initially because I thought it might be too slow – it’s already a bit slow.

Speaking about slow, let’s talk about performance.

some notes on performance

Like I mentioned before, this backend is running on a pretty small instance (1 shared CPU, 256 MB RAM). Here are some quick notes on that:

1. the frontend runs on a CDN, so the backend only gets used when someone actually executes an nginx config. That takes a lot of pressure off the tiny backend.
2. According to the server logs, each request seems to take about 400ms right now. That’s not too bad!
3. It’s running on a server in Toronto right now, so I guess it’s be slower for people far away from Toronto. I could fix that by running more fly servers in more locations though.
4. I used a Go clone of httpbin instead of the original Python version, since I figured the Go version would be lighter weight
5. The frontend performance isn’t great – the CSS and JS is all in separate files. I didn’t want to use an npm build step to combine them because I’m pretty bad at Javascript and I’m always worried my Javascript build will break and then I’ll be too lazy to fix it and then it’ll be impossible for me to deploy changes.
6. I added a little rocket ship gif that plays while the backend is running to make it a little more fun to wait

The silliest performance problem I had was that I was originally stopping the nginx worker processes by sending them a SIGKILL signal. But that killed just the main process and not the worker processes, so then I was leaking nginx worker processes, which eventually made the instance run out of memory. Sending nginx processes a SIGTERM instead made it shut down everything correctly and fixed that problem.

the design

The design basically just copies jsfiddle and codepen.

In particular JSFiddle does a nice simple thing where it calculates the height of the main area as calc(100vh - 60px) and the header has height 60px. I wouldn’t have thought of that on my own but it seems to work really well.

I used CodeMirror for syntax highlighting because that’s what jsfiddle and codepen both seem to do, it was super easy to set up, and it turns out it even has an nginx and a shell mode! It’s everything I could have dreamed of :)

The main hiccup with CodeMirror was that I initially wanted to use a vue-codemirror integration and there wasn’t one for Vue 3, but I decided that it was unnecessary and just wrote my own tiny integration that updates Vue when the textbox is updated. (basically just this.nginx_codemirror.on('change', cm => this.nginx_config = cm.getValue()))

You can see the Javascript code at script.js but there’s really not a lot of it.

still to do: add more example nginx configs

I’m still not quite sure what the examples should be, but I think I want to provide a few more template nginx configs to use as starting points.

this was easier to build than I thought

This was a lot easier to build than I thought it would be! It makes me want to build playgrounds for other programs too, though I’m not sure which one would be next. HAProxy seems like an obvious similar choice.

Hello! This post (like patterns in confusing explanations) is another one in the “julia attempts to articulate her teaching strategy” series.

I’ll start out by talking about a “backwards” approach to learning that I think a lot of you will recognize (do projects first without fully understanding what you’re doing, fill in missing knowledge after), and then talk about I try to teach in a way that works better with that mode of learning.

how I’ve learned programming: backwards

There are a lot of “basic” concepts in programming, like memory management, networking, operating systems, assembly, etc. You could imagine that you learn programming by first starting with the basic concepts and then learning abstractions on top of those concepts.

But, even though I have a very traditional CS background, that’s not how it’s usually worked for me. Instead, here’s a real example from how I learned about websites:

1. Set up a website (for example using Apache)
2. Three years later, realize “wait, I have no idea how that worked at all”
3. Learn the basics of HTTP

Of course, to really understand how websites work, you do need to understand the basics of HTTP. But at the same time, you can make lots of great websites without knowing anything about HTTP except that 404 means “not found”.

it’s fun to build a lot of cool stuff quickly

Obviously I don’t actually think it’s bad to make websites without knowing HTTP. It’s fun! You can make a website and set up nginx or Apache by copying some stuff you found on Stack Overflow and didn’t fully understand, and then you have a website on the internet! I find it very fun to start out with a technology by quickly building a bunch of stuff with it without understanding exactly how it works (like making this shader).

it’s fun to fill in the information you’ve missed

Now that you have your website, here’s a way you could imagine learning about HTTP headers:

1. Your website suddenly gets really popular, so you put it behind a CDN to save money
2. But some of your pages are dynamic and you don’t want to cache them
3. So then you need to learn about browser caching and the Cache-Control HTTP header
4. And maybe then you learn about headers in general and find some other headers you want to set

I think that learning about HTTP headers after you already have a website is way more fun and a lot easier than learning about them before you have a website.

It’s much more obvious why you might want to use them, and you’re a lot more likely to remember the information later.

learning abstractions first results in knowledge gaps

Let’s say you start learning to make websites by learning Ruby on Rails. Rails abstracts away a lot of things, like SQL (through its ORM), sockets, and HTTP.

So if you learn Rails, you’re not likely to “naturally” pick up HTTP or sockets or SQL the same way you would if you were writing a webserver from scratch.

jobs today have more abstractions than they used to

I asked on twitter what people feel was easier to learn 15 years ago. One example a lot of people mentioned was the command line.

I was initially a bit surprised by this, but when I thought about it makes sense – if you were a web developer 15 years ago, it’s more likely that you’d be asked to set up a Linux server. That means installing packages, editing config files, and all kinds of things that would get you fluent at the command line. But today a lot of that is abstracted away and not as big a part of people’s jobs. For example if your site is running on Heroku, you barely have to know that there’s a server there at all.

I think this applies to a lot more things than the command line – networking is more abstracted away than it used to be too! In a lot of web frameworks, you just set up some routes and functions to handle those routes, and you’re done!

Abstractions are great, but they’re also leaky, and to do great work you sometimes need to learn about what lives underneath the abstraction.

how can we help people understand what’s underneath the abstractions?

Here some of you might be tempted to go on a rant about how “kids these days” don’t understand databases or networking or the command line or whatever thing has most annoyed you.

But that’s boring, we’ve had that argument a billion times, and it’s ineffective – people are going to keep learning abstractions first! It’s fun to learn abstractions first! You can get a lot of work done without understanding sockets!

But obviously I do think it’s important to fill in some of those knowledge gaps when it’s relevant to your job – that’s what this whole blog is about!

So let’s talk about how we can help people fill in some of their knowledge gaps (when they want/need to!), but by actually providing useful information instead of ranting about what “kids these days” should know :)

my steps for explaining something

I don’t actually have explicit steps I use when teaching, but I spent some time thinking about my approach and here’s what I think I do.

Instead of starting “at the beginning” when explaining a topic, what I usually do when writing a blog post or comic is:

1. notice a useful idea that I think is on the edge of a lot of people’s knowledge
2. make a bunch of assumptions about what I think that group of people generally know (usually based on what I know and what I think my friends/coworkers know)
3. explain the idea using those assumptions

Here are a few more guidelines that I use when teaching “backwards”.

• use people’s existing knowledge when teaching
• explain just one concept at a time
• use bugs to peel back abstractions

Let’s talk more about each of those!

use people’s existing knowledge when teaching

One mistake I see people make a lot when they notice someone has a knowledge gap is:

1. notice that someone is missing a fundamental concept X (like knowing how binary works or something)
2. assume “wow, you don’t know how binary works, you must not know ANYTHING”
3. give an explanation that assumes the person basically doesn’t know how to program at all
4. waste a bunch of their time explaining things they already know

But this is a huge missed opportunity when explaining a concept to a professional programmer – they already know a lot! Any professional programmer definitely knows a lot of things that are very related to how binary works.

explain just one concept at a time

Because I don’t have a lot of information about who I’m writing for, I try to:

1. only address one major gap at a time
2. make it super clear what the gap I’m addressing is

Then people can easily tell if what I’m writing is relevant to them, get the information they’re interested in, and leave.

It might seem like this wouldn’t work because everyone has a totally unique set of knowledge, but actually there are a lot of interesting facts that happen to be on the edge of a LOT of people’s knowledge. So it’s all about identifying patterns (“hmm, I didn’t realize X for a long time, I bet other people didn’t either”).

bugs are an amazing way to learn what’s under an abstraction

My favourite way to learn what’s underneath an abstraction is bugs! Bugs often break through abstraction boundaries, like this performance issue that was caused by a TCP setting.

So they’re a great excuse to learn what’s underneath, they show you exactly how the underlying thing (like TCP) is connected to the abstraction (like an HTTP request). And they inherently help you learn one step at a time, because each bug will usually teach you about just one or two things.

teaching by filling in knowledge gaps can be faster

One thing I love about teaching this way is that if the reader is already using a tool, I can assume that they know a bunch of basic things about it and skip straight to the interesting parts.

For example, let’s look at this bash cheat sheet.

This cheat sheet assumes that you know:

• what bash is
• that variables are referenced in bash with $VAR
• what “stdout” means
• what it means to search and replace in a string
• what an array is

And I’d guess that most people who are writing bash scripts know those things! But I’d also guess that many people who write bash scripts don’t know that:

• /usr/bin/[ is a program (actually in bash [ is a builtin, but it’s a builtin that behaves the same way as the program /usr/bin/[)
• the curly brace in a{.png.svg} has a totally different meaning from the one in { command1; command2 }
• you can search and replace in a string with ${}

example of teaching by filling in knowledge gaps: bite size bash

Last year I wrote a zine called bite size bash about how bash scripting works. (the above cheat sheet is part of it)

You might think that I’d explain some basic things like “what bash is” and the usual syntax of a bash script. But I didn’t. That’s because I wrote the zine for people who were already using bash.

Instead, I went through the most things bash script writers don’t understand about bash (variable assignment, how if statements work, how to quote things, etc) and explained how those things actually work in bash.

filling in knowledge gaps at the drop-in tutoring center

To me teaching and writing in this way feels very natural. But people have told me they find it really difficult to write without explaining everything from the beginning.

I think my teaching style might be informed in some way by my experience working as a tutor as a drop-in math tutoring centre. I’ve had 5 or so part time jobs (from age 15-20) doing math or physics tutoring where I basically had to talk to random students who were confused about something for 20 minutes. I could probably only teach them only 1 or 2 things in those 20 minutes, and I might only ever meet the person once. So I had to quickly figure out what they already knew and what kind of information might help them.

I learned to listen a lot and not jump to conclusions about what the person didn’t know – often they already actually understood a lot about the math class they were taking and I just needed to explain one or two small things.

I think of writing on the internet as being kind of similar. I have no idea who’s reading my writing or what they already know, and they’re probably going to only spend 10 minutes or so reading it. So I can only intervene in a really limited way.

I think unpeeling abstractions is what I’m trying to do in my writing

I think “show people what lives underneath their abstractions” is a big part of what I’m trying to do with my writing. This might be why I’m so obsessed with bugs – they’re such a natural way to learn about what’s hiding under your abstractions in a way that’s actually relevant to your work.

Thanks to Miles, Jake, Dan, Matthew, and Kamal for reading a draft of this post.

Hello! I was talking to a Python programmer friend yesterday about debugging, and I mentioned that I really like debugging using a REPL. He said he’d never tried it and that it sounded fun, so I thought I’d write a quick post about it.

This debugging method doesn’t work in a lot of languages, but it does work in Python and Ruby and kiiiiiind of in C (via gdb).

what’s a REPL?

REPL stands for “read eval print loop”. A REPL is a program that:

1. reads some input from you like print(f"2 + 2 = {2+2}") (read)
2. evaluates the input (eval)
3. print out the result (print)
4. and then goes back to step 1 (loop)

Here’s an example of me using the IPython REPL to run a print statement. (also it demonstrates f-strings, my favourite Python 3 feature)

$ ipython3
Python 3.9.5 (default, May 24 2021, 12:50:35) 
Type 'copyright', 'credits' or 'license' for more information
IPython 7.24.1 -- An enhanced Interactive Python. Type '?' for help.

In [1]: print(f"2 + 2 = {2+2}")
2 + 2 = 4

In [2]: 

you can start a REPL at a breakpoint

There are 2 ways to use a REPL when debugging.

Way 1: Open an empty REPL (like IPython, pry, or a browser Javascript console) to test out something.

This is great but it’s not what I’m talking about in this post.

Way 2: Set a breakpoint in your program, and start a REPL at that breakpoint.

This is the one we’re going to be talking about. I like doing this because it gives me both:

1. all the variables in scope at the breakpoint, so I can print them out interactively
2. easy access to all the functions in my program, so I can call them to try to find issues

how to get a REPL in Python: ipdb.set_trace()

Here’s a program called test.py that sets a breakpoint on line 5 using import ipdb; ipdb.set_trace().

import requests

def make_request():
    result = requests.get("https://google.com")
    import ipdb; ipdb.set_trace()

make_request()

And here’s what it looks like when you run it: you get a REPL where you can inspect the result variable or do anything else you want.

python3 test.py
--Return--
None
> /home/bork/work/homepage/test.py(5)make_request()
      4     result = requests.get("https://google.com")
----> 5     import ipdb; ipdb.set_trace()
      6 

ipdb> result.headers
{'Date': 'Thu, 16 Sep 2021 13:11:19 GMT', 'Expires': '-1', 'Cache-Control': 'private, max-age=0', 'Content-Type': 'text/html; charset=ISO-8859-1', 'P3P': 'CP="This is not a P3P policy! See g.co/p3phelp for more info."', 'Content-Encoding': 'gzip', 'Server': 'gws', 'X-XSS-Protection': '0', 'X-Frame-Options': 'SAMEORIGIN', 'Set-Cookie': '1P_JAR=2021-09-16-13; expires=Sat, 16-Oct-2021 13:11:19 GMT; path=/; domain=.google.com; Secure, NID=223=FXhKNT7mgxX7Fjhh6Z6uej9z13xYKdm9ZuAU540WDoIwYMj9AZzWTgjsVX-KJF6GErxfMijl-uudmjrJH1wgH3c1JjudPcmDMJovNuuAiJqukh1dAao_vUiqL8ge8pSIXRx89vAyYy3BDRrpJHbEF33Hbgt2ce4_yCZPtDyokMk; expires=Fri, 18-Mar-2022 13:11:19 GMT; path=/; domain=.google.com; HttpOnly', 'Alt-Svc': 'h3=":443"; ma=2592000,h3-29=":443"; ma=2592000,h3-T051=":443"; ma=2592000,h3-Q050=":443"; ma=2592000,h3-Q046=":443"; ma=2592000,h3-Q043=":443"; ma=2592000,quic=":443"; ma=2592000; v="46,43"', 'Transfer-Encoding': 'chunked'}

You have to install ipdb to make this work, but I think it’s worth it – import pdb; pdb.set_trace() will work too (and is built into Python) but ipdb is much nicer. I just learned that you can also use breakpoint() in Python 3 to get a breakpoint, but that puts you in pdb too which I don’t like.

how to get a REPL in Ruby: binding.pry

Here’s the same thing in Ruby – I wrote a test.rb program:

require 'net/http'
require 'pry'

def make_request()
  result = Net::HTTP.get_response('example.com', '/')
  binding.pry
end

make_request()

and here’s what it looks like when I run it:

$ ruby test.rb
From: /home/bork/work/homepage/test.rb:6 Object#make_request:

    4: def make_request()
    5:   result = Net::HTTP.get_response('example.com', '/')
 => 6:   binding.pry
    7: end

[1] pry(main)> result.code
=> "200"

you can also do get a REPL in the middle of an HTTP request

Rails also lets you start a REPL in the middle of a HTTP request and poke around and see what’s happening. I assume you can do this in Flask and Django too – I’ve only really done this in Sinatra (in Ruby).

GDB is sort of like a REPL for C

I was talking to another friend about REPLs, and we agreed that GDB is a little bit like a REPL for C.

Now, obviously this is sort of not true – C is a compiled language, and you can’t just type in arbitrary C expressions in GDB and have them work.

But you can do a surprising number of things like:

• call functions
• inspect structs if your program has debugging symbols (p var->field->subfield)

This stuff only works in gdb because the gdb developers put in a lot of work doing Very Weird Things to make it easier to get a REPL-like experience. I wrote a blog post a few years called how does gdb call functions? about how surprising it is that gdb can call functions, and how it does that.

This is the only way I use gdb when looking at C programs – I never set watchpoints or do anything fancy, I just set a couple of breakpoints in the program and then poke around at those points.

where this method works

languages where this works:

• Python
• Ruby
• probably PHP, but I don’t know
• C, sort of, in a weird way (though you might disagree :))
• in Javascript: it seems like you can use debugger; to get a REPL through either node inspect or the browser console. There seem to be some limitations on what you can do (like node won’t let me use await in its REPL), but I haven’t done enough JS to fully understand this.
• In Java, apparently IntelliJ lets you evaluate arbitrary expressions at a breakpoint, which isn’t quite a REPL but is cool

languages where this doesn’t work:

• most compiled languages

REPL debugging is easy for me to remember how to do

There are (at least) 4 different ways of debugging:

1. Lots of print statements
2. a debugger
3. getting a REPL at a breakpoint
4. inspect your program with external tools like strace

I think part of the reason I like this type of REPL debugging more than using a more traditional debugger is – it’s so easy to remember how to do it! I can just set a breakpoint, and then run code to try to figure out what’s wrong.

With debuggers, I always forget how to use the debugger (probably partly because I switch programming languages a lot) and I get confused about what features it has and how they work, so I never use it.

Hello! I was talking to a friend yesterday who was studying for a programming interview and trying to learn some algorithms basics.

The topic of quadratic-time vs linear-time algorithms came up, I thought this would be fun to write about here because avoiding quadratic-time algorithms isn’t just important in interviews – it’s sometimes good to know about in real life too! I’ll explain what a “quadratic-time algorithm is” in a minute :)

here are the 3 things we’ll talk about:

1. quadratic time functions are WAY WAY WAY slower than linear time functions
2. sometimes you can make a quadratic algorithm into a linear algorithm by using a hashmap
3. this is because hashmaps lookups are very fast (instant!)

I’m going to try to keep the math jargon to a minimum and focus on real code examples and how fast/slow they are.

our problem: intersect two lists

Let’s talk about a simple interview-style problem: getting the intersection of 2 lists of numbers. For example, intersect([1,2,3], [2,4,5]) should return [2].

This problem is also somewhat realistic – you could imagine having a real program where you need to take the intersection of 2 lists of IDs.

the “obvious” solution:

Let’s write some code to take the intersection of 2 lists. Here’s a program that does it, called quadratic.py.

import sys

# the actual code
def intersection(list1, list2):
    result = []
    for x in list1:
        for y in list2:
            if x == y:
                result.append(y)
    return result

# some boilerplate so that we can run it from the command line on lists of
# different sizes
def run(n):
    # make 2 lists of n+1 elements
    list1 = list(range(3, n)) + [2]
    list2 = list(range(n+1, 2*n)) + [2]
    # intersect them and print out the result
    print(list(intersection(list1, list2)))

# Run with the program's first command line argument
run(int(sys.argv[1]))

The reason it’s called quadratic.py is that if list1 and list2 have size n, then the inner loop (if x == y) will run n^2 times. And in math, functions like x^2 are called “quadratic” functions.

how slow is quadratic.py?

Let’s run this program with a bunch of lists of different lengths. The intersection of the two lists is always the same: [2].

$ time python3 quadratic.py 10
[2]

real	0m0.037s
$ time python3 quadratic.py 100
[2]

real	0m0.053s
$ time python3 quadratic.py 1000
[2]

real	0m0.051s
$ time python3 quadratic.py 10000 # 10,000
[2]

real	0m1.661s

So far none of this is too bad – it’s still taking less than 2 seconds.

Then I ran it on two lists with 100,000 elements, and I had to wait a LONG time. Here’s the result:

$ time python3 quadratic.py 100000 # 100,000
[2]

real	2m41.059s

This is very slow! It’s 160 seconds, which is almost exactly 100x longer than it did to run on 10,000 elements (which was 1.6 seconds). So we can see that after a certain point, every time we make the list 10x bigger, the program takes about 100x longer to run.

I didn’t try to run this program on 1,000,000 elements, because I knew it would take 100x longer again – probably about 3 hours. I don’t have time for that!

You can probably see now why quadratic time algorithms can be a problem – even this very simple program starts getting very slow pretty quickly.

let’s write a fast version: linear.py

Okay, so let’s write a fast version of the program. First I’ll show you the program, then I’ll explain it.

import sys

# the actual algorithm
def intersection(list1, list2):
    set1 = set(list1) # this is a hash set
    result = []
    for y in list2:
        if y in set1:
            result.append(y)
    return result

# some boilerplate so that we can run it from the command line on lists of
# different sizes
def run(n):
    # make 2 lists of n+1 elements
    list1 = range(3, n) + [2]
    list2 = range(n+1, 2*n) + [2]
    # print out the intersection
    print(intersection(list1, list2))

run(int(sys.argv[1]))

(this isn’t the most idiomatic Python, but I wanted to write it without using too many python-isms so that people who don’t know Python could understand it more easily)

We’ve done 2 things differently here than our slow program:

1. convert list1 into a set called set1
2. only use one for loop instead of two for loops

let’s see how fast this linear.py program is

Before we talk about why this program is fast, let’s first prove that it’s fast by running it on some big lists. Here it is running on lists of size 10 to 10,000,000. (remember that our original program started getting SUPER slow when run on 100,000 elements)

$ time python3 linear.py 100
[2]

real	0m0.056s
$ time python3 linear.py 1000
[2]

real	0m0.036s
$ time python3 linear.py 10000 # 10,000
[2]

real	0m0.028s
$ time python3 linear.py 100000 # 100,000 
[2]

real	0m0.048s <-- quadratic.py took 2 minutes in this case! we're doing it in 0.04 seconds now!!! so fast!
$ time python3 linear.py 1000000 # 1,000,000
[2]

real	0m0.178s
$ time python3 linear.py 10000000 # 10,000,000
[2]

real	0m1.560s

running linear.py on an extremely big list

If we try to run it on a very very big list (10 billion / 10,000,000,000 elements), then actually we run into a different problem: it’s fast enough (that list is only 100x bigger than the list that took 4.2 seconds, so we could probably do it in 420 seconds), but my computer doesn’t have enough memory to store all of the elements of the list and so the program crashes before it gets there.

$ time python3 linear.py 10000000000
Traceback (most recent call last):
  File "/home/bork/work/homepage/linear.py", line 18, in <module>
    run(int(sys.argv[1]))
  File "/home/bork/work/homepage/linear.py", line 13, in run
    list1 = [1] * n + [2]
MemoryError

real	0m0.090s
user	0m0.034s
sys	0m0.018s

We’re not talking about memory usage in this blog post though, so let’s ignore that.

okay, why is linear.py fast?

Now I’ll try to explain why linear.py is fast.

Here’s the code again:

def intersection(list1, list2):
    set1 = set(list1) # this is a hash set
    result = []
    for y in list2:
        if y in set1:
            result.append(y)
    return result

Let’s say that list1 and list2 are both lists of about 10,000,000 different elements. That’s kind of a lot of elements!

So why is this able to run so fast? HASHMAPS!!!

hashmap lookups are instant (“constant time”)

Let’s look at this if statement from our fast program:

if y in set1:
    result.append(y)

You might think that this check – if y in set1 – would be slower if the set1 contains 10 million elements than it is if set1 contains 1000 elements. But it’s not! It always takes basically the same amount of time (SUPER FAST), no matter how big set1 gets.

This is because set1 is a hash set, which is a type of hashmap/hashtable which only has keys and no values.

I’m not going to explain why hashmap lookups are instant in this post, but the amazing Vaidehi Joshi’s basecs series has explanations of hash tables and hash functions which talk about it.

accidentally quadratic: real life quadratic algorithms!

This issue that we saw where quadratic time algorithms are really slow is actually a problem that shows up in real life – Nelson Elhage has a great blog called accidentally quadratic with stories about performance problems caused by code that accidentally ran in quadratic time.

quadratic time algorithms can kind of sneak up on you

The weird thing about quadratic time algorithms is that when you run them on a small number of elements (like 1000), it doesn’t seem so bad! It’s not that slow! But then if you throw 1,000,000 elements at it, it can really take hours to run.

So I think it’s worth being broadly aware of them, so you can avoid writing them by accident. Especially if there’s an easy way to write a linear-time algorithm instead (like using a hashmap).

sometimes the “slow” algorithm is actually faster

If you’re doing serious performance work, for example on an embedded system, it’s also important to realize that a “faster” algorithm like this example of using a hashmap will often actually be slower on a small number of elements. (I’ve never run into this myself, but friends have told me that it comes up)

For example, this great blog post Linear vs Binary Search has some benchmarks showing that linear search is faster than binary search for small arrays (up to 100 elements!)

hashmaps always feel a little magical to me

Hashmaps aren’t magic of course (you can learn the math behind why hashmap lookups are instant! it’s cool!), but it always feels a little magical to me, and every time I use hashmaps in a program to speed things up it makes me happy :)

Hello! Recently I’ve been thinking about why I explain things the way I do. The usual way I write is:

1. Try to learn a topic
2. Read a bunch of explanations that I find confusing
3. Eventually understand the topic
4. Write an explanation that makes sense to me, to help others

So why do I find all these explanations so confusing? I decided to try and find out! I came up with a list of 13 patterns that make explanations hard for me to understand. For each pattern I’ll also explain what I like to do instead to avoid the issue.

these patterns are very normal

This list isn’t meant to make you feel bad about your writing. I’ve probably done all of these things! I’m certainly going to do them again! I even did at least one of them while writing this post!

But knowing that I’m likely to accidentally do these things makes it easier for me to avoid them, and it makes me more receptive to critique when people point out issues with my writing (“Julia, this is assuming a lot of knowledge that I don’t have!“).

Being aware of these patterns also helps me when reading a confusing explanation: “oh, I’m not confused by this explanation because I’m stupid, I’m confused because it’s introduced 6 new-to-me concepts and it hasn’t explained what any of them is yet!“.

why this post is framed in a negative way

I practically always write in a positive way (“X is a good practice!”) instead of in a negative way (“Y is a bad practice!”). So why am I writing about confusing patterns instead of writing about positive patterns?

Writing clearly is a LOT of work. A big part of what motivates me to put in the work to write clearly is my frustration with confusing technical explanations (“ugh, everything I read about Linux containers was SO confusing, I wish someone had just told me X Y Z…“).

But, if I’m not careful, it’s easy to reproduce the exact same confusing patterns in my own writing! And the problem with positive patterns (like “avoid introducing unnecessary jargon”) is that they seem so obvious that I trick myself into thinking I’m following them, even when I’m not! So I’m writing these down to try to keep myself honest and hopefully help you avoid some of these patterns as well.

now for the patterns!

Now that I’ve explained my motivation, let’s explain the patterns! Here’s a quick index of all of them. They’re not in any particular order.

1. pattern 1: making outdated assumptions about the audience’s knowledge
2. pattern 2: having inconsistent expectations of the reader’s knowledge
3. pattern 3: strained analogies
4. pattern 4: fun illustrations on dry explanations
5. pattern 5: unrealistic examples
6. pattern 6: jargon that doesn’t mean anything
7. pattern 7: missing key information
8. pattern 8: introducing too many concepts at a time
9. pattern 9: starting out abstract
10. pattern 10: unsupported statements
11. pattern 11: no examples
12. pattern 12: explaining the “wrong” way to do something without saying it’s wrong
13. pattern 13: “what” without “why”

pattern 1: making outdated assumptions about the audience’s knowledge

I see a lot of writing, especially systems writing, that makes outdated assumptions about what the reader knows. For example, here’s a paragraph from this Git book comparing Git’s implementation of branching to other version control tools.

Nearly every VCS has some form of branching support. […] In many VCS tools, this is a somewhat expensive process, often requiring you to create a new copy of your source code directory, which can take a long time for large projects.

The outdated assumption here is that you (the reader) know how other version control systems implement branching, and that comparing other tools’ implementation of branching to Git’s implementation will help you understand branching.

But if you’re reading this and you’ve never used another version control system and never plan to, this explanation is useless! Who cares about how other version control systems implement branching? You just want to understand how Git works!

The reason this explanation is written this way is probably that the first edition of the book was published in 2009, and this assumption was probably true in 2009! Many people learning Git shortly after it was released were switching from Subversion or CVS or something and found comparisons like this helpful.

But in 2021 Git has been the dominant version control system for a long time, and most people learning Git for the first time won’t have any experience with version control other than Git.

I also sometimes see this “outdated assumptions about the audience’s knowledge” problem with newer writing. It generally happens when the writer learned the concept many years ago, but doesn’t have a lot of experience explaining it in the present. So they give the type of explanation that assumes that the reader knows approximately the same things they and their friends knew in 2005 and don’t realize that most people learning it today have a different set of knowledge.

instead: test your explanations!

Usually if I learned a concept a long time ago, it means that I’ve lost touch with what it’s like to learn it for the first time today. So running an explanation by a few people who don’t already know the concept helps to catch incorrect assumptions I’ve made.

(I bolded “don’t already know the concept” because it’s tempting to ask someone who already understands the concept for a review. But they might have the exact same blind spots as you!)

pattern 2: having inconsistent expectations of the reader’s knowledge

For example, a new language tutorial might explain a concept that almost all programmers would know, like how a for loop is used for iteration, while the writing that immediately follows implicitly assumes knowledge that many people don’t have, like how the stack works, how malloc works, etc. (thanks to Dan Luu for this example!)

The problem with this is that are probably zero people who understand malloc but don’t understand how a for loop works! And even though it sounds silly, it’s easy to accidentally write like this if you don’t have a clear idea of who you’re writing for.

instead: pick 1 specific person and write for them!

You can pick a friend, a coworker, or just a past version of yourself. Writing for just 1 person might feel insufficiently general (“what about all the other people??“) but writing that’s easy to understand for 1 person (other than you!) has a good chance of being easy to understand for many other people as well.

pattern 3: strained analogies

Sometimes when trying to explain a complex technical concept, an author will start with a real-world concept that the reader definitely understands and use a very involved analogy to compare them.

Here’s an example I made up:

Imagine our event system is like the Mississippi River. It travels through many different ecosystems, and some rain particles don’t make it all the way. Sometimes it flows at different speeds depending on environmental conditions. The Mississippi River ends in many different tributaries.

Many different kinds of fish live in the event system. Different fish have different destinations. Humans decide to live along the river and use it for different purposes. They construct dams to control the flow.

This example is a parody, but I always find this type of analogy confusing because I end up wasting a lot of time trying to analyze exactly how an event stream is different / the same as the Mississippi river instead of just learning technical facts about event streams:

I think authors do this because.. it’s kind of fun to write these Big Weird Analogies! Like, is there something in a stream processing system that’s like a dam? Maybe! It’s kind of fun to think about! But even though these can be fun to write, they’re not as fun to read – it’s a struggle to extract the actual technical facts you want to know.

instead: keep analogies to a single idea

Instead of using “big” analogies where I explain in depth exactly how an event processing system is like a river, I prefer to explain the analogy in one or two sentences to make a specific point and then leave the analogy behind.

Here are 2 ways to do that.

option 1: use “implicit” metaphors

For example, if we’re talking about streams, I might write:

Every event in a stream flows from a producer to a consumer.

Here I’m using the word “flow”, which is definitely a water metaphor. I think this is great – it’s an efficient way to evoke an idea of directionality and the idea that there are potentially a large number of events.

I put together a bunch more metaphors in this style in Metaphors in man pages.

option 2: use a very limited analogy

For example, here’s a nice explanation from When costs are nonlinear, keep it small by Jessica Kerr that explains batching using an analogy to doing your laundry in a batch.

We like batching. Batching is more efficient: doing ten at once is faster than doing one, one, two, one, one, etc. I don't wash my socks as soon as I take them off, because lumping them in with the next load is free.

This analogy is very clear! I think it works well because batching in laundry works for the same reasons as batching in computing – batching your laundry works because there’s a low incremental cost to adding another pair of socks to the load. And it’s only used to illustrate one idea – that batching is a good choice when there’s a low incremental cost for adding a new item.

pattern 4: fun illustrations on dry explanations

Sometimes I see authors put fun illustrations with a very dry explanation to make the explanation seem more appealing and approachable.

The goal isn’t generally to trick the reader into expecting a more friendly explanation – I think the logic is usually more like “people like fun illustrations! let’s add some!“. But no matter what the intent, the problem is that the reader can end up feeling misled.

instead: make the design reflect the style of the explanation

There are lots of great examples of illustrated explanations where the writing is in a clear and friendly style:

• how dns works
• why’s (poignant) guide to ruby
• how do calculators even

On the other hand, dry explanations are useful too! Nobody expects the Intel instruction-set reference to be light reading! The writing is dry and technical, and the design is very utilitarian, which matches the style of the writing.

pattern 5: unrealistic examples

Here’s an unrealistic example of how to use lambda in Python:

numbers = [1, 2, 3, 4]
squares = map(lambda x: x * x, numbers)

This example is unrealistic because most people don’t use map in Python – you’d use list comprehensions to do this instead.

Here’s another unrealistic example: an interface example from the Oracle docs on interfaces.

interface Bicycle {
    //  wheel revolutions per minute
    void changeCadence(int newValue);
    void changeGear(int newValue);
    void speedUp(int increment);
    void applyBrakes(int decrement);
}

This kind of “real world example” is super common in object oriented programming explanations but I find it quite confusing – I’ve never implemented a bicycle or car in my code! It doesn’t tell me anything about what interfaces are useful for!

instead: write realistic examples!

Here’s a more realistic example of Python lambdas, which sorts a list of children by their age. (from my post Write good examples by starting with real code) This is how I use Python lambdas the most in practice.

children = [
    {"name": "ashwin", "age": 12},
    {"name": "radhika", "age": 3},
]
sorted_children = sorted(children, key=lambda x: x['age'])

Here’s a more realistic example of Java interfaces.

The Comparable interface (from the JDK source) just has one method -- here's its full implementation.

public interface Comparable<T> {
    int compareTo(T o);
}

To implement this interface, you just need to implement the compareTo method. And if you write a class that implements this interface (like a Money class for example), then you get all sorts of useful things for free! You can sort an array of Money objects with Arrays.sort! You can put them in a SortedSet!

In this Java example, of course it’s not enough to explain built-in Java interfaces – you also need realistic examples of how to create and use your own interfaces. But this post isn’t about Java interfaces so let’s move on.

pattern 6: jargon that doesn’t mean anything

Let’s talk about this sentence from this chapter on commit signing:

Git is cryptographically secure, but it’s not foolproof.

“Cryptographically secure” is unclear here because it sounds like it should have a specific technical meaning, but it’s not explained anywhere what’s actualy meant. Is it saying that Git uses SHA-1 to hash commits and it’s difficult to generate SHA-1 hash collisions? I don’t know!

Using jargon in a meaningless way like this is confusing because it can trick the reader into thinking something specific is being said, when the information they need is not actually there. (the chapter doesn’t explain anywhere what’s meant by “cryptographically secure” in this context)

instead: Avoid jargon where it’s not needed

A lot of the time I find I can communicate what I need to without using any jargon at all! For example, I’d explain why commit signing is important like this:

When making a Git commit, you can set any name and email you want! For example, I can make a commit right now saying I'm Linus Torvalds like this:

git commit -m"Very Serious Kernel Update" \
 --author='Linus Torvalds <torvalds@linux-foundation.org>'
 

pattern 7: missing key information

Sometimes explanations of a concept are missing the most important idea to understand the concept. For example, take this explanation from this chapter on the Git object model (which by the way has a nice concrete example of how to explore Git’s object model):

Git is a content-addressable filesystem. Great. What does that mean? It means that at the core of Git is a simple key-value data store. What this means is that you can insert any kind of content into a Git repository, for which Git will hand you back a unique key you can use later to retrieve that content.

This paragraph is missing what to me is the main idea of content-addressable storage – that the key for a piece of content is a deterministic function of the content, usually a hash (though the page does later say that Git uses a SHA-1 hash). It’s important that the key is a function of the content and not just any random unique key because the idea is that the content is addressed by itself – if the content changes, then its key also has to change.

This pattern is hard to recognize as a reader because – how are you supposed to recognize that there’s a key idea missing when you don’t know what the key ideas are? So this is a case where a reviewer who understands the subject well can be really helpful.

pattern 8: introducing too many concepts at a time

Here’s an explanation of linkers from this page that I find confusing:

During the link process, the linker will pick up all the object modules specified on the command line, add some system-specific startup code in front and try to resolve all external references in the object module with external definitions in other object files (object files can be specified directly on the command line or may implicitly be added through libraries). It will then assign load addresses for the object files, that is, it specifies where the code and data will end up in the address space of the finished program. Once it’s got the load addresses, it can replace all the symbolic addresses in the object code with “real”, numerical addresses in the target’s address space. The program is ready to be executed now.

Here are the concepts in this paragraph:

• object modules (.o files)
• external references
• symbolic addresses
• load addresses
• system-specific startup code

It’s too much!

instead: give each concept some space to breathe

For example, I might explain “external references” like this:

if you run objdump -d myfile.o on an object file you can see that the call function calls are missing a target address, so that's why the linker needs to fill that in.

  33:   e8 00 00 00 00          call   38 
           ^^^^^^^^^^^
             this address is all 0s -- it needs to be filled in by the linker!
             with the actual function that's going to be called!
  38:   84 c0                   test   %al,%al
  3a:   74 3b                   je     77 
  3c:   48 83 7d f8 00          cmpq   $0x0,-0x8(%rbp)
  

There’s still a lot of missing information here (how does the linker know what address to fill in?), but it’s a clear starting point and gives you questions to ask.

pattern 9: starting out abstract

Imagine I try to explain to you what a Unix signal using the definition from Wikipedia.

Signals are a limited form of inter-process communication (IPC), typically used in Unix, Unix-like, and other POSIX-compliant operating systems. A signal is an asynchronous notification sent to a process or to a specific thread within the same process to notify it of an event. Signals originated in 1970s Bell Labs Unix and were later specified in the POSIX standard.

By itself, this probably isn’t going to help you understand signals if you’ve never heard of them before! It’s very abstract and jargon-heavy (“asynchonous notification”, “inter-process communication”) and doesn’t have any information about what Unix signals are used for in practice.

Of course, the Wikipedia explanation isn’t “bad” exactly – it’s probably written like that because teaching people about signals for the first time isn’t really the goal of the Wikipedia article on signals.

instead: start out concrete

For example, I wrote this page explaining signals a few years ago.

I start out by relating signals to the reader’s experience (“have you used kill? you’ve used signals!“) before explaining how they work.

pattern 10: unsupported statements

Here’s an explanation of C header files, from this page.

In modern C, header files are crucial tools that must be designed and used correctly. They allow the compiler to cross-check independently compiled parts of a program.

Headers declare types, functions, macros etc that are needed by the consumers of a set of facilities. All the code that uses any of those facilities includes the header. All the code that defines those facilities includes the header. This allows the compiler to check that the uses and definitions match.

This says “In modern C, header files are crucial tools…” (which is true), but it doesn’t explain why header files are crucial. This of course wouldn’t be a problem if the audience already understood why header files in C are important (it’s a very fundamental concept!). But the whole point here is to explain header files, so it needs to be explained.

instead: Prove that your statements are true!

For example, I might write:

Almost every C program includes header files. For example, if you've ever written #include <stdio.h> at the beginning of a C program, stdio.h is a header file. #include basically tells the C preprocessor to paste the contents of stdio.h at the beginning of the program.

One reason header files are important is that they define types and constants you need in your programs. For example, this code by itself will fail to compile with the error error: unknown type name 'FILE', because the FILE type is undefined.

int main() {
    FILE *fp;
    fp  = fopen("data.txt", "w");
}

FILE is defined in stdio.h and if you add a #include <stdio.h>, at the top, then the program will compile.

This example program lets the reader actually run that program themselves and verify that it doesn’t compile if they want – they don’t have to take my word for it!

pattern 11: no examples

Another problem with the previous explanation of header files is – there aren’t any examples! Leaving out examples makes it harder for the reader to relate the new terminology to their own experiences.

Almost anyone who’s ever written a C program has definitely used header files, so a simple example (like mentioning stdio.h) can really help.

In that header files example, I replaced

In modern C, header files are crucial tools…

with an explanation that includes a simple example:

Almost every C program includes header files -- if you've ever seen something like #include at the beginning of a C program, stdio.h is a header file.

pattern 12: explaining the “wrong” way to do something without saying it’s wrong

Here’s a pattern I see sometimes in tutorials (though unfortunately I don’t have an example):

1. Explain the “wrong” way of doing something without saying up front that it’s wrong
2. Later on, show the consequences of doing the “wrong” thing
3. Explain the “right” way

I think the intention of this is to imitate the real-life experience of making mistakes. Usually when you make a mistake, you don’t know that it’s wrong at the time!

But often the reader will end up feeling misled or confused about which way is actually “correct”. And it’s possible that they would never even have made that particular mistake on their own!

instead: here are four options for presenting mistakes

Here are a few ways of accomplishing the same thing without misleading the reader:

1. Frame the “wrong” thing as an experiment (“what if we try doing it X way?”)
2. State an incorrect belief the reader might have: (“You might think that the command line tool would need to run as root (because it’s talking to the kernel, but…“)
3. Explain a common mistake (for example “Avoid Striding and Slicing in a Single Expression” in Effective Python)
   
4. Tell a story about a mistake you made and why it caused problems (here’s one of mine: Why Ruby’s Timeout is dangerous (and Thread.raise is terrifying))

Talking about mistakes is very important, just say up front that the thing is a mistake!

pattern 13: “what” without “why”

I very often see people introduce new technologies with a list of features instead of explaining why people choose the technology.

For example, the kubernetes homepage lists a bunch of Kubernetes’ features: automated rollouts and rollbacks, service discovery and load balancing, storage orchestration, secret and configuration management, automatic bin packing, etc. This kind of feature list is pretty common on a project homepage, but by itself it doesn’t help someone understand whether Kubernetes is right for them.

I think one reason writers leave out the “why” is that it’s hard to write a simple universal answer to “why do people use Kubernetes?”. There are a lot of reasons! And if you get the “why” wrong, it’s very noticeable and it feels embarrassing. So it feels safer to just list some features and move on.

But as a reader, I find that a weak “why” is much better than no “why”. I’d rather read “well, we use Kubernetes because it provides a decent basic deployment system and GKE means we don’t have to think about servers” than an attempt at covering every single company’s business reasons for using Kubernetes.

instead: talk about your reasons for using the technology

Of course, if you have a clear universal explanation of the problems a technology solves, that’s great. But I think a lot of the time authors (including me!!) just don’t have a great grasp of why other people are choosing a given technology. That’s okay!

If you don’t feel you can give a universal “why”, I think it’s better to just be open about that and give an example from your personal experience.

For example, I might say about Kubernetes:

The only problem I've solved with Kubernetes was: we had a distributed cron job system (Chronos) that wasn't working reliably (cron jobs would sometimes just not run), and we replaced the system with Kubernetes. Kubernetes' distributed cron was a lot more reliable.

This isn’t a good explanation of why people in general use Kubernetes! But I find reading many specific personal stories like this WAY more helpful than an attempt at cramming “here’s what’s Kubernetes is for” into a few paragraphs.

I want to be clear here that even just explaining your own personal experience isn’t that easy. Technology projects can be messy, and sometimes their goals change in the middle. I started out trying to give an example of why I’ve used Envoy and I realized I would need to think about it for several hours and have a few conversations with old coworkers to explain it coherently so I decided to use a different example.

that’s all for now!

Originally I thought it would be simple to put together these patterns (“there are so many confusing explanations!“) but it was surprisingly hard to articulate exactly what was confusing to me about each explanation in a convincing way.

It’s definitely incomplete, but I’ve already spent two weeks and 3000 words on it so I’ll stop here and I’d love to hear what I’ve missed :)

thanks to Laura, Dan, Kamal, Alyssa, Lindsey, Paul, Ivan, Edith, Hazem, Anton, and John for helping improve this post

translations

here is a translation:

• Korean