Dwayne.xyz

Reading List

The most recent articles from a list of feeds I subscribe to.

New talk: Making Hard Things Easy

Julia Evans

A few weeks ago I gave a keynote at Strange Loop called Making Hard Things Easy. It’s about why I think some things are hard to learn and ideas for how we can make them easier.

Here’s the video, as well as the slides and a transcript of (roughly) what I said in the talk.

the video

the transcript

Hello, Strange Loop! Strange Loop is one of the first places I spoke almost 10 years ago and I'm so honored to be back here today for the last one. Can we have one more round of applause for the organizers?

I often give talks about things that I'm excited about, or that I think are really fun.

But today, I want to talk about something that I'm a little bit mad about, which is that sometimes things that seem like they should be basic take me 10 years or 20 years to learn, way longer than it seems like they should.

One thing that took me a long time to learn was DNS, which is this question of -- what's the IP address for a domain name like example.com? This feels like it should be a straightforward thing.

But seven years into learning DNS, I'd be setting up a website. And I'd feel like things should be working. I thought I understood DNS. But then I'd run into problems, like my domain name wouldn't work. And I'd wonder -- why not? What's happening?

And sometimes this would feel kind of personal! This shouldn't be so hard for me! I should understand this already. It's been seven years!

And this "it's just me" attitude is often encouraged -- when I write about finding things hard to learn on the Internet, Internet strangers will sometimes tell me: "yeah, this is easy! You should get it already! Maybe you're just not very smart!"

But luckily I have a pretty big ego so I don't take the internet strangers too seriously. And I have a lot of patience so I'm willing to keep coming back to a topic I'm confused about. There were maybe four different things that were going wrong with DNS in my life and eventually I figured them all out.

So, hooray! I understood DNS! I win! But then I see some of my friends struggling with the exact same things.

They're wondering, hey, my DNS isn't working. Why not?

And it doesn't end. We're still having the same problems over and over and over again. And it's frustrating! It feels redundant! It makes me mad. Especially when friends take it personally, and they feel like "hey I should really understand this already".

Because everyone is going through this. From the sounds of recognition I hear, I think a lot of you have been through some of these same problems with DNS.

I got so mad about this that I decided to make it my job.

I started a little publishing company called Wizard Zines where --

(applause)

Wow. Where I write about some of these topics and try to demystify them.

Here are a few of the zines I've published. I want to talk today about a few of these topics and what makes them so hard and how we can make them easier.

We're going to talk about bash, HTTP, SQL, and DNS.

For each of them, we're going to talk a little bit about:

a. what's so hard about it?

b. what are some things we can do to make it a little bit easier for each other?

Let's start with Bash.

What's so hard about it?

So, bash is a programming language, right? But it's one of the weirdest programming languages that I work with.

To understand why it's weird, let's do a little small demo of bash.

First, let's run this script, bad.sh:

mv ./*.txt /tmmpp
echo "success!"

This moves a file and prints "success!". And with most of the programming languages that I use, if there's a problem, the program will stop.

[laughter from audience]

But I think a lot of you know from maybe sad experience that bash does not stop, right? It keeps going. And going... and sometimes very bad things happen to your computer in the process.

When I run this program, here's the output:

mv: cannot stat './*.txt': No such file or directory
success!

It didn't stop after the failed mv.

Eventually I learned that you can write set -e at the top of your program, and that will make bash stop if there's a problem.

When we run this new program with set -e at the top, here's the output:

mv: cannot stat './*.txt': No such file or directory

Great. We're happy. Everything is good. But every time I think I've learned everything that go wrong with bash, I'll find out -- surprise! There are more bad things that can happen! Let's look at another program as an example.

Here we've put our code in a function. And if the function fails, we want to echo "failed".

So use set -e at the beginning, and you might think everything should be okay.

But if we run it... this is the output we get

mv: cannot stat './*.txt': No such file or directory
success

We get the "success" message again! It didn't stop, it just kept going. This is because the "or" (|| echo "failed") globally disables set -e in the function.

Which is certainly not what I wanted, and not what I would expect. But this is not a bug in bash, it's is the documented behavior.

And I think one reason this is tricky is a lot of us don't use bash very often. Maybe you write a bash script every six months and don't look at it again.

When you use a system very infrequently and it's full of a lot of weird trivia and gotchas, it's hard to use the system correctly.

So how can we make this easier? What can we do about it?

One thing that I sometimes hear is -- a newcomer will say "this is hard", and someone more experienced will say "Oh, yeah, it's impossible to use bash. Nobody knows how to use it."

But I would say this is factually untrue. How many of you are using bash?

A lot of us ARE using it! And it doesn't always work perfectly, but often it gets the job done.

We have a lot of bash programs that are mostly working, and there's a big community of us who are using bash mostly successfully despite all the problems.

The way I think this is -- you have some people on the left in this diagram who are confused about bash, who think it seems awful and incomprehensible.

And some people on the right who know how to make the bash work for them, mostly.

So how do we move people from the left to the right, from being overwhelmed by a pile of impossible gotchas to being able to mostly use the system correctly?

Well, bash has a giant pile of trivia to remember. But who's good at remembering giant piles of trivia?

Not me! I can't memorize all of the weird things about bash. But computers! Computers are great at memorizing trivia!

And for bash, we have this incredible tool called shellcheck.

[ Applause ]

Yes! Shellcheck is amazing! And shellcheck knows a lot of things that can go wrong and can tell you "oh no, you don't want to do that. You're going to have a bad time."

I'm very grateful for shellcheck, it makes it much easier for me to write tiny bash scripts from time to time.

Now let's do a shellcheck demo!

$ shellcheck -o all bad-again.sh
In bad-again.sh line 7:
f || echo "failed!"
^-- SC2310 (info): This function is invoked in an || condition so set -e will be disabled. Invoke separately if failures should cause the script to exit.

Shellcheck gives us this lovely error message. The message isn't completely obvious on its own (and this check is only run if you invoke shellcheck with -o all). But shellcheck tells you "hey, there's this problem, maybe you should be worried about that".

And I think it's wonderful that all these tips live in this linter.

I'm not trying to tell you to write linters, though I think that some of you probably will write linters because this is that kind of crowd.

I've personally never written a linter, and I'm definitely not going to create something as cool as shellcheck!

But instead, the way I write linters is I tell people about shellcheck from time to time and then I feel a little like I invented shellcheck for those people. Because some people didn't know about the tool until I told them about it!

I didn't find out about shellcheck for a long time and I was kind of mad about it when I found out. I felt like -- excuse me? I could have been using shellcheck this whole time? I didn't need to remember all of this stuff in my brain?

So I think an incredible thing we can do is to reflect on the tools that we're using to reduce our cognitive load and all the things that we can't fit into our minds, and make sure our friends or coworkers know about them.

I also like to warn people about gotchas and some of the terrible things computers have done to me.

I think this is an incredibly valuable community service. The example I shared about how set -e got disabled is something I learned from my friend Jesse a few weeks ago.

They told me how this thing happened to them, and now I know and I don't have to go through it personally.

One way I see people kind of trying to share terrible things that their computers have done to them is by sharing "best practices".

But I really love to hear the stories behind the best practices!

If someone has a strong opinion like "nobody should ever use bash", I want to hear about the story! What did bash do to you? I need to know.

The reason I prefer stories to best practices is if I know the story about how the bash hurt you, I can take that information and decide for myself how I want to proceed.

Maybe I feel like -- the computer did that to you? That's okay, I can deal with that problem, I don't mind.

Or I might instead feel like "oh no, I'm going to do the best practice you recommended, because I do not want that thing to happen to me".

These bash stories are a great example of that: my reaction to them is "okay, I'm going to keep using bash, I'll just use shellcheck and keep my bash scripts pretty simple". But other people see them and decide "wow, I never want to use bash for anything, that's awful, I hate it".

Different people have different reactions to the same stories and that's okay.

That's all for bash. Next up we're gonna talk about HTTP.

I was talking to Marco Rogers at some point, many years ago, and he mentioned some new developers he was working with were struggling with HTTP.

And at first, I was a little confused about this -- I didn't understand what was hard about HTTP.

The way I was thinking about it at the time was that if you have an HTTP response, it has a few parts: a response code, some headers, and a body.

I felt like -- that's a pretty simple structure, what's the problem? But of course there was a problem, I just couldn't see what it was at first.

So, I talked to a friend who was newer to HTTP. And they asked "why does it matter what headers you set?"

And I said: "well, the browser..."

But then I thought... the browser?

the browser?

The browser!

Firefox is 20 million lines of code! It's been evolving since the '90s. There have been as I understand it, 1 million changes to the browser security model as people have discovered new and exciting exploits and the web has become a scarier and scarier place.

The browser is really a lot to understand.

One trick for understanding why a topic is hard is -- if the implementation if the thing involves 20 million lines of code, maybe that's why people are confused!

Though that 20 million lines of code also involves CSS and JS and many other things that aren't HTTP, but still.

Once I thought of it in terms of how complex a modern web browser is, it made so much more sense! Of course newcomers are confused about HTTP if you have to understand what the browser is doing!

Then my problem changed from "why is this hard?" to "how do I explain this at all?"

So how do we make it easier? How do we wrap our minds around this 20 million lines of code?

One way I think about this for HTTP is: here are some of the HTTP request headers. That's kind of a big list there are 43 headers there.

There are more unofficial headers too.

My brain does not contain all of those headers, I have no idea what most of them are.

When I think about trying to explain big topics, I think about -- what is actually in my brain, which only contains a normal human number of things?

This is a comic I drew about HTTP request headers. You don't have to read the whole thing. This has 15 request headers.

I wrote that these are "the most important headers", but what I mean by "most important" here is that these are the ones that I know about and use. It's a subjective list.

I wrote about 12 words about each one, which I think is approximately the amount of information about each header that lives in my mind.

For example I know that you can set Accept-Encoding to gzip and then you might get back a compressed response. That's all I know, and that's usually all I need to know!

This very small set of information is working pretty well for me.

The general way I think about this trick is "turn a big list into a small list".

Turn the set of EVERY SINGLE THING into just the things I've personally used. I find it helps a lot.

Another example of this "turn a big list into a small list" trick is command line arguments.

I use a lot of command line tools, the number of arguments they have can be overwhelming, and I've written about them a fair amount over the years.

Here are all the flags for grep, from its man page. That's too much! I've been using grep for 20 years but I don't know what all that stuff is.

But when I look at the grep man page, this is what I see.

I think it's very helpful to newcomers when a more experienced person says "look, I've been using this system for a while, I know about 7 things about it, and here's what they are".

We're just pruning those lists down to a more human scale. And it can even help other more experienced people -- often someone else will know a slightly different set of 7 things from me.

But what about the stuff that doesn't fit in my brain?

Because I have a few things about HTTP stored in my brain. But sometimes I need other information which is hard to remember, like maybe the exact details of how CORS works.

And so, that's where we come to references. Where do we find the information that we can't remember?

I often have trouble finding the right references.

For example I've been trying to learn CSS off and on for 20 years. I've made a lot of progress -- it's going well!

But only in the last 2 years or so I learned about this wonderful website called CSS Tricks.

And I felt kind of mad when I learned about CSS Tricks! Why didn't I know about this before? It would have helped me!

But anyway, I'm happy to know about CSS Tricks now. (though sadly they seem to have stopped publishing in April after the acquisition, I'm still happy the older posts are there)

For HTTP, I think a lot of us use the Mozilla Developer Network.

Another HTTP reference I love is the official RFC, RFC 9110 (also 9111, 9112, 9113, 9114)

It's a new authoritative reference for HTTP and it was written just last year, in 2022! They decided to organize all the information really nicely. So if you want to know exactly what the Connection header does, you can look it up.

This is not really my top reference. I'm usually on MDN. But I really appreciate that it's available.

So I love to share my favorite references.

I do sometimes find it tempting to kind of lie about references. Not on purpose. But I'll see something on the internet, and I'll think it's kind of cool, and tell a friend about. But then my friend might ask me -- "when have you used this?" And I'll have to admit "oh, never, I just thought it seemed cool".

I think it's important to be honest about what the references that I'm actually using in real life are. Even if maybe the real references I use are a little "embarrassing", like maybe w3schools or something.

So that's HTTP! Next we're going to talk about SQL.

The case of the mysterious execution order.

I started thinking about SQL because someone mentioned they're trying to learn SQL. I get most of my zine ideas that way, one person will make an offhand comment and I'll decide "ok, I'm going to spend 4 months writing about that". It's a weird process.

So I was wondering -- what's hard about SQL? What gets in the way of trying to learn that?

I want to say that when I'm confused about what's hard about something, that's a fact about me. It's not usually that the thing is easy, it's that I need to work on understanding what's hard about it. It's easy to forget when you've been using something for a while.

So, I was used to reading SQL queries. For example this made up query that tries to find people who own exactly two cats. It felt straightforward to me, SELECT, FROM, WHERE, GROUP BY.

But then I was talking to a friend about these queries who was new to SQL. And my friend asked -- what is this doing?

I thought, hmm, fair point.

And I think the point my friend was making was that the order that this SQL query is written in, is not the order that it actually happens in. It happens in a different order, and it's not immediately obvious what that is.

So how do we make this easier?

I like to think about: what does the computer do first? What actually happens first chronologically?

Computers actually do live in the same timeline as us. Things happen. Things happen in an order. So what happens first?

The way I think about an SQL query is: is you start with a table like cats.

Then maybe you filter it, you remove some stuff.

Then you make some groups.

Then you filter the groups, remove some of them.

Then you do some aggregation. There's two things in each group.

And you sort it. And you can also limit the results.

So, that's how I think about SQL. The way a query runs is first FROM, then WHERE, GROUP BY, HAVING, SELECT, ORDER BY, LIMIT.

At least conceptually. Real life databases have optimizations and it's more complicated than that. But this is the mental model that I use most of the time and it works for me. Everything is in the same order as you write it, except SELECT is fifth.

I've really gotten a lot out of this trick where you try to tell the chronological story of what the computer is doing. I want to talk about a couple other examples.

One is CORS, in HTTP.

This comic is way too small to read on the slide. But the idea is if you're making a cross-origin request in your browser, you can write down every communication that's happening between your browser and the server, in chronological order.

And I think writing down everything in chronological order makes it a lot easier to understand and more concrete.

"What happens in chronological order?" is a very straightforward structure, which is what I like about it. "What happens first?" feels like it should be easy to answer. But it's not!

I've found that it's actually very hard to know what our computers is doing, and it's a really fun question to explore.

As an example of how this is hard: I wrote a blog post recently called "Behind Hello World on Linux". It's about what happens when you run "hello world" on a Linux computer. I wrote a bunch about it, and I was really happy with it.

But after I wrote the post, I thought -- haven't I written about this before? Maybe 10 years ago?

And sure enough, I'd tried to write a similar post 10 years before.

I think this is really cool. Because the 2013 version of this post was about 6 times shorter. This isn't because Linux is more complicated than it was 10 years ago -- I think everything in the 2023 post was probably also true in 2013. The 2013 post just has a lot less information in it.

The reason the 2023 post is longer is that I didn't know what was happening chronologically on my computer in 2013 very well, and in 2023 I know a lot more. Maybe in 2033 I'll know even more!

I think a lot of us -- like me in 2013 and honestly me now, often don't know the facts of what's happening on our computers. It's very hard, which is what makes it such a fun question to try and discuss.

I think it's cool that all of us have different knowledge about what is happening chronologically on our computers and we can all chip in to this conversation.

For example when I posted this blog post about Hello World on Linux, some people mentioned that they had a lot of thoughts about what happens exactly in your terminal, or more details about the filesystem, or about what's happening internally in the Python interpreter, or any number of things. You can go really deep.

I think it's just a really fun collaborative question.

I've seen "what happens chronologically?" work really well as an activity with coworkers, where you're ask: "when a request comes into this API endpoint we run, how does that work? What happens?"

What I've seen is that someone will understand some part of the system, like "X happens, then Y happens, then it goes over to the database and I have no idea how that works". And then someone else can chime in and say "ah, yes, with the database A B C happens, but then there's a queue and I don't know about that".

I think it's really fun to get together with people who have different specializations and try to make these little timelines of what the computers are doing. I've learned a lot from doing that with people.

That's all for SQL.

So, now we've arrived at DNS which is where we started the talk.

Even though I struggled with DNS. Once I got figured it out, I felt like "dude, this is easy!". Even though it just took me 10 years to learn how it works.

But of course, DNS was pretty hard for me to learn. So -- why is that? Why did it take me so long?

So, I have a little chart here of how I think about DNS.

You have your browser on the left. And over on the right there's the authoritative nameservers, the source of truth of where the DNS records for a domain live.

In the middle, there's a function that you call and a cache. So you have browser, function, cache, source of truth.

One problem is that there are a lot of things in this diagram that are totally hidden from you.

The library code that you're using where you make a DNS request -- there are a lot of different libraries you could be using, and it's not straightforward to figure out which one is being used. That was the source of some of my confusion.

There's a cache which has a bunch of cached data. That's invisible to you, you can't inspect it easily and you have no control over it. that

And there's a conversation between the cache and the source of truth, these two red arrows which also you can't see at all.

So this is kind of tough! How are you supposed to develop an intuition for a system when it's mostly things that are completely hidden from you? Feels like a lot to expect.

So, what do we do about this?

So: let's talk about these red arrows on the right.

We have our cache and then we have the source of truth. This conversation is normally hidden from you because you often don't control either of these servers. Usually they're too busy doing high-performance computing to report to you what they're doing.

But I thought: anyone can write an authoritative nameserver! In particular, I could write one that reports back every single message that it receives to its users. So, with my friend Marie, we wrote a little DNS server.

(demo of messwithdns.net)

This is called Mess With DNS. The idea is I have a domain name and you can do whatever you want with it. We're going to make a DNS record called strangeloop, and we're going to make a CNAME record pointing at orange.jvns.ca, which is just a picture of an orange. Because I like oranges.

And then over here, every time a request comes in from a resolver, this will -- this will report back what happened. So, if we click on this link, we can see -- a Canadian DNS resolver, which is apparently what my browser is configured to use, is requesting an IPv4 record and an IPv6 record, A and AAAA.

(at this point in the demo everyone in the audience starts visiting the link and it gets a bit chaotic, it's very funny)

So the trick here is to find ways to show people parts of what the computer is doing that are normally hidden.

Another great example of showing things that are hidden is this website called float.exposed by Bartosz Ciechanowski who makes a lot of incredible visualizations.

So if you look at this 32-bit floating point number and click the "up" button on the significand, it'll show you the next floating point number, which is 2 more. And then as you make the number bigger and bigger (by increasing the exponent), you can see that the floating point numbers get further and further apart.

Anyway, this is not a talk about floating point. I could do an entire talk about this site and how we can use it to see how floating point works, but that's not this talk.

Another thing that makes DNS confusing is that it's a giant distributed system -- maybe you're confused because there are 5 million computers involved (really, more!). Most of which you have no control over, and some are doing not what they're supposed to do.

So that's another trick for understanding why things are hard, check to see if there are actually 5 million computers involved.

So what else is hard about DNS?

We've talked about how most of the system is hidden from you, and about how it's a big distributed system.

One problem I've run into is that the tools are confusing.

One of the hidden things I talked about was: the resolver has cached data, right? And you might be curious about whether a certain domain name is cached or not by your resolver right now.

Just to understand what's happening: am I getting this result because it was cached? What's the deal?

I said this was hidden, but there are a couple of ways to query a resolver to see what it has cached, and I want to show you one of them.

The tool I usually use for making DNS queries is called dig, and it has a flag called +norecurse. You can use it to query a resolver and ask it to only return results it already has cached.

With dig +norecurse jvns.ca, I'm kind of asking -- how popular is my website? Is it popular enough that someone has visited it in the last 5 minutes? Because my records are not cached for that long, only for 5 minutes.

But when I look at this response, I feel like "please! What is all this?"

And when I show newcomers this output, they often respond by saying "wow, that's complicated, this DNS thing must be really complicated". But really this is just not a great output format, I think someone just made some relatively arbitrary choices about how to print this stuff out in the 90s and it's stayed that way ever since.

So a bad output format can mislead newcomers into thinking that something is more complicated than it actually is.

What can we do about confusing output like this?

One of my favorite tricks, I call eraser eyes.

Because when I look at that output, I'm not looking at all of it, I'm just looking at a few things. My eyes are ignoring the rest of it.

When I look at the output, this is what I see: it says SERVFAIL. That's the DNS response code.

Which as I understand it is a very unintuitive way of it saying, "I do not have that in my cache". So nobody has asked that resolver about my domain name in the last 5 minutes, which isn't very surprising.

I've learned so much from people doing a little demo of a tool, and showing how they use it and which parts of the output or UI they pay attention to, and which parts they ignore.

Becuase usually we ignore most of what's on our screens!

I really love to use dig even though it's a little hairy because it has a lot of features (I don't know of another DNS debugging that supports this +norecurse trick), it's everywhere, and it hasn't changed in a long time. And I know if I learn its weird output format once I can know that forever. Stability is really valuable to me.

So we've talked about these four technologies. Let's talk a little more about how we can make things easier for each other.

What can we do to move folks from "I really don't get it" to "okay, I can mostly deal with this, at least 90% of the time, it's fine"? For bash or HTTP or DNS or anything else.

We've talked about some tricks I use to bring people over, like:

sharing useful tools
sharing references
telling a chronological story of what happens on your computer
turning a big list into a small list of the things you actually use
showing the hidden things
demoing a confusing tool and telling folks which parts I pay attention to

When I practiced this talk, I got some feedback from people saying "julia! I don't do those things! I don't have a blog, and I'm not going to start one!"

And it's true that most people are probably not going to start programming blogs.

But I really don't think you need to have a public presence on the internet to tell the people around you a little bit about how you use computers and how you understand them.

My experience is that a lot of people (who do not have blogs!) have helped me understand how computers work and have shared little pieces of their experience with computers with me.

I've learned a lot from my friends and my coworkers and honestly a lot of random strangers on the Internet too. I'm pretty sure some of you here today have helped me over the years, maybe on Twitter or Mastodon.

So I want to talk about some archetypes of helpful people

One kind of person who has really helped me is the grumpy old-timer. I'll say "this is so cool". And they'll reply yes, however, let me tell you some stories of how this has gone wrong in my life.

And those stories have sometimes helped spare me some suffering.

We have the loud newbie, who asks questions like "wait, how does that work?" And then everyone else feels relieved -- "oh, thank god. It's not just me."

I think it's especially valuable when the person who takes the "loud newbie" role is actually a pretty senior developer. Because when you're more secure in your position, it's easier to put yourself out there and say "uh, I don't get this" because nobody is going to judge you for that and think you're incompetent.

And then other people who feel more like they might be judged for not knowing something can ride along on your coattails.

Then we have the bug chronicler. Who decides "ok, that bug. This can never happen again".

"I'm gonna make sure we understand what happened. Because I want this to end now."

We have the tool builder, whose attitude is more like "I see people struggling with something, and I don't feel like explaining it. But I can write code to just make it easier permanently for everyone."

There's this "today I learned" person who's into sharing cool new tools they learned about, a bug that they ran into, or a great new-to-them library feature.

There's the person who has read the entire Internet and has 700 tabs open. If you want to know where to find something, there's a good chance they already have it open in their browser.

We have the person who is just willing to answer questions! "Yeah, I can tell you how that works!"

And at the end of all this, sometimes you have someone who likes to write some things down so that other people can read it and can find it later.

But all of us have different roles and we need to work together. I'm into writing but a lof of the stuff I've written about, I only know about because someone told me about it or explained it to me.

To end: the one thing I would like to convince you of is: if you're struggling with something that feels basic, it's not just you! You're not alone. We're all struggling with a lot of these things that feel like they should be "basic".

And we're struggling with these things for a lot of the same reasons as each other.

And much like when debugging a computer program, when you have a bug, you want to understand why the bug is happening if you're gonna fix it.

If we're all struggling with the same things together for the same reasons, if we can figure out what those reasons are, we can do a better job of fixing them.

Some of the reasons we've talked about were:

a giant pile of trivia and gotchas.
or maybe there's 20 million lines of code somewhere.
Maybe a big part of the system is being hidden from you.
Maybe the tool's output is extremely confusing and no UI designer has ever worked on improving it

And there are a lot more reasons.

I don't have all the answers for why things are hard. For example I don't really understand why Git is hard, that's something I've been thinking about recently.

But that's something I'm excited to keep working on and keep trying to figure out.

And that's all I have for you. Thank you.

I brought some zines to the conference, if you come to the signing later on you can get one.

some thanks

This was the last ever Strange Loop and I’m really grateful to Alex Miller and the whole organizing team for making such an incredible conference for so many years. Strange Loop accepted one of my first talks (you can be a kernel hacker) 9 years ago when I had almost no track record as a speaker so I owe a lot to them.

Thanks to Sumana for coming up with the idea for this talk, and to Marie, Danie, Kamal, Alyssa, and Maya for listening to rough drafts of it and helping make it better, and to Dolly, Jesse, and Marco for some of the conversations I mentioned.

Also after the conference Nick Fagerland wrote a nice post with thoughts on why git is hard in response to my “I don’t know why git is hard” comment and I really appreciated it. It had some new-to-me ideas and I’d love to read more analyses like that.

blogs
tech

In a git repository, where do your files live?

Julia Evans

Hello! I was talking to a friend about how git works today, and we got onto the topic – where does git store your files? We know that it’s in your .git directory, but where exactly in there are all the versions of your old files?

For example, this blog is in a git repository, and it contains a file called content/post/2019-06-28-brag-doc.markdown. Where is that in my .git folder? And where are the old versions of that file? Let’s investigate by writing some very short Python programs.

git stores files in `.git/objects`

Every previous version of every file in your repository is in .git/objects. For example, for this blog, .git/objects contains 2700 files.

$ find .git/objects/ -type f | wc -l
2761

note: .git/objects actually has more information than “every previous version of every file in your repository”, but we’re not going to get into that just yet

Here’s a very short Python program (find-git-object.py) that finds out where any given file is stored in .git/objects.

import hashlib
import sys


def object_path(content):
    header = f"blob {len(content)}\0"
    data = header.encode() + content
    digest = hashlib.sha1(data).hexdigest()
    return f".git/objects/{digest[:2]}/{digest[2:]}"


with open(sys.argv[1], "rb") as f:
    print(object_path(f.read()))

What this does is:

read the contents of the file
calculate a header (blob 16673\0) and combine it with the contents
calculate the sha1 sum (e33121a9af82dd99d6d706d037204251d41d54 in this case)
translate that sha1 sum into a path (.git/objects/e3/3121a9af82dd99d6d706d037204251d41d54)

We can run it like this:

$ python3 find-git-object.py content/post/2019-06-28-brag-doc.markdown
.git/objects/8a/e33121a9af82dd99d6d706d037204251d41d54

jargon: “content addressed storage”

The term for this storage strategy (where the filename of an object in the database is the same as the hash of the file’s contents) is “content addressed storage”.

One neat thing about content addressed storage is that if I have two files (or 50 files!) with the exact same contents, that doesn’t take up any extra space in Git’s database – if the hash of the contents is aabbbbbbbbbbbbbbbbbbbbbbbbb, they’ll both be stored in .git/objects/aa/bbbbbbbbbbbbbbbbbbbbb.

how are those objects encoded?

If I try to look at this file in .git/objects, it gets a bit weird:

$ cat .git/objects/8a/e33121a9af82dd99d6d706d037204251d41d54
x^A<8D><9B>}s<E3>Ƒ<C6><EF>o|<8A>^Q<9D><EC>ju<92><E8><DD>\<9C><9C>*<89>j<FD>^...

What’s going on? Let’s run file on it:

$ file .git/objects/8a/e33121a9af82dd99d6d706d037204251d41d54
.git/objects/8a/e33121a9af82dd99d6d706d037204251d41d54: zlib compressed data

It’s just compressed! We can write another little Python program called decompress.py that uses the zlib module to decompress the data:

import zlib
import sys

with open(sys.argv[1], "rb") as f:
    content = f.read()
    print(zlib.decompress(content).decode())

Now let’s decompress it:

$ python3 decompress.py .git/objects/8a/e33121a9af82dd99d6d706d037204251d41d54 
blob 16673---
title: "Get your work recognized: write a brag document"
date: 2019-06-28T18:46:02Z
url: /blog/brag-documents/
categories: []
---
... the entire blog post ...

So this data is encoded in a pretty simple way: there’s this blob 16673\0 thing, and then the full contents of the file.

there aren’t any diffs

One thing that surprised me here is the first time I learned it: there aren’t any diffs here! That file is the 9th version of that blog post, but the version git stores in the .git/objects is the whole file, not the diff from the previous version.

Git actually sometimes also does store files as diffs (when you run git gc it can combine multiple different files into a “packfile” for efficiency), but I have never needed to think about that in my life so we’re not going to get into it. Aditya Mukerjee has a great post called Unpacking Git packfiles about how the format works.

what about older versions of the blog post?

Now you might be wondering – if there are 8 previous versions of that blog post (before I fixed some typos), where are they in the .git/objects directory? How do we find them?

First, let’s find every commit where that file changed with git log:

$ git log --oneline  content/post/2019-06-28-brag-doc.markdown
c6d4db2d
423cd76a
7e91d7d0
f105905a
b6d23643
998a46dd
67a26b04
d9999f17
026c0f52
72442b67

Now let’s pick a previous commit, let’s say 026c0f52. Commits are also stored in .git/objects, and we can try to look at it there. But the commit isn’t there! ls .git/objects/02/6c* doesn’t have any results! You know how we mentioned “sometimes git packs objects to save space but we don’t need to worry about it?“. I guess now is the time that we need to worry about it.

So let’s take care of that.

let’s unpack some objects

So we need to unpack the objects from the pack files. I looked it up on Stack Overflow and apparently you can do it like this:

$ mv .git/objects/pack/pack-adeb3c14576443e593a3161e7e1b202faba73f54.pack .
$ git unpack-objects < pack-adeb3c14576443e593a3161e7e1b202faba73f54.pack

This is weird repository surgery so it’s a bit alarming but I can always just clone the repository from Github again if I mess it up, so I wasn’t too worried.

After unpacking all the object files, we end up with way more objects: about 20000 instead of about 2700. Neat.

find .git/objects/ -type f | wc -l
20138

back to looking at a commit

Now we can go back to looking at our commit 026c0f52. You know how we said that not everything in .git/objects is a file? Some of them are commits! And to figure out where the old version of our post content/post/2019-06-28-brag-doc.markdown is stored, we need to dig pretty deep into this commit.

The first step is to look at the commit in .git/objects.

commit step 1: look at the commit

The commit 026c0f52 is now in .git/objects/02/6c0f5208c5ea10608afc9252c4a56c1ac1d7e4 after doing some unpacking and we can look at it like this:

$ python3 decompress.py .git/objects/02/6c0f5208c5ea10608afc9252c4a56c1ac1d7e4
commit 211tree 01832a9109ab738dac78ee4e95024c74b9b71c27
parent 72442b67590ae1fcbfe05883a351d822454e3826
author Julia Evans <julia@jvns.ca> 1561998673 -0400
committer Julia Evans <julia@jvns.ca> 1561998673 -0400

brag doc

We can also get same information with git cat-file -p 026c0f52, which does the same thing but does a better job of formatting the data. (the -p option means “format it nicely please”)

commit step 2: look at the tree

This commit has a tree. What’s that? Well let’s take a look. The tree’s ID is 01832a9109ab738dac78ee4e95024c74b9b71c27, and we can use our decompress.py script from earlier to look at that git object. (though I had to remove the .decode() to get the script to not crash)

$ python3 decompress.py .git/objects/01/832a9109ab738dac78ee4e95024c74b9b71c27
b'tree 396\x00100644 .gitignore\x00\xc3\xf7`$8\x9b\x8dO\x19/\x18\xb7}|\xc7\xce\x8e:h\xad100644 README.md\x00~\xba\xec\xb3\x11\xa0^\x1c\xa9\xa4?\x1e\xb9\x0f\x1cfG\x96\x0b

This is formatted in kind of an unreadable way. The main display issue here is that the commit hashes (\xc3\xf7$8\x9b\x8dO\x19/\x18\xb7}|\xc7\xce\…) are raw bytes instead of being encoded in hexadecimal. So we see \xc3\xf7$8\x9b\x8d instead of c3f76024389b8d. Let’s switch over to using git cat-file -p which formats the data in a friendlier way, because I don’t feel like writing a parser for that.

$ git cat-file -p 01832a9109ab738dac78ee4e95024c74b9b71c27
100644 blob c3f76024389b8d4f192f18b77d7cc7ce8e3a68ad	.gitignore
100644 blob 7ebaecb311a05e1ca9a43f1eb90f1c6647960bc1	README.md
100644 blob 0f21dc9bf1a73afc89634bac586271384e24b2c9	Rakefile
100644 blob 00b9d54abd71119737d33ee5d29d81ebdcea5a37	config.yaml
040000 tree 61ad34108a327a163cdd66fa1a86342dcef4518e	content <-- this is where we're going next
040000 tree 6d8543e9eeba67748ded7b5f88b781016200db6f	layouts
100644 blob 22a321a88157293c81e4ddcfef4844c6c698c26f	mystery.rb
040000 tree 8157dc84a37fca4cb13e1257f37a7dd35cfe391e	scripts
040000 tree 84fe9c4cb9cef83e78e90a7fbf33a9a799d7be60	static
040000 tree 34fd3aa2625ba784bced4a95db6154806ae1d9ee	themes

This is showing us all of the files I had in the root directory of the repository as of that commit. Looks like I accidentally committed some file called mystery.rb at some point which I later removed.

Our file is in the content directory, so let’s look at that tree: 61ad34108a327a163cdd66fa1a86342dcef4518e

commit step 3: yet another tree

$ git cat-file -p 61ad34108a327a163cdd66fa1a86342dcef4518e

040000 tree 1168078878f9d500ea4e7462a9cd29cbdf4f9a56	about
100644 blob e06d03f28d58982a5b8282a61c4d3cd5ca793005	newsletter.markdown
040000 tree 1f94b8103ca9b6714614614ed79254feb1d9676c	post <-- where we're going next!
100644 blob 2d7d22581e64ef9077455d834d18c209a8f05302	profiler-project.markdown
040000 tree 06bd3cee1ed46cf403d9d5a201232af5697527bb	projects
040000 tree 65e9357973f0cc60bedaa511489a9c2eeab73c29	talks
040000 tree 8a9d561d536b955209def58f5255fc7fe9523efd	zines

Still not done…

commit step 4: one more tree….

The file we’re looking for is in the post/ directory, so there’s one more tree:

$ git cat-file -p 1f94b8103ca9b6714614614ed79254feb1d9676c	
.... MANY MANY lines omitted ...
100644 blob 170da7b0e607c4fd6fb4e921d76307397ab89c1e	2019-02-17-organizing-this-blog-into-categories.markdown
100644 blob 7d4f27e9804e3dc80ab3a3912b4f1c890c4d2432	2019-03-15-new-zine--bite-size-networking-.markdown
100644 blob 0d1b9fbc7896e47da6166e9386347f9ff58856aa	2019-03-26-what-are-monoidal-categories.markdown
100644 blob d6949755c3dadbc6fcbdd20cc0d919809d754e56	2019-06-23-a-few-debugging-resources.markdown
100644 blob 3105bdd067f7db16436d2ea85463755c8a772046	2019-06-28-brag-doc.markdown <-- found it!!!!!

Here the 2019-06-28-brag-doc.markdown is the last file listed because it was the most recent blog post when it was published.

commit step 5: we made it!

Finally we have found the object file where a previous version of my blog post lives! Hooray! It has the hash 3105bdd067f7db16436d2ea85463755c8a772046, so it’s in git/objects/31/05bdd067f7db16436d2ea85463755c8a772046.

We can look at it with decompress.py

$ python3 decompress.py .git/objects/31/05bdd067f7db16436d2ea85463755c8a772046 | head
blob 15924---
title: "Get your work recognized: write a brag document"
date: 2019-06-28T18:46:02Z
url: /blog/brag-documents/
categories: []
---
... rest of the contents of the file here ...

This is the old version of the post! If I ran git checkout 026c0f52 content/post/2019-06-28-brag-doc.markdown or git restore --source 026c0f52 content/post/2019-06-28-brag-doc.markdown, that’s what I’d get.

this tree traversal is how `git log` works

This whole process we just went through (find the commit, go through the various directory trees, search for the filename we wanted) seems kind of long and complicated but this is actually what’s happening behind the scenes when we run git log content/post/2019-06-28-brag-doc.markdown. It needs to go through every single commit in your history, check the version (for example 3105bdd067f7db16436d2ea85463755c8a772046 in this case) of content/post/2019-06-28-brag-doc.markdown, and see if it changed from the previous commit.

That’s why git log FILENAME is a little slow sometimes – I have 3000 commits in this repository and it needs to do a bunch of work for every single commit to figure out if the file changed in that commit or not.

how many previous versions of files do I have?

Right now I have 1530 files tracked in my blog repository:

$ git ls-files | wc -l
1530

But how many historical files are there? We can list everything in .git/objects to see how many object files there are:

$ find .git/objects/ -type f | grep -v pack | awk -F/ '{print $3 $4}' | wc -l
20135

Not all of these represent previous versions of files though – as we saw before, lots of them are commits and directory trees. But we can write another little Python script called find-blobs.py that goes through all of the objects and checks if it starts with blob or not:

import zlib
import sys

for line in sys.stdin:
    line = line.strip()
    filename = f".git/objects/{line[0:2]}/{line[2:]}"
    with open(filename, "rb") as f:
        contents = zlib.decompress(f.read())
        if contents.startswith(b"blob"):
            print(line)

$ find .git/objects/ -type f | grep -v pack | awk -F/ '{print $3 $4}' | python3 find-blobs.py | wc -l
6713

So it looks like there are 6713 - 1530 = 5183 old versions of files lying around in my git repository that git is keeping around for me in case I ever want to get them back. How nice!

that’s all!

Here’s the gist with all the code for this post. There’s not very much.

I thought I already knew how git worked, but I’d never really thought about pack files before so this was a fun exploration. I also don’t spend too much time thinking about how much work git log is actually doing when I ask it to track the history of a file, so that was fun to dig into.

As a funny postscript: as soon as I committed this blog post, git got mad about how many objects I had in my repository (I guess 20,000 is too many!) and ran git gc to compress them all into packfiles. So now my .git/objects directory is very small:

$ find .git/objects/ -type f | wc -l
14

tech
blogs

Notes on using a single-person Mastodon server

Julia Evans

I started using Mastodon back in November, and it’s the Twitter alternative where I’ve been spending most of my time recently, mostly because the Fediverse is where a lot of the Linux nerds seem to be right now.

I’ve found Mastodon quite a bit more confusing than Twitter because it’s a distributed system, so here are a few technical things I’ve learned about it over the last 10 months. I’ll mostly talk about what using a single-person server has been like for me, as well as a couple of notes about the API, DMs and ActivityPub.

I might have made some mistakes, please let me know if I’ve gotten anything wrong!

what’s a mastodon instance?

First: Mastodon is a decentralized collection of independently run servers instead of One Big Server. The software is open source.

In general, if you have an account on one server (like ruby.social), you can follow people on another server (like hachyderm.io), and they can follow you.

I’m going to use the terms “Mastodon server” and “Mastodon instance” interchangeably in this post.

on choosing a Mastodon instance

These were the things I was concerned about when choosing an instance:

An instance name that I was comfortable being part of my online identity. For example, I probably wouldn’t want to be @b0rk@infosec.exchange because I’m not an infosec person.
The server’s stability. Most servers are volunteer-run, and volunteer moderation work can be exhausting – will the server really be around in a few years? For example mastodon.technology and mastodon.lol shut down.
The admins’ moderation policies.
That server’s general reputation with other servers. I started out on mastodon.social, but some servers choose to block or limit mastodon.social for various reasons
The community: every Mastodon instance has a local timeline with all posts from users on that instance, would I be interested in reading the local timeline?
Whether my account would be a burden for the admin of that server (since I have a lot of followers)

In the end, I chose to run my own mastodon server because it seemed simplest – I could pick a domain I liked, and I knew I’d definitely agree with the moderation decisions because I’d be in charge.

I’m not going to give server recommendations here, but here’s a list of the top 200 most common servers people who follow me use.

using your own domain

One big thing I wondered was – can I use my own domain (and have the username @b0rk@jvns.ca or something) but be on someone else’s Mastodon server?

The answer to this seems to be basically “no”: if you want to use your own domain on Mastodon, you need to run your own server. (you can kind of do this, but it’s more like an alias or redirect – if I used that method to direct b0rk@jvns.ca to b0rk@mastodon.social, my posts would still show up as being from b0rk@mastodon.social)

There’s also other ActivityPub software (Takahē) that supports people bringing their own domain in a first-class way.

notes on having my own server

I really wanted to have a way to use my own domain name for identity, but to share server hosting costs with other people. This isn’t possible on Mastodon right now, so I decided to set up my own server instead.

I chose to run a Mastodon server (instead of some other ActivityPub implementation) because Mastodon is the most popular one. Good managed Mastodon hosting is readily available, there are tons of options for client apps, and I know for sure that my server will work well with other people’s servers.

I use masto.host for Mastodon hosting, and it’s been great so far. I have nothing interesting to say about what it’s like to operate a Mastodon instance because I know literally nothing about it. Masto.host handles all of the server administration and Mastodon updates, and I never think about it at all.

Right now I’m on their $19/month (“Star”) plan, but it’s possible I could use a smaller plan with no problems. Right now their cheapest plan is $6/month and I expect that would be fine for someone with a smaller account.

Some things I was worried about when embarking on my own Mastodon server:

I wanted to run the server at social.jvns.ca, but I wanted my username to be b0rk@jvns.ca instead of b0rk@social.jvns.ca. To get this to work I followed these Setting up a personal fediverse ID directions from Jacob Kaplan-Moss and it’s been fine.
The administration burden of running my own server. I imported a small list of servers to block/defederate from but didn’t do anything else. That’s been fine.
Reply and profile visibility. This has been annoying and we’ll talk about it next

downsides to being on a single-person server

Being on a 1-person server has some significant downsides. To understand why, you need to understand a little about how Mastodon works.

Every Mastodon server has a database of posts. Servers only have posts that they were explicitly sent by another server in their database.

Some reasons that servers might receive posts:

someone on the server follows a user
a post mentions someone on the server

As a 1-person server, my server does not receive that many posts! I only get posts from people I follow or posts that explicitly mention me in some way.

The causes several problems:

when I visit someone’s profile on Mastodon who I don’t already follow, my server will not fetch the profile’s content (it’ll fetch their profile picture, description, and pinned posts, but not any of their post history). So their profile appears as if they’ve never posted anything
bad reply visibility: when I look at the replies to somebody else’s post (even if I follow them!), I don’t see all of the replies, only the ones which have made it to my server. If you want to understand the exact rules about who can see which replies (which are quite complicated!), here’s a great deep dive by Sebastian Jambor. I think it’s possible to end up in a state where no one person can see all of the replies, including the original poster.
favourite and boost accounts are inaccurate – usually posts show up having at most 1 or 2 favourites / boosts, even if the post was actually favourite or boosted hundreds of times. I think this is because it only counts favourites/boosts from people I follow.

All of these things will happen to users of any small Mastodon server, not just 1-person servers.

bad reply visibility makes conversations harder

A lot of people are on smaller servers, so when they’re participating in a conversation, they can’t see all the replies to the post.

This means that replies can get pretty repetitive because people literally cannot see each other’s replies. This is especially annoying for posts that are popular or controversial, because the person who made the post has to keep reading similar replies over and over again by people who think they’re making the point for the first time.

To get around this (as a reader), you can click “open link to post” or something in your Mastodon client, which will open up the page on the poster’s server where you can read all of the replies. It’s pretty annoying though.

As a poster, I’ve tried to reduce repetitiveness in replies by:

putting requests in my posts like “(no need to reply if you don’t remember, or if you’ve been using the command line comfortably for 15 years — this question isn’t for you :) )”
occasionally editing my posts to include very common replies
very occasionally deleting the post if it gets too out of hand

The Mastodon devs are extremely aware of these issues, there are a bunch of github issues about them:

My guess is that there are technical reasons these features are difficult to add because those issues have been open for 5-7 years.

The Mastodon devs have said that they plan to improve reply fetching, but that it requires a significant amount of work.

some visibility workarounds

Some people have built workarounds for fetching profiles / replies.

Also, there are a couple of Mastodon clients which will proactively fetch replies. For iOS:

Mammoth does it automatically
Mona will fetch posts if I click “load from remote server” manually

I haven’t tried those yet though.

other downsides of running your own server: discovery is much harder

Mastodon instances have a “local timeline” where you can see everything other people on the server are posting, and a “federated timeline” which shows sort of a combined feed from everyone followed by anyone on the server. This means that you can see trending posts and get an idea of what’s going on and find people to follow. You don’t get that if you’re on a 1-person server – it’s just me talking to myself! (plus occasional interjections from my reruns bot).

Some workarounds people mentioned for this:

you can populate your federated timeline with posts from another instance by using a relay. I haven’t done this but someone else said they use FediBuzz and I might try it out.
some mastodon clients (like apparently Moshidon on Android) let you follow other instances

If anyone else on small servers has suggestions for how to make discovery easier I’d love to hear them.

account migration

When I moved to my own server from mastodon.social, I needed to run an account migration to move over my followers. First, here’s how migration works:

Account migration does not move over your posts. All of my posts stayed on my old account. This is part of why I moved to running my own server – I didn’t want to ever lose my posts a second time.
Account migration does not move over the list of people you follow/mute/block. But you can import/export that list in your Mastodon settings so it’s not a big deal. If you follow private accounts they’ll have to re-approve your follow request.
Account migration does move over your followers

The follower move was the part I was most worried about. Here’s how it turned out:

over ~24 hours, most of my followers moved to the new account
one or two servers did not get the message about the account migration for some reason, so about 2000 followers were “stuck” and didn’t migrate. I fixed this by waiting 30 days and re-running the account migration, which moved over most of the remaining followers. There’s also a tootctl command that the admin of the old instance can run to retry the migration
about 200 of my followers never migrated over, I think because they’re using ActivityPub software other than Mastodon which doesn’t support account migration. You can see the old account here

using the Mastodon API is great

One thing I love about Mastodon is – it has an API that’s MUCH easier to use than Twitter’s API. I’ve always been frustrated with how difficult it is to navigate large Twitter threads, so I made a small mastodon thread view website that lets you log into your Mastodon account. It’s pretty janky and it’s really only made for me to use, but I’ve really appreciated the ability to write my own janky software to improve my Mastodon experience.

Some notes on the Mastodon API:

You can build Mastodon client software totally on the frontend in Javascript, which is really cool.
I couldn’t find a vanilla Javascript Mastodon client, so I wrote a crappy one
API docs are here
Here’s a tiny Python script I used to list all my Mastodon followers, which also serves as a simple example of how easy using the API is.
The best documentation I could find for which OAuth scopes correspond to which API endpoints is this github issue

Next I’ll talk about a few general things about Mastodon that confused or surprised me that aren’t specific to being on a single-person instance.

DMs are weird

The way Mastodon DMs work surprised me in a few ways:

Technically DMs are just regular posts with visibility limited to the people mentioned in the post. This means that if you accidentally mention someone in a DM (“@x is such a jerk”), it’s possible to accidentally send the message to them
DMs aren’t very private: the admins on the sending and receiving servers can technically read your DMs if they have access to the database. So they’re not appropriate for sensitive information.
Turning off DMs is weird. Personally I don’t like receiving DMs from strangers – it’s too much to keep track of and I’d prefer that people email me. On Twitter, I can just turn it off and people won’t see an option to DM me. But on Mastodon, when I turn off notifications for DMs, anyone can still “DM” me, but the message will go into a black hole and I’ll never see it. I put a note in my profile about this.

defederation and limiting

There are a couple of different ways for a server to block another Mastodon server. I haven’t really had to do this much but people talk about it a lot and I was confused about the difference, so:

A server can defederate from another server (this seems to be called suspend in the Mastodon docs). This means that nobody on a server can follow someone from the other server.
A server can limit (also known as “silence”) a user or server. This means that content from that user is only visible to that user’s followers – people can’t discover the user through retweets (aka “boosts” on Mastodon).

One thing that wasn’t obvious to me is that who servers defederate / limit is sometimes hidden, so it’s hard to suss out what’s going on if you’re considering joining a server, or trying to understand why you can’t see certain posts.

there’s no search for posts

There’s no way to search past posts you’ve read. If I see something interesting on my timeline and want to find it later, I usually can’t. (Mastodon has a Elasticsearch-based search feature, but it only allows you to search your own posts, your mentions, your favourites, and your bookmarks)

These limitations on search are intentional (and a very common source of arguments) – it’s a privacy / safety issue. Here’s a summary from Tim Bray with lots of links.

It would be personally convenient for me to be able to search more easily but I respect folks’ safety concerns so I’ll leave it at that.

My understanding is that the Mastodon devs are planning to add opt-in search for public posts relatively soon.

other ActivityPub software

We’ve been talking about Mastodon a lot, but not everyone who I follow is using Mastodon: Mastodon uses a protocol called ActivityPub to distribute messages.

Here are some examples of other software I see people talking about, in no particular order:

I’m probably missing a bunch of important ones.

what’s the difference between Mastodon and other ActivityPub software?

This confused me for a while, and I’m still not super clear on how ActivityPub works. What I’ve understood is:

ActivityPub is a protocol (you can explore how it works with blinry’s nice JSON explorer)
Mastodon servers communicate with each other (and with other ActivityPub servers) using ActivityPub
Mastodon clients communicate with their server using the Mastodon API, which is its own thing
There’s also software like GoToSocial that aims to be compatible with the Mastodon API, so that you can use a Mastodon client with it

more mastodon resources

Fedi.Tips seems to be a great introduction
I think you can still use FediFinder to find folks you followed on Twitter on Mastodon
I’ve been using the Ivory client on iOS, but there are lots of great clients. Elk is an alternative web client that folks seem to like.

I haven’t written here about what Mastodon culture is like because other people have done a much better job of talking about it than me, but of course it’s is the biggest thing that affects your experience and it was the thing that took me longest to get a handle on. A few links:

Erin Kissane on frictions people run into when joining Mastodon
Kyle Kingsbury wrote some great moderation guidelines for woof.group (note: woof.group is a LGBTQ+ leather instance, be prepared to see lots of NSFW posts if you visit it)
Mekka Okereke writes lots of great posts about issues Black people encounter on Mastodon (though they’re all on Mastodon so it’s a little hard to navigate)

that’s all!

I don’t regret setting up a single-user server – even though it’s inconvenient, it’s important to me to have control over my social media. I think “have control over my social media” is more important to me than it is to most other people though, because I use Twitter/Mastodon a lot for work.

I am happy that I didn’t start out on a single-user server though – I think it would have made getting started on Mastodon a lot more difficult.

Mastodon is pretty rough around the edges sometimes but I’m able to have more interesting conversations about computers there than I am on Twitter (or Bluesky), so that’s where I’m staying for now.

tech
blogs

What helps people get comfortable on the command line?

Julia Evans

Sometimes I talk to friends who need to use the command line, but are intimidated by it. I never really feel like I have good advice (I’ve been using the command line for too long), and so I asked some people on Mastodon:

if you just stopped being scared of the command line in the last year or three — what helped you?

(no need to reply if you don’t remember, or if you’ve been using the command line comfortably for 15 years — this question isn’t for you :) )

This list is still a bit shorter than I would like, but I’m posting it in the hopes that I can collect some more answers. There obviously isn’t one single thing that works for everyone – different people take different paths.

I think there are three parts to getting comfortable: reducing risks, motivation and resources. I’ll start with risks, then a couple of motivations and then list some resources.

ways to reduce risk

A lot of people are (very rightfully!) concerned about accidentally doing some destructive action on the command line that they can’t undo.

A few strategies people said helped them reduce risks:

regular backups (one person mentioned they accidentally deleted their entire home directory last week in a command line mishap, but it was okay because they had a backup)
For code, using git as much as possible
Aliasing rm to a tool like safe-rm or rmtrash so that you can’t accidentally delete something you shouldn’t (or just rm -i)
Mostly avoid using wildcards, use tab completion instead. (my shell will tab complete rm *.txt and show me exactly what it’s going to remove)
Fancy terminal prompts that tell you the current directory, machine you’re on, git branch, and whether you’re root
Making a copy of files if you’re planning to run an untested / dangerous command on them
Having a dedicated test machine (like a cheap old Linux computer or Raspberry Pi) for particularly dangerous testing, like testing backup software or partitioning
Use --dry-run options for dangerous commands, if they’re available
Build your own --dry-run options into your shell scripts

a “killer app”

A few people mentioned a “killer command line app” that motivated them to start spending more time on the command line. For example:

ripgrep
jq
wget / curl
git (some folks found they preferred the git CLI to using a GUI)
ffmpeg (for video work)
yt-dlp
hard drive data recovery tools (from this great story)

A couple of people also mentioned getting frustrated with GUI tools (like heavy IDEs that use all your RAM and crash your computer) and being motivated to replace them with much lighter weight command line tools.

inspiring command line wizardry

One person mentioned being motivated by seeing cool stuff other people were doing with the command line, like:

explain shell

Several people mentioned explainshell where you can paste in any shell incantation and get it to break it down into different parts.

history, tab completion, etc:

There were lots of little tips and tricks mentioned that make it a lot easier to work on the command line, like:

up arrow to see the previous command
Ctrl+R to search your bash history
navigating inside a line with Ctrl+w (to delete a word), Ctrl+a (to go to the beginning of the line), Ctrl+e (to go to the end), and Ctrl+left arrow / Ctrl+right arrow (to jump back/forward a word)
setting bash history to unlimited
cd - to go back to the previous directory
tab completion of filenames and command names
learning how to use a pager like less to read man pages or other large text files (how to search, scroll, etc)
backing up configuration files before editing them
using pbcopy/pbpaste on Mac OS to copy/paste from your clipboard to stdout/stdin
on Mac OS, you can drag a folder from the Finder into the terminal to get its path

fzf

Lots of mentions of using fzf as a better way to fuzzy search shell history. Some other things people mentioned using fzf for:

picking git branches (git checkout $(git for-each-ref --format='%(refname:short)' refs/heads/ | fzf))
quickly finding files to edit (nvim $(fzf))
switching kubernetes contexts (kubectl config use-context $(kubectl config get-contexts -o name | fzf --height=10 --prompt="Kubernetes Context> "))
picking a specific test to run from a test suite

The general pattern here is that you use fzf to pick something (a file, a git branch, a command line argument), fzf prints the thing you picked to stdout, and then you insert that as the command line argument to another command.

You can also use fzf as an tool to automatically preview the output and quickly iterate, for example:

automatically previewing jq output (echo '' | fzf --preview "jq {q} < YOURFILE.json")
or for sed (echo '' | fzf --preview "sed {q} YOURFILE")
or for awk (echo '' | fzf --preview "awk {q} YOURFILE")

You get the idea.

In general folks will generally define an alias for their fzf incantations so you can type gcb or something to quickly pick a git branch to check out.

raspberry pi

Some people started using a Raspberry Pi, where it’s safer to experiment without worrying about breaking your computer (you can just erase the SD card and start over!)

a fancy shell setup

Lots of people said they got more comfortable with the command line when they started using a more user-friendly shell setup like oh-my-zsh or fish. I really agree with this one – I’ve been using fish for 10 years and I love it.

A couple of other things you can do here:

some folks said that making their terminal prettier helped them feel more comfortable (“make it pink!”).
set up a fancy shell prompt to give you more information (for example you can make the prompt red when a command fails). Specifically transient prompts (where you set a super fancy prompt for the current command, but a much simpler one for past commands) seem really nice.

Some tools for theming your terminal:

I use base16-shell
powerlevel10k is a popular fancy zsh theme which has transient prompts
starship is a fancy prompt tool
on a Mac, I think iTerm2 is easier to customize than the default terminal

a fancy file manager

A few people mentioned fancy terminal file managers like ranger or nnn, which I hadn’t heard of.

a helpful friend or coworker

Someone who can answer beginner questions and give you pointers is invaluable.

shoulder surfing

Several mentions of watching someone more experienced using the terminal – there are lots of little things that experienced users don’t even realize they’re doing which you can pick up.

aliases

Lots of people said that making their own aliases or scripts for commonly used tasks felt like a magical “a ha!” moment, because:

they don’t have to remember the syntax
then they have a list of their most commonly used commands that they can summon easily

cheat sheets to get examples

A lot of man pages don’t have examples, for example the openssl s_client man page has no examples. This makes it a lot harder to get started!

People mentioned a couple of cheat sheet tools, like:

tldr.sh
cheat (which has the bonus of being editable – you can add your own commands to reference later)
um (an incredibly minimal system that you have to build yourself)

For example the cheat page for openssl is really great – I think it includes almost everything I’ve ever actually used openssl for in practice (except the -servername option for openssl s_client).

One person said that they configured their .bash_profile to print out a cheat sheet every time they log in.

don’t try to memorize

A couple of people said that they needed to change their approach – instead of trying to memorize all the commands, they realized they could just look up commands as needed and they’d naturally memorize the ones they used the most over time.

(I actually recently had the exact same realization about learning to read x86 assembly – I was taking a class and the instructor said “yeah, just look everything up every time to start, eventually you’ll learn the most common instructions by heart”)

Some people also said the opposite – that they used a spaced repetition app like Anki to memorize commonly used commands.

vim

One person mentioned that they started using vim on the command line to edit files, and once they were using a terminal text editor it felt more natural to use the command line for other things too.

Also apparently there’s a new editor called micro which is like a nicer version of pico/nano, for folks who don’t want to learn emacs or vim.

use Linux on the desktop

One person said that they started using Linux as their main daily driver, and having to fix Linux issues helped them learn. That’s also how I got comfortable with the command too back in ~2004 (I was really into installing lots of different Linux distributions to try to find my favourite one), but my guess is that it’s not the most popular strategy these days.

being forced to only use the terminal

Some people said that they took a university class where the professor made them do everything in the terminal, or that they created a rule for themselves that they had to do all their work in the terminal for a while.

workshops

A couple of people said that workshops like Software Carpentry workshops (an introduction to the command line, git, and Python/R programming for scientists) helped them get more comfortable with the command line.

You can see the software carpentry curriculum here.

books & articles

a few that were mentioned:

articles:

The Terminal
command line kung fu (has a mix of Unix and Windows command line tips)

books:

effective linux at the command line
unix power tools (which might be outdated)
The Linux Pocket guide

videos:

CLI tools aren’t inherently user-hostile by Mindy Preston
Gary Bernhardt’s destroy all software screencasts
DistroTube

blogs
tech

Some tactics for writing in public

Julia Evans

Someone recently asked me – “how do you deal with writing in public? People on the internet are such assholes!”

I’ve often heard the advice “don’t read the comments”, but actually I’ve learned a huge amount from reading internet comments on my posts from strangers over the years, even if sometimes people are jerks. So I want to explain some tactics I use to try to make the comments on my posts more informative and useful to me, and to try to minimize the number of annoying comments I get.

talk about facts

On here I mostly talk about facts – either facts about computers, or stories about my experiences using computers.

For example this post about tcpdump contains some basic facts about how to use tcpdump, as well as an example of how I’ve used it in the past.

Talking about facts means I get a lot of fact-based comments like:

people sharing their own similar (or different) experiences (“I use tcpdump a lot to look at our RTP sequence numbers”)
pointers to other resources (“the documentation from F5 about tcpdump is great”)
other interesting related facts I didn’t mention (“you can use tcpdump -X too”, “netsh on windows is great”, “you can use sudo tcpdump -s 0 -A 'tcp[((tcp[12:1] & 0xf0) >> 2):4] = 0x47455420' to filter for HTTP GET requests)
potential problems or gotchas (“be careful about running tcpdump as root, try just setting the required capabilities instead”)
questions (“Is there a way to place the BPF filter after IP packet reassembly?” or “what’s the advantage of tcpdump over wireshark?”)
mistakes I made

In general, I’d say that people’s comments about facts tend to stay pretty normal. The main kinds of negative comments I get about facts are:

occasionally people get a little rude about facts I didn’t mention (“Didn’t use -n in any of the examples…please…“). I think I didn’t mention -n in that post because at the time I didn’t know why the -n flag was useful (it’s useful because it turns off this annoying reverse DNS lookup that tcpdump does by default so you can see the IP addresses).
people are also sometimes weird about mistakes. I mostly try to head this off by trying to be self-aware about my knowledge level on a topic, and saying “I’m not sure…” when I’m not sure about something.

stories are great

I think stories encourage pretty good discussion. For example, why you should understand (a little) about TCP is a story about a time it was important for me to understand how TCP worked.

When I share stories about problems I solved, the comments really help me understand how what I learned fits into a bigger context. For example:

is this a common problem? people will often comment saying “this happened to me too!”
what are other common related problems that come up?
are there other possible solutions I didn’t consider?

Also I think these kinds of stories are incredibly important – that post describes a bug that was VERY hard for me to solve, and the only reason I was able to figure it out in the first place was that I read this blog post.

ask technical questions

Often in my blog posts I ask technical questions that I don’t know the answer to (or just mention “I don’t know X…”). This helps people focus their replies a little bit – an obvious comment to make is to provide an answer to the question, or explain the thing I didn’t know!

This is fun because it feels like a guaranteed way to get value out of people’s comments – people LOVE answering questions, and so they get to look smart, and I get the answer to a question I have! Everyone wins!

fix mistakes

I make a lot of mistakes in my blog posts, because I write about a lot of things that are on the edge of my knowledge. When people point out mistakes, I often edit the blog post to fix it.

Usually I’ll stay near a computer for a few hours after I post a blog post so that I can fix mistakes quickly as they come up.

Some people are very careful to list every single error they made in their blog posts (“errata: the post previously said X which was wrong, I have corrected it to say Y”). Personally I make mistakes constantly and I don’t have time for that so I just edit the post to fix the mistakes.

ask for examples/experiences, not opinions

A lot of the time when I post a blog post, people on Twitter/Mastodon will reply with various opinions they have about the thing. For example, someone recently replied to a blog post about DNS saying that they love using zone files and dislike web interfaces for managing DNS records. That’s not an opinion I share, so I asked them why.

They explained that there are some DNS record types (specifically TLSA) that they find often aren’t supported in web interfaces. I didn’t know that people used TLSA records, so I learned something! Cool!

I’ve found that asking people to share their experiences (“I wanted to use X DNS record type and I couldn’t”) instead of their opinions (“DNS web admin interfaces are bad”) leads to a lot of useful information and discussion. I’ve learned a lot from it over the years, and written a lot of tweets like “which DNS record types have you needed?” to try to extract more information about people’s experiences.

I try to model the same behaviour in my own work when I can – if I have an opinion, I’ll try to explain the experiences I’ve had with computers that caused me to have that opinion.

start with a little context

I think internet strangers are more likely to reply in a weird way when they have no idea who you are or why you’re writing this thing. It’s easy to make incorrect assumptions! So often I’ll mention a little context about why I’m writing this particular blog post.

For example:

A little while ago I started using a Mac, and one of my biggest frustrations with it is that often I need to run Linux-specific software.

I’ve started to run a few more servers recently (nginx playground, mess with dns, dns lookup), so I’ve been thinking about monitoring.

Last night, I needed to scan some documents for some bureaucratic reasons. I’d never used a scanner on Linux before and I was worried it would take hours to figure out…

avoid causing boring conversations

There are some kinds of programming conversations that I find extremely boring (like “should people learn vim?” or “is functional programming better than imperative programming?“). So I generally try to avoid writing blog posts that I think will result in a conversation/comment thread that I find annoying or boring.

For example, I wouldn’t write about my opinions about functional programming: I don’t really have anything interesting to say about it and I think it would lead to a conversation that I’m not interested in having.

I don’t always succeed at this of course (it’s impossible to predict what people are going to want to comment about!), but I try to avoid the most obvious flamebait triggers I’ve seen in the past.

There are a bunch of “flamebait” triggers that can set people off on a conversation that I find boring: cryptocurrency, tailwind, DNSSEC/DoH, etc. So I have a weird catalog in my head of things not to mention if I don’t want to start the same discussion about that thing for the 50th time.

Of course, if you think that conversations about functional programming are interesting, you should write about functional programming and start the conversations you want to have!

Also, it’s often possible to start an interesting conversation about a topic where the conversation is normally boring. For example I often see the same talking points about IPv6 vs IPv4 over and over again, but I remember the comments on Reasons for servers to support IPv6 being pretty interesting. In general if I really care about a topic I’ll talk about it anyway, but I don’t care about functional programming very much so I don’t see the point of bringing it up.

preempt common suggestions

Another kind of “boring conversation” I try to avoid is suggestions of things I have already considered. Like when someone says “you should do X” but I already know I could have done X and chose not to because of A B C.

So I often will add a short note like “I decided not to do X because of A B C” or “you can also do X” or “normally I would do X, here I didn’t because…”. For example, in this post about nix, I list a bunch of Nix features I’m choosing not to use (nix-shell, nix flakes, home manager) to avoid a bunch of helpful people telling me that I should use flakes.

Listing the things I’m not doing is also helpful to readers – maybe someone new to nix will discover nix flakes through that post and decide to use them! Or maybe someone will learn that there are exceptions to when a certain “best practice” is appropriate.

set some boundaries

Recently on Mastodon I complained about some gross terminology (“domain information groper”) that I’d just noticed in the dig man page on my machine. A few dudes in the replies (who by now have all deleted their posts) asked me to prove that the original author intended it to be offensive (which of course is besides the point, there’s just no need to have a term widely understood to be referring to sexual assault in the dig man page) or tried to explain to me why it actually wasn’t a problem.

So I blocked a few people and wrote a quick post:

man so many dudes in the replies demanding that i prove that the person who named dig “domain information groper” intended it in an offensive way. Big day for the block button I guess :)

I don’t do this too often, but I think it’s very important on social media to occasionally set some rules about what kind of behaviour I won’t tolerate. My goal here is usually to drive away some of the assholes (they can unfollow me!) and try to create a more healthy space for everyone else to have a conversation about computers in.

Obviously this only works in situations (like Twitter/Mastodon) where I have the ability to garden my following a little bit over time – I can’t do this on HN or Reddit or Lobsters or whatever and wouldn’t try.

As for fixing it – the dig maintainers removed the problem language years ago, but Mac OS still has a very outdated version for license reasons.

(you might notice that this section is breaking the “avoid boring conversations” rule above, this section was certain to start a very boring argument, but I felt it was important to talk about boundaries so I left it in)

don’t argue

Sometimes people seem to want to get into arguments or make dismissive comments. I don’t reply to them, even if they’re wrong. I dislike arguing on the internet and I’m extremely bad at it, so it’s not a good use of my time.

analyze negative comments

If I get a lot of negative comments that I didn’t expect, I try to see if I can get something useful out of it.

For example, I wrote a toy DNS resolver once and some of the commenters were upset that I didn’t handle parsing the DNS packet. At the time I thought this was silly (I thought DNS parsing was really straightforward and that it was obvious how to do it, who cares that I didn’t handle it?) but I realized that maybe the commenters didn’t think it was easy or obvious, and wanted to know how to do it. Which makes sense! It’s not obvious at all if you haven’t done it before!

Those comments partly inspired implement DNS in a weekend, which focuses much more heavily on the parsing aspects, and which I think is a much better explanation how to write a DNS resolver. So ultimately those comments helped me a lot, even if I found them annoying at the time.

(I realize this section makes me sound like a Perfectly Logical Person who does not get upset by negative public criticism, I promise this is not at all the case and I have 100000 feelings about everything that happens on the internet and get upset all the time. But I find that analyzing the criticism and trying to take away something useful from it helps a bit)

that’s all!

Thanks to Shae, Aditya, Brian, and Kamal for reading a draft of this.

Some other similar posts I’ve written in the past:

blogs
tech

Reading List

New talk: Making Hard Things Easy

the video

the transcript

some thanks

In a git repository, where do your files live?

git stores files in .git/objects

jargon: “content addressed storage”

how are those objects encoded?

there aren’t any diffs

what about older versions of the blog post?

let’s unpack some objects

back to looking at a commit

commit step 1: look at the commit

commit step 2: look at the tree

commit step 3: yet another tree

commit step 4: one more tree….

commit step 5: we made it!

this tree traversal is how git log works

how many previous versions of files do I have?

that’s all!

Notes on using a single-person Mastodon server

what’s a mastodon instance?

on choosing a Mastodon instance

using your own domain

notes on having my own server

downsides to being on a single-person server

bad reply visibility makes conversations harder

some visibility workarounds

other downsides of running your own server: discovery is much harder

account migration

using the Mastodon API is great

DMs are weird

defederation and limiting

there’s no search for posts

other ActivityPub software

what’s the difference between Mastodon and other ActivityPub software?

more mastodon resources

that’s all!

What helps people get comfortable on the command line?

ways to reduce risk

a “killer app”

inspiring command line wizardry

explain shell

history, tab completion, etc:

fzf

raspberry pi

a fancy shell setup

a fancy file manager

a helpful friend or coworker

shoulder surfing

aliases

cheat sheets to get examples

don’t try to memorize

vim

use Linux on the desktop

being forced to only use the terminal

workshops

books & articles

Some tactics for writing in public

talk about facts

stories are great

ask technical questions

fix mistakes

ask for examples/experiences, not opinions

start with a little context

avoid causing boring conversations

preempt common suggestions

set some boundaries

don’t argue

analyze negative comments

that’s all!

git stores files in `.git/objects`

this tree traversal is how `git log` works