Hello! Over the holidays I decided it might be fun to run NixOS on one of my servers, as part of my continuing experiments with Nix.

My motivation for this was that previously I was using Ansible to provision the server, but then I’d ad hoc installed a bunch of stuff on the server in a chaotic way separately from Ansible, so in the end I had no real idea of what was on that server and it felt like it would be a huge pain to recreate it if I needed to.

This server just runs a few small personal Go services, so it seemed like a good candidate for experimentation.

I had trouble finding explanations of how to set up NixOS and I needed to cobble together instructions from a bunch of different places, so here’s a very short summary of what worked for me.

why NixOS instead of Ansible?

I think the reason NixOS feels more reliable than Ansible to me is that NixOS is the operating system. It has full control over all your users and services and packages, and so it’s easier for it to reliably put the system into the state you want it to be in.

Because Nix has so much control over the OS, I think that if I tried to make any ad-hoc changes at all to my Nix system, Nix would just blow them away the next time I ran nixos-rebuild. But with Ansible, Ansible only controls a few small parts of the system (whatever I explicitly tell it to manage), so it’s easy to make changes outside Ansible.

That said, here’s what I did to set up NixOS on my server and run a Go service on it.

step 1: install NixOS with nixos-infect

To install NixOS, I created a new Hetzner instance running Ubuntu, and then ran nixos-infect on it to convert the Ubuntu installation into a NixOS install, like this:

curl https://raw.githubusercontent.com/elitak/nixos-infect/master/nixos-infect | PROVIDER=hetznercloud NIX_CHANNEL=nixos-23.11 bash 2>&1 | tee /tmp/infect.log

I originally tried to do this on DigitalOcean, but it didn’t work for some reason, so I went with Hetzner instead and that worked.

This isn’t the only way to install NixOS (this wiki page lists options for setting up NixOS cloud servers), but it seemed to work. It’s possible that there are problems with installing that way that I don’t know about though. It does feel like using an ISO is probably better because that way you don’t have to do this transmogrification of Ubuntu into NixOS.

I definitely skipped Step 1 in nixos-infect’s README (“Read and understand the script”), but I didn’t feel too worried because I was running it on a new instance and I figured that if something went wrong I’d just delete it.

step 2: copy the generated Nix configuration

Next I needed to copy the generated Nix configuration to a new local Git repository, like this:

scp root@SERVER_IP:/etc/nixos/* .

This copied 3 files: configuration.nix, hardware-configuration.nix, and networking.nix. configuration.nix is the main file. I didn’t touch anything in hardware-configuration.nix or networking.nix.

step 3: create a flake

I created a flake to wrap configuration.nix. I don’t remember why I did this (I have some idea of what the advantages of flakes are, but it’s not clear to me if any of them are actually relevant in this case) but it seems to work. Here’s my flake.nix:

{ inputs.nixpkgs.url = "github:NixOS/nixpkgs/23.11";

  outputs = { nixpkgs, ... }: {
    nixosConfigurations.default = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";

      modules = [ ./configuration.nix ];
    };
  };
}

The main gotcha about flakes that I needed to remember here was that you need to git add every .nix file you create otherwise Nix will pretend it doesn’t exist.

The rules about git and flakes seem to be:

• you do need to git add your files
• you don’t need to commit your changes
• unstaged changes to files are also fine, as long as the file has been git added

These rules feel very counterintuitive to me (why require that you git add files but allow unstaged changes?) but that’s how it works. I think it might be an optimization because Nix has to copy all your .nix files to the Nix store for some reason, so only copying files that have been git added makes the copy faster. There’s a GitHub issue tracking it here so maybe the way this works will change at some point.

step 4: figure out how to deploy my configuration

Next I needed to figure out how to deploy changes to my configuration. There are a bunch of tools for this, but I found the blog post Announcing nixos-rebuild: a “new” deployment tool for NixOS that said you can just use the built-in nixos-rebuild, which has --target-host and --build-host options so that you can specify which host to build on and deploy to, so that’s what I did.

I wanted to be able to get Go repositories and build the Go code on the target host, so I created a bash script that runs this command:

nixos-rebuild switch --fast --flake .#default --target-host my-server --build-host my-server --option eval-cache false

Making --target-host and --build-host the same machine is certainly not something I would do for a Serious Production Machine, but this server is extremely unimportant so it’s fine.

This --option eval-cache false is because Nix kept not showing me my errors because they were cached – it would just say error: cached failure of attribute 'nixosConfigurations.default.config.system.build.toplevel' instead of showing me the actual error message. Setting --option eval-cache false turned off caching so that I could see the error messages.

Now I could run bash deploy.sh on my laptop and deploy my configuration to the server! Hooray!

step 5: update my ssh config

I also needed to set up a my-server host in my ~/.ssh/config. I set up SSH agent forwarding so that the server could download the private Git repositories it needed to access.

Host my-server
   Hostname MY_IP_HERE
   User root
   Port 22
   ForwardAgent yes

AddKeysToAgent yes

step 6: set up a Go service

The thing I found the hardest was to figure out how to compile and configure a Go web service to run on the server. The norm seems to be to define your package and define your service’s configuration in 2 different files, but I didn’t feel like doing that – I wanted to do it all in one file. I couldn’t find a simple example of how to do this, so here’s what I did.

I’ve replaced the actual repository name with my-service because it’s a private repository and you can’t run it anyway.

{ pkgs ? (import <nixpkgs> { }), lib, stdenv, ... }: 
let myservice = pkgs.callPackage pkgs.buildGoModule {
  name = "my-service";
  src = fetchGit {
    url = "git@github.com:jvns/my-service.git";
    rev = "efcc67c6b0abd90fb2bd92ef888e4bd9c5c50835"; # put the right git sha here
  };
  vendorHash = "sha256-b+mHu+7Fge4tPmBsp/D/p9SUQKKecijOLjfy9x5HyEE"; # nix will complain about this and tell you the right value
}; in { 
  services.caddy.virtualHosts."my-service.example.com".extraConfig = ''
    reverse_proxy localhost:8333
  '';

  systemd.services.my-service = {
    enable = true;
    description = "my-service";
    after = ["network.target"];
    wantedBy = ["multi-user.target"];
    script = "${myservice}/bin/my-service";
    environment = {
      DB_FILENAME = "/var/lib/my-service/db.sqlite";
    };
    serviceConfig = {
      DynamicUser = true;
      StateDirectory = "my-service"; # /var/lib/my-service
    };
  };
}

Then I just needed to do 2 more things:

1. add ./my-service.nix to the imports section of configuration.nix
2. add services.caddy.enable = true; to configuration.nix to enable Caddy

and everything worked!!

Some notes on this service configuration file:

1. I used extraConfig to configure Caddy because I didn’t feel like learning Nix’s special Caddy syntax – I wanted to just be able to refer to the Caddy documentation directly.
2. I used systemd’s DynamicUser to create a user dynamically to run the service. I’d never used this before but it seems like a great simple way to create a different user for every service without having to write a bunch of repetitive boilerplate and being really careful to choose unique UID and GIDs. The blog post Dynamic Users with systemd talks about how it works.
3. I used StateDirectory to get systemd to create a persistent directory where I could store a SQLite database. It creates a directory at /var/lib/my-service/

I’d never heard of DynamicUser or StateDirectory before Kamal told me about them the other day but they seem like cool systemd features and I wish I’d known about them earlier.

why Caddy?

One quick note on Caddy: I switched to Caddy a while back from nginx because it automatically sets up Let’s Encrypt certificates. I’ve only been using it for tiny hobby services, but it seems pretty great so far for that, and its configuration language is simpler too.

problem: “fetchTree requires a locked input”

One problem I ran into was this error message:

error: in pure evaluation mode, 'fetchTree' requires a locked input, at «none»:0

I found this really perplexing – what is fetchTree? What is «none»:0? What did I do wrong?

I learned 4 things from debugging this (with help from the folks in the Nix discord):

1. In Nix, fetchGit calls an internal function called fetchTree. So errors that say fetchTree might actually be referring to fetchGit.
2. Nix truncates long stack traces by default. Sometimes you can get more information with --show-trace.
3. It seems like Nix doesn’t always give you the line number in your code which caused the error, even if you use --show-trace. I’m not sure why this is. Some people told me this is because fetchTree is a built in function but – why can’t I see the line number in my nix code that called that built in function?
4. Like I mentioned before, you can pass --option eval-cache false to turn off caching so that Nix will always show you the error message instead of error: cached failure of attribute 'nixosConfigurations.default.config.system.build.toplevel'

Ultimately the problem turned out to just be that I forgot to pass the Github revision ID (rev = "efcc67c6b0abd90fb2bd92ef888e4bd9c5c50835";) to fetchGit which was really easy to fix.

nix syntax is still pretty confusing to me

I still don’t really understand the nix language syntax that well, but I haven’t felt motivated to get better at it yet – I guess learning new language syntax just isn’t something I find fun. Maybe one day I’ll learn it. My plan for now with NixOS is to just keep copying and pasting that my-service.nix file above forever.

some questions I still have

I think my main outstanding questions are:

• When I run nixos-rebuild, Nix checks that my systemd services are still working in some way. What does it check exactly? My best guess is that it checks that the systemd service starts successfully, but if the service starts and then immediately crashes, it won’t notice.
• Right now to deploy a new version of one of my services, I need to manually copy and paste the Git SHA of the new revision. There’s probably a better workflow but I’m not sure what it is.

that’s all!

I really do like having all of my service configuration defined in one file, and the approach Nix takes does feel more reliable than the approach I was taking with Ansible.

I just started doing this a week ago and as with all things Nix I have no idea if I’ll end up liking it or not. It seems pretty good so far though!

I will say that I find using Nix to be very difficult and I really struggle when debugging Nix problems (that fetchTree problem I mentioned sounds simple, but it was SO confusing to me at the time), but I kind of like it anyway. Maybe because I’m not using Linux on my laptop right now I miss having linux evenings and Nix feels like a replacement for that :)

The tech news is chockablock¹ with antitrust rumblings and slow-motion happenings. Eagle-eyed press coverage, regulatory reports, and legal discovery have comprehensively documented the shady dealings of Apple and Google's app stores. Pressure for change has built to an unsustainable level. Something's gotta give.

This is the backdrop to the biggest app store story nobody is writing about: on pain of steep fines, gatekeepers are opening up to competing browsers. This, in turn, will enable competitors to replace app stores with directories of Progressive Web Apps. Capable browsers that expose web app installation and powerful features to developers can kickstart app portability, breaking open the mobile duopoly.

But you'd never know it reading Wired or The Verge.

With shockingly few exceptions, coverage of app store regulation assumes the answer to crummy, extractive native app stores is other native app stores. This unexamined framing shapes hundreds of pieces covering regulatory events, including by web-friendly authors. The tech press almost universally fails to mention the web as a substitute for native apps and fail to inform readers of its potential to disrupt app stores.

As Cory Doctorow observed:

"An app is just a web-page wrapped in enough IP to make it a crime to defend yourself against corporate predation."

The implication is clear: browsers unchained can do to mobile what the web did to desktop, where more than 70% of daily "jobs to be done" happen on the web.

Replacing mobile app stores will look different than the web's path to desktop centrality, but the enablers are waiting in the wings. It has gone largely unreported that Progressive Web Apps (PWAs) have been held back by Apple and Google denying competing browsers access to essential APIs.²

Thankfully, regulators haven't been waiting on the press to explain the situation. Recent interventions into mobile ecosystems include requirements to repair browser choice, and the analysis backing those regulations takes into account the web's role as a potential competitor (e.g., Japan's JFTC (pdf)).

Regulators seem to understand that:

• App stores protect proprietary ecosystems through preferential discovery and capabilities.
• Stores then extract rents from developers dependent on commodity capabilities duopolists provide only through proprietary APIs.
• App portability threatens the proprietary agenda of app stores.
• The web can interrupt this model by bringing portability to apps and over-the-top discovery through search. This has yet to happen because...
• The duopolists, in different ways, have kneecapped competing browsers along with their own, keeping the web from contesting the role of app stores.

Apple and Google saw what the web did to desktop, and they've laid roadblocks to the competitive forces that would let history repeat on smartphones.

The Buried Lede

The web's potential to disrupt mobile is evident to regulators, advocates, and developers. So why does the tech news fail to explain the situation?

Consider just one of the many antitrust events of recent months. It was covered by The Verge, Mac Rumors, Apple Insider, and more.

None of the linked articles note browser competition's potential to upend app stores. Browsers unshackled have the potential to free businesses from build-it-twice proprietary ecosystems, end rapacious app store taxes, pave the way for new OS entrants — all without the valid security concerns side-loading introduces.

Lest you think this an isolated incident, this article on the impact of the EU's DMA lacks any hint of the web's potential to unseat app stores. You can repeat this trick with any DMA story from the past year. Or spot-check coverage of the NTIA's February report.

Reporters are "covering" these stories in the lightest sense of the word. Barrels of virtual ink has been spilt documenting unfair app store terms, conditions, and competition. And yet.

Disruption Disrupted

In an industry obsessed with "disruption," why is this David vs. Goliath story going untold? Some theories, in no particular order.

First, Mozilla isn't advocating for a web that can challenge native apps, and none of the other major browser vendors are telling the story either. Apple and Google have no interest in seeing their lucrative proprietary platforms supplanted, and Microsoft (your narrator's employer) famously lacks sustained mobile focus.

Next, it's hard to overlook that tech reporters live like wealthy people, iPhones and all. From that vantage point, it's often news that the web is significantly more capable on other OSes (never mind that they spend much of every day working in a desktop browser). It's hard to report on the potential of something you can't see for yourself.

Also, this might all be Greek. Reporters and editors aren't software engineers, so the potential of browser competition can remain understandably opaque. Stories that include mention of "alternative app stores" generally fail to mention that these stores may not be as safe, or that OS restrictions on features won't disappear just because of a different distribution mechanism, or that the security track records of the existing duopolist app stores are sketchy at best. Under these conditions, it's asking a lot to expect details-based discussion of alternatives, given the many technical wrinkles. Hopefully, someone can walk them through it.

Further, market contestability theory has only recently become a big part of the tech news beat. Regulators have been writing reports to convey their understanding of the market, and to shape effective legislation that will unchain the web, but smart folks unversed in both antitrust and browser minutiae might need help to pick up what regulators are putting down.

Lastly, it hasn't happened yet. Yes, Progressive Web Apps have been around for a few years, but they haven't had an impact on the iPhones that reporters and their circles almost universally carry. It's much easier to get folks to cover stories that directly affect them, and this is one that, so far, largely hasn't.

Green Shoots

The seeds of web-based app store dislocation have already been sown, but the chicken-and-egg question at the heart of platform competition looms.

On the technology side, Apple has been enormously successful at denying essential capabilities to the web through a strategy of compelled monoculture combined with strategic foot-dragging.

As an example, the eight-year delay in implementing Push Notifications for the web³ kept many businesses from giving the web a second thought. If they couldn't re-engage users at the same rates as native apps, the web might as well not exist on phones. This logic has played out on a loop over the last decade, category-by-category, with gatekeepers preventing competing browsers from bringing capabilities to web apps that would let them supplant app stores^2:1 while simultaneously keeping them from being discovered through existing stores.

Proper browser choice could upend this situation, finally allowing the web to provide "table stakes" features in a compelling way. For the first time, developers could bring the modern web's full power to wealthy mobile users, enabling the "write once, test everywhere" vision, and cut out the app store middleman — all without sacrificing essential app features or undermining security.

Sunsetting the 30% tax requires a compelling alternative, and Apple's simultaneous underfunding of Safari and compelled adoption of its underpowered engine have interlocked to keep the web out of the game. No wonder Apple is massively funding lobbyists, lawyers, and astroturf groups to keep engine diversity at bay while belatedly battening the hatches.

On the business side, managers think about "mobile" as a category. Rather than digging into the texture of iOS, Android, and the differing web features available on each, businesses tend to bulk accept or reject the app store model. One sub-segment of "mobile" growing the ability to route around highway robbery Ts & Cs is tantalising, but not enough to change the game; the web, like other metaplatforms, is only a disruptive force when pervasive and capable.⁴

A prohibition on store discovery for web apps has buttressed Apple's denial of essential features to browsers:

Google's answer to web apps in Play is a dog's breakfast, but it does at least exist for developers willing to put in the effort, or for teams savvy enough to reach for PWA Builder.

Recent developments also point to a competitive future for capable web apps.

First, browser engine choice should become a reality on iOS in the EU in 2024, thanks to the plain language of the DMA. Apple will, of course, attempt to delay the entry of competing browsers through as-yet-unknown strategies, but the clock is ticking. Once browsers can enable capable web apps with easier distribution, the logic of the app store loses a bit of its lustre.

Work is also underway to give competing browsers a chance to facilitate PWAs that can install other PWAs. Web App Stores would then become a real possibility through browsers that support them, and we should expect that regulatory and legislative interventions will facilitate this in the near future. Removed from the need to police security (browsers have that covered) and handle distribution (websites update themselves), PWA app stores like store.app can become honest-to-goodness app management surfaces that can safely facilitate discovery and sync.

It's no surprise that Apple and Google have kept private the APIs needed to make this better future possible. They built the necessary infrastructure for the web to disrupt native, then kept it to themselves. This potential has remained locked away within organisations politically hamstrung by native app store agendas. But all of that is about to change.

This begs the question: where's the coverage? This is the most exciting moment in more than 15 years for the web vs. native story, but the tech press is whiffing it.

A New Hope

2024 will be packed to the gills with app store and browser news, from implementation of the DMA, to the UK's renewed push into mobile browsers and cloud gaming, to new legislation arriving in many jurisdictions, to the first attempts at shipping iOS ports of Blink and Gecko browsers. Each event is a chance to inform the public about the already-raging battle for the future of the phone.

It's still possible to reframe these events and provide better context. We need a fuller discussion about what it will mean for mobile OSes to have competing native app stores when the underlying OSes are foundationally insecure. There are also existing examples of ecosystems with this sort of choice (e.g., China), and more needs to be written about the implications for users and developers. Instead of nirvana, the insecure status quo of today's mobile OSes, combined with (even more) absentee app store purveyors, turns side-loading into an alternative form of lock-in, with a kicker of added insecurity for users. With such a foundation, the tech-buying public could understand why a browser's superior sandboxing, web search's better discovery, and frictionless links are better than dodgy curation side-deals and "beware of dog" sign security.

The more that folks understand the stakes, the more likely tech will genuinely change for the better. And isn't that what public interest journalism is for?

FOOTNOTES

Hello! This was my 4th year working full time on Wizard Zines! Here are a few of the things I worked on this year.

a zine!

I published How Integers and Floats Work, which I worked on with Marie.

This one started out its life as “how your computer represents things in memory”, but once we’d explained how integers and floats were represented in memory the zine was already long enough, so we just kept it to integers and floats.

This zine was fun to write: I learned about why signed integers are represented in memory the way they are, and I’m really happy with the explanation of floating point we ended up with.

a playground: memory spy!

When explaining to people how your computer represents people in memory, I kept wanting to open up gdb or lldb and show some example C programs and how the variables in those C programs are represented in memory.

But gdb is kind of confusing if you’re not used to looking at it! So me and Marie made a cute interface on top of lldb, where you can put in any C program, click on a line, and see what the variable looks like. It’s called memory spy and here’s what it looks like:

a playground: integer exposed!

I got really obsessed with float.exposed by Bartosz Ciechanowski for seeing how floats are represented in memory. So with his permission, I made a copy of it for integers called integer.exposed.

Here’s a screenshot:

It was pretty straightforward to make (copying someone else’s design is so much easier than making your own!) but I learned a few CSS tricks from analyzing how he implemented it.

Implement DNS in a Weekend

I’ve been working on a big project to show people how to implement a working networking stack (TCP, TLS, DNS, UDP, HTTP) in 1400 lines of Python, that you can use to download a webpage using 100% your own networking code. Kind of like Nand to Tetris, but for computer networking.

This has been going VERY slowly – writing my own working shitty implementations was relatively easy (I finished that in October 2022), but writing clear tutorials that other people can follow is not.

But in March, I released the first part: Implement DNS in a Weekend. The response was really good – there are dozens of people’s implementations on GitHub, and people have implemented it in Go, C#, C, Clojure, Python, Ruby, Kotlin, Rust, Typescript, Haskell, OCaml, Elixir, Odin, and probably many more languages too. I’d like to see more implementations in less systems-y languages like vanilla JS and PHP, need to think about what I can do to encourage that.

I think “Implement IPv4 in a Weekend” might be the next one I release. It’s going to come with bonus guides to implementing ICMP and UDP too.

a talk: Making Hard Things Easy!

I gave a keynote at Strange Loop this year called Making Hard Things Easy (video + transcript), about why some things are so hard to learn and how we can make them easier. I’m really proud of how it turned out.

a lot of blog posts about Git!

In September I decided to work on a second zine about Git, focusing more on how Git works. This is one of the hardest projects I’ve ever worked on, because over the last 10 years of using it I’d completely lost sight of what’s hard about Git.

So I’ve been doing a lot of research to try to figure out why Git is hard, and I’ve been writing a lot of blog posts. So far I’ve written:

• In a git repository, where do your files live?
• Some miscellaneous git facts
• Confusing git terminology
• git rebase: what can go wrong?
• How git cherry-pick and revert use 3-way merge
• git branches: intuition & reality
• Mounting git commits as folders with NFS

What’s been most surprising so far is that I originally thought “to understand Git, people just need to learn git’s internal data model!”. But the more I talk to people about their struggles with Git, the less I think that’s true. I’ll leave it at that for now, but there’s a lot of work still to do.

some Git prototypes!

I worked on a couple of fun Git tools this year:

• git-commit-folders: a way to mount your Git commits as (read-only) folders using FUSE or NFS. This one came about because someone mentioned that they think of Git commits as being folders with old versions of the code, and it made me wonder – why can’t you just have a virtual folder for every commit? It turns out that it can and it works pretty well.
• git-oops: an experimental prototype of an undo system for git. This one came out of me wondering “why can’t we just have a git undo?”. I learned a bunch of things about why that’s not easy through writing the prototype, I might write a longer blog post about it later.

I’ve been trying to put a little less pressure on myself to release software that’s Amazing and Perfect – sometimes I have an idea that I think is cool but don’t really have the time or energy to fully execute on it. So I decided to just put these both on Github in a somewhat unfinished state, so I can come back to them if later if I want. Or not!

I’m also working on another Git software project, which is a collaboration with a friend.

hired an operations manager!

This year I hired an Operations Manager for Wizard Zines! Lee is incredible and has done SO much to streamline the logistics of running the company, so that I can focus more on writing and coding. I don’t talk much about the mechanics of running the business on here, but it’s a lot and I’m very grateful to have some help.

A few of the many things Lee has made possible:

• run a Black Friday sale!
• we added a review system to the website! (it’s so nice to hear about how people loved getting zines for Christmas!)
• the store has been reorganized to be way clearer!
• we’re more consistent about sending out the new comics newsletter!
• I can take a vacation and not worry about support emails!

migrated to Mastodon!

I spent 10 years building up a Twitter presence, but with the Recent Events, I spent a lot of time in 2023 working on building up a Mastodon account. I’ve found that I’m able to have more interesting conversations about computers on Mastodon than on Twitter or Bluesky, so that’s where I’ve been spending my time. We’ve been having a lot of great discussions about Git there recently.

I’ve run into a few technical issues with Mastodon (which I wrote about at Notes on using a single-person Mastodon server) but overall I’m happy there and I’ve been spending a lot more time there than on Twitter.

some questions for 2024

one of my questions for 2022 was:

• What’s hard for developers about learning to use the Unix command line in 2022? What do I want to do about it?

Maybe I’ll work on that in 2024! Maybe not! I did make a little bit of progress on that question this year (I wrote What helps people get comfortable on the command line?).

Some other questions I’m thinking about on and off:

• Could man pages be a more useful form of documentation? Do I want to try to do anything about that?
• What format do I want to use for this “implement all of computer networking in Python” project? (is it a website? a book? is there a zine? what’s the deal?) Do I want to run workshops?
• What community guidelines do I want to have for discussions on Mastodon?
• Could I be marketing Mess With DNS (from 2021) more? How do I want to do that?

moving slowly is okay

I’ve started to come to terms with the fact that projects always just take longer than I think they will. I started working this “implement your own terrible networking stack” project in 2022, and I don’t know if I’ll finish it in 2024. I’ve been working on this Git zine since September and I still don’t completely understand why Git is hard yet. There’s another small secret project that I initally thought of 5 years ago, made a bunch of progress on this year, but am still not done with. Things take a long time and that’s okay.

As always, thanks for reading and for making it possible for me to do this weird job.