Reading List
The most recent articles from a list of feeds I subscribe to.
What's involved in getting a "modern" terminal setup?
Hello! Recently I ran a terminal survey and I asked people what frustrated them. One person commented:
There are so many pieces to having a modern terminal experience. I wish it all came out of the box.
My immediate reaction was “oh, getting a modern terminal experience isn’t that hard, you just need to….”, but the more I thought about it, the longer the “you just need to…” list got, and I kept thinking about more and more caveats.
So I thought I would write down some notes about what it means to me personally to have a “modern” terminal experience and what I think can make it hard for people to get there.
what is a “modern terminal experience”?
Here are a few things that are important to me, with which part of the system is responsible for them:
- multiline support for copy and paste: if you paste 3 commands in your shell, it should not immediately run them all! That’s scary! (shell, terminal emulator)
- infinite shell history: if I run a command in my shell, it should be saved forever, not deleted after 500 history entries or whatever. Also I want commands to be saved to the history immediately when I run them, not only when I exit the shell session (shell)
- a useful prompt: I can’t live without having my current directory and current git branch in my prompt (shell)
- 24-bit colour: this is important to me because I find it MUCH easier to theme neovim with 24-bit colour support than in a terminal with only 256 colours (terminal emulator)
- clipboard integration between vim and my operating system so that when I copy in Firefox, I can just press
p
in vim to paste (text editor, maybe the OS/terminal emulator too) - good autocomplete: for example commands like git should have command-specific autocomplete (shell)
- having colours in
ls
(shell config) - a terminal theme I like: I spend a lot of time in my terminal, I want it to look nice and I want its theme to match my terminal editor’s theme. (terminal emulator, text editor)
- automatic terminal fixing: If a programs prints out some weird escape codes that mess up my terminal, I want that to automatically get reset so that my terminal doesn’t get messed up (shell)
- keybindings: I want
Ctrl+left arrow
to work (shell or application) - being able to use the scroll wheel in programs like
less
: (terminal emulator and applications)
There are a million other terminal conveniences out there and different people value different things, but those are the ones that I would be really unhappy without.
how I achieve a “modern experience”
My basic approach is:
- use the
fish
shell. Mostly don’t configure it, except to:- set the
EDITOR
environment variable to my favourite terminal editor - alias
ls
tols --color=auto
- set the
- use any terminal emulator with 24-bit colour support. In the past I’ve used GNOME Terminal, Terminator, and iTerm, but I’m not picky about this. I don’t really configure it other than to choose a font.
- use
neovim
, with a configuration that I’ve been very slowly building over the last 9 years or so (the last time I deleted my vim config and started from scratch was 9 years ago) - use the base16 framework to theme everything
A few things that affect my approach:
- I don’t spend a lot of time SSHed into other machines
- I’d rather use the mouse a little than come up with keyboard-based ways to do everything
- I work on a lot of small projects, not one big project
some “out of the box” options for a “modern” experience
What if you want a nice experience, but don’t want to spend a lot of time on configuration? Figuring out how to configure vim in a way that I was satisfied with really did take me like ten years, which is a long time!
My best ideas for how to get a reasonable terminal experience with minimal config are:
- shell: either
fish
orzsh
with oh-my-zsh - terminal emulator: almost anything with 24-bit colour support, for example all of these are popular:
- linux: GNOME Terminal, Konsole, Terminator, xfce4-terminal
- mac: iTerm (Terminal.app doesn’t have 256-colour support)
- cross-platform: kitty, alacritty, wezterm, or ghostty
- shell config:
- set the
EDITOR
environment variable to your favourite terminal text editor - maybe alias
ls
tols --color=auto
- set the
- text editor: this is a tough one, maybe micro or helix? I haven’t used
either of them seriously but they both seem like very cool projects and I
think it’s amazing that you can just use all the usual GUI editor commands
(
Ctrl-C
to copy,Ctrl-V
to paste,Ctrl-A
to select all) in micro and they do what you’d expect. I would probably try switching to helix except that retraining my vim muscle memory seems way too hard. Also helix doesn’t have a GUI or plugin system yet.
Personally I wouldn’t use xterm, rxvt, or Terminal.app as a terminal emulator, because I’ve found in the past that they’re missing core features (like 24-bit colour in Terminal.app’s case) that make the terminal harder to use for me.
I don’t want to pretend that getting a “modern” terminal experience is easier than it is though – I think there are two issues that make it hard. Let’s talk about them!
issue 1 with getting to a “modern” experience: the shell
bash and zsh are by far the two most popular shells, and neither of them provide a default experience that I would be happy using out of the box, for example:
- you need to customize your prompt
- they don’t come with git completions by default, you have to set them up
- by default, bash only stores 500 (!) lines of history and (at least on Mac OS) zsh is only configured to store 2000 lines, which is still not a lot
- I find bash’s tab completion very frustrating, if there’s more than one match then you can’t tab through them
And even though I love fish, the fact that it isn’t POSIX does make it hard for a lot of folks to make the switch.
Of course it’s totally possible to learn how to customize your prompt in bash
or whatever, and it doesn’t even need to be that complicated (in bash I’d
probably start with something like export PS1='[\u@\h \W$(__git_ps1 " (%s)")]\$ '
, or maybe use starship).
But each of these “not complicated” things really does add up and it’s
especially tough if you need to keep your config in sync across several
systems.
An extremely popular solution to getting a “modern” shell experience is oh-my-zsh. It seems like a great project and I know a lot of people use it very happily, but I’ve struggled with configuration systems like that in the past – it looks like right now the base oh-my-zsh adds about 3000 lines of config, and often I find that having an extra configuration system makes it harder to debug what’s happening when things go wrong. I personally have a tendency to use the system to add a lot of extra plugins, make my system slow, get frustrated that it’s slow, and then delete it completely and write a new config from scratch.
issue 2 with getting to a “modern” experience: the text editor
In the terminal survey I ran recently, the most popular terminal text editors
by far were vim
, emacs
, and nano
.
I think the main options for terminal text editors are:
- use vim or emacs and configure it to your liking, you can probably have any feature you want if you put in the work
- use nano and accept that you’re going to have a pretty limited experience (for example I don’t think you can select text with the mouse and then “cut” it in nano)
- use
micro
orhelix
which seem to offer a pretty good out-of-the-box experience, potentially occasionally run into issues with using a less mainstream text editor - just avoid using a terminal text editor as much as possible, maybe use VSCode, use
VSCode’s terminal for all your terminal needs, and mostly never edit files in
the terminal. Or I know a lot of people use
code
as theirEDITOR
in the terminal.
issue 3: individual applications
The last issue is that sometimes individual programs that I use are kind of
annoying. For example on my Mac OS machine, /usr/bin/sqlite3
doesn’t support
the Ctrl+Left Arrow
keyboard shortcut. Fixing this to get a reasonable
terminal experience in SQLite was a little complicated, I had to:
- realize why this is happening (Mac OS won’t ship GNU tools, and “Ctrl-Left arrow” support comes from GNU readline)
- find a workaround (install sqlite from homebrew, which does have readline support)
- adjust my environment (put Homebrew’s sqlite3 in my PATH)
I find that debugging application-specific issues like this is really not easy and often it doesn’t feel “worth it” – often I’ll end up just dealing with various minor inconveniences because I don’t want to spend hours investigating them. The only reason I was even able to figure this one out at all is that I’ve been spending a huge amount of time thinking about the terminal recently.
A big part of having a “modern” experience using terminal programs is just
using newer terminal programs, for example I can’t be bothered to learn a
keyboard shortcut to sort the columns in top
, but in htop
I can just click
on a column heading with my mouse to sort it. So I use htop instead! But discovering new more “modern” command line tools isn’t easy (though
I made a list here),
finding ones that I actually like using in practice takes time, and if you’re
SSHed into another machine, they won’t always be there.
everything affects everything else
Something I find tricky about configuring my terminal to make everything “nice” is that changing one seemingly small thing about my workflow can really affect everything else. For example right now I don’t use tmux. But if I needed to use tmux again (for example because I was doing a lot of work SSHed into another machine), I’d need to think about a few things, like:
- if I wanted tmux’s copy to synchronize with my system clipboard over SSH, I’d need to make sure that my terminal emulator has OSC 52 support
- if I wanted to use iTerm’s tmux integration (which makes tmux tabs into iTerm tabs), I’d need to change how I configure colours – right now I set them with a shell script that I run when my shell starts, but that means the colours get lost when restoring a tmux session.
and probably more things I haven’t thought of. “Using tmux means that I have to change how I manage my colours” sounds unlikely, but that really did happen to me and I decided “well, I don’t want to change how I manage colours right now, so I guess I’m not using that feature!”.
It’s also hard to remember which features I’m relying on – for example maybe my current terminal does have OSC 52 support and because copying from tmux over SSH has always Just Worked I don’t even realize that that’s something I need, and then it mysteriously stops working when I switch terminals.
change things slowly
Personally even though I think my setup is not that complicated, it’s taken me 20 years to get to this point! Because terminal config changes are so likely to have unexpected and hard-to-understand consequences, I’ve found that if I change a lot of terminal configuration all at once it makes it much harder to understand what went wrong if there’s a problem, which can be really disorienting.
So I usually prefer to make pretty small changes, and accept that changes can
might take me a REALLY long time to get used to. For example I switched from
using ls
to eza a year or two ago and
while I like it (because eza -l
prints human-readable file sizes by default)
I’m still not quite sure about it. But also sometimes it’s worth it to make a
big change, like I made the switch to fish (from bash) 10 years ago and I’m
very happy I did.
getting a “modern” terminal is not that easy
Trying to explain how “easy” it is to configure your terminal really just made me think that it’s kind of hard and that I still sometimes get confused.
I’ve found that there’s never one perfect way to configure things in the terminal that will be compatible with every single other thing. I just need to try stuff, figure out some kind of locally stable state that works for me, and accept that if I start using a new tool it might disrupt the system and I might need to rethink things.
"Rules" that terminal programs follow
Recently I’ve been thinking about how everything that happens in the terminal is some combination of:
- Your operating system’s job
- Your shell’s job
- Your terminal emulator’s job
- The job of whatever program you happen to be running (like
top
orvim
orcat
)
The first three (your operating system, shell, and terminal emulator) are all kind of known quantities – if you’re using bash in GNOME Terminal on Linux, you can more or less reason about how how all of those things interact, and some of their behaviour is standardized by POSIX.
But the fourth one (“whatever program you happen to be running”) feels like it could do ANYTHING. How are you supposed to know how a program is going to behave?
This post is kind of long so here’s a quick table of contents:
- programs behave surprisingly consistently
- these are meant to be descriptive, not prescriptive
- it’s not always obvious which “rules” are the program’s responsibility to implement
- rule 1: noninteractive programs should quit when you press
Ctrl-C
- rule 2: TUIs should quit when you press
q
- rule 3: REPLs should quit when you press
Ctrl-D
on an empty line - rule 4: don’t use more than 16 colours
- rule 5: vaguely support readline keybindings
- rule 5.1:
Ctrl-W
should delete the last word - rule 6: disable colours when writing to a pipe
- rule 7:
-
means stdin/stdout - these “rules” take a long time to learn
programs behave surprisingly consistently
As far as I know, there are no real standards for how programs in the terminal should behave – the closest things I know of are:
- POSIX, which mostly dictates how your terminal emulator / OS / shell should
work together. I think it does specify a few things about how core utilities like
cp
should work but AFAIK it doesn’t have anything to say about how for examplehtop
should behave. - these command line interface guidelines
But even though there are no standards, in my experience programs in the terminal behave in a pretty consistent way. So I wanted to write down a list of “rules” that in my experience programs mostly follow.
these are meant to be descriptive, not prescriptive
My goal here isn’t to convince authors of terminal programs that they should follow any of these rules. There are lots of exceptions to these and often there’s a good reason for those exceptions.
But it’s very useful for me to know what behaviour to expect from a random new terminal program that I’m using. Instead of “uh, programs could do literally anything”, it’s “ok, here are the basic rules I expect, and then I can keep a short mental list of exceptions”.
So I’m just writing down what I’ve observed about how programs behave in my 20 years of using the terminal, why I think they behave that way, and some examples of cases where that rule is “broken”.
it’s not always obvious which “rules” are the program’s responsibility to implement
There are a bunch of common conventions that I think are pretty clearly the program’s responsibility to implement, like:
- config files should go in
~/.BLAHrc
or~/.config/BLAH/FILE
or/etc/BLAH/
or something --help
should print help text- programs should print “regular” output to stdout and errors to stderr
But in this post I’m going to focus on things that it’s not 100% obvious are
the program’s responsibility. For example it feels to me like a “law of nature”
that pressing Ctrl-D
should quit a REPL, but programs often
need to explicitly implement support for it – even though cat
doesn’t need
to implement Ctrl-D
support, ipython
does. (more about that in “rule 3” below)
Understanding which things are the program’s responsibility makes it much less surprising when different programs’ implementations are slightly different.
rule 1: noninteractive programs should quit when you press Ctrl-C
The main reason for this rule is that noninteractive programs will quit by
default on Ctrl-C
if they don’t set up a SIGINT
signal handler, so this is
kind of a “you should act like the default” rule.
Something that trips a lot of people up is that this doesn’t apply to
interactive programs like python3
or bc
or less
. This is because in
an interactive program, Ctrl-C
has a different job – if the program is
running an operation (like for example a search in less
or some Python code
in python3
), then Ctrl-C
will interrupt that operation but not stop the
program.
As an example of how this works in an interactive program: here’s the code in prompt-toolkit (the library that iPython uses for handling input)
that aborts a search when you press Ctrl-C
.
rule 2: TUIs should quit when you press q
TUI programs (like less
or htop
) will usually quit when you press q
.
This rule doesn’t apply to any program where pressing q
to quit wouldn’t make
sense, like tmux
or text editors.
rule 3: REPLs should quit when you press Ctrl-D
on an empty line
REPLs (like python3
or ed
) will usually quit when you press Ctrl-D
on an
empty line. This rule is similar to the Ctrl-C
rule – the reason for this is
that by default if you’re running a program (like cat
) in “cooked mode”, then
the operating system will return an EOF
when you press Ctrl-D
on an empty
line.
Most of the REPLs I use (sqlite3, python3, fish, bash, etc) don’t actually use cooked mode, but they all implement this keyboard shortcut anyway to mimic the default behaviour.
For example, here’s the code in prompt-toolkit that quits when you press Ctrl-D, and here’s the same code in readline.
I actually thought that this one was a “Law of Terminal Physics” until very recently because I’ve basically never seen it broken, but you can see that it’s just something that each individual input library has to implement in the links above.
Someone pointed out that the Erlang REPL does not quit when you press Ctrl-D
,
so I guess not every REPL follows this “rule”.
rule 4: don’t use more than 16 colours
Terminal programs rarely use colours other than the base 16 ANSI colours. This
is because if you specify colours with a hex code, it’s very likely to clash
with some users’ background colour. For example if I print out some text as
#EEEEEE
, it would be almost invisible on a white background, though it would
look fine on a dark background.
But if you stick to the default 16 base colours, you have a much better chance that the user has configured those colours in their terminal emulator so that they work reasonably well with their background color. Another reason to stick to the default base 16 colours is that it makes less assumptions about what colours the terminal emulator supports.
The only programs I usually see breaking this “rule” are text editors, for example Helix by default will use a purple background which is not a default ANSI colour. It seems fine for Helix to break this rule since Helix isn’t a “core” program and I assume any Helix user who doesn’t like that colorscheme will just change the theme.
rule 5: vaguely support readline keybindings
Almost every program I use supports readline
keybindings if it would make
sense to do so. For example, here are a bunch of different programs and a link
to where they define Ctrl-E
to go to the end of the line:
- ipython (Ctrl-E defined here)
- atuin (Ctrl-E defined here)
- fzf (Ctrl-E defined here)
- zsh (Ctrl-E defined here)
- fish (Ctrl-E defined here)
- tmux’s command prompt (Ctrl-E defined here)
None of those programs actually uses readline
directly, they just sort of
mimic emacs/readline keybindings. They don’t always mimic them exactly: for
example atuin seems to use Ctrl-A
as a prefix, so Ctrl-A
doesn’t go to the
beginning of the line.
Also all of these programs seem to implement their own internal cut and paste
buffers so you can delete a line with Ctrl-U
and then paste it with Ctrl-Y
.
The exceptions to this are:
- some programs (like
git
,cat
, andnc
) don’t have any line editing support at all (except for backspace,Ctrl-W
, andCtrl-U
) - as usual text editors are an exception, every text editor has its own approach to editing text
I wrote more about this “what keybindings does a program support?” question in entering text in the terminal is complicated.
rule 5.1: Ctrl-W should delete the last word
I’ve never seen a program (other than a text editor) where Ctrl-W
doesn’t
delete the last word. This is similar to the Ctrl-C
rule – by default if a
program is in “cooked mode”, the OS will delete the last word if you press
Ctrl-W
, and delete the whole line if you press Ctrl-U
. So usually programs
will imitate that behaviour.
I can’t think of any exceptions to this other than text editors but if there are I’d love to hear about them!
rule 6: disable colours when writing to a pipe
Most programs will disable colours when writing to a pipe. For example:
rg blah
will highlight all occurrences ofblah
in the output, but if the output is to a pipe or a file, it’ll turn off the highlighting.ls --color=auto
will use colour when writing to a terminal, but not when writing to a pipe
Both of those programs will also format their output differently when writing
to the terminal: ls
will organize files into columns, and ripgrep will group
matches with headings.
If you want to force the program to use colour (for example because you want to
look at the colour), you can use unbuffer
to force the program’s output to be
a tty like this:
unbuffer rg blah | less -R
I’m sure that there are some programs that “break” this rule but I can’t think
of any examples right now. Some programs have an --color
flag that you can
use to force colour to be on, in the example above you could also do rg --color=always | less -R
.
rule 7: -
means stdin/stdout
Usually if you pass -
to a program instead of a filename, it’ll read from
stdin or write to stdout (whichever is appropriate). For example, if you want
to format the Python code that’s on your clipboard with black
and then copy
it, you could run:
pbpaste | black - | pbcopy
(pbpaste
is a Mac program, you can do something similar on Linux with xclip
)
My impression is that most programs implement this if it would make sense and I can’t think of any exceptions right now, but I’m sure there are many exceptions.
these “rules” take a long time to learn
These rules took me a long time for me to learn because I had to:
- learn that the rule applied anywhere at all ("
Ctrl-C
will exit programs") - notice some exceptions (“okay,
Ctrl-C
will exitfind
but notless
”) - subconsciously figure out what the pattern is ("
Ctrl-C
will generally quit noninteractive programs, but in interactive programs it might interrupt the current operation instead of quitting the program") - eventually maybe formulate it into an explicit rule that I know
A lot of my understanding of the terminal is honestly still in the “subconscious pattern recognition” stage. The only reason I’ve been taking the time to make things explicit at all is because I’ve been trying to explain how it works to others. Hopefully writing down these “rules” explicitly will make learning some of this stuff a little bit faster for others.
Why pipes sometimes get "stuck": buffering
Here’s a niche terminal problem that has bothered me for years but that I never really understood until a few weeks ago. Let’s say you’re running this command to watch for some specific output in a log file:
tail -f /some/log/file | grep thing1 | grep thing2
If log lines are being added to the file relatively slowly, the result I’d see is… nothing! It doesn’t matter if there were matches in the log file or not, there just wouldn’t be any output.
I internalized this as “uh, I guess pipes just get stuck sometimes and don’t
show me the output, that’s weird”, and I’d handle it by just
running grep thing1 /some/log/file | grep thing2
instead, which would work.
So as I’ve been doing a terminal deep dive over the last few months I was really excited to finally learn exactly why this happens.
why this happens: buffering
The reason why “pipes get stuck” sometimes is that it’s VERY common for programs to buffer their output before writing it to a pipe or file. So the pipe is working fine, the problem is that the program never even wrote the data to the pipe!
This is for performance reasons: writing all output immediately as soon as you can uses more system calls, so it’s more efficient to save up data until you have 8KB or so of data to write (or until the program exits) and THEN write it to the pipe.
In this example:
tail -f /some/log/file | grep thing1 | grep thing2
the problem is that grep thing1
is saving up all of its matches until it has
8KB of data to write, which might literally never happen.
programs don’t buffer when writing to a terminal
Part of why I found this so disorienting is that tail -f file | grep thing
will work totally fine, but then when you add the second grep
, it stops
working!! The reason for this is that the way grep
handles buffering depends
on whether it’s writing to a terminal or not.
Here’s how grep
(and many other programs) decides to buffer its output:
- Check if stdout is a terminal or not using the
isatty
function- If it’s a terminal, use line buffering (print every line immediately as soon as you have it)
- Otherwise, use “block buffering” – only print data if you have at least 8KB or so of data to print
So if grep
is writing directly to your terminal then you’ll see the line as
soon as it’s printed, but if it’s writing to a pipe, you won’t.
Of course the buffer size isn’t always 8KB for every program, it depends on the implementation. For grep
the buffering is handled by libc, and libc’s buffer size is
defined in the BUFSIZ
variable. Here’s where that’s defined in glibc.
(as an aside: “programs do not use 8KB output buffers when writing to a terminal” isn’t, like, a law of terminal physics, a program COULD use an 8KB buffer when writing output to a terminal if it wanted, it would just be extremely weird if it did that, I can’t think of any program that behaves that way)
commands that buffer & commands that don’t
One annoying thing about this buffering behaviour is that you kind of need to remember which commands buffer their output when writing to a pipe.
Some commands that don’t buffer their output:
- tail
- cat
- tee
I think almost everything else will buffer output, especially if it’s a command where you’re likely to be using it for batch processing. Here’s a list of some common commands that buffer their output when writing to a pipe, along with the flag that disables block buffering.
- grep (
--line-buffered
) - sed (
-u
) - awk (there’s a
fflush()
function) - tcpdump (
-l
) - jq (
-u
) - tr (
-u
) - cut (can’t disable buffering)
Those are all the ones I can think of, lots of unix commands (like sort
) may
or may not buffer their output but it doesn’t matter because sort
can’t do
anything until it finishes receiving input anyway.
Also I did my best to test both the Mac OS and GNU versions of these but there are a lot of variations and I might have made some mistakes.
programming languages where the default “print” statement buffers
Also, here are a few programming language where the default print statement will buffer output when writing to a pipe, and some ways to disable buffering if you want:
- C (disable with
setvbuf
) - Python (disable with
python -u
, orPYTHONUNBUFFERED=1
, orsys.stdout.reconfigure(line_buffering=False)
, orprint(x, flush=True)
) - Ruby (disable with
STDOUT.sync = true
) - Perl (disable with
$| = 1
)
I assume that these languages are designed this way so that the default print function will be fast when you’re doing batch processing.
Also whether output is buffered or not might depend on how you print, for
example in C++ cout << "hello\n"
buffers when writing to a pipe but cout << "hello" << endl
will flush its output.
when you press Ctrl-C
on a pipe, the contents of the buffer are lost
Let’s say you’re running this command as a hacky way to watch for DNS requests
to example.com
, and you forgot to pass -l
to tcpdump:
sudo tcpdump -ni any port 53 | grep example.com
When you press Ctrl-C
, what happens? In a magical perfect world, what I would
want to happen is for tcpdump
to flush its buffer, grep
would search for
example.com
, and I would see all the output I missed.
But in the real world, what happens is that all the programs get killed and the
output in tcpdump
’s buffer is lost.
I think this problem is probably unavoidable – I spent a little time with
strace
to see how this works and grep
receives the SIGINT
before
tcpdump
anyway so even if tcpdump
tried to flush its buffer grep
would
already be dead.
After a little more investigation, there is a workaround: if you find
tcpdump
’s PID and kill -TERM $PID
, then tcpdump will flush the buffer so
you can see the output. That’s kind of a pain but I tested it and it seems to
work.
redirecting to a file also buffers
It’s not just pipes, this will also buffer:
sudo tcpdump -ni any port 53 > output.txt
Redirecting to a file doesn’t have the same “Ctrl-C
will totally destroy the
contents of the buffer” problem though – in my experience it usually behaves
more like you’d want, where the contents of the buffer get written to the file
before the program exits. I’m not 100% sure whether this is something you can
always rely on or not.
a bunch of potential ways to avoid buffering
Okay, let’s talk solutions. Let’s say you’ve run this command:
tail -f /some/log/file | grep thing1 | grep thing2
I asked people on Mastodon how they would solve this in practice and there were 5 basic approaches. Here they are:
solution 1: run a program that finishes quickly
Historically my solution to this has been to just avoid the “command writing to pipe slowly” situation completely and instead run a program that will finish quickly like this:
cat /some/log/file | grep thing1 | grep thing2 | tail
This doesn’t do the same thing as the original command but it does mean that you get to avoid thinking about these weird buffering issues.
(you could also do grep thing1 /some/log/file
but I often prefer to use an
“unnecessary” cat
)
solution 2: remember the “line buffer” flag to grep
You could remember that grep has a flag to avoid buffering and pass it like this:
tail -f /some/log/file | grep --line-buffered thing1 | grep thing2
solution 3: use awk
Some people said that if they’re specifically dealing with a multiple greps
situation, they’ll rewrite it to use a single awk
instead, like this:
tail -f /some/log/file | awk '/thing1/ && /thing2/'
Or you would write a more complicated grep
, like this:
tail -f /some/log/file | grep -E 'thing1.*thing2'
(awk
also buffers, so for this to work you’ll want awk
to be the last command in the pipeline)
solution 4: use stdbuf
stdbuf
uses LD_PRELOAD to turn off libc’s buffering, and you can use it to turn off output buffering like this:
tail -f /some/log/file | stdbuf -o0 grep thing1 | grep thing2
Like any LD_PRELOAD
solution it’s a bit unreliable – it doesn’t work on
static binaries, I think won’t work if the program isn’t using libc’s
buffering, and doesn’t always work on Mac OS. Harry Marr has a really nice How stdbuf works post.
solution 5: use unbuffer
unbuffer program
will force the program’s output to be a TTY, which means
that it’ll behave the way it normally would on a TTY (less buffering, colour
output, etc). You could use it in this example like this:
tail -f /some/log/file | unbuffer grep thing1 | grep thing2
Unlike stdbuf
it will always work, though it might have unwanted side
effects, for example grep thing1
’s will also colour matches.
If you want to install unbuffer, it’s in the expect
package.
that’s all the solutions I know about!
It’s a bit hard for me to say which one is “best”, I think personally I’m
mostly likely to use unbuffer
because I know it’s always going to work.
If I learn about more solutions I’ll try to add them to this post.
I’m not really sure how often this comes up
I think it’s not very common for me to have a program that slowly trickles data into a pipe like this, normally if I’m using a pipe a bunch of data gets written very quickly, processed by everything in the pipeline, and then everything exits. The only examples I can come up with right now are:
- tcpdump
tail -f
- watching log files in a different way like with
kubectl logs
- the output of a slow computation
what if there were an environment variable to disable buffering?
I think it would be cool if there were a standard environment variable to turn
off buffering, like PYTHONUNBUFFERED
in Python. I got this idea from a
couple of blog posts by Mark Dominus
in 2018. Maybe NO_BUFFER
like NO_COLOR?
The design seems tricky to get right; Mark points out that NETBSD has environment variables called STDBUF
, STDBUF1
, etc which gives you a
ton of control over buffering but I imagine most developers don’t want to
implement many different environment variables to handle a relatively minor
edge case.
I’m also curious about whether there are any programs that just automatically flush their output buffers after some period of time (like 1 second). It feels like it would be nice in theory but I can’t think of any program that does that so I imagine there are some downsides.
stuff I left out
Some things I didn’t talk about in this post since these posts have been getting pretty long recently and seriously does anyone REALLY want to read 3000 words about buffering?
- the difference between line buffering and having totally unbuffered output
- how buffering to stderr is different from buffering to stdout
- this post is only about buffering that happens inside the program, your operating system’s TTY driver also does a little bit of buffering sometimes
- other reasons you might need to flush your output other than “you’re writing to a pipe”
Importing a frontend Javascript library without a build system
I like writing Javascript without a build system and for the millionth time yesterday I ran into a problem where I needed to figure out how to import a Javascript library in my code without using a build system, and it took FOREVER to figure out how to import it because the library’s setup instructions assume that you’re using a build system.
Luckily at this point I’ve mostly learned how to navigate this situation and either successfully use the library or decide it’s too difficult and switch to a different library, so here’s the guide I wish I had to importing Javascript libraries years ago.
I’m only going to talk about using Javacript libraries on the frontend, and only about how to use them in a no-build-system setup.
In this post I’m going to talk about:
- the three main types of Javascript files a library might provide (ES Modules, the “classic” global variable kind, and CommonJS)
- how to figure out which types of files a Javascript library includes in its build
- ways to import each type of file in your code
the three kinds of Javascript files
There are 3 basic types of Javascript files a library can provide:
- the “classic” type of file that defines a global variable. This is the kind
of file that you can just
<script src>
and it’ll Just Work. Great if you can get it but not always available - an ES module (which may or may not depend on other files, we’ll get to that)
- a “CommonJS” module. This is for Node, you can’t use it in a browser at all without using a build system.
I’m not sure if there’s a better name for the “classic” type but I’m just going to call it “classic”. Also there’s a type called “AMD” but I’m not sure how relevant it is in 2024.
Now that we know the 3 types of files, let’s talk about how to figure out which of these the library actually provides!
where to find the files: the NPM build
Every Javascript library has a build which it uploads to NPM. You might be thinking (like I did originally) – Julia! The whole POINT is that we’re not using Node to build our library! Why are we talking about NPM?
But if you’re using a link from a CDN like https://cdnjs.cloudflare.com/ajax/libs/Chart.js/4.4.1/chart.umd.min.js, you’re still using the NPM build! All the files on the CDNs originally come from NPM.
Because of this, I sometimes like to npm install
the library even if I’m not
planning to use Node to build my library at all – I’ll just create a new temp
folder, npm install
there, and then delete it when I’m done. I like being able to poke
around in the files in the NPM build on my filesystem, because then I can be
100% sure that I’m seeing everything that the library is making available in
its build and that the CDN isn’t hiding something from me.
So let’s npm install
a few libraries and try to figure out what types of
Javascript files they provide in their builds!
example library 1: chart.js
First let’s look inside Chart.js, a plotting library.
$ cd /tmp/whatever
$ npm install chart.js
$ cd node_modules/chart.js/dist
$ ls *.*js
chart.cjs chart.js chart.umd.js helpers.cjs helpers.js
This library seems to have 3 basic options:
option 1: chart.cjs
. The .cjs
suffix tells me that this is a CommonJS
file, for using in Node. This means it’s impossible to use it directly in the
browser without some kind of build step.
option 2:chart.js
. The .js
suffix by itself doesn’t tell us what kind of
file it is, but if I open it up, I see import '@kurkle/color';
which is an
immediate sign that this is an ES module – the import ...
syntax is ES
module syntax.
option 3: chart.umd.js
. “UMD” stands for “Universal Module Definition”,
which I think means that you can use this file either with a basic <script src>
, CommonJS,
or some third thing called AMD that I don’t understand.
how to use a UMD file
When I was using Chart.js I picked Option 3. I just needed to add this to my code:
<script src="./chart.umd.js"> </script>
and then I could use the library with the global Chart
environment variable.
Couldn’t be easier. I just copied chart.umd.js
into my Git repository so that
I didn’t have to worry about using NPM or the CDNs going down or anything.
the build files aren’t always in the dist
directory
A lot of libraries will put their build in the dist
directory, but not
always! The build files’ location is specified in the library’s package.json
.
For example here’s an excerpt from Chart.js’s package.json
.
"jsdelivr": "./dist/chart.umd.js",
"unpkg": "./dist/chart.umd.js",
"main": "./dist/chart.cjs",
"module": "./dist/chart.js",
I think this is saying that if you want to use an ES Module (module
) you
should use dist/chart.js
, but the jsDelivr and unpkg CDNs should use
./dist/chart.umd.js
. I guess main
is for Node.
chart.js
’s package.json
also says "type": "module"
, which according to this documentation
tells Node to treat files as ES modules by default. I think it doesn’t tell us
specifically which files are ES modules and which ones aren’t but it does tell
us that something in there is an ES module.
example library 2: @atcute/oauth-browser-client
@atcute/oauth-browser-client
is a library for logging into Bluesky with OAuth in the browser.
Let’s see what kinds of Javascript files it provides in its build!
$ npm install @atcute/oauth-browser-client
$ cd node_modules/@atcute/oauth-browser-client/dist
$ ls *js
constants.js dpop.js environment.js errors.js index.js resolvers.js
It seems like the only plausible root file in here is index.js
, which looks
something like this:
export { configureOAuth } from './environment.js';
export * from './errors.js';
export * from './resolvers.js';
This export
syntax means it’s an ES module. That means we can use it in
the browser without a build step! Let’s see how to do that.
how to use an ES module with importmaps
Using an ES module isn’t an easy as just adding a <script src="whatever.js">
. Instead, if
the ES module has dependencies (like @atcute/oauth-browser-client
does) the
steps are:
- Set up an import map in your HTML
- Put import statements like
import { configureOAuth } from '@atcute/oauth-browser-client';
in your JS code - Include your JS code in your HTML like this:
<script type="module" src="YOURSCRIPT.js"></script>
The reason we need an import map instead of just doing something like import { BrowserOAuthClient } from "./oauth-client-browser.js"
is that internally the module has more import statements like import {something} from @atcute/client
, and we need to tell the browser where to get the code for @atcute/client
and all of its other dependencies.
Here’s what the importmap I used looks like for @atcute/oauth-browser-client
:
<script type="importmap">
{
"imports": {
"nanoid": "./node_modules/nanoid/bin/dist/index.js",
"nanoid/non-secure": "./node_modules/nanoid/non-secure/index.js",
"nanoid/url-alphabet": "./node_modules/nanoid/url-alphabet/dist/index.js",
"@atcute/oauth-browser-client": "./node_modules/@atcute/oauth-browser-client/dist/index.js",
"@atcute/client": "./node_modules/@atcute/client/dist/index.js",
"@atcute/client/utils/did": "./node_modules/@atcute/client/dist/utils/did.js"
}
}
</script>
Getting these import maps to work is pretty fiddly, I feel like there must be a tool to generate them automatically but I haven’t found one yet. It’s definitely possible to write a script that automatically generates the importmaps using esbuild’s metafile but I haven’t done that and maybe there’s a better way.
I decided to set up importmaps yesterday to get github.com/jvns/bsky-oauth-example to work, so there’s some example code in that repo.
Also someone pointed me to Simon Willison’s download-esm, which will download an ES module and rewrite the imports to point to the JS files directly so that you don’t need importmaps. I haven’t tried it yet but it seems like a great idea.
problems with importmaps: too many files
I did run into some problems with using importmaps in the browser though – it needed to download dozens of Javascript files to load my site, and my webserver in development couldn’t keep up for some reason. I kept seeing files fail to load randomly and then had to reload the page and hope that they would succeed this time.
It wasn’t an issue anymore when I deployed my site to production, so I guess it was a problem with my local dev environment.
Also one slightly annoying thing about ES modules in general is that you need to
be running a webserver to use them, I’m sure this is for a good reason but it’s
easier when you can just open your index.html
file without starting a
webserver.
Because of the “too many files” thing I think actually using ES modules with importmaps in this way isn’t actually that appealing to me, but it’s good to know it’s possible.
how to use an ES module without importmaps
If the ES module doesn’t have dependencies then it’s even easier – you don’t need the importmaps! You can just:
- put
<script type="module" src="YOURCODE.js"></script>
in your HTML. Thetype="module"
is important. - put
import {whatever} from "https://example.com/whatever.js"
inYOURCODE.js
alternative: use esbuild
If you don’t want to use importmaps, you can also use a build system like esbuild. I talked about how to do that in Some notes on using esbuild, but this blog post is about ways to avoid build systems completely so I’m not going to talk about that option here. I do still like esbuild though and I think it’s a good option in this case.
what’s the browser support for importmaps?
CanIUse says that importmaps are in
“Baseline 2023: newly available across major browsers” so my sense is that in
2024 that’s still maybe a little bit too new? I think I would use importmaps
for some fun experimental code that I only wanted like myself and 12 people to
use, but if I wanted my code to be more widely usable I’d use esbuild
instead.
example library 3: @atproto/oauth-client-browser
Let’s look at one final example library! This is a different Bluesky auth
library than @atcute/oauth-browser-client
.
$ npm install @atproto/oauth-client-browser
$ cd node_modules/@atproto/oauth-client-browser/dist
$ ls *js
browser-oauth-client.js browser-oauth-database.js browser-runtime-implementation.js errors.js index.js indexed-db-store.js util.js
Again, it seems like only real candidate file here is index.js
. But this is a
different situation from the previous example library! Let’s take a look at
index.js
:
There’s a bunch of stuff like this in index.js
:
__exportStar(require("@atproto/oauth-client"), exports);
__exportStar(require("./browser-oauth-client.js"), exports);
__exportStar(require("./errors.js"), exports);
var util_js_1 = require("./util.js");
This require()
syntax is CommonJS syntax, which means that we can’t use this
file in the browser at all, we need to use some kind of build step, and
ESBuild won’t work either.
Also in this library’s package.json
it says "type": "commonjs"
which is
another way to tell it’s CommonJS.
how to use a CommonJS module with esm.sh
Originally I thought it was impossible to use CommonJS modules without learning a build system, but then someone Bluesky told me about esm.sh! It’s a CDN that will translate anything into an ES Module. skypack.dev does something similar, I’m not sure what the difference is but one person mentioned that if one doesn’t work sometimes they’ll try the other one.
For @atproto/oauth-client-browser
using it seems pretty simple, I just need to put this in my HTML:
<script type="module" src="script.js"> </script>
and then put this in script.js
.
import { BrowserOAuthClient } from "https://esm.sh/@atproto/oauth-client-browser@0.3.0"
It seems to Just Work, which is cool! Of course this is still sort of using a build system – it’s just that esm.sh is running the build instead of me. My main concerns with this approach are:
- I don’t really trust CDNs to keep working forever – usually I like to copy dependencies into my repository so that they don’t go away for some reason in the future.
- I’ve heard of some issues with CDNs having security compromises which scares me.
- I don’t really understand what esm.sh is doing.
esbuild can also convert CommonJS modules into ES modules
I also learned that you can also use esbuild
to convert a CommonJS module
into an ES module, though there are some limitations – the import { BrowserOAuthClient } from
syntax doesn’t work. Here’s a github issue about that.
I think the esbuild
approach is probably more appealing to me than the
esm.sh
approach because it’s a tool that I already have on my computer so I
trust it more. I haven’t experimented with this much yet though.
summary of the three types of files
Here’s a summary of the three types of JS files you might encounter, options for how to use them, and how to identify them.
Unhelpfully a .js
or .min.js
file extension could be any of these 3
options, so if the file is something.js
you need to do more detective work to
figure out what you’re dealing with.
- “classic” JS files
- How to use it::
<script src="whatever.js"></script>
- Ways to identify it:
- The website has a big friendly banner in its setup instructions saying “Use this with a CDN!” or something
- A
.umd.js
extension - Just try to put it in a
<script src=...
tag and see if it works
- How to use it::
- ES Modules
- Ways to use it:
- If there are no dependencies, just
import {whatever} from "./my-module.js"
directly in your code - If there are dependencies, create an importmap and
import {whatever} from "my-module"
- or use download-esm to remove the need for an importmap
- Use esbuild or any ES Module bundler
- If there are no dependencies, just
- Ways to identify it:
- Look for an
import
orexport
statement. (notmodule.exports = ...
, that’s CommonJS) - An
.mjs
extension - maybe
"type": "module"
inpackage.json
(though it’s not clear to me which file exactly this refers to)
- Look for an
- Ways to use it:
- CommonJS Modules
- Ways to use it:
- Use https://esm.sh to convert it into an ES module, like
https://esm.sh/@atproto/oauth-client-browser@0.3.0
- Use a build somehow (??)
- Use https://esm.sh to convert it into an ES module, like
- Ways to identify it:
- Look for
require()
ormodule.exports = ...
in the code - A
.cjs
extension - maybe
"type": "commonjs"
inpackage.json
(though it’s not clear to me which file exactly this refers to)
- Look for
- Ways to use it:
it’s really nice to have ES modules standardized
The main difference between CommonJS modules and ES modules from my perspective is that ES modules are actually a standard. This makes me feel a lot more confident using them, because browsers commit to backwards compatibility for web standards forever – if I write some code using ES modules today, I can feel sure that it’ll still work the same way in 15 years.
It also makes me feel better about using tooling like esbuild
because even if
the esbuild project dies, because it’s implementing a standard it feels likely
that there will be another similar tool in the future that I can replace it
with.
the JS community has built a lot of very cool tools
A lot of the time when I talk about this stuff I get responses like “I hate javascript!!! it’s the worst!!!”. But my experience is that there are a lot of great tools for Javascript (I just learned about https://esm.sh yesterday which seems great! I love esbuild!), and that if I take the time to learn how things works I can take advantage of some of those tools and make my life a lot easier.
So the goal of this post is definitely not to complain about Javascript, it’s to understand the landscape so I can use the tooling in a way that feels good to me.
questions I still have
Here are some questions I still have, I’ll add the answers into the post if I learn the answer.
- Is there a tool that automatically generates importmaps for an ES Module that I have set up locally? (apparently yes: jspm)
- How can I convert a CommonJS module into an ES module on my computer, the way https://esm.sh does? (apparently esbuild can sort of do this, though named exports don’t work)
- When people normally build CommonJS modules into regular JS code, what’s code is doing that? Obviously there are tools like webpack, rollup, esbuild, etc, but do those tools all implement their own JS parsers/static analysis? How many JS parsers are there out there?
- Is there any way to bundle an ES module into a single file (like
atcute-client.js
), but so that in the browser I can still import multiple different paths from that file (like both@atcute/client/lexicons
and@atcute/client
)?
all the tools
Here’s a list of every tool we talked about in this post:
- Simon Willison’s download-esm which will download an ES module and convert the imports to point at JS files so you don’t need an importmap
- https://esm.sh/ and skypack.dev
- esbuild
- JSPM can generate importmaps
Writing this post has made me think that even though I usually don’t want to
have a build that I run every time I update the project, I might be willing to
have a build step (using download-esm
or something) that I run only once
when setting up the project and never run again except maybe if I’m updating my
dependency versions.
that’s all!
Thanks to Marco Rogers who taught me a lot of the things in this post. I’ve probably made some mistakes in this post and I’d love to know what they are – let me know on Bluesky or Mastodon!
New microblog with TILs
I added a new section to this site a couple weeks ago called TIL (“today I learned”).
the goal: save interesting tools & facts I posted on social media
One kind of thing I like to post on Mastodon/Bluesky is “hey, here’s a cool thing”, like the great SQLite repl litecli, or the fact that cross compiling in Go Just Works and it’s amazing, or cryptographic right answers, or this great diff tool. Usually I don’t want to write a whole blog post about those things because I really don’t have much more to say than “hey this is useful!”
It started to bother me that I didn’t have anywhere to put those things: for example recently I wanted to use diffdiff and I just could not remember what it was called.
the solution: make a new section of this blog
So I quickly made a new folder called /til/, added some
custom styling (I wanted to style the posts to look a little bit like a tweet),
made a little Rake task to help me create new posts quickly (rake new_til
), and
set up a separate RSS Feed for it.
I think this new section of the blog might be more for myself than anything, now when I forget the link to Cryptographic Right Answers I can hopefully look it up on the TIL page. (you might think “julia, why not use bookmarks??” but I have been failing to use bookmarks for my whole life and I don’t see that changing ever, putting things in public is for whatever reason much easier for me)
So far it’s been working, often I can actually just make a quick post in 2 minutes which was the goal.
inspired by Simon Willison’s TIL blog
My page is inspired by Simon Willison’s great TIL blog, though my TIL posts are a lot shorter.
I don’t necessarily want everything to be archived
This came about because I spent a lot of time on Twitter, so I’ve been thinking about what I want to do about all of my tweets.
I keep reading the advice to “POSSE” (“post on your own site, syndicate elsewhere”), and while I find the idea appealing in principle, for me part of the appeal of social media is that it’s a little bit ephemeral. I can post polls or questions or observations or jokes and then they can just kind of fade away as they become less relevant.
I find it a lot easier to identify specific categories of things that I actually want to have on a Real Website That I Own:
- blog posts here!
- comics at https://wizardzines.com/comics/!
- now TILs at https://jvns.ca/til/)
and then let everything else be kind of ephemeral.
I really believe in the advice to make email lists though – the first two (blog posts & comics) both have email lists and RSS feeds that people can subscribe to if they want. I might add a quick summary of any TIL posts from that week to the “blog posts from this week” mailing list.