You have likely participated in several Devographics surveys before, such as State of CSS, or State of JS. These surveys have become the primary source of unbiased data for the practices of front-end developers today (there is also the Web Almanac research, but because this studies what is actually used on the web, it takes a lot longer for changes in developer practices to propagate).

You may remember that last summer, Google sponsored me to be Survey Design Lead for State of CSS 2022. It went really well: we got 60% higher response rate than the year before, which gave browsers a lot of actionable data to prioritize their work. The feedback from these surveys is a prime input into the Interop project, where browsers collaborate to implement the most important features for developers interoperably.

So this summer, Google trusted me with a much bigger project, a brand new survey: State of HTML!

For some of you, a State of HTML survey may be the obvious next step, the remaining missing piece. For others, the gap this is filling may not be as clear. No, this is not about whether you prefer <div> or <span>! It turns out, just like JavaScript and CSS, HTML is actually going through an evolution of its own! New elements like <selectmenu> and <breadcrumb> are on the horizon, or cool new features like popovers and declarative Shadow DOM. There are even JS APIs that are intrinsically tied to HTML, such as e.g. Imperative slot assignment or DOM APIs like input.showPicker() Historically, these did not fit in any of these surveys. Some were previously asked in State of JS, some in State of CSS, but it was always a bit awkward. This new survey aims to fill these gaps, and finish surveying the core technologies of the Web, which are HTML, CSS and JavaScript.

Designing a brand new survey is a more daunting task than creating the new edition of an existing survey, but also an exciting one, as comparability with the data from prior years is not a concern, so there is a lot more freedom.

Each State of X survey consists of two parts: Part 1 is a quiz: a long list of lesser-known and/or cutting-edge (or even upcoming) features where respondents select one of three options:

Starting with State of CSS 2022, respondents could also add freeform comments to provide more context about their answer through the little speech bubble icon. One of my goals this year is to make this feature quicker to use for common types of feedback, and to facilitate quantitative analysis of the responses (to some degree).

At the end of the survey, respondents even get a knowledge score based on their answers, which provides immediate value and motivation which reduces survey fatigue.

Part 2 is more freeform, and usually includes multiple-choice questions about tools and resources, freeform questions about pain points, and of course, demographics.

One of the novel things I tried in the 2022 State of CSS survey was to involve the community in the design process, with one-click voting for the features to ask about. These were actually GitHub Issues with certain labels. Two years prior I had released MaVoice: an app to facilitate one click voting on Issues in any repo, and it fit the bill perfectly here.

This process worked exceptionally well for uncovering blind spots: it turned out there were a bunch of CSS features that would be good to ask about, but were simply not on our radar. This is one of the reasons I strongly believe in transparency and co-design: no one human or small team can ever match the collective intelligence of the community.

Predictably, I plan to try the same approach for State of HTML. Instead of using MaVoice, this year I’m trying GitHub Discussions. These allow one click voting from the GitHub interface itself, without users having to authorize a separate app. They also allow for more discussion, and do not clutter Issues, which are better suited for – well – actual issues.

I have created a Discussions category for this and seeded it with 55 features spanning 12 focus areas (Forms & Editing, Making Web Components, Consuming Web Components, ARIA & Accessibility APIs, Embedding, Multimedia, Interactivity, Semantic HTML, Templating, Bridging the gap with native, Performance, Security & Privacy). These initial ideas and focus areas came from a combination of personal research, as well as several brainstorming sessions with the WebDX CG.

Vote on Features for State of HTML 2023!

You can also see a (read-only) summary of the proposed features with their metadata here though keep in mind that it’s manually updated so it may not not include new proposals.

If you can think of features we missed, please post a new Discussion in this category. There is also a more general 💬 State of HTML 2023 Design category, for meta-discussions on Part 1 of the survey, and design brainstorming on Part 2.

Some things to keep in mind when voting and generally participating in these discussions:

• The votes and proposals collected through this process are only one of the many variables that feed into deciding what to ask about, and are non-binding.
• There are two goals to balance here:
  1. The survey needs to provide value to developers – and be fun to fill in!
  2. The survey needs to provide value to browsers, i.e. get them actionable feedback they can use to help prioritize what to work on. This is the main way that these surveys have impact on the web platform, and is at least as important as (1).
• While the title is “State of HTML”, certain JS APIs or even CSS syntax is also relevant, especially those very close to HTML, such as DOM, ARIA, Web Components, PWAs etc.
• Stable features that have existed for a long time and are widely known are generally less likely to make it to the survey.

Now go vote! 🗳

WordPress has been with me since my very first post in 2009. There is a lot to love about it: It’s open source, it has a thriving ecosystem, a beautiful default theme, and a revolutionary block editor that makes my inner UX geek giddy. Plus, WP made building a website and publishing content accessible to everyone. No wonder it’s the most popular CMS in the world, by a huge margin.

However, for me, the bad had started to outweigh the good:

• Things I could do in minutes in a static site, in WP required finding a plugin or tweaking PHP code.
• It was slow and bloated.
• Getting a draft out of it and into another medium was a pain.
• Despite having never been hacked, I was terrified about it, given all the horror stories.
• I was periodically getting “Error establishing a database connection” errors, whose frequency kept increasing.

It was time to move on. It’s not you WP, it’s me.

It seemed obvious that the next step would be a statically generated blog. I had been using Eleventy for a while on a variety of sites at that point and loved it, so using that was a no-brainer. In fact, my blog was one of my last remaining non-JAMstack sites, and by far the biggest. I had built a simple 11ty blog for my husband a year ago, and was almost jealous of the convenience and simplicity. There are so many conveniences that just come for free with this workflow: git, Markdown, custom components, even GitHub Copilot as you write your prose! And if you can make the repo public, oooooh, the possibilities! People could even file PRs and issues for your blog posts!

Using Netlify as a platform was also a no-brainer: I had been using it for years, for over 30 sites at this point! I love their simplicity, their focus on developer experience, and their commitment to open source. I also happen to know a bunch of folks there, and they have a great culture too.

However, I was dreading the amount of work it would take to migrate 14 years of content, plugins, and styling. The stroke that broke the camel’s back was a particularly bad db outage. I tweeted about my frustration, but I had already made up my mind.

I reviewed the list of plugins I had installed on WP to estimate the amount of work. Nearly all fell in one of two categories:

1. Solving problems I wouldn’t have if I wasn’t using WP (e.g. SVG support, Don’t Muck My Markup)
2. Giving me benefits I could get in 11ty with very little code (e.g. Prism syntax highlighting, Custom Body Class, Disqus, Unlist Posts & Pages, Widget CSS classes)
3. Giving me benefits I could get with existing Eleventy plugins (e.g. Add Anchor Links, Easy Table of Contents)

This could actually work!

Public or private repo?

One of the hardest dilemmas was whether to make the repo for this website public or private.

Overall, I was happy to have most files be public, but there were a few things I wanted to keep private:

• Drafts (some drafts I’m ok to share publicly, but not all)
• Unlisted pages and posts (posts with publicly accessible URLs, but not linked from anywhere)
• Private pages (e.g. in the previous site I had a password-protected page with my details for conference organizers)

Unfortunately, right now it’s all-or-nothing, even if only one file needs to be private, the whole repo needs to be private.

Making the repo public does have many advantages:

• Transparency is one of my core values, and this is in line with it.
• People can learn from my code and avoid going down the same rabbit holes I did.
• People can file issues for problems.
• People can send PRs to fix both content and functionality.
• I wouldn’t need to use a separate public repo for the data that populates my Speaking, Publications, and Projects pages.

I went back and forth quite a lot, but in the end I decided to make it public. In fact, I fully embraced it, by making it as easy as possible to file issues and submit PRs.

Licensing

Note that a public repo is not automatically open source. As you know, I have a long track record of open sourcing my code. I love seeing people learning from it, using it in their own projects, and blogging about what they’ve learned. So, MIT-licensing the code part of this website is a no-brainer. CC-BY also seems like a no-brainer for content, because, why not?

Where it gets tricky is the design. I’m well aware that neither my logo nor the visual style of this website would win any design awards; I haven’t worked as a graphic designer for many years, and it shows. However, it’s something I feel is very personal to me, my own personal brand, which by definition needs to be unique. Seeing another website with the same logo and/or visual style would feel just as unsettling as walking into a house that looks exactly like mine. I’m speaking from experience: I’ve had my logo and design copied many times, and it always felt like a violation.

I’m not sure how to express this distinction in a GitHub LICENSE file, so I haven’t yet added one, but I did try to outline it in the Credits & Making Of page.

It’s still difficult to draw the line precisely, especially when it comes to CSS code. I’m basically happy for people to copy as much of my CSS code as they want (following MIT license rules), as long as the end result doesn’t scream “Lea Verou” to anyone who has seen this site. But how on Earth do you express that? 🤔

Migrating content to Markdown

The title of this section says “to Markdown” because that’s one of the benefits of this approach: static site generators are largely compatible with each other, so if I ever needed to migrate again, it would be much easier.

Thankfully, travelers on this road before me had already paved it. Many open source scripts out there to migrate WP to Markdown! The one that worked well for me was lonekorean/wordpress-export-to-markdown (though I later discovered there’s a more updated fork now)

It was still a bumpy road. First, it kept getting stuck on parsing the WP XML export, specifically in comments. I use Disqus for comments, but it mirrors comments in the internal WP system. Also, WP seems to continue recording trackbacks even if they are not displayed anywhere. Turns out I had hundreds of thousands of spam trackbacks, which I spent hours cleaning up (it was quite a meditative experience). In the end I got the total comments + trackbacks from 290K down to 26K which reduced the size of the XML export from 210 MB to a mere 31 MB. This did not fix the parsing issue, but allowed me to simply open the file in VS Code and delete the problematic comments manually. It also fixed the uptime issues I was having: I never got another “Error establishing a database connection” error after that, despite taking my own sweet time to migrate (started in April 2023, and finished in July!). Ideally, I wish WP had an option to export without comments, but I guess that’s not a common use case.

While this importer is great, and allowed me to configure the file structure in a way that preserved all my URLs, I did lose a few things:

• “Read more” separators (filed it as issue #93)
• Figures (they are imported as just images with text underneath) (filed it as issue #94)
• Drafts (#16)
• Pages (I had to manually copy them over, but it was only a handful)
• Any custom classes were gone (e.g. a "view-demo" class I used to create “call to action” links)

A few other issues:

• It downloaded all images, but did not update the URLs in the Markdown files. This was easy to fix with a regex find and replace from https?://lea.verou.me/wp-content/uploads/(\d{4}/\d{2})/([\w\.-]+\.(?:png|gif|jpe?g)) to images/$2.
• Some images from some posts were not downloaded – I still have no idea why.
• It did not download any non-media uploads, e.g. zip files. Thankfully, these were only a couple, so I could detect and port over manually.
• Older posts included code directly in the content, without code blocks, which meant it was being parsed as HTML, often with disastrous results (e.g. the post just cutting off in the middle of a sentence because it mentioned <script>, which opened an actual <script> element and ate up the rest of the content). I fixed a few manually, but I’m sure there’s more left.
• Because code was just included as content, the importer also escaped all Markdown special symbols, so adding code blocks around it was not enough, I also had to remove a bunch of backslashes manually.

Rethinking Categorization

While the importer preserved both tags and categories, this was a good opportunity to rethink whether I need them both, and to re-evaluate how I use them.

This spun off into a separate post: Rethinking Categorization.

Migrating comments

Probably one of the hardest parts of this migration was preserving Disqus comments. In fact, it was so hard that I procrastinated on it for three months, being stuck in a limbo where I couldn’t blog because I’d have to port the new post manually.

I’ve documented the process in a separate blog post, as it was quite involved, including some thoughts about what system to use in the future, as I eventually hope to migrate away from Disqus.

Keeping URLs cool

I wanted to preserve the URL structure of my old site as much as possible, both for SEO, but also because cool URLs don’t change. The WP importer I used allowed me to preserve the /year/month/slug structure of my URLs.

I did want to have the blog in its own directory though. This site started as a blog, but I now see it as more of a personal site with a blog. Thankfully, redirecting these URLs to corresponding /blog/ URLs was a one liner using Netlify redirects:

/20* /blog/20:splat 301

Going forwards, I also decided to do away with the month being part of the URL, as it complicates the file structure for no discernible benefit and I don’t blog nearly as much now as I did in 2009, e.g. compare 2009 vs 2022: 38 vs 7! I do think I will start blogging more again now, not only due to the new site, but also due to new interests and a long backlog of ideas (just look at July 2023 so far!). However, I doubt I will ever get back to the pre-2014 levels, I simply don’t have that kind of time anymore (coincidentally, it appears my blogging frequency dropped significantly after I started my PhD).

I also wanted to continue having nice, RESTful, usable URLs, which also requires:

URLs that are “hackable” to allow users to move to higher levels of the information architecture by hacking off the end of the URL

In practice, this means it’s not enough if tags/foo/ shows all posts tagged “foo”, tags/ should also show all tags. Similarly, it’s not enough if /blog/2023/04/private-fields-considered-harmful/ links to the corresponding blog post, but also:

• /blog/2023/04/ should show all posts from April 2023
• /blog/2023/ should show all posts from 2023
• and of course /blog/ should show all posts

This proved quite tricky to do with Eleventy, and spanned an entirely different blog post.

Overall impressions

Overall, I’m happy with the result, and the flexibility. I’ve had a lot of fun with this project, and it was a great distraction during a very difficult time in my life, due to dealing with some serious health issues in my immediate family.

However, there are a few things that are now more of a hassle than they were in WP, mainly around the editing flow:

• In WP, editing a blog post I was looking at in my browser was a single click (provided I was logged in). I guess I could still do that by editing through GitHub, but now I’m spoiled, I want an easy way to edit in my own editor (VS Code, which has a lot of nice features for Markdown editing), however the only way to do that is to either painfully traverse the directory structure, or …search to find the right *.md file, neither of which is ideal.
• Previewing a post I was editing was also a single click, whereas now I need to run a local server and manually type the URL in (or browse the website to find it).
• Making edits now requires me to think of a suitable commit message. Sure, this is useful sometimes, but most of the time, I want the convenience of just saving my changes and being done with it.

Open file in VS Code from the browser?

There is a way to solve the first problem: VS Code supports a vscode:// protocol that allows you to open a file in VS Code from the browser. This means, this link would open the file for this blog post in VS Code:

<a href="vscode://file/Users/leaverou/Documents/lea.verou.me/blog/2023/going-lean/index.md">Edit in VS Code</a>

See the issue? I cannot add a button to the UI that only works for me! However, I don’t need to add a button to the UI: as long as I expose the input path of the current page (Eleventy’s page.inputPath) in the HTML somehow, I can just add a bookmarklet to my own browser that just does:

location.href = `vscode://file/Users/leaverou/Documents/lea.verou.me/${document.documentElement.dataset.inputpath}`;

In fact, here it is, ready to be dragged to the bookmarks bar: Edit in VS Code

Now, if only I could find a way to do the opposite: open the localhost URL that corresponds to the Markdown file I’m editing — and my workflow would be complete!

What’s next?

Obviously, there’s a lot of work left to do, and I bet people will find a lot more breakage than I had noticed. I also have a backlog of blog post ideas that I can’t wait to write about.

But I’ve also been toying around with the idea of porting over my personal (non-tech) blog posts, and keep them in an entirely separate section of the website. I don’t like that my content is currently hostage to Tumblr (2012-2013) and Medium (2017-2021), and would love to own it too, though I’m a bit concerned that properly separating the two would take a lot of work.

Anyhow, 'nuff said. Ship it, squirrel! 🚢🐿️

This is the third spinoff post in the migration saga of this blog from WordPress to 11ty.

Migrating was a good opportunity to rethink the information architecture of my site, especially around categorization.

Categories vs Tags

Just like most WP users, I was using both categories and tags, simply because they came for free. However the difference between them was a bit fuzzy, as evidenced by how inconsistently they are used, both here and around the Web. I was mainly using Categories for the type of article (Articles, Rants, Releases, Tips, Tutorials, News, Thoughts), however there were also categories that were more like content tags (e.g. CSS WG, Original, Speaking, Benchmarks).

This was easily solved by moving the latter to actual tags. However, tags are no panacea, there are several issues with them as well.

Problems with tags

Tag aliases

First, there were many tags that were synonyms of each other, and posts were fragmented across them, or had to include both (e.g. JS and Javascript). I addressed this by defining aliases in a global data file, and using Eleventy to dynamically build Netlify redirects for them.

# Tag aliases
{% for alias, tag in tag_aliases %}/tags/{{ alias }}/ /tags/{{ tag }}/ 301
{% endfor %}

Turns out I’m not the first to think of building the Netlify _redirects file dynamically, some googling revealed this blog post from 2021 that does the same thing.

I’ve also decided to expose these aliases in the tags index:

“Orphan” tags

Lastly, another issue is what I call “orphan tags”: Tags that are only used in a single post. The primary use case for both tags and categories is to help you discover related content. Tags that are only used once clutter the list of tags, but serve no actual purpose.

It is important to note that orphan tags are not (always) an authoring mistake. While some tags are definitely too specific and thus unlikely to be used again, the vast majority of orphan tags are tags that could plausibly be used again, but it simply hasn’t happened.

I definitely removed a bunch of overly specific tags from the content, but was still left with more orphan tags than tags with more than one post (103 vs 78 as I write these lines).

For (1), the best course of action is probably to remove the tags from the content altogether. However for (2), there are two things to consider.

How to best display orphan tags in the tag index?

For the tag index, I’ve separated orphan tags from the rest, and I’m displaying them in a <details> element at the end, that is collapsed by default.

Each tag is a link to the post that uses it instead of a tags page, since there is only one post that uses it.

How to best display orphan tags in the post itself?

This is a little trickier. For now, I’ve refrained from making them links, and I’m displaying them faded out to communicate this.

Another alternative I’m contemplating is to hide them entirely. Not as a punitive measure because they have failed at their one purpose in life 😅, but because this would allow me to use tags liberally, and only what sticks would be displayed to the end user.

A third, intermediate solution, would be to have a “and 4 orphan tags” message at the end of the list of tags, which can be clicked to show them.

These are not just UX/IA improvements, they are also performance improvements. Not linking orphan tags to tag pages means I don’t need to generate these tag pages at all. Since the majority of tags are orphan tags, this allowed me to substantially reduce the number of pages that need to be generated, and cut down build time by a whopping 40%, from 2.7s to 1.7s (on average).

Tag hierarchies?

The theory is that categories are a taxonomy and tags a folksonomy. Taxonomies can be hierarchical, but folksonomies are, by definition, flat. However, in practice, tags almost always have an implicit hierarchy, which is also what research on folksonomies in the wild tends to find.

Examples from this very blog:

• There is a separate tag for ES (ECMAScript), and a separate one for JS. However, any post tagged ES should also be tagged JS – though the opposite is not true.
• There is a tag for CSS, tags for specific CSS specifications (e.g. CSS Backgrounds & Borders), and even tags for specific CSS functions or properties (e.g. background-attachment, background-size). However, these are not orthogonal: posts tagged with specific CSS features should also be tagged with the CSS spec that contains them, as well as a general “CSS” tag.

I have yet to see a use case for tagging that does not result in implicit hierarchies. Yet, all UIs for entering tags assume that they are flat. Instead, it’s up to each individual post to maintain these relationships, which is tedious and error prone. In practice, the more general tags are often left out, but not intentionally or predictably.

It would be much better to be able to define this hierarchy in a central place, and have it automatically applied to all posts. In 11ty, it could be as simple as a data file for each tag’s “parent” tag. Every time the tag is used, its parent is also added to the post automatically, recursively all the way up to the root (at build time). I have not tried this yet, but I’m excited to experiment with it once I have a bit more time.

Categories vs Tags: Reprise

Back to our original dilemma: Do I still need categories, especially if I eventually implement tag hierarchies? It does seem that the categories I used in WP for the article type (Articles, Rants, Releases, Tips, Tutorials, News, Thoughts etc) are somewhat distinct from my usage of tags, which are more about the content of the article. However, it is unclear whether this is the best use of categories, or whether I should just use tags for this as well. Another common practice is to use tags for more specific content tags, and categories for broader areas (e.g. “Software engineering”, “Product”, “HCI”, “Personal” etc). Skipping past the point that tag hierarchies make it easy to use tags for this too, this makes me think: maybe what is needed is actually metadata, not categories. Instead of deciding that categories hold the article type, or the broader domain, what if we had certain attributes for both of these things. Then, we could have a “type” attribute, and a “domain” attribute, and use them both for categorization, and for filtering. Since Eleventy already supports arbitrary metadata, this is just a matter of implementation.

Lots to think about, but one thing seems clear: Categories do not have a clear purpose, and thus I’m doing away with them. For now, I have converted all past categories to tags, so that the additional metadata is not lost, and I will revisit how to best expose this metadata in the future.

This is a second spinoff post in the migration saga of this blog from WordPress to 11ty.

On good URLs

It was important to me to have good, RESTful, usable, hackable URLs. While a lot of that is easy and comes for free, following this principle with Eleventy proved quite hard:

URLs that are “hackable” to allow users to move to higher levels of the information architecture by hacking off the end of the URL

What does this mean in practice? It means it’s not enough if tags/foo/ shows all posts tagged “foo”, tags/ should also show all tags. Similarly, it’s not enough if /blog/2023/04/private-fields-considered-harmful/ links to the corresponding blog post, but also:

• /blog/2023/04/ should show all posts from April 2023
• /blog/2023/ should show all posts from 2023
• /blog/ should show all posts

Eleventy “Pagination” Primer

Eleventy has a pagination feature, which actually does a lot more than pagination: it’s used every time you want to generate several pages from a single template by chunking object keys and using them in permalinks.

One of the most common non-pagination use cases for it is tag pages. The typical /tags/tagname/ page is generated by a deceptively simple template:

---
pagination:
  data: collections
  size: 1
  alias: tag
  filter: ["blog", "all"]
permalink: /blog/tags/{{ tag }}/
override:tags: []
eleventyComputed:
  title: "{{ collections[ tag ].length | pluralize('post') }} on {{ tag | format_tag }}"
---

{% set taglist = collections[ tag ] | reverse %}
{# ... Loop over taglist here ... #}

That was it, then you just loop over taglist (or collections[ tag ] | reverse directly) to template the posts under each tag in reverse chronological order. Simple, right? But what about the indices? As it currently stands, visiting /blog/tags/ will just produce a 404.

Index of all tags

Creating an index of all tags only involves a single page, so it does not involve contorting the pagination feature to mind-bending levels, like the rest of this post. However, we need to do some processing to sort the tags by post count, and remove those that are not “real” tags.

There are many ways to go about with this.

The quick and dirty way

The quick and dirty way is to just iterate over collections and count the posts for each tag:

<ol>
{% for tag, posts in collections %}
	<li>{{ tags.one(tag) }}
		({{ posts.length }} posts)
	</li>
{% endfor %}
</ol>

The problem is that this does not produce the tags in any particular order, and you usually want frequently used tags to come first. You could actually fix that with CSS:

<ol>
{% for tag, posts in collections %}
	<li style="order: {{ collections.all.length - posts.length }}">
		{{ tags.one(tag) }}
		({{ posts.length }} posts)
	</li>
{% endfor %}
</ol>

The only advantage of this approach is that this is entirely doable via templates and doesn’t require any JS, but there are several drawbacks. First, it limits what styling you can use: for the order property to actually have an effect, you need to be using either Flexbox or Grid layout. But worse, the order property does not affect the order screen readers read your content one iota.

Dynamic postsByTag collection

To do it all in Eleventy, the most common way is a dynamic collection, added via eleventyConfig.addCollection():

config.addCollection("postsByTag", (collectionApi) => {
	const posts = collectionApi.getFilteredByTag("blog");
	let ret = {};

	for (let post of posts) {
		for (let tag of post.data.tags) {
			ret[tag] ??= [];
			ret[tag].push(post);
		}
	}

	// Now sort, and reconstruct the object
	ret = Object.fromEntries(Object.entries(ret).sort((a, b) => b[1].length - a[1].length));

	return ret;
});

That we then use in the template:

<ol>
{% for tag, posts in collections.postsByTag %}
	<li>
		{{ tags.one(tag) }} ({{ posts }} posts)
	</li>
{% endfor %}
</ol>

Custom taglist filter

Another way is a custom filter:

config.addFilter("taglist" (collections) => {
	let tags = Object.keys(collections).filter(filters.is_real_tag);
	tags.sort((a, b) => collections[b].length - collections[a].length);

	return Object.fromEntries(tags.map(tag => [tag, collections[tag].length]));
});

used like this:

<ol>
{% for tag, posts in collections | taglist %}
	<li>
		{{ tags.one(tag) }} ({{ posts }} posts)
	</li>
{% endfor %}
</ol>

Usually, filters are meant for more broadly usable utility functions, and are not a good fit here. However, the filter approach can be more elegant if your use case is more complicated and involves many different outputs. For the vast majority of use cases, a dynamic collection is more appropriate.

Index of posts by year

Generating yearly indices can be quite similar as generating tag pages. The main difference is that for tags the collection already exists (collections[tag]) whereas for years you have to build it yourself, using addCollection() in your config file.

This seems to come up pretty frequently, both for years and months (the next section):

• https://github.com/11ty/eleventy/issues/502
• https://github.com/tomayac/blogccasion/issues/19
• https://github.com/11ty/eleventy/issues/316#issuecomment-441053919
• https://github.com/11ty/eleventy/issues/1284
• Group posts by year in Eleventy (Blog post)

This is what I did, after spending a pretty long time reading discussions and blog posts:

eleventyConfig.addCollection("postsByYear", (collectionApi) => {
	const posts = collectionApi.getFilteredByTag("blog").reverse();
	const ret = {};

	for (let post of posts) {
		let key = post.date.getFullYear();
		ret[key] ??= [];
		ret[key].push(post);
	}

	return ret;
});

and then, in blog/year-index.njk:

---
pagination:
  data: collections.postsByYear
  size: 1
  alias: year
permalink: /blog/{{ year }}/
override:tags: []
eleventyComputed:
  title: "Posts from {{ year }}"
---

{% import "_posts.njk" as posts %}
{{ posts.list(collections.postsByYear[year], {style: "compact"}) }}

You can see an example of such a page here: Posts from 2010.

Bonus, because this collection is more broadly useful, I was able to utilize it to make a little yearly archives bar chart!

Index of posts by month

Pagination only works on one level: You cannot paginate a paginated collection (though Zach has a workaround for that that I’m still trying to wrap my head around). This also means that you cannot easily paginate tag or year index pages. I worked around that by simply showing a more compact post list if there are more than 10 posts.

However, it also means you cannot process the postsByYear collection and somehow paginate by month. You need to create another collection, this time with the year + month as the key:

config.addCollection("postsByMonth", (collectionApi) => {
    const posts = collectionApi.getFilteredByTag("blog").reverse();
    const ret = {};

    for (let post of posts) {
        let key = filters.format_date(post.date, "iso").substring(0, 7); // YYYY-MM
        ret[key] ??= [];
        ret[key].push(post);
    }

    return ret;
});

And a separate blog/month-index.njk file:

---
pagination:
  data: collections.postsByMonth
  size: 1
  alias: month
permalink: /blog/{{ month | replace("-", "/") }}/
override:tags: []
eleventyComputed:
  title: "Posts from {{ month | format_date({month: 'long', year: 'numeric'}) }}"
---

{% import "_posts.njk" as posts %}
{{ posts.list(collections.postsByMonth[month], {style: "compact"}) }}

You can see an example of such a page here: Posts from December 2010.

So I recently ported my 14 year old blog from WordPress to Eleventy.

I had been using Disqus for comments for years, so I didn’t want to lose them, even if I ended up using a different solution for the future (or no comments at all).

Looking around for an existing solution did not yield many results. There’s Zach’s eleventy-import-disqus but it’s aimed at importing Disqus comments as static copies, but I wanted to have the option to continue using Disqus.

Looking at the WP generated HTML source, I noticed that Disqus was using the WP post id (a number that is not displayed in the UI) to link its threads to the posts. However, the importer I used did not preserve the post ids as metadata (filed issue #95). What to do?

Getting the WP post id

My first thought was to add the post id to each post manually, but use a HEAD request to my existing blog to read it from the Link header, possibly en masse. My second thought was, if I can use JS to get it, maybe I can include Disqus dynamically, through JS, after it procures this number. Then I remembered that 11ty can handle any number of different data sources, and combines them all into a single data object. If I could build an index of slug → post id as another data file, I could add a post id via JS in the 11ty config.

My last epiphany was realizing I didn’t need any HTTP requests to get the post id: it was all in the exported sitemap XML, just unused by the importer! Indeed, each <item> included a <wp:post_id> element with the post id and a <link> element with the URL. I tried to open it in Chrome so I could run some JS on it and build the index, but it complained of parse errors. When I fixed them, the tab crashed under its sheer size.

I needed to remove non-relevant data, and I needed to do it fast. All I really needed was the post id, the slug, and the containing <item> element. Since this did not just contain posts, but also other types of content, such as attachments or custom blocks, we also needed to retain <wp:post_type> so we can filter these out. I copied the XML over to a separate file, and run a series of find & replaces in VS Code:

1. ^(?!.*(wp:post_id|wp:post_type|</?item>|</?link>)).+\n (regex) with empty string
2. \n{3,} (regex) with \n (to remove empty lines)
3. wp:post with post to remove namespaces and make the XML easier to handle
4. https://lea.verou.me/ with empty string and </link> with </link> to keep just the yyyy/mm/slug part of the URL

Then added <?xml version="1.0" encoding="UTF-8" ?> at the top and wrapped everything in a <root> element to make it valid XML.

This resulted in a series of <item> elements that looked like this:

<item>
	<link>2023/04/private-fields-considered-harmful</link>
	<post_id>3599</post_id>
	<post_type><![CDATA[post]]></post_type>
</item>
<item>
	<link>2023/04/private-fields-considered-harmful/image-27</link>
	<post_id>3600</post_id>
	<post_type><![CDATA[attachment]]></post_type>
</item>

At this point, we have exuahsted the capabilities of find & replace; it’s time for some JS!

I opened the file in Chrome and ran:

copy(Object.assign({}, ...[...document.querySelectorAll("item")]
	.filter(item => item.querySelector("post_type").textContent === "post")
	.map(item => ({ [item.querySelector("link").textContent]: item.querySelector("post_id").textContent } ))));

which copies JSON like this to the clipboard, ready to be pasted in a JSON file (I used wpids.json):

{
	...
	"2022/11/tag-2": "3531",
	"2023/03/contrast-ratio-new-home": "3592",
	"2023/04/private-fields-considered-harmful": "3599"
}

Some cleanup was still needed, but this was basically good to go.

Adding the post id to the posts

To inject a wpid property to each post, I added a blog.11tydata.js file with the following:

module.exports = {
	eleventyComputed: {
		postUrlStem: data => {
			return data.page.filePathStem.replace(/^\/blog\/|\/index$/g, "");
		},
		wpid: data => {
			return data.wpids[data.postUrlStem];
		}
	}
};

Linking to Disqus

We now have the post id, and we can use it in our template. Adapting the code from the Universal Embed Code, we get:

{% if wpid %}
<div id="disqus_thread"></div>
<script>
    /**
     *  RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT
     *  THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR
     *  PLATFORM OR CMS.
     *
     *  LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT:
     *  https://disqus.com/admin/universalcode/#configuration-variables
     */

    var disqus_config = function () {
        // Replace PAGE_URL with your page's canonical URL variable
        this.page.url = 'https://lea.verou.me/{{ postUrlStem }}/';

        // Replace PAGE_IDENTIFIER with your page's unique identifier variable
        this.page.identifier = "{{ wpid }} https:\/\/lea.verou.me\/?p={{ wpid }}";
    };

    (function() {  // REQUIRED CONFIGURATION VARIABLE: EDIT THE SHORTNAME BELOW
        var d = document, s = d.createElement('script');

        // IMPORTANT: Replace EXAMPLE with your forum shortname!
        s.src = 'https://leaverou.disqus.com/embed.js';

        s.setAttribute('data-timestamp', +new Date());
        (d.head || d.body).appendChild(s);
    })();
</script>
{% endif %}

That’s it! This now works and displays the Disqus threads correctly!

Using Disqus on new posts as well

Note that as it currently stands, this will not display the Disqus UI on new posts, since they won’t have a wpid. Even if I switch to something else in the future, Disqus is better than nothing meanwhile (for me – many people would disagree: switching to no comments at all seems very common when people switch to a SSG blog).

So, new posts don’t have a wpid, but they don’t need one either. As long as we pass some kind of unique identifier to Disqus, we have a comment thread. The easiest way to do this is to use the post’s path, e.g. 2023/preserve-disqus for this one, as this is guaranteed to be unique.

We also want to be able to disable comments on a per-post basis, so we need a way to do that.

So instead of dealing with wpid directly in templates, I added another computed property in blog.11tydata.js:

disqus_id: data => {
	let wpid = data.wpid;

	if (wpid) {
		return `${ wpid } https:\/\/lea.verou.me\/?p=${ wpid }`;
	}
	else if (data.disqus !== false) {
		return typeof data.disqus !== "string"? data.postUrlStem : data.disqus;
	}
}

Note that this allows us to pass a custom identifier to Disqus by using a string, disable it by using false, or just get the automatic behavior by using true or not specifying it at all. The custom identifier can be useful if we want to change the URL of a post without losing the comments.

Then, I updated the template to use disqus_id instead of wpid.

What’s next?

I don’t know if I will continue using Disqus. It’s convenient, but also heavyweight, and there are privacy concerns around it.

However, I’m not sure what I would use instead. Any third party SaaS service would have the same privacy issues. Not necessarily now, but quite likely in the future.

I’ve looked into Webmentions, but the end-user experience does not compare to a regular comment system, and it seems like quite a hassle to implement.

Utterances is a really cool idea: it uses GitHub issues as a backend for a comment system. Having myself (ab)used the GitHub API as a storage backend many a times (even as early as 2012), I can see the appeal. This may be a viable path forwards, though I need to verify that GitHub Issues can be easily exported, so that I’m not locked in.

On a similar vein, I really loved Gisqus seems great too: It’s like Utterances, but uses GitHub Discussions instead of Issues. What holds me back from switching to it is that Discussions cannot yet be exported, and I think portability is important here.

People don’t even really use comments much anymore, they post on social media instead. I would have loved some way to simply collect the discussions about the post from various social media and display them underneath, but with API prices getting out of control (1 2), that doesn’t seem feasible.

If there are any options I missed, please let me know in the (Disqus, for now 😕) comments!