11ty: Index ALL the things!

Lea Verou

This is a second spinoff post in the migration saga of this blog from WordPress to 11ty.

On good URLs

It was important to me to have good, RESTful, usable, hackable URLs. While a lot of that is easy and comes for free, following this principle with Eleventy proved quite hard:

URLs that are “hackable” to allow users to move to higher levels of the information architecture by hacking off the end of the URL

What does this mean in practice? It means it’s not enough if tags/foo/ shows all posts tagged “foo”, tags/ should also show all tags. Similarly, it’s not enough if /blog/2023/04/private-fields-considered-harmful/ links to the corresponding blog post, but also:

/blog/2023/04/ should show all posts from April 2023
/blog/2023/ should show all posts from 2023
/blog/ should show all posts

Eleventy “Pagination” Primer

Eleventy has a pagination feature, which actually does a lot more than pagination: it’s used every time you want to generate several pages from a single template by chunking object keys and using them in permalinks.

One of the most common non-pagination use cases for it is tag pages. The typical /tags/tagname/ page is generated by a deceptively simple template:

---
pagination:
  data: collections
  size: 1
  alias: tag
  filter: ["blog", "all"]
permalink: /blog/tags/{{ tag }}/
override:tags: []
eleventyComputed:
  title: "{{ collections[ tag ].length | pluralize('post') }} on {{ tag | format_tag }}"
---

{% set taglist = collections[ tag ] | reverse %}
{# ... Loop over taglist here ... #}

That was it, then you just loop over taglist (or collections[ tag ] | reverse directly) to template the posts under each tag in reverse chronological order. Simple, right? But what about the indices? As it currently stands, visiting /blog/tags/ will just produce a 404.

Index of all tags

Creating an index of all tags only involves a single page, so it does not involve contorting the pagination feature to mind-bending levels, like the rest of this post. However, we need to do some processing to sort the tags by post count, and remove those that are not “real” tags.

There are many ways to go about with this.

The quick and dirty way

The quick and dirty way is to just iterate over collections and count the posts for each tag:

<ol>
{% for tag, posts in collections %}
	<li>{{ tags.one(tag) }}
		({{ posts.length }} posts)
	</li>
{% endfor %}
</ol>

Unfamiliar with the tags.one() syntax above? It’s using Nunjucks macros (there’s a {% import "_tags.njk" as tags %} earlier in the template too). Macros allow you to create parameterized templates snippets, and I’ve come to love them during this migration project.

The problem is that this does not produce the tags in any particular order, and you usually want frequently used tags to come first. You could actually fix that with CSS:

<ol>
{% for tag, posts in collections %}
	<li style="order: {{ collections.all.length - posts.length }}">
		{{ tags.one(tag) }}
		({{ posts.length }} posts)
	</li>
{% endfor %}
</ol>

The only advantage of this approach is that this is entirely doable via templates and doesn’t require any JS, but there are several drawbacks. First, it limits what styling you can use: for the order property to actually have an effect, you need to be using either Flexbox or Grid layout. But worse, the order property does not affect the order screen readers read your content one iota.

Dynamic `postsByTag` collection

To do it all in Eleventy, the most common way is a dynamic collection, added via eleventyConfig.addCollection():

config.addCollection("postsByTag", (collectionApi) => {
	const posts = collectionApi.getFilteredByTag("blog");
	let ret = {};

	for (let post of posts) {
		for (let tag of post.data.tags) {
			ret[tag] ??= [];
			ret[tag].push(post);
		}
	}

	// Now sort, and reconstruct the object
	ret = Object.fromEntries(Object.entries(ret).sort((a, b) => b[1].length - a[1].length));

	return ret;
});

That we then use in the template:

<ol>
{% for tag, posts in collections.postsByTag %}
	<li>
		{{ tags.one(tag) }} ({{ posts }} posts)
	</li>
{% endfor %}
</ol>

Another way is a custom filter:

config.addFilter("taglist" (collections) => {
	let tags = Object.keys(collections).filter(filters.is_real_tag);
	tags.sort((a, b) => collections[b].length - collections[a].length);

	return Object.fromEntries(tags.map(tag => [tag, collections[tag].length]));
});

used like this:

<ol>
{% for tag, posts in collections | taglist %}
	<li>
		{{ tags.one(tag) }} ({{ posts }} posts)
	</li>
{% endfor %}
</ol>

Usually, filters are meant for more broadly usable utility functions, and are not a good fit here. However, the filter approach can be more elegant if your use case is more complicated and involves many different outputs. For the vast majority of use cases, a dynamic collection is more appropriate.

Index of posts by year

Generating yearly indices can be quite similar as generating tag pages. The main difference is that for tags the collection already exists (collections[tag]) whereas for years you have to build it yourself, using addCollection() in your config file.

This seems to come up pretty frequently, both for years and months (the next section):

This is what I did, after spending a pretty long time reading discussions and blog posts:

eleventyConfig.addCollection("postsByYear", (collectionApi) => {
	const posts = collectionApi.getFilteredByTag("blog").reverse();
	const ret = {};

	for (let post of posts) {
		let key = post.date.getFullYear();
		ret[key] ??= [];
		ret[key].push(post);
	}

	return ret;
});

and then, in blog/year-index.njk:

---
pagination:
  data: collections.postsByYear
  size: 1
  alias: year
permalink: /blog/{{ year }}/
override:tags: []
eleventyComputed:
  title: "Posts from {{ year }}"
---

{% import "_posts.njk" as posts %}
{{ posts.list(collections.postsByYear[year], {style: "compact"}) }}

You can see an example of such a page here: Posts from 2010.

Bonus, because this collection is more broadly useful, I was able to utilize it to make a little yearly archives bar chart!

Bar chart of posts by year, with 2012 selected and text above it reading "28 posts"

Index of posts by month

Pagination only works on one level: You cannot paginate a paginated collection (though Zach has a workaround for that that I’m still trying to wrap my head around). This also means that you cannot easily paginate tag or year index pages. I worked around that by simply showing a more compact post list if there are more than 10 posts.

However, it also means you cannot process the postsByYear collection and somehow paginate by month. You need to create another collection, this time with the year + month as the key:

config.addCollection("postsByMonth", (collectionApi) => {
    const posts = collectionApi.getFilteredByTag("blog").reverse();
    const ret = {};

    for (let post of posts) {
        let key = filters.format_date(post.date, "iso").substring(0, 7); // YYYY-MM
        ret[key] ??= [];
        ret[key].push(post);
    }

    return ret;
});

And a separate blog/month-index.njk file:

---
pagination:
  data: collections.postsByMonth
  size: 1
  alias: month
permalink: /blog/{{ month | replace("-", "/") }}/
override:tags: []
eleventyComputed:
  title: "Posts from {{ month | format_date({month: 'long', year: 'numeric'}) }}"
---

{% import "_posts.njk" as posts %}
{{ posts.list(collections.postsByMonth[month], {style: "compact"}) }}

You can see an example of such a page here: Posts from December 2010.

tech
blogs

Migrating Disqus from WP to 11ty

Lea Verou

So I recently ported my 14 year old blog from WordPress to Eleventy.

I had been using Disqus for comments for years, so I didn’t want to lose them, even if I ended up using a different solution for the future (or no comments at all).

Looking around for an existing solution did not yield many results. There’s Zach’s eleventy-import-disqus but it’s aimed at importing Disqus comments as static copies, but I wanted to have the option to continue using Disqus.

Looking at the WP generated HTML source, I noticed that Disqus was using the WP post id (a number that is not displayed in the UI) to link its threads to the posts. However, the importer I used did not preserve the post ids as metadata (filed issue #95). What to do?

Getting the WP post id

My first thought was to add the post id to each post manually, but use a HEAD request to my existing blog to read it from the Link header, possibly en masse. My second thought was, if I can use JS to get it, maybe I can include Disqus dynamically, through JS, after it procures this number. Then I remembered that 11ty can handle any number of different data sources, and combines them all into a single data object. If I could build an index of slug → post id as another data file, I could add a post id via JS in the 11ty config.

My last epiphany was realizing I didn’t need any HTTP requests to get the post id: it was all in the exported sitemap XML, just unused by the importer! Indeed, each <item> included a <wp:post_id> element with the post id and a <link> element with the URL. I tried to open it in Chrome so I could run some JS on it and build the index, but it complained of parse errors. When I fixed them, the tab crashed under its sheer size.

I needed to remove non-relevant data, and I needed to do it fast. All I really needed was the post id, the slug, and the containing <item> element. Since this did not just contain posts, but also other types of content, such as attachments or custom blocks, we also needed to retain <wp:post_type> so we can filter these out. I copied the XML over to a separate file, and run a series of find & replaces in VS Code:

^(?!.*(wp:post_id|wp:post_type|</?item>|</?link>)).+\n (regex) with empty string
\n{3,} (regex) with \n (to remove empty lines)
wp:post with post to remove namespaces and make the XML easier to handle
https://lea.verou.me/ with empty string and </link> with </link> to keep just the yyyy/mm/slug part of the URL

Then added <?xml version="1.0" encoding="UTF-8" ?> at the top and wrapped everything in a <root> element to make it valid XML.

This resulted in a series of <item> elements that looked like this:

<item>
	<link>2023/04/private-fields-considered-harmful</link>
	<post_id>3599</post_id>
	<post_type><![CDATA[post]]></post_type>
</item>
<item>
	<link>2023/04/private-fields-considered-harmful/image-27</link>
	<post_id>3600</post_id>
	<post_type><![CDATA[attachment]]></post_type>
</item>

At this point, we have exuahsted the capabilities of find & replace; it’s time for some JS!

I opened the file in Chrome and ran:

copy(Object.assign({}, ...[...document.querySelectorAll("item")]
	.filter(item => item.querySelector("post_type").textContent === "post")
	.map(item => ({ [item.querySelector("link").textContent]: item.querySelector("post_id").textContent } ))));

which copies JSON like this to the clipboard, ready to be pasted in a JSON file (I used wpids.json):

{
	...
	"2022/11/tag-2": "3531",
	"2023/03/contrast-ratio-new-home": "3592",
	"2023/04/private-fields-considered-harmful": "3599"
}

Some cleanup was still needed, but this was basically good to go.

Adding the post id to the posts

To inject a wpid property to each post, I added a blog.11tydata.js file with the following:

module.exports = {
	eleventyComputed: {
		postUrlStem: data => {
			return data.page.filePathStem.replace(/^\/blog\/|\/index$/g, "");
		},
		wpid: data => {
			return data.wpids[data.postUrlStem];
		}
	}
};

Linking to Disqus

We now have the post id, and we can use it in our template. Adapting the code from the Universal Embed Code, we get:

{% if wpid %}
<div id="disqus_thread"></div>
<script>
    /**
     *  RECOMMENDED CONFIGURATION VARIABLES: EDIT AND UNCOMMENT
     *  THE SECTION BELOW TO INSERT DYNAMIC VALUES FROM YOUR
     *  PLATFORM OR CMS.
     *
     *  LEARN WHY DEFINING THESE VARIABLES IS IMPORTANT:
     *  https://disqus.com/admin/universalcode/#configuration-variables
     */

    var disqus_config = function () {
        // Replace PAGE_URL with your page's canonical URL variable
        this.page.url = 'https://lea.verou.me/{{ postUrlStem }}/';

        // Replace PAGE_IDENTIFIER with your page's unique identifier variable
        this.page.identifier = "{{ wpid }} https:\/\/lea.verou.me\/?p={{ wpid }}";
    };

    (function() {  // REQUIRED CONFIGURATION VARIABLE: EDIT THE SHORTNAME BELOW
        var d = document, s = d.createElement('script');

        // IMPORTANT: Replace EXAMPLE with your forum shortname!
        s.src = 'https://leaverou.disqus.com/embed.js';

        s.setAttribute('data-timestamp', +new Date());
        (d.head || d.body).appendChild(s);
    })();
</script>
{% endif %}

That’s it! This now works and displays the Disqus threads correctly!

Using Disqus on new posts as well

Note that as it currently stands, this will not display the Disqus UI on new posts, since they won’t have a wpid. Even if I switch to something else in the future, Disqus is better than nothing meanwhile (for me – many people would disagree: switching to no comments at all seems very common when people switch to a SSG blog).

So, new posts don’t have a wpid, but they don’t need one either. As long as we pass some kind of unique identifier to Disqus, we have a comment thread. The easiest way to do this is to use the post’s path, e.g. 2023/preserve-disqus for this one, as this is guaranteed to be unique.

We also want to be able to disable comments on a per-post basis, so we need a way to do that.

So instead of dealing with wpid directly in templates, I added another computed property in blog.11tydata.js:

disqus_id: data => {
	let wpid = data.wpid;

	if (wpid) {
		return `${ wpid } https:\/\/lea.verou.me\/?p=${ wpid }`;
	}
	else if (data.disqus !== false) {
		return typeof data.disqus !== "string"? data.postUrlStem : data.disqus;
	}
}

Note that this allows us to pass a custom identifier to Disqus by using a string, disable it by using false, or just get the automatic behavior by using true or not specifying it at all. The custom identifier can be useful if we want to change the URL of a post without losing the comments.

Then, I updated the template to use disqus_id instead of wpid.

What’s next?

I don’t know if I will continue using Disqus. It’s convenient, but also heavyweight, and there are privacy concerns around it.

However, I’m not sure what I would use instead. Any third party SaaS service would have the same privacy issues. Not necessarily now, but quite likely in the future.

I’ve looked into Webmentions, but the end-user experience does not compare to a regular comment system, and it seems like quite a hassle to implement.

Utterances is a really cool idea: it uses GitHub issues as a backend for a comment system. Having myself (ab)used the GitHub API as a storage backend many a times (even as early as 2012), I can see the appeal. This may be a viable path forwards, though I need to verify that GitHub Issues can be easily exported, so that I’m not locked in.

On a similar vein, I really loved Gisqus seems great too: It’s like Utterances, but uses GitHub Discussions instead of Issues. What holds me back from switching to it is that Discussions cannot yet be exported, and I think portability is important here.

People don’t even really use comments much anymore, they post on social media instead. I would have loved some way to simply collect the discussions about the post from various social media and display them underneath, but with API prices getting out of control (1 2), that doesn’t seem feasible.

If there are any options I missed, please let me know in the (Disqus, for now 😕) comments!

tech
blogs

JS private class fields considered harmful

Lea Verou

Reading Time: 2 minutes Today I mourn. What am I mourning? Encapsulation. At least in my projects. As a library author, I’ve decided to avoid private class fields from now on and gradually refactor them out of my existing libraries. Why did I make such a drastic decision? It all started a few days ago, when I was building […]

tech
blogs

JS private class fields considered harmful

Lea Verou

Today I mourn. What am I mourning? Encapsulation. At least in my projects.

As a library author, I’ve decided to avoid private class fields from now on and gradually refactor them out of my existing libraries.

Why did I make such a drastic decision?

It all started a few days ago, when I was building a Vue 3 app that used Color.js Color objects. For context, Vue 3 uses proxies to implement its reactivity system, just like Mavo did back in 2016 (the first one to do so as far as I’m aware). I was getting several errors and upon tracking them down I had a very sad realization: instances of classes that use private fields cannot be proxied.

I will let that sink in for a bit. Private fields, proxies, pick one, you can’t have both. Here is a reduced testcase illustrating the problem.

Basically, because a Proxy creates a different object, it breaks both strict equality (obj1 === obj2), as well as private properties. MDN even has a whole section on this. Unfortunately, the workaround proposed is no help when proxies are used to implement reactivity, so when I tried to report this as a Vue bug, it was (rightly) closed as wontfix. It would not be possible to fix this in Mavo either, for the same reason.

I joined TC39 fairly recently, so I was not aware of the background when proxies or private class fields were designed. Several fellow TC39 members filled me in on the discussions from back then. A lot of the background is in this super long thread, some interesting tl;drs as replies to my tweet:

https://twitter.com/LeaVerou/status/1650562320702099457

After a lot of back and forth, I decided I cannot justify using private properties going forwards. The tradeoff is simply not worth it. There is no real workaround for proxy-ability, whereas for private fields there is always private-by-convention. Does it suck? Absolutely. However, a sucky workaround is better than a nonexistent workaround.

Also, I control the internal implementation of my classes, whereas proxying happens by other parties. As a library user, it must be incredibly confusing to have to deal with errors about access to private fields in a class you did not write.

This was one of the saddest PRs I have ever written https://github.com/LeaVerou/color.js/pull/306. It feels like such a huge step backwards. I’ve waited years for private fields to be supported everywhere and relished when they got there. I was among the first library authors to adopt them in library code, before a lot of tooling even parsed them properly (and some still don’t).

Sure, they were kind of annoying to use (you usually want protected, i.e. visible to subclasses, not actually private), but they were better than nothing. I was not joking in the first paragraph; I am literally grieving.

I may still use private fields on a case by case basis, where I cannot imagine objects being proxied being very useful, for example in web components. But from now on I will not reach to them without thought, like I have been for the past couple of years.

tech
blogs

JS private class fields considered harmful

Lea Verou

Today I mourn. What am I mourning? Encapsulation. At least in my projects.

As a library author, I’ve decided to avoid private class fields from now on and gradually refactor them out of my existing libraries.

Why did I make such a drastic decision?

It all started a few days ago, when I was building a Vue 3 app that used Color.js Color objects. For context, Vue 3 uses proxies to implement its reactivity system, just like Mavo did back in 2016 (the first one to do so as far as I’m aware). I was getting several errors and upon tracking them down I had a very sad realization: instances of classes that use private fields cannot be proxied.

I will let that sink in for a bit. Private fields, proxies, pick one, you can’t have both. Here is a reduced testcase illustrating the problem.

Basically, because a Proxy creates a different object, it breaks both strict equality (obj1 === obj2), as well as private properties. MDN even has a whole section on this. Unfortunately, the workaround proposed is no help when proxies are used to implement reactivity, so when I tried to report this as a Vue bug, it was (rightly) closed as wontfix. It would not be possible to fix this in Mavo either, for the same reason.

I joined TC39 fairly recently, so I was not aware of the background when proxies or private class fields were designed. Several fellow TC39 members filled me in on the discussions from back then. A lot of the background is in this super long thread, some interesting tl;drs as replies to my tweet:

https://twitter.com/LeaVerou/status/1650562320702099457

After a lot of back and forth, I decided I cannot justify using private properties going forwards. The tradeoff is simply not worth it. There is no real workaround for proxy-ability, whereas for private fields there is always private-by-convention. Does it suck? Absolutely. However, a sucky workaround is better than a nonexistent workaround.

Also, I control the internal implementation of my classes, whereas proxying happens by other parties. As a library user, it must be incredibly confusing to have to deal with errors about access to private fields in a class you did not write.

This was one of the saddest PRs I have ever written https://github.com/LeaVerou/color.js/pull/306. It feels like such a huge step backwards. I’ve waited years for private fields to be supported everywhere and relished when they got there. I was among the first library authors to adopt them in library code, before a lot of tooling even parsed them properly (and some still don’t).

Sure, they were kind of annoying to use (you usually want protected, i.e. visible to subclasses, not actually private), but they were better than nothing. I was not joking in the first paragraph; I am literally grieving.

I may still use private fields on a case by case basis, where I cannot imagine objects being proxied being very useful, for example in web components. But from now on I will not reach to them without thought, like I have been for the past couple of years.

blogs
tech

Reading List