Fetching Emails in Go With JMAP

I wrote an article a while back about switching my email provider from Gmail to Fastmail. Back when I was trying to figure out which service to switch to, I came across a blog post (written in 2019) from Fastmail about publishing the standard for their JMAP protocol. I knew about POP3 and IMAP just from trying to configure mail clients over the years, but I never implemented anything on the server or client for email.

I still don’t have that much of a reason to use JMAP with anything since 1. I’m not really a heavy email user and 2. there’s already an email app on every device I use. But I was still curious about the protocol for some reason so I decided to build some “small” email features into the website so I can read and archive my email from the admin section.

Here’s what I ended up doing.

About JMAP

Most email providers support both IMAP and some kind of custom API (which each provider promises is the fastest and most efficient API possible) for access to email. Fastmail introduced JMAP as their “better than IMAP” API back in 2014.

Fastmail decided to publish the JMAP specs as a standard to encourage other email providers to use it too. I don’t know of any other services using it yet, but I figured it’s probably still a good thing for me to know anyway. No matter how many different products and services are created for messaging and communication, email is always something to fall back on.

Now I just had to think of something useful/interesting to build.

The Feature

I made a start page for myself a little while ago. If you’re logged in as an admin (I’m the only one that can log in as an admin), it shows most recently updated notes and site notifications and stuff. I have it set to my new tab page in my browser.

Since I already had notifications on the page, I thought it made sense to use Fastmail’s JMAP API to show the latest three emails in my inbox on the page too.

At first I was planning to have the main link for each of the three emails go to the Fastmail web client page for it, but by the time I was done, I also added the ability to display a whole email thread and the inbox itself (with pagination¹).

I figured this would be a good start to building even more useful email related features as the website continues to evolve.

Golang and JMAP

This website is very backend/server driven. I’ve been doing a lot of frontend React/Redux/Vue stuff over the years, and I wanted to switch it up for this project and go back to fully server side rendered pages and minimal client-side/Javascript features. So before I even knew exactly what I was gonna use JMAP for, I knew it would require figuring out how to make JMAP formatted requests from Go (golang).

The libraries page of the JMAP website doesn’t list any for Go, but after all the code I wrote to format and parse both JSON (weather, Steam, and Spotify APIs) and XML (RSS feed) requests and responses, I figured it wouldn’t be too difficult to deal with this the same way. I was mostly right until I actually tried parsing the JMAP responses.

JSON (and XML) Parsing in Go

Both JSON and XML parsing are both done the same way in Go (using the standard library). You describe all the fields of a struct with named attributes (they look like json:"userId"), and then pass both the JSON/XML data and an empty struct to the xml.Unmarshal method and it’ll fill the struct with the parsed data.

So for example, on my About Me page, I use the Steam API to get the list of games. The JSON response from Steam looks like this:

 1{
 2    "response": {
 3        "total_count": 5,
 4        "games": [
 5            {
 6                "appid": 289070,
 7                "name": "Sid Meier's Civilization VI",
 8                "playtime_2weeks": 1015,
 9                "playtime_forever": 60809,
10                "img_icon_url": "9dc914132fec244adcede62fb8e7524a72a7398c",
11                "img_logo_url": "356443a094f8e20ce21293039d7226eac3d3b4d9",
12                "playtime_windows_forever": 20608,
13                "playtime_mac_forever": 0,
14                "playtime_linux_forever": 0
15            },
16            {
17                "appid": 860510,
18                "name": "Little Nightmares II",
19                "playtime_2weeks": 755,
20                "playtime_forever": 755,
21                "img_icon_url": "8875efb23a0fc1f079f227508d981f2cba82d735",
22                "img_logo_url": "19b6e7fc830b2a752891a9a23afdc3fe94dc5d6b",
23                "playtime_windows_forever": 755,
24                "playtime_mac_forever": 0,
25                "playtime_linux_forever": 0
26            },
27        ]
28    }
29}

So I created these structs:

 1type SteamGame struct {
 2    AppID           int    `json:"appid"`
 3    Name            string `json:"name"`
 4    PlayTime        int    `json:"playtime_2weeks"`
 5    PlayTimeForever int    `json:"playtime_forever"`
 6    ImageIconURL    string `json:"img_icon_url"`
 7    ImageLogoURL    string `json:"img_logo_url"`
 8    PlayTimeString  string `json:"playtime_string"`
 9}
10
11type SteamRecentGamesResponseObject struct {
12    Total int         `json:"total_count"`
13    Games []SteamGame `json:"games"`
14}
15
16type SteamRecentGamesResponse struct {
17    Response SteamRecentGamesResponseObject `json:"response"`
18}

And then I can parse the JSON like this:

 1var r SteamRecentGamesResponse
 2
 3res, err := http.Get(url)
 4if err != nil {
 5    return nil, err
 6}
 7
 8defer res.Body.Close()
 9body, err := ioutil.ReadAll(res.Body)
10if err != nil {
11    return nil, err
12}
13
14json.Unmarshal(body, &r)
15for _, game := range r.Response.Games {
16    ...
17}

I started to do something similar for the responses from JMAP, but there’s a problem right from the start. The main data type for the requests and responses is something they call an Invocation. An Invocation is a tuple², of values, which is usually a string, then a JSON object (or map of strings to any JSON value), and then another string. If your HTTP request contains multiple Invocations, you’ll get more than one tuple back, so in JSON, it’s represented as an array of arrays.

Here’s a truncated example of one JMAP request with three Invocations:

 1[
 2    [ "Email/query", {
 3        "accountId": "...",
 4        "filter": {
 5            "inMailbox": ""
 6        }
 7    }, "0" ],
 8    
 9    [ "Email/get", {
10        "accountId": "...",
11        "#ids": {
12            "name": "Email/query",
13            "path": "/ids",
14            "resultOf": "0"
15        }
16    }, "1" ],
17    
18    [ "Thread/get", {
19        "accountId": "...",
20        "#ids": {
21            "name": "Email/get",
22            "path": "/list/*/threadId",
23            "resultOf": "1"
24        }
25    }, "2" ],
26]

The first tuple is for the Email/query request, the second for Email/get, and the last one is for Thread/get. The second value for each tuple is a JSON object/attribute map that act as the arguments for the call.

The result for a call like this would contain three Invocations in response:

 1[
 2    [ "Email/query", {
 3        "accountId": "...",
 4        "ids": [
 5            "..."
 6        ],
 7        "queryState": "123:0",
 8        "total": 10
 9    }, "0" ],
10
11    [ "Email/get", {
12        "accountId": "...",
13        "list": [
14            { "id": "...", "threadId": "..." }
15        ],
16        "state": "123",
17        "notFound": []
18    }, "1" ],
19
20    [ "Thread/get", {
21        "accountId": "u123456",
22        "list": [
23            { "id": "...", "emailIds": [] },
24        ],
25        "state": "123"
26    }, "2" ],
27
28]

It’s a pretty good format that allows you to combine requests and have Invocations reference each other (the third tuple elements are Invocation IDs you provide for reference; see the resultOf attribute in the last request Invocation above).

Unfortunately there isn’t a great way to describe the tuples in these response types in Go.

Go’s type system

As you can see from the Steam API example above, Go is strongly typed. Before you can do anything with a variable, you have to know/describe its type. Arrays are always lists of one specific type of data. That type might be the “anything” type interface{}³.

So it’s possible to declare an array of strings, an array of attribute maps, or an array of “anything”, but not a strongly typed Invocation tuple.

Here’s an example struct with some fields one might think to use:

1type ExampleStruct struct {
2	StrArray []string
3	AttrMapArray []map[string]interface{}
4
5	AnythingArray []interface{}
6}

The first two fields, StrArray and AttrMapArray, wouldn’t work at all in this case (it would fail to parse). AnythingArray would work, but a type assertion would be needed on each element before use (which is a problem with the AttrMapArray too). Here’s an example of what that might look like:

 1var s ExampleStruct
 2
 3// The first value might be something like "Email/get"
 4strValue := s.AnythingArray[0].(string)
 5
 6// Response values as an attribute map
 7mapValue := s.AnythingArray[1].(map[string]interface{})
 8
 9// The list of emails in that attribute map
10list := mapValue["list"].([]interface{})
11
12// The string ID of the first email in the list
13id := list[0]["id"].(string)

But it’s not good enough. It doesn’t even feel strongly typed anymore. The compiler can’t really check anything here, and it would be very easy to get any of the types wrong. It also looks weird.

I wanted a solution that’s at least somewhat type-safe and actually takes advantage of the features of Go.

Here’s what I came up with:

Requests and Responses

First, I made Invocation and APIRequest structs. The second element of an Invocation is a interface{} type. That value is always either a map[string]interface{} type or a JSON ready struct.

APIRequest is the struct I use to json.Marshal the entire request body. It describes the list of Invocations to send as an array of interface{} arrays.

1type APIRequest struct {
2	Using       []string        `json:"using"`
3	MethodCalls [][]interface{} `json:"methodCalls"`
4}

APIResponse is similar:

1type APIResponse struct {
2	SessionState    string          `json:"sessionState"`
3	MethodResponses [][]interface{} `json:"methodResponses"`
4}

Marshalling and Unmarshalling the Response

When I json.Unmarshal the response body, I get the Invocation responses as expected, but they still need type assertions. My solution was to add methods onto the APIResponse object to (1) find an Invocation by name (the first element of each of the MethodResponses arrays) and then (2) json.Marshal the value back into a byte array and then (3) json.Unmarshal that value back into the actual struct it should be.

 1func (r *APIResponse) GetEmailResponse() *EmailResponse {
 2	for _, response := range r.MethodResponses {
 3		// (1) Type assertion for the name of the invocation
 4		name := response[0].(string)
 5		if name == "Email/get" {
 6			var r EmailResponse
 7			// (2) Marshal from interface{} back to byte array
 8			value, err := json.Marshal(response[1])
 9			if err != nil {
10				logger.Logf("error marshalling json: %v", err)
11				return nil
12			}
13			// (3) Unmarshal to the correct struct (EmailResponse)
14			json.Unmarshal(value, &r)
15			return &r
16		}
17	}
18	return nil
19}

End Result

So now making a request looks something like this:

 1// FetchEmail returns an email thread.
 2func FetchEmail(id, threadID, accountID string) ([]*Email, error) {
 3	calls := []*Invocation{
 4		{
 5			Name: "Thread/get",
 6			Value: GenericJSON{
 7				"accountId": accountID,
 8				"ids": []string{
 9					threadID,
10				},
11			},
12			ID: "0",
13		},
14		{
15			Name: "Email/get",
16			Value: GenericJSON{
17				...
18			},
19			ID: "1",
20		},
21	}
22
23	r, err := Request(calls)
24	if err != nil {
25		return nil, err
26	}
27
28	value := r.GetEmailResponse()
29	return value.List, nil
30}

The Value field for the Invocations that are passed to Request() are interface{} values, but using a map[string]interface{} type (which is what GenericJSON is) works. The compiler (and my editor) knows that the value variable returned from r.GetEmailResponse() is a *EmailResponse type, and that value.List is a []*Email type.

I like how this turned out so far. Adding other email functionality like marking threads as read, archiving/deleting mail, and adding caching (a start page is supposed to load as quickly as possible of course⁴) were all easy to add on top of this base.

I don’t feel like I have to go much further with this since I can do everything I need to do with email from my computer, phone, and watch, but it’s nice to know it wouldn’t be too hard to write an entire email client into this website if I wanted to.