Cloudflare CEO Matthew Prince:

The issue was not caused, directly or indirectly, by a cyber attack or malicious activity of any kind. Instead, it was triggered by a change to one of our database systems’ permissions which caused the database to output multiple entries into a “feature file” used by our Bot Management system. That feature file, in turn, doubled in size. The larger-than-expected feature file was then propagated to all the machines that make up our network.

The software running on these machines to route traffic across our network reads this feature file to keep our Bot Management system up to date with ever changing threats. The software had a limit on the size of the feature file that was below its doubled size. That caused the software to fail.

After we initially wrongly suspected the symptoms we were seeing were caused by a hyper-scale DDoS attack, we correctly identified the core issue and were able to stop the propagation of the larger-than-expected feature file and replace it with an earlier version of the file. Core traffic was largely flowing as normal by 14:30. We worked over the next few hours to mitigate increased load on various parts of our network as traffic rushed back online. As of 17:06 all systems at Cloudflare were functioning as normal.

We are sorry for the impact to our customers and to the Internet in general. Given Cloudflare’s importance in the Internet ecosystem any outage of any of our systems is unacceptable. That there was a period of time where our network was not able to route traffic is deeply painful to every member of our team. We know we let you down today.

This post is an in-depth recount of exactly what happened and what systems and processes failed. It is also the beginning, though not the end, of what we plan to do in order to make sure an outage like this will not happen again.

Everything about this incident exemplifies why Cloudflare is one of my favorite companies in the world. Ideally, it wouldn’t have happened, but shit does happen. Among the things to note about Cloudflare’s response:

• They identified and fixed the issue quickly.
• They issued frequent updates to their status site while the incident remained ongoing.
• They published this postmortem within 24 hours. (That’s remarkable, given the technical breadth of the postmortem. Publishing this tomorrow, within 48 hours of the incident, would have been a praise-worthy accomplishment.) Update: Actually, according to Prince, commenting on Hacker News, the postmortem was published less than 12 hours after the incident began. Amazing.
• The postmortem starts with a cogent, well-written layperson’s explanation of what happened and why.
• The postmortem expands to include very specific technical details, including source code.

Lastly, it’s worth noting that Prince put his own name on the postmortem (and wrote much of it himself, using BBEdit), and closed with this apology, taking personal responsibility:

An outage like today is unacceptable. We’ve architected our systems to be highly resilient to failure to ensure traffic will always continue to flow. When we’ve had outages in the past it’s always led to us building new, more resilient systems.

On behalf of the entire team at Cloudflare, I would like to apologize for the pain we caused the Internet today.

This is how it’s done.

The New York Times:

The world’s richest man. One of the world’s most famous soccer players. The president of soccer’s governing body. Dozens of executives from the finance, tech and energy sectors. These are some of the guests who attended President Trump’s black-tie dinner for Crown Prince Mohammed bin Salman of Saudi Arabia at the White House on Tuesday evening.

The red carpet welcome for Prince Mohammed is an extraordinary moment in diplomatic relations with Saudi Arabia. It is his first visit to the United States since the 2018 killing of the Washington Post columnist Jamal Khashoggi, which U.S. intelligence determined the prince ordered. Prince Mohammed has denied involvement.

Yours truly, back in August, after Tim Cook’s Oval Office gift of gold to Trump:

It is disturbing to think that the leader of a beloved, trusted, and widely believed-to-be-ethical company like Apple has succumbed to avarice. That Tim Cook feels no qualms about — or perhaps even delights in — participating in a quid-pro-quo-driven corrupt administration in which flattery, fealty, gifts, and barely-concealed bribes are rewarded. That the United States devolving into kleptocracy suits Tim Cook just fine, because Apple’s pockets are deep enough to pay the vig.

But the alternative is more disturbing.

What if Tim Cook is, in fact, strong, proud, and driven by a keen sense of moral and ethical clarity? Perhaps Cook declined Trump’s invitation to join his Middle East entourage in May only because he was otherwise busy. But I believe there are bridges he will not cross — and that trip, especially its implicit and explicit praise and sanctification of the Saudi regime in general, and MBS in particular, was one of them. The whole trip was grotesque, and made a mockery of traditional American values.

MBS being feted in the White House is even more grotesque.

See also: Karen Attiah, who in her previous job as editor of The Washington Post’s global opinion section hired Jamal Khashoggi after he was exiled from Saudi Arabia, in The Guardian: “The Saudification of America Is Under Way”.

Special guest Dan Frommer returns to the show. Topics include the indie media business, the iPhone Pocket, the iPhone Air (including rumors about the second generation model), AI “personalities”, and five years of Apple Silicon Macs. Also, six years of Dan’s site, The New Consumer.

Sponsored by:

• Factor: Healthy eating, made easy. Get 50% off your first box, plus free breakfast for 1 year, with code talkshow50off.
• Notion: The AI workspace where teams and AI agents get more done together.
• Squarespace: Save 10% off your first purchase of a website or domain using code talkshow.

Jonathan V. Last, writing for The Bulwark:

The president of the United States gave a speech yesterday before a group of McDonald’s corporate workers and franchise owners. I’m going to quote a few sections of his remarks at great length, because if you have not listened to Trump speaking recently, the decline in his cognitive abilities is a bit shocking.

The point of this exercise is not to clown on Trump, but to give everyone a baseline understanding of where he is, with the mentals, as we try to understand how he will respond to increasing pressures in the coming months.

The video of his remarks is here and I’ll include timestamps for each section, in case you want to see what he looks and sounds like.

Bottom line: This is a man in noticeable mental decline.

The whole world is inured to listening to Trump speak like this — it sounds normal even to those of us who see that Trump is spiraling into dementia. Ah, that’s just Trump being Trump. But it’s not normal. Reading the transcript is jarring. Also jarring: listening to Juliet Jeske’s performative reading of a particularly nonsensical portion of Trump’s McDonald’s speech. It’s just gibberish.

President Donald Trump, today in the Oval Office alongside his “very good friend” Saudi crown prince Mohammed bin Salman, after ABC News reporter Mary Bruce had the temerity to pose a question regarding Mohammed having ordered the brutal murder of Washington Post columnist Jamal Khashoggi in 2018:

People are wise to your hoax. ABC, your company, your crappy company, is one of the perpetrators. And I’ll tell you something, I think the license should be taken away from ABC because your news is so fake, and it’s so wrong. And we have a great commissioner, a chairman, who should look at that.

Also from Trump, regarding Khashoggi’s murder:

A lot of people didn’t like that gentleman that you’re talking about. Whether you like him or didn’t like him, things happen.

Things happen, indeed.

Disney (ABC’s owner), a year ago settled a lawsuit Trump filed against ABC News — a lawsuit most experts agreed ABC would have won — for $16 million, in what clearly amounted to a bribe.

Yours truly, last month, in a post on Apple’s capitulation to Trump regarding the ICEBlock app for iOS:

When you give a bully your lunch money, they always come back for more.

I think Bob Iger gets that now. “Fuck you, make me” remains the correct response to these threats.