They’re an upstate New York tradition. READ MORE...

The research field known as cognitive systems engineering (CSE) emerged in the early 1980s as a subfield of safety science. That time period saw a dramatic rise in the use of computer-based automation for controlling safety-critical systems. One of the big ideas of CSE, I would even call it the big idea of the field, is that introducing new automation changes the nature of the work that humans do. It’s not that automation means that humans now have less to do: instead, it means that humans face a different kind of work, some of it being genuinely novel.

It’s in the spirit of CSE that I read with interest the recent Anthropic white paper from Nathan Wilmers, Regina Catipon, Maxim Massenkoff, and Peter McCrory titled How employment is changing at Anthropic. It’s a study of how work is changing inside of Anthropic with the embrace of LLM agents. Here are some thoughts I had reading this.

Attention is all used up

Keeping the agents busy is a challenge: these agents spend more time waiting for human input than they do getting work done. Notably, there’s no discussion in the paper about whether agent utilization is genuinely important to the business or not. But what’s clear from the paper is that humans are the bottleneck. More specifically, they point out that human attention is the bottleneck.

Figure 6 in the paper has a fun graph about how much time it would take a human to simply read all of the output generated by their agents, given the average number of tokens outputted per session, assuming 0.75 words per token, and a reading speed of 250 words per minute. For marketing and comms, it would take 12 hours for a person to read the output generated by a week of agent usage, which is 30% of a 40-hour work week (assuming 40 hour work weeks is naive, I know, but let’s go with it). For legal, policy & trust, it’s 26 hours, which is 65% of a 40-hour work week. For finance and accounting, it’s 43 hours of reading, or 108% of a work week. And for engineering and research, it would take 424 hours(!). I’m not going to compute the work week percentage for this one, but the discriminating reader will note that there are only 168 hours in an entire week. What this means is that there is literally not enough time in a week for engineers to read all of the Claude output, even if they work 24×7.

They’re like, I need my juice box opened and it’s an emergency that my juice box is opened. And that is voiced with the same level of concern as the one who has sliced their finger and is bleeding.

The quote above is taken from an engineer pointing out that not all agent requests for human attention are created equal, but that the agents themselves do a poor job at coordinating with the humans to effectively communicate the level of urgency involved in a request. Here I want to reference a famous CSE paper, now over twenty years old, called Ten challenges for making automation a “team player” in joint human-agent activity, written by Gary Klein, David Woods, Jeffrey Bradshaw, Robert Hoffman, and Paul Feltovich. Two of the challenges mentioned in the paper are:

• An agent must have a model of the behavior and beliefs of the other people involved in the joint activity
• An agent must be able to observe the behavior of the humans and use that information to identify their current states and intentions

These are about how agents need to have a model of the humans they are interacting with, in order to effectively coordinate with them In CSE terms, these agents are not good team players. Because the agents don’t have a good model of the engineer’s current goals and state, they aren’t able to make decisions about when it’s appropriate to vie for the engineer’s attention.

From engineers to managers (but still engineers)

One solution to this deluge is rising automation. But this requires work of a different kind.

The paper describes how the work at Anthropic is shifting to meta-work, more commonly referred to as management. More and more of their engineers’ time is spent on coordinating work in dealing with the agents. The paper has some fun examples of this, including attempts to automate it: an agent that triages requests from other agents, doing performance reviews of agents, writing specs for multi-agent runs, looping one agent to review the output of another agent.

The paper even tries to estimate the amount of time spent on meta-work, as shown in figure 7, which has the fantastic title “Becoming managers of Claudes”. The group with the highest meta-work overhead is infrastructure, at 29%.

Judgment and autonomy: the decline of the junior hire

“the thing that raised eyebrows was the uplift for performance engineers in particular and it basically overnight changed what we were looking for.”

One of the main themes in the paper is a hiring trend: they are hiring fewer junior engineers. However, this drop in junior hiring isn’t related to the fact that seniors have deeper domain expertise than juniors. In fact, they specifically call out that domain expertise is becoming less of an issue for hiring managers. Instead, they note that hiring managers are looking for candidates who exhibit good judgment and are capable of working autonomously. Junior engineers are typically responsible for executing on well-defined tasks, and it sounds like they see have internalized the idea that LLM agents can replace juniors at doing this sort of work. But they aren’t yet able to use LLM agents to replace the judgment of senior engineers, and they don’t yet trust their LLM agents to work at the level of autonomy that a senior engineer works at.

This is both unsurprising and depressing. Senior software engineers don’t spring fully formed from Zeus’s forehead; if the industry stops hiring junior engineers, eventually there won’t be senior engineers around anymore. I’d had a sense that it was getting harder for junior devs to find positions, and this is a data point in support of that theory.

I also wonder if Anthropic will tailor their interviews to focus on assessing the quality of judgment of their senior candidates, as well as their ability to work autonomously. I bet if you asked them today, they’d say “we already do assess that”. But I also bet they don’t just have an explicit “judgment” interview and an explicit “autonomy” interview, that they try to gather this info from the existing interview tropes, like coding and system design. Would be very interesting if they transitioned entirely to focusing on experience for these senior hiring interview rounds, although there’s no evidence in the paper about this happening.

Grab bag of trends

A break from managing the agents typically looks like jumping into a meeting.

Time spent in face-to-face meetings is trending upwards. The authors hypothesize that engineers spend more time on strategic work, and resolving questions of strategy requires more conversation among the humans.

The percentage of non-engineers is dropping at Anthropic. They note that the ratio of engineers to product managers is increasing over time: currently the ratio is seventeen engineers to one product manager. This surprised me: I would have assumed that product managers would be endlessly busy coming up with new product ideas, and so the demand for product managers would be going up relative to engineers rather than going down. Then again, I don’t really know what product managers do at Anthropic.

There is organizational pressure to have work discussions happen in places where the agents can observe. This means having them in public Slack channels, as opposed to private DMs or in unrecorded meetings. They want to ensure that the work is visible so that it is legible to the agents.

Outro

It was fun to see Anthropic make an attempt to observe itself like this. I hope more companies look at how AI is changing the work that their employees do, and what new challenges their employees face. I also hope people discover CSE research and how relevant it is to the age of LLM-based automation.

This video from Practical Engineering was very illuminating to me, a naive person who thought there were only three types of electrical outlets used in the US — the two-prong, the three-prong, and the 240V — and not the real number, which is more than 35 types of electrical outlets! Oh and “two-prong”? Not that plug’s actual name! (via unsung)

Tags: electricity · engineering · video

The Washington State Department of Transportation (WSDOT) will no longer be sending out text alerts for I-90 travel notifications.

I’m using this post to gather together some common threads I’ve noticed after reading write-ups of major cloud software incidents. By cloud software, I’m referring to software-as-a-service (do people even say that anymore? in the cloud. This doesn’t just apply to cloud providers, although it does apply to them as well.

Here’s an outline of the topics in this post:

• problem areas
  • saturation
    • example: databases
  • networking (traffic routing failure)
    • example: DNS
  • security (deny valid access)
    • example: SSL certificates
• essential non-standard changes
  • mitigating an operational issue
  • migration
• essential increase in essential complexity
  • reliability subsystem
  • migration

I think of all of these as omnipresent availability risks: I think these are fundamentally unavoidable, and will be contributing to software incidents until the end of time; or, at the very least, until the end of my own career in software.

There are three general areas that most major incidents seem to fall into: saturation, networking, and security. So, let’s start with those.

Saturation

Saturation is probably the topic I talk about most frequently, both on this blog and elsewhere (e.g.,: the saturation post I wrote for the Resilience in Software Foundation and my saturation talk at the Software Should Work conference). The system becomes saturated when it reaches a limit. That’s a pretty generic description, but there are many limits!

Databases

Many major incidents involve some system component becoming saturated in one form or another. I personally worry about database saturation the most. That’s because it’s difficult to recover from an overloaded production database. In addition, because database systems are such complex beasts, it can be quite difficult to even determine what the specific performance issue actually is. This is why having in-house database operational expertise is critical.

Saturation is an omnipresent risk because the finite nature of resources is a hard constraint in the world that we live in. Eventually, some resource in your system is going to run out.

Example: GitHub Incident, Aug 26, 2026

Networking

While I prattle on endlessly about saturation, not every major incident involves saturation. You can encounter scenarios where all of your internal subsystems are reporting healthy, but from your customer’s point of view, your site is down: they can’t use it. One way this can happen is if your users can’t even reach your site, and that’s where the networking problem area comes in.

A networking problem can lead to packets being misrouted. These requests might be black-holed (i.e., silently dropped), or they might be incorrectly routed to a service that doesn’t have the capacity to respond to all of these requests, in which case you’ve got both a network routing issue and a saturation issue.

DNS

DNS issues are an example of this kind of network-related failure mode. There’s no way those packets are going to make it to their destination if the client can’t even determine which IP address to send them to. And when DNS breaks, that’s what happens.

I bring DNS up because it’s bitten folks enough times that there’s a famous haiku:

More generally, networking is an omnipresent risk because cloud software is inherently distributed, so networking is always a critical service. Now, I don’t work in networking, but from the outside, networking just feels like a dangerous domain to do operational stuff in. The blast radius of a networking issue can be very large. And, because network behavior is inherently distributed, reasoning about the behavior of operational changes is just inherently difficult. Honestly, that’s probably why I don’t work in networking.

And, so, I predict we’ll continue to see networking issues contribute to large-scale incidents.

Example: Buildkite incident, Aug 25, 2026

Security

There’s a fundamental tradeoff between availability and security: availability is about ensuring that the good people can access the system. Security is about ensuring that the bad people cannot access the system. This means that there’s always a risk that a security system designed to prevent bad actors from accessing the system can lead to good actors also being blocked. Consider this scenario: there’s an internal security subsystem that goes unhealthy (possibly due to saturation). Is your policy to fail closed or fail open in the event that this subsystem is erroring? Answering that requires making an availability-security tradeoff.

SSL certificate expiration

Another example of this failure mode, which keeps biting our industry again and again, is SSL certificate expiration. Here you have the behavior of a security system that is preventing legitimate access because the cert wasn’t renewed.

And so, my claim is availability incidents that involve security subsystems will continue to be a thing forever.

Example: Bazel incident, Sep 27, 2025

Essential uncommon changes

Your system is constantly undergoing change. Heck, if you stopped making changes, the system would eventually stop working properly. Now, there are some changes that your org does very frequently. Hopefully, you’re deploying often, flipping feature flags a lot, and so on. But there are other changes that your org has less experience with, because they happen much less often. That means that there hasn’t been as much investment in tooling to support these sorts of changes, and it means that the people making these changes don’t have the same level of expertise as they do with the more common changes. That makes these sorts of changes more dangerous: less mature tooling and less experienced humans.

Mitigating an operational issue

A few years ago, I wrote a post titled a conjecture on why reliable systems fail where I speculated on two common contributors to major incidents. One of those contributors was a manual intervention that was intended to mitigate a minor incident. Now, it may be that you frequently have to do manual interventions to mitigate system issues, in which case you’ll have a lot of experience with those sorts of interventions. But you’ll also be more motivated to put in the engineering effort to automate away those sorts of common issues.

It’s exactly the uncommon issues that require a human operator to intervene to mitigate that are dangerous, because they are uncommon. But they’re essential: there’s a problem in the system, and you need to fix it! But because all practitioner actions are gambles, the manual mitigation carries risk that you could make the problem even worse. And, eventually, this will happen to you.

Example: Azure Regional Outage, Jul 23, 2026

Migration

If you’re at a tech company, unless it’s a start-up, you’ll be dealing with migrations, as old tech gets replaced by newer tech that is better suited to the problems that your org is currently facing. While migrations as a general category are extremely common, each migration is itself a snowflake. This means that the specific details of the migration work is an uncommon change. The work of migration involves making a kind of change to your system that you haven’t made before.

To make things worse, one of the dangers of migration is that, as you go along, you start to build confidence that your changes are safe, but there are actually hidden dangers lurking in the system for the next migration. The confidence in the safety of the work exceeds the actual safety. I mean, you made n-1 changes as part of the migration, and none of those changes had negative consequences. It’s natural to assume that the same outcome will occur with the nth change.

Example: Rogers Network outage, Jul 8, 2022

Essential increase in essential complexity

The late American computer scientist Fred Brooks wrote a famous software engineering essay titled No Silver Bullet where he drew a distinction between accidental complexity and essential complexity. The general idea was that there was some amount of complexity in a software system that didn’t need to be there (accidental complexity) and some amount that was just inherent to the nature of the problem space and solution space and so could not be removed (essential complexity).

Reliability subsystem

We’ve developed multiple techniques to improve the reliability of software systems, including retries, concurrency limiting, autoscaling, automated failover, circuit breakers, health checks, canaries, outlier detection, the list goes on and on. There’s one thing that all of these techniques have in common: they increase the complexity of the overall system! And they do this because they have to increase complexity in order to do their job. This is a consequence of Ashby’s Law, which states that if you want to build a control system that handles more scenarios, you have to increase the complexity of the controller itself.

This means that reliability subsystems result in a complexity trade-off. On the one hand, our system can now automatically recover from failure modes that previously required manual intervention. On the other hand, as we all know, increase in complexity is itself dangerous because it can introduce entirely new failure modes that weren’t there before.

Going back to my conjecture blog post, the second contributor I posited was: unexpected behavior of a subsystem whose primary purpose was to improve reliability. And this is exactly why. Adding reliability subsystems improves the robustness of our system, but it adds essential complexity to our system, which can lead to novel incidents.

Example: OpenAI incident, Dec 11, 2024

Migration

Like all engineers, I’m a big fan of giving the answer “it depends” if somebody asks me a question about whether they should do X or Y. However, if someone came up to me and said, “Lorin, I’m preparing to do a migration at my company, and I’m trying to decide whether to do a big-bang migration or an incremental one”, then I would almost certainly say, “For the love of God, please do an incremental migration!”. Sometimes big-bang migrations are unavoidable, but when given a choice, I’m going to go for the incremental migration as the safer option.

However, when you do an incremental migration, it means that you need to simultaneously support the old system and the new system at the same time while you’re doing the migration. This means that even if the new system yields a net decrease in overall complexity over the old system, while the migration is happening, you’re going to see an increase in system complexity. And that means that you’ll see incidents arise as a byproduct of this increased complexity.

Example: Cloudflare incident, Jul 14, 2025

Incidents are inevitable, so you’d better be ready

To reiterate, I think all of the risks mentioned here are omnipresent: they are fundamental to the nature of cloud software. I don’t think that any of these risks can be eliminated. That’s why I believe so strongly in the value of getting better at incident response. Because, if you prepare, you can get better at dealing with problems that arise as a result of these risks.

Transform and roll out with these highlights from the career of the iconic voice of Optimus Prime. Continue reading…