Who’s in charge? The fraught union of automation and human behavior

Automation is ubiquitous, from the autopilot in airplanes to the cycles of the washing machine in your home. And with the modern advent of artificial intelligence, computers and machines are taking on ever more tasks.

That doesn’t necessarily mean human jobs go away, but they change — from doing the task itself to supervising the machine doing it, says Ron McLeod of Glasgow, Scotland. A retired specialist in the field of “human factors,” McLeod spent his career studying how human brains and automation work together — or fail to — in industrial settings like oil refinery control rooms. McLeod wrote a book, published in February 2026, on this topic: Transitioning to Autonomy: The Psychology of Human Supervisory Control.

At work, AI, robots and automation offer potential gains, but also limitations, as explained by Harvard University sociologists Ya-Wen Lei and Rachel Kim in the Annual Review of Sociology. AI tools, they note, might be opaque or just plain wrong, and human workers can’t always override their decisions, leaving employees disengaged and frustrated.

Transitioning to supervision brings challenges both technological and psychological, McLeod posits. The human mind gets bored and distracted and isn’t always ready to spring into action if the machine needs help. But that’s only part of it. Systems, be they control rooms or cockpits or automatic sports referees, have to be designed to work effectively with humans, not to cut them out of the loop. And engineers and designers don’t necessarily understand how to balance the sensors and actions of automated systems with the needs and abilities of the people keeping watch.

When supervision of automation goes wrong, either in the moment or in the earlier design or training stages, the results can be disastrous.

Knowable Magazine spoke with McLeod about the past, present and future of automation supervision as it evolves in the AI age. This interview has been edited for length and clarity.

What inspired you to write this book?

Since I was an undergraduate student in the late 1970s, I’ve been interested in the applied discipline of human factors. It’s a combination of psychology, engineering, sociology and other sciences. We try to understand how humans perform in association with technical systems, and how we can support them best.

As part of this work, I was aware of supervisory control; I taught people about it as part of my role at Shell International. But the first time I really experienced that transition for myself was in 2022, when I took ownership of a car that could drive itself.

I got no training from the company. They gave me the keys, showed me how to use the indicators (or turn signals) and turn the radio on, but they told me absolutely nothing about the automation. It was only over the next few days, as I tried to study the manual and in-car support, that I realized how this option was supposed to work.

But I was still confused. There’s a motorway near my home, it’s 70 miles per hour, and there’s a tight bend where it goes down to 40. I know that’s coming — does the car know that’s coming? If it’s icy, is the car going to slow down? I don’t know. I have to be constantly paying attention, making these mental judgments: Do I intervene, or don’t I?

The problem is that automation is really reliable, and there’s not much to do. The brain isn’t engaged in the task. And it’s difficult to pay attention, to acquire information, to understand what’s happening, if you’re not mentally engaged in the task.

That transition period into the supervisory role, which is mentally demanding, is what interests me. Manufacturers need to start getting this stuff right. That’s what drove me to write the book.

How does AI complicate the supervision of automation?

Automation may or may not include AI. AI certainly brings new elements.

There are four possible levels at which to automate something: acquiring information, making sense of information, making decisions and taking actions. AI can be applied to any of those. Generative AI, for example, is mainly in the domain of acquiring information.

There’s an intriguing piece of work on AI published recently by Steven Shaw and Gideon Nave at the Wharton School of the University of Pennsylvania. They built on the two styles of thinking as defined by the late psychologist and Nobel laureate Daniel Kahneman in his book Thinking, Fast and Slow.

System 1 is intuitive; it takes no effort. System 2 is slow, taking time to think things through. Shaw and Nave set up an AI that gave wrong answers, and more than half the time, people just accepted it. They suggest that when you’re using AI as a tool, there’s this third kind of thinking, and they call it “cognitive surrender” — where you just give up your thinking and let AI tell you the answer.

I’m most interested in cases where the automation and the person are controlling physical things in real time, which means the human has the chance to supervise and, if they think things are going a bit awry, to intervene. This is happening all over the place.

We’ve talked about self-driving cars. Where else might supervision of automation become an issue?

In industry and manufacturing. I spent 10 years working in oil and gas, and they have a long history of automating the control room.

And anything about the control of vehicles. Whether it’s cars or ships or planes, they’re all going through the same process.

Another domain is healthcare: for example, where AI will be helping radiologists and they may not have the confidence to overrule it. I recently met an anesthesiologist who said he was originally trained to manually monitor a patient’s life signs and apply anesthetics, but nowadays, it’s all automatic. He just sits there and monitors numbers.

And in surgery, they’ll also be bringing in automation. Today, as I understand it, the surgeon still has control, but we’re not far away from the authority to perform an operation being given to some technology. And surgeons will have to go through the same transition, from manually performing surgery to being supervisors of the technology, with exactly the same issues as other cases of automation.

What kind of history informs your understanding of the challenges in supervising automation in diverse industries?

In 1979, with the Three Mile Island meltdown, that was the first time these human factors issues were really taken seriously.

This meltdown resulted from a series of failures in the plant’s cooling system, but a major contributor was a badly designed control room. A pressure-relief valve got stuck open, but the indicator in the control room showed it was closed. The operators had no indicator at all for the level of water covering the reactor core, because this level was not supposed to change.

With incomplete and inaccurate information, the controllers thought there was a risk of flooding the system, so they shut off the emergency water. The core became uncovered and overheated. Fortunately, no one was hurt.

If they’d done nothing, the plant would have safely shut down automatically. Initially, the operators were blamed, but it was later realized that the operators did perfectly logical things, based on the information they had. So that was a first real eye-opener about some of the psychological complexity.

Another example was the loss of the Air France Airbus that crashed as it was going from Rio de Janeiro to Paris in 2009. There were a number of human-factor problems, including cockpit design that didn’t give the pilots necessary information as well as ineffective pilot training and emergency protocols.

I found several cases like these. Again and again and again, when systems aren’t easy to use, things go wrong and people make mistakes. The company leaders that install automation assume the design will be easy based on common sense, but that isn’t always the case.

What other specific issues come up when people are supervising systems that are mostly automated?

One of the core issues is about the balance of authority between the people and the automation. Who’s actually in control?

This came up during the tennis championships at Wimbledon in 2025. This was the first time Wimbledon gave full authority to an automated line-calling system, called Hawk-Eye, to decide if balls were “in” or “out.”

It was the fourth round of the ladies’ singles, Anastasia Pavlyuchenkova versus Sonay Kartal. Kartal hit a ball that was clearly out — but the automation did not call it as out. The players could see it, the umpires could see it, the whole world watching on TV could see it was out, but the umpire didn’t have the authority to overrule the automated system. The umpire made the pair replay the point, and Pavlyuchenkova lost that game, though she went on to win the match.

It turned out to be human error: The system had accidentally been turned off. The authorities were very embarrassed, and their knee-jerk reaction — rather than acknowledging that they were dealing with a complex system that relied on human behavior — was to give a blanket reassurance that they had made changes to prevent the same error. But another miscall happened the next day. They hadn’t recognized the right lesson: The system is not 100 percent reliable, so you still need the people.

This is a common issue: Under what conditions is it allowable for humans to intervene with the automation, or for the automation to overrule humans? In the car, for example, when the autopilot is on, if I press my foot on the brake, it cuts out and I’m back in control. It lets me “over-vene.”

What should carmakers be doing to ease this transition for drivers?

So in my car, as soon as I take my hands off the wheel, I’m in that supervisory role. I need different information. I don’t need to know what gear I’m in; I don’t even need to know what speed I’m doing. I need to know, who’s in control, me or the car? And I need to know, what is the car thinking, can it see an upcoming threat?

At the very least, whether it’s driving a car, or working as doctor, or supervising a control room, I can’t think of any reason why there shouldn’t be an online training package that will take you through the basic information and put you through a few scenarios.

For cars specifically, the National Transportation Safety Board in the United States recently held a meeting following investigations of two fatal 2024 collisions in which Ford Mustang SUVs in partial automation mode hit vehicles that were stopped on the highway. In both cases, the Ford drivers were impaired or distracted from supervising the SUV’s automated driving. The NTSB recommended automakers install systems to reduce drivers becoming disengaged or complacent with automation.

These systems would be monitoring the human supervisors that are supposed to be overseeing the automated driving. There has been a lot of research and development put into products for tracking their state of alertness. And they do that, for example, by monitoring things like eye movements, head movements.

A related example is that for many years, the rail industry in some countries relied on a “vigilance” device to ensure the train driver has not fallen asleep, given how boring their task is. It consists of an audible alarm that sounds every 30 seconds. The driver is required to press a button to cancel the sound. Pretty low tech, but seemingly effective in making sure the driver is awake!

And not everyone is going to have the cognitive skills that you need to drive as a supervisor. Should there be something in the driving test about, do you have the mental capacity to make those judgments? I don’t know, but there should be more research there.

What else can be done to help human supervisors stay engaged and be effective in other automated situations?

The core issue here isn’t actually that difficult. It’s just about good design. If you realize you are changing people from a manual role to a supervisory role, then on the system end, you design the user interface based on understanding: What is the human’s task? What information do they need? How do we optimize that?

In the case of my car, the interface was designed for manual drivers, not for me as a supervisor. Part of the problem was that there was so much information on the screens, I didn’t know where to look. I didn’t know what the symbols meant. There was a lot of text, a lot of it abbreviated. Some of it, I couldn’t read it anyway, because it was in a location I couldn’t see.

Stay in the Know
Sign up for the Knowable Magazine newsletter today

For the user, it’s important to consider training, incentive schemes, workload and competing tasks. Companies must ensure people who are supervising automation are trained so that they understand what is involved in their supervisory role: for example, what kinds of signs or information they should be looking for and where that information is, and what could happen if things went wrong.

And, for example, not having incentives that discourage the supervisor from making the best decisions. When I was working in oil and gas, there were a number of cases where people in control rooms had intervened to shut down or slow down production — which has a dollar value attached to it. And they’d actually done the right thing, but they were blamed for it.

When you add in the AI dimension, there are even more potential issues, like the cognitive surrender. We still need people — it’s just about how to make sure they’re supported. It isn’t rocket science; it’s just about recognizing the complexity of this new role.