What we learned from MLB’s spring robot-umpire test: Players, managers, execs weigh in

CLEARWATER, Fla. — Do you miss those robots yet?

We’ve spent the past five weeks watching them pop out of hiding, every time a catcher, hitter or pitcher tapped his head to ask his favorite robot umps: Where the heck was that pitch?

Then, almost instantaneously, a six-second animation would roll on the ballpark videoboard — and baseball’s spring training experiment with the Automated Ball-Strike (ABS) challenge system would work its magic.

So now that it’s time to send those robots back from whence they came — namely, the minor leagues — as spring training fades into the rearview mirror, how’d that experiment go? That’s what we’re here to tell you, because over the past month, we’ve pretty much seen it all.

We’ve seen walks turn into strikeouts.

We’ve seen strikeouts turn into walks.

We’ve seen strikeouts turn into home runs.

We’ve seen fans boo those poor, well-meaning robots.

And in the midst of it all, Alex Cora revealed his worst robotic nightmare.

“My first thought,” the Red Sox manager said this spring, “was, like: Bases loaded … 3-2 count … ninth inning in the World Series … tie game.”

We’ll pause here to let you cue that imaginary 4K video in your brain. Now … here comes that pitch.

“Yeah!” Cora bellowed, pumping his fist. “Strike three! (A smile. A shake of the head.) No! Ball four! That’s where my mind went. And I was, like: Oh, shoot!”

OK, hold on. Don’t shoot. This was only a test. No World Series games were played this spring. It was only spring training, with an electronic ball-strike challenge system running in baseball’s test lab. It was all just for feedback purposes. We won’t see this technology in real games before next year at the earliest — and probably longer.

But that scene in Cora’s brain could happen someday, if — OK, let’s say when — this challenge system gets unleashed in games that count. So is that what we want? Is that what baseball wants? We’ll discuss those questions shortly.

First, though, we’re here to answer all your big questions — not to mention all the big questions you forgot to ask. So what did we learn from watching those robot umps this spring? Let’s discuss!

Does the technology work?

Technology is a beautiful thing. You cue it up. You program the robots. And it does what it’s built to do.

A strike is a strike. A ball is a ball. And if you don’t like how the humans called that pitch, you challenge — and the robots end the debate.

Major League Baseball declined The Athletic’s request for specific data from this spring. But it has spent four years honing the ABS system in the minor leagues — and before that in the independent Atlantic League.

It has tweaked the definition of the strike zone multiple times. It experimented with different ways to match the height of the zone to the height of the players. And after all those tweaks, the league was comfortable that the technology was ready for its midterm exam.

“We have made a lot of progress in the way the system works,” MLB’s vice president of on-field strategy, Joe Martinez, said at a media-demonstration session last month, “and also the way we weave the system into the gameplay. And we’re at a point in Triple A where we have a system that the players like, the coaches like, the umpires like and the fans like.”

So this was the logical next step — to find out whether big leaguers liked it, but also to get feedback on what they didn’t like. I can help with that, because I’ve been asking the same questions all spring.

I thought the most interesting responses came from catchers, because they lived in Robot World on both offense and defense.

“I like it as a hitter,” said the Phillies’ J.T. Realmuto. “I don’t really like it as a catcher as much, just because I think it takes part of the game away, part of the catcher position. Framing is still going to matter, but it’s not going to be as big of a deal. So that part I don’t like. But as a hitter, I do like having the consistent strike zone.”

Tigers catcher Jake Rogers also thought that as the ABS strike zone establishes its presence in the sport, hitters will be the biggest fans — but not for the same reason.

“There are going to be a lot more balls called than normal, I think,” Rogers said, “because the strike zone is a lot smaller.”

Since veteran major-league umpires were told by the league just to call their normal zone, to get ready for a season without ABS, players found themselves living in a world with two different strike zones this spring.

“With ABS, I think the bottom of the zone drastically changes,” Rogers said. “It was a lot higher, and I think the top goes a lot lower. … So on the pitches that are close to the edge, the umpires are just going to call a ball and rely on us to challenge.”

But remember, the robot umps don’t know any of that. They just call the balls and strikes they’re programmed to call, whether Laz Diaz would call them the same or not. And hey, that reminds us of something important:

Big-league umpires are better at this than you think.

FanGraphs’ Davy Andrews pointed me toward Statcast data on Baseball Savant that shows how much ball-strike calls — by human umps — have gotten better in recent years. Here’s a look, in five-year increments.

(Source: Baseball Savant)

In 2024, that accuracy percentage actually ticked down — from 92.81 percent in 2023 to 92.53. Nevertheless, there were over 28,000 more correct calls last season than in 2008, the first year of available data. So let’s give those humans a big hand. That works out to over 1,000 more correct calls a week!

So is it worth using technology to chip away at the 7.5 percent or so that human umps are still getting wrong? That might depend on the answer to our next question …

Does the strike zone feel like the strike zone?

What’s a strike?

For almost 150 years, that has been the question that defined baseball. Hasn’t it? It all starts with the strike zone.

Too bad the strike zone you’ll find in your rule book isn’t the same thing as the strike zone that has been called by humans for pretty much that entire century and a half. Essentially, human umps call an oval-shaped zone — no matter how many times they’ve seen that rectangle in the rule book.

So why did baseball just spend spring training fiddling with a system that will not be used in a single regular-season game this year? It was all about the robot-ump strike-zone experience — “to get people some reps of experiencing the system,” Martinez said, “and seeing how it feels, particularly how the strike zone feels.”

In other words, it all comes down to this:

Every hitter, catcher and pitcher has an idea in his head of what a strike is and what a ball is. So for ABS to work — really work — the electronic strike zone has to feel essentially like the zone baseball players have in their heads.

You know what won’t work? If that zone feels just like some sort of technological creation.

So which was it this spring? Uh, let’s just say it’s a work in progress.

Rogers said that Tigers players had a chance to speak to commissioner Rob Manfred in person this spring. Guess what they asked about most when ABS came up? The strike zone.

“That’s the big thing we talked about with Rob,” Rogers said. “It becomes about: What is the strike zone? It’s the oldest rule in baseball. And then, when you start kind of tweaking it a little bit, it becomes different.”

Here’s what confused players the most, he said. As part of the ABS education process this spring, players were given an iPad that allowed them to review every pitch of a game, to see if the robot umps would have called it a strike or a ball. But they continued to get their usual postgame reports, looking at whether those same pitches were in or out of the zone — and they weren’t the same.

“It’s crazy,” Rogers said, “because on ABS, you look at the iPad … and (the pitch is) half an inch below the zone. And then we get our report back with the old strike zone, and it’s a full ball in the zone. So it’s like, wow, it looks like a strike. It feels like a strike. And all of a sudden, you’re thinking: Do you challenge, or do you not challenge? So you go back and look at it, and it’s a ball (on ABS).”

There’s a reason for that — even beyond the different technologies involved. Human umps are still being graded with a buffer zone that gives them wiggle room if a pitch is barely off the rule-book strike zone. But robot umps don’t know a buffer from a muffler. So a ball is either in the zone or out. Period.

Is that good? Is that what we all want? You won’t be surprised to learn that one player with some thoughts on that topic is Max Scherzer.

In his recent appearance on the Starkville podcast with me and Doug Glanville, Scherzer said one thing he’d like to see is “a buffer zone, maybe around the challenge system. So hey, if you challenge and it’s in the buffer, the call stands. So you keep human power, the human element, still with the umpire.

“I’m OK changing the call when it’s an egregious call,” the Blue Jays’ future Hall of Famer said. “But when we’re talking about a quarter of an inch that you can’t really detect it, I don’t necessarily know if that makes the game better.”

But when I mentioned Scherzer’s idea to an executive from an American League club, he swatted it away like a mosquito. What’s the point of honing the best technology, he wondered, if we’re then going to ignore it by dropping in a buffer zone?

“I think they’ve done enough research on it to come up with the right zone,” he said. “So I don’t think there needs to be a buffer zone. I think this ABS zone is very clear, very definitive. So I would not be in favor of a buffer zone. It’s just, that’s the strike zone. You’ve got to draw the line somewhere.”

What the powers that be would say is: What’s the difference between this and tennis? If a ball lands a 16th of an inch outside the line in the U.S. Open finals, you know what they’d call that — using the same Hawk-Eye technology? Out.

But when I ran that logic past one player this spring, he said: “This is not tennis. Tennis doesn’t have a catcher, right? There’s a reason why.”

Did fans really boo the robots?

Boooooo.

Wait. Did I just hear what I thought I heard? It was a Feb. 28 game between the Phillies and Red Sox. Phillies center fielder Brandon Marsh had just taken a 2-and-2 fastball, slightly off the inside corner, from Red Sox ace Garret Crochet. So the count was full. Or was it?

Boston catcher Seby Zavala tapped his helmet. The robot-ump challenge cartoon played on the videoboard — and turned that ball three into strike three, much to the delight of …

Nobody watching this in the Phillies’ home ballpark, apparently — at least judging by that sound: Boooooo.

An inning later, there were still more boos when Realmuto and Red Sox prospect Marcelo Mayer challenged back-to-back pitches in the same at-bat, and both were called balls by the robot ump, handing Mayer a five-pitch walk.

“Did I really hear those fans booing the robot?” I asked Realmuto later.

“Well, it’s Philly, so there’s no telling what they were booing,” he said. “I think they were booing the batter for challenging. I know that they booed that. And then they also booed when I challenged and it ended up being a ball. So yeah, I guess they were booing the robot.”

How cool is it to win a challenge? The thrill of victory!

A minor-league manager told me last year that there was a lot more riding on these challenges than balls and strikes.

“It’s like a baseball IQ test for the players,” he said. “You really find out who knows the strike zone and who doesn’t.”

So what did we learn about baseball IQ this spring? That catchers (of course) are baseball geniuses, the group that got the most challenges correct … and pitchers, um, not so much. But the hitters? According to preliminary data, they were somewhere in the middle.

That isn’t all we learned, though. For the hitters who consistently aced their challenges, that thrill of victory was the coolest thing ever.

Heading into this spring, Orioles outfielder Dylan Carlson didn’t figure he’d challenge much. But that was before he turned into the MVC (Most Valuable Challenger) of the whole darned spring.

In a span of a week and a half, he challenged four called third strikes — and got all four overturned. Two of them came in one game against the Twins. Three of them were on 3-2 pitches, so they magically transformed strikeouts into walks. And the fourth was on a 2-2 pitch, which canceled a strikeout and led to another walk.

And after that, he said with a laugh, “They kind of gave me the green light to use it whenever I wanted.”

So Carlson went into the final weekend of spring training with more than twice as many walks (eight) as strikeouts (three). If this hadn’t been The Spring of the Robot Ump, he’d have had more strikeouts (seven) than walks (four). And his on-base percentage would have been over 100 points lower (at .351 instead of .459).

You think that guy is a fan of this robot-ump thing, or what?

“For me personally,” he said, “just the swings — going from strikeouts to getting on base — a lot of those were leading off innings. So it kind of changes the whole inning in some ways. You get a leadoff runner on, right? Then it’s go time. So it’s been interesting, for sure.”

All spring, I’ve watched players challenge, then turn their attention to the ABS cartoon on the videoboard. They couldn’t have been paying closer attention if that board was telling them whether they’d just won 10 million bucks. So I asked Carlson to describe the feeling — of paying rapt attention to that animation and then finding out he was right … again.

“So actually, every time I’ve challenged,” he said, “I’ve asked the catcher too: ‘Hey, was that a strike?’ And every time they’ve all said yes. So I’ve been a little nervous looking up at the board. But fortunately, they’ve all gone my way.”

Which beats the alternative, obviously. Speaking of which …

How brutal is it to lose a big challenge? The agony of defeat!

Even in subdued spring training, real humans are playing. And sometimes, their future is riding on these games. So when you’re a pitcher, on the fringe of a big-league roster (or hoping to get there someday), and you lose a challenge, is that just a “get-the-call-right” moment? Or is it more?

Yankees manager Aaron Boone thinks it can often be more. There is always an emotional component to these games. And just last week, he saw a game change on a challenge — and a swing of emotion.

It was the sixth inning of a game against the Red Sox. Right-hander Geoff Hartlieb was on the mound. Pitching with a two-run lead, Hartlieb got the first two hitters out, then thought he’d dotted the outside corner with a 2-and-2 fastball to Kristian Campbell.

The home-plate ump, Roberto Ortiz, pumped his fist. Strike three. Campbell had other ideas and tapped his helmet. The ABS animation rolled. The robots ruled the pitch was a millimeter off the plate. So this inning wasn’t over. Want to guess what happened next?

On the next pitch, Campbell walked. Then the next hitter, Masataka Yoshida, pumped a game-tying homer. Boone was steaming.

“Strike three, pitcher walking off the mound,” Boone grumbled the next day. “Oh. Challenge. Overturned. So it’s like, you’ve got a pitcher getting out of an inning. Makes a pitch. Walking off. Boom. Oh, no. Next pitch, home run.

“I just don’t like it, honestly,” he said. “I feel like the umpires are getting so good — and look, I know I’m the poster child, sometimes, for arguing — but literally, sometimes I’m arguing when they’re missing by, like, (a fraction of an inch). But I feel like more and more, these umpires are really good. And just the frivolous challenge — like the 1-1 pitch in the second inning. It’s stop … ball … challenge … 2-and-1 … no wait, 1-and-2. I just don’t like it.”

So, because he brought this up …

A moment of silence for managerial ejections

There once was a time when I would have paid to go to the ballpark and watch Lou Piniella, Bobby Cox or Don Zimmer get booted from a game by their favorite men in blue. Old-school baseball didn’t get more entertaining than that.

But now? We’re in serious jeopardy of never seeing another manager ejected from any game again — if those robot umps take over the world.

We’ve already gotten a preview of where this could be heading, thanks to instant replay. As recently as 2011, there were nearly 50 ejections across MLB solely because of calls on the bases, according to research by Retrosheet founder Dave Smith. Then replay arrived in 2014 — and by 2019, there was only one ejection, over a call on the bases, all season.

So the good old-fashioned managerial ejection tirade was already in trouble even before the looming arrival of ABS in any form. Ejection legend Bobby Cox once got thrown out of 11 games in one season. Dave Roberts has been managing for a decade — and has 12 ejections in his whole career.

According to Retrosheet, more than 70 percent of all ejections these days revolve around ball/strike “disagreements.” So what happens in a world where we suddenly start getting all the “big” ball/strike calls right? Will any of these managers ever get heaved out of a game again?

“I only get ejected once or twice a year anyway,” said the Tigers’ A.J. Hinch. “But you know, it’s funny. All the veteran managers have always told me: ‘Your job is to manage the game, not manage a game from your desk in your office.’ So yeah, you’re right. It’s getting harder and harder to get ejected. I mean, what do I argue about?”

But you’ll be heartened to know that Boone — the self-professed “poster boy” for arguing, and a guy who has led his league in ejections four seasons in a row — is not ready to concede his tirade days are over.

If this happens, I told him, he’ll never get thrown out of another game.

“That’s not true,” he promised. “I’ll find something.”

Phew!

What will broadcasters talk about?

You think it’s easy talking about baseball games into a microphone for three hours every night? It’s a great gig, but that doesn’t mean it’s easy. So perhaps you’ve noticed that there’s a certain topic that’s pretty much guaranteed to come up every darned game. By which I mean …

C’mon. That pitch is not a strike/ball. How could he miss that pitch?

Have you ever wondered how many broadcast hours the work of the home-plate umpire has consumed over the past 50 years? Might be a million hours. Possibly a billion. Does Statcast track that stuff? It should.

But now imagine how different the life of the average broadcaster might be if some sort of technology was invented that would get every ball-strike call right — or at least the biggest ball-strike calls right. I can assure you that thought occurred to many a broadcaster this spring … when that technology suddenly arrived.

So I sought out Larry Andersen, the always-entertaining radio color analyst for the Phillies, a guy who has been known to express a few (affectionate) opinions about men standing behind home plate wearing blue jackets.

The Athletic: “I don’t want to jump to any conclusions, but I’ve listened to you call games a few times. You don’t seem like you’re normally that happy with the umpires’ work beyond the plate. Would that be fair?”

Andersen: “That would be fair. I would say I’m not. But this is going to sound crazy, because they drive me nuts, but it’s gotten worse since they put the (strike-zone) box on TV.”

That, he said, is because now that there’s a depiction of the strike zone on the screen, it’s hard not to notice when a pitch misses that box “by 6 inches” and still gets called a strike. So guess what? He might mention that.

But Andersen also isn’t convinced (with good reason) that those TV boxes are totally accurate. Whereas the ABS rectangle is going to be basically 100 percent accurate, even if it might not correspond exactly to how humans call balls and strikes. So back to our original question.

TA: “So if we suddenly got every ball/strike call right in this sport, what would you talk about all night?”

Andersen: “Well, it would probably turn my wrath onto the players. And I don’t want to do that.”

TA: “Right. At least when you’re ripping the umpires, you’re a man of the people, because the people are with you on that. So if you weren’t ripping the umpires every night, what would happen?

Andersen: “It would really put a damper on my broadcasting career.”

So let’s ask this again, in a different context: Is that what you want? You know where to find me if you have some thoughts on that.

Are we sure this is what we want?

All right, let’s end where we began — with Alex Cora’s Game 7 dream/nightmare.

Game 7 of the World Series. Two outs in the ninth. Bases loaded. Here comes the 3-2 pitch. It pops the mitt. Strike three. You just won the World Series. Or did you?

Here comes the challenge animation. Was that pitch really 1-78th of an inch off the plate? According to ABS, it was. So don’t pop that champagne yet. This game is tied – thanks to those emotionless robot umps.

“I think most people would say: Well, it’s a ball, so it should be called a ball, and if that’s what determines the World Series, you should still call it,” said an executive of one contender.

He then rolled back the clock two springs and brought up the arrival of the pitch clock. Didn’t we hear the same stuff about the pitch clock? Would we really let a postseason game end on a clock violation?

“So what you described could be a possibility,” he said. “But I think the likelihood of that happening is pretty low.”

Then again, it’s not zero. Do we really want a World Series decided by a pitch that’s literally the width of a hair off the plate? I asked that question of an AL exec. He swatted it away like a piece of lint.

“Maybe just get the call right,” he said. “I mean, that’s not the ideal use of the system, but I heard the same argument when we went to replay. Hey, play at the plate. It’s bang, bang. The umpire rules him out. But we go to a replay. We wait. No, he’s actually safe. Game’s over. You win the World Series. What’s the difference? It’s the exact same thing.”

Is it, though? Of course, most want to use technology to make the game better. Of course, we want to get as many calls right as possible. Why wouldn’t we?

But I’ve spent all spring listening to people wonder whether we’re really making the sport better by using technology to decide ball/strike calls that are so close, the human eye can’t even detect them. It all depends, said another AL exec, on what the true goal is.

“I think the functionality of the system is great,” he said. “And it has worked seamlessly. But I do wonder a little bit: What are we trying to accomplish?”

When the sport first began testing ABS, he said, one of the big goals was to try to create a fixed strike zone that could reduce the strikeout rate and create more balls in play. Great idea. Never got there. No matter how the league adjusted the zone, no one ever figured out a variation that made more action happen.

“So that was the initial intent of the automated strike zone,” the exec said. “It has now morphed to (something else.) So I’m wondering: Is the focus now to try to get as many calls right as possible? Because if that’s really the goal, is it worth it? I’m not quite sure.”

You know who else is asking that question? Max Scherzer.

“Go back and look at the (2024) postseason,” Scherzer said on Starkville. “Are we really talking about (anything that) happened with the home-plate umpires and strikes or balls? No, I don’t think so. So what problem are we really solving?”

He, too, has seen the data that shows home-plate umpires are more accurate than ever. So use this technology, he said. Just use it to grade the umpires and make them even more accurate. But don’t fix what isn’t broken.

“We’re not saying there’s a problem,” Scherzer said. “We said in the postseason, there wasn’t a problem. Do we really need to be trying to change a fundamental part of the baseball experience? Pitching? Catching? I don’t think so. So that’s where I’m skeptical. That’s where I think, as an industry, we just need to have a conversation.”

But guess what? No matter how that conversation goes, it’s not going to keep the robot umps from invading one of these years. So let’s embrace those robots — and even boo them just to see if robots have feelings. But between now and the day they arrive for good, let’s also ask that fundamental question:

What’s the true goal here? What are we trying to accomplish?

Technology is awesome. Robots are the future. And right calls are better than wrong calls. But is the sport truly better off if a World Series gets decided on a pitch 1-78th of an inch outside a robotized strike zone? The answers are so much harder than the questions.

(Top photo: Christian Petersen / Getty Images)