« Roger Federer is Houdini, Congrats to Roddick, Too | Main | Singing The Paul Krugman Blues »

July 06, 2009

Is Tennis' Hawk-Eye Infallible?

I'm a big fan of Tennis' computerized instant-replay system. Whatever else you say about the invisible powers that run Tennis, they've done a much better job than other sports of implementing new technology to review calls. 

Not only has reply made calls more accurate, it has all but eliminated the silly tantrums spoiled tyrants such as John McEnroe and Jimmy Connors used to disrupt games and cheat their rivals. Replay has also speeded games (although much of the benefit is lost to prodigious stallers like Rafael Nadal, Novak Djkokovic et al). 
ReplayTennis Plus, personally, I get a lot of amusement from watching inflated self-promoters such as McEnroe and Brad Gilbert repeatedly proved wrong when they arrogantly announce from broadcast booth that, "He's just hoping. That was obviously out. Oh, it was well in.", and "That was a horrible call. Chalk flew. He has to call for a review. Oh, it was half-a-foot wide, but hit the baseline." How many times have their ilk gotten fans steamed over nothing before replay? How many times did they vulgarly insult officials when they were wrong? 

Still, I seriously doubt that Hawk-Eye (formerly known as Shot Spot) is remotely as accurate as its promoters' claim.

At this year's Wimbledon, the male players challenged 314 calls, but only 29.62 percent were over-turned. This shows what a great job the lines-people do. Granted, some challenges are throw-aways, made simply because the point is big and it's near the end of the set. But, players only challenge very close calls and their objections are wrong more than twice as often as they are right. 

Yet on many of the most critical calls, whether the ball was in or out was determined by something close to the margin of error the manufacturer apparently claims. I say, apparently, because there is some ambiguity there. From an ESPN report

On Hawk-Eye's Web site, an analysis of  the disputed call (a replay Roger Federer criticized) states that it was "likely" that the ball was in by 1 millimeter (0.04 inches). 

Paul Hawkins, managing director of Hawk-Eye technology, says the line-calling system has gone through more than 1,000 tests. "We've gotten every single one of the tests correct," he said. ...

Hawkins said that Hawk-Eye's margin of error averaged about 3.6 millimeters (0.14 inches) and that the system was around 99.9 percent accurate.

"Hawk-Eye isn't infallible, but it's pretty damned close," he said. ... 

According to the ITF criteria, any electronic line-calling system must be able to judge a ball in or out within 5 millimeters (0.20 inches). Incorrect calls are allowed, so long as they are not more than 10 millimeters (0.40 inches) off.

"On no occasion have we said that this technology is perfect," said Stuart Miller, head of science and technical issues at the ITF.

Miller said that accompanying Hawk-Eye's rulings with a disclaimer that the system's reconstructions were only a best guess of what happened would only confuse the public.

So, Hawk-Eye is actually accurate to somewhere between a tenth and four-tenths of an inch. A full 0.4 inches in either direction seems like a big error margin to me. It is arguable whether that is more accurate than looking at a ball mark. 

Put aside the potential for human error: In a Murray-Ljubicic match the umpire, who has to pick the item to replay on a hand-held device, erroneously chose the second impact point of the ball on a point where the ball landed out on the first bounce, and on the second bounced landed nearby, but in. That may have cost the Bosnian the match.  

Let's try to boil down what Hawk-Eye does without diving into too much boring technology. Hawk-Eye is basically a computer simulation. Multiple cameras track the flight of the ball, triangulate its location and create a simulation of the path. 

Then, here's what I think is Hawk-Eye's vulnerability: It does not capture the impact point on camera, it creates a simulation of what that would look like. If you've looked at ball marks on your own court you know how much variation there is. Some are wider, other longer, depending on the velocity, angle and spin. 

On the call Federer disputed, a critical point that cost Fed a game against Nadal in a Wimbldon final, the ball was going slowly through the air, then came down almost straight down because of the top spin. The TV replay appeared to show the ball landed out to my eye, but it was not definitive. Hawk-Eye showed a long mark, with a long, thin tail that touched the line. It isn't logical that a slow moving, vertical impact would leave a long streak, which is more characteristic of a fast, flat shot. 

I have no idea what their algorithm is, but I suspect it incorrectly picked or calculated the shape of the ball's impact on the court. That leads me to believe Hawk-Eye was wrong and adversely affected that match. 

Regardless, Hawk-Eye is not infallible and should not be presented as such by the TV announcers. 

Replay should be used, not as the ultimate arbiter, but as a tool for the chair umpire. Back to the ESPN article: 

Collins and Evans said that while what Hawk-Eye achieves is remarkable, its use in tennis needs to be refined. They're right. 

"It should be used like a spell-checker on your computer," Collins said. "It's not right all the time, but it's a useful adviser."

Wimbledon 09 Tennis Challenge Stats

Reader Comments

Miller said that accompanying Hawk-Eye's rulings with a disclaimer that the system's reconstructions were only a best guess of what happened would only confuse the public.

Verify your Comment

Previewing your Comment

This is only a preview. Your comment has not yet been posted.

Your comment could not be posted. Error type:
Your comment has been posted. Post another comment

The letters and numbers you entered did not match the image. Please try again.

As a final step before posting your comment, enter the letters and numbers you see in the image below. This prevents automated programs from posting comments.

Having trouble reading this image? View an alternate.