Nieman Foundation at Harvard
The Vikings were losing, again. It was late in the fourth quarter and Minnesota was down 10 against the Denver Broncos and a cardboard cutout of Peyton Manning. This is a familiar scenario to me, one I have trouble dealing with objectively. So when the Vikes decided to go for it on 4th and 1, I had no small amount of terror climbing up my back.
Clearly, I should have been listening to The New York Times’ 4th Down Bot, who knew the Vikings had a 67 percent chance of getting the first down. And after they converted, they went on to score.
They still lost the game. Some things bots still can’t fix.
But like any third-year NFL veteran, 4th Down Bot has gotten a little smarter about its game. This fall, the team behind the bot has made some upgrades to the way the system reads games and makes calls on the field. The bot is now set up to make faster calls, take better account for the power of underdogs, and do more to predict the behavior of one of the most important players on the field: the kicker.
“The older model was built taking advantage of the past 15 years of data, but it wasn’t accounting for a lot of the improvements that kickers have made,” said Josh Katz, a graphics editor on the Times Upshot team who helped reboot the bot.
Yes, that’s right, kickers, seemingly the least important player on the bench. The position you likely pick next to last on your fantasy team. It’s become one of the most important jobs on the field as kickers have become increasingly accurate. Ben Morris, writing in FiveThirtyEight, sums it up:
For all the talk of West Coast offenses, the invention of the pro formation, the wildcat, 5-wide sets, the rise of the pass-catching tight-end, Bill Walsh, the Greatest Show On Turf, and the general recognition that passing, passing and more passing is the best way to score in football, half the improvement in scoring in the past 50-plus years of NFL history has come solely from field-goal kickers kicking more accurately.
Now the bot takes into account the improved accuracy and distance of kickers, but also additional variables like the stadium (indoor vs. outdoor) and weather forecast (clear skies or blustery).
The 4th Down Bot now also accounts for the relative strength of a team against given opponents. Specifically the bot looks out for underdogs, giving a one-touchdown underdog a 22 percent chance of winning at the start of the game. That makes a difference to the bot because underdogs may be more likely to be aggressive on fourth-down plays, said Trey Causey, a data scientist who worked with the Times on upgrading the 4th Down Bot.
The secret? Gamblers. The bot’s new team strength predictor is based on the point spread of individual games, Causey told me. “You can’t do much better than hundreds or thousands of people who have made their thoughts about who they think is going to win available,” he said.
The trickiest part of the updates to the 4th Down Bot was speeding up its play-calling ability. This fall they introduced “live calls,” where the bot makes a suggestion before a team makes its fourth-down play. As Causey explains, the bot runs through the likelihood of success or failure on a punt, field goal, or going for it and makes a call right after the third down is complete.
It’s 4th-and-4 for the Steelers from the Chargers’ 42. I would go for it.
— NYT 4th Down Bot (@NYT4thDownBot) October 13, 2015
But the live calls also introduce an element of risk for the bot. There are some variables the 4th Down Bot can’t account for, like penalties or coaches’ challenges that change a team’s field position. The live calls are automated, meaning the bot could base its call off incomplete information before the next snap of the ball.
Causey and Katz say they’re still testing out the live calls feature and can turn it on or off during the season. “I think it’s fun,” Causey said. “I also think it gives good insight into how fast these decisions have to be made and how much of the process is done on the fly.”
The upgrade to the bot came about partially because they wanted to refresh the project, but also because Brian Burke, who created the original model the bot uses, went to work for ESPN.
“Brian going to ESPN forced us to change some things about how we were running the bot,” Katz said. “But at the same time, there were a few aspects of the bot we wanted to change.”
Those changes also presented a new opportunity to give other people ways to improve the 4th Down Bot by making the system open. Well, open with a small caveat: The code responsible for the bot’s prediction model is now available on GitHub, but it will only really work with a subscription to NFL data from Armchair Analysis.
Do you like @NYT4thDownBot? Want to make it better? Submit a pull request:
https://t.co/YscGODYat4
— Josh Katz (@jshkatz) October 1, 2015
Katz said the Upshot team likes to make code open if there’s no reason, legal or otherwise, not to make it available online. (One example of the benefits of being open: Developer Ben Dilday has already forked the bot’s code to create a version that works with freely available NFL play-by-play data.)
In the case of the bot, putting the code on GitHub is a way of being transparent about how the system works to all the people who have come to use it.
“It turns something from what would be a black-box mystery to: Here’s exactly where these numbers come from,” Katz said. “It’s not this mysterious, mystical, process. It’s just math.”
