A couple of weeks ago I posted an article looking at whether we are smarter than ESPN when it comes to selecting our starting lineup for Fantasy Football. Today, I want to update and expand upon that post. Specifically, I am adding in FantasyPros's expert consensus ranking and Reddit Consensus Rankings as compiled by u/blackjack30000 (I also want to shout out u/eatmoreartichokes who provided half-PPR fantasy pros data that I am working on getting enough ESPN leagues to compare to). So today we can explore across different ranking sources and see if there is one ranking that is better than others.
I am also correcting one of the limitations that was a common complaint from the first article. As I mentioned in the limitations section, using public ESPN leagues has some issues as people tend to drop out. While this wouldn't matter for comparing across the different ranking systems, it does matter when comparing against the average fantasy manager (whose performance is artificially worse due to managers dropping out). To account for this, I marked any lineup where a player was started while being projected by ESPN for 0 points to be a bad lineup. This would normally indicate the player was on bye or injured and would be indicative that the fantasy manager did not pay attention that week (yes this is not perfect, but should account for the worst of it). So for all following analysis, I excluded any team that had a bad lineup from all weekly analysis and any team that had 2 or more bad lineups from all season long analysis (we all have a bad week from time to time).
I collected Espn lineups from leagues with full-PPR scoring (to match the fantasy pros and Reddit Consensus rankings) in 2019. After excluding bad lineups, I ended up with 5722 unique weekly lineups and 1081 unique teams for season-long analysis. (Note: I am surprised by the number of teams that were excluded when accounting for bad lineups and does suggest that the issue of dropout teams was larger than I thought). For each team, I calculated how many points they scored if the starting lineup was determined by the coach, ESPN, Fantasy Pros, and Reddit's Consensus rankings, respectively, as well as what the optimal lineup would have scored in a given week. Because Reddit rankings only include QB, RB, WR, and TE, I only included lineup slots that you could play one of those positions in the starting lineup (even though the starting lineup was determined by each league's settings beyond that, allowing some teams started 2 WRs while others started 3 WRs as per their league rules for instance). There will be some overall inaccuracy due to this (as maybe one ranking source is better at K or DST than the rest), but I think given the volatility of these positions, the gist of this article still holds.
Weekly Score Differences
The first thing to look at is how these ranking systems perform on a weekly basis. Are they scoring more points than your average fantasy manager?
Boxplots of weekly point differences between what the fantasy manager did and what different ranking sources would have done. The red line marks the level of equivalence to the fantasy manager and the black x's mark the mean of each group.
As we can see, ESPN outperforms both the fantasy managers and the other ranking sources. By the numbers, ESPN produced a mean of 3 points more than the fantasy manager per week though no change with the median (Note this is less than the mean of 6 and median of 1.5 points I described previously, likely due to bad lineups and not taking K and DST into account). This is better than Fantasy Pros (mean of -1.7 points, median of 0) and Reddit Consensus (mean of -3.05, median of -0.3). Though it should be noted that none of these are optimal, which suggests fantasy managers miss a mean of 20 points and a median of 17.7 points from their optimal lineup.
Overall, I was surprised to see ESPN beating out fantasy pros (which by integrating a wide range of experts I figured would be more accurate). Also, we can see the number of fantasy managers that are just following ESPN's suggestions in the plot above (given that the median and 25 percentile are both essentially 0 (no difference between the selected lineup and ESPN's suggested lineup), whereas we get a full range above and below the coach's decision for fantasy pros and Reddit Consensus rankings.
Coaching efficiency
Another way to look at the effectiveness of different rankings is to score each weekly score as a percentage of the optimal lineup. Coaching Efficiency is therefore how close to optimal the lineup was (with 100% being perfect and anything below 100% being points missed).
Boxplots of Coaching Efficiency, black x's represent the mean of each group.
Here we can see as expected that espn's lineup is the most efficient (mean of 87.8%, median of 86.4% efficiency). Fantasy managers (with a mean of 85% and median of 83%) is a little worse while the Fantasy Pros (mean of 80%), and Reddit Consensus rankings (mean of 79%) are worse still. This is in line with the data above (as expected because this is essentially the same metric just presented differently) and continues to support ESPN.
Billing Department
Season-long wins
As I outlined in the last post, points are good but at the end of the day, it is wins that matter. I got some pushback on this point after the last post so I want to elaborate. On a player by player basis, you are trying to get the most points, that is absolutely true. Additionally, given that you cannot stop your opponent from scoring, points are the metric that you can control. However, most leagues do not compete for who scores the most points, we compete for who has the most wins. Because there is some luck that comes into play when determining wins (see my post on luck here), if the team didn't have a chance to win, it does not matter which ranking system got you the best team, you still lost. So while I do think looking at points is useful (which is why I post it above), I think the be all end all is still win.
So let's look at the differences in terms of wins across the entire season
Boxplots of wins over the fantasy manager. The red line would be equal to the fantasy manager and the black x's are mean values for the group.
Again we can see that ESPN seems to outperform the fantasy manager, fantasy pros, and Reddit consensus. Specifically, ESPN outpaces the individual fantasy managers by a mean of 0.2 wins over the course of the season (though the median is the same as the fantasy manager). Similar to the weekly point differences, this is less than the 0.9 wins above coach I previously reported, which I again chalk up to a combination of removing the bad lineups and not taking defense and kicker into account. Surprisingly, neither Fantasy Pros nor Reddit Consensus rankings are able to match the fantasy manager (actually costing the team 1.7 and 1.8 wins respectively over the course of the season. Additionally, none of these options are optimal, as the optimal lineup could have secured 2.9 wins more than the fantasy manager (assuming the opponents did not also optimize their lineups.
So is ESPN (and are we) really better than Fantasy Pros and Reddit Consensus?
I am not so sure. ESPN has an advantage over the other ranking systems, and that is personalization. The ESPN projections are altered by the scoring settings of each individual league, while the fantasy pros and Reddit consensus rankings are generic to all leagues. I am not convinced on an even playing field, ESPN's projections are a full win better than fantasy pros or Reddit consensus. But at the same time, we normally do not interact with these things on an even playing field (I know fantasy pros has an option to import your league with its scoring settings, but not everyone does this and this is not an option for Reddit's rankings). If anything, this shows the value of the fantasy manager over these systems by taking the information and adjusting them to match their own league settings.
Do better managers trust ESPN more?
Finally, we can look at how well people follow ESPN vs where they finish as before.
Boxplots broken up by standing of how the coaches lineup compared to ESPN's lineup. Notice this is flipped from previous graphs and shows that the coach underperforms ESPN on average (the red line represents ESPN's score).
Interestingly, unlike last time where players who finish lower followed ESPN less, this seems to suggest that the numbers are about the same regardless of finish. Importantly though, it does not suggest that you need to beat ESPN to finish top and in fact, 75% of first place finishers would be better if they followed ESPN. So while the strength of my advice from last time may have changed, the message has not, follow ESPN's projections and spend your energy elsewhere.
In Conclusion
While the effect is less than I previously reported, once again, ESPN comes out on top. Even if this is due to the personalization, it is also still outperforming the coach (even after accounting for bad lineups). Therefore I am sticking with my recommendation to follow espn's projections and spend your energy on other aspects of Fantasy Football.
I will be more clear this time though that ESPN is not perfect. ESPN still leaves 17 points per week on the table, costing a team about 2.5 wins over the course of the season. This is better than the average fantasy manager but does leave room to try to beat ESPN. Additionally, some of the fun of fantasy football is selecting your lineup and I do not mean to take away from that, you can always beat ESPN but the average fantasy manager does not do so over the entire season.
I will continue to follow ESPN's suggestions, whether you do the same is up to you.
Questions? Comments? Let me know at ac@alexcates.com. Want to read more breakdowns like this? sign up for my newsletter. Finally, like what I do? Consider supporting me on buy me a coffee.