One MTGA Player Conceded Hundreds of Matches to Probe the Ranking System – Here are the Results.

MTG: Arena player and Twitter user @Hareeb_AlSaq made waves on social media over the weekend by publishing a long and extremely detailed report about how they used multiple Arena accounts in an attempt to reverse engineer the game’s ranking system- an opaque system which has had very little official explanation on how it actually works.

It has come to my attention that some people have noticed a research project I've been running for awhile. I have information on the k-value for Bo1 and Bo3, how it changes through the month, info on the #1500 rating throughout the month, the pairing system, and more 1/2.

— Hareeb al-Saq (@Hareeb_alSaq) May 23, 2021

Misleading and outright false. I've also discovered several clear flaws in their implementation that affect both struggle-to-diamond and T-1200 contenders. Hopefully my work will lead to a better system for everybody soon. A report will be out by tomorrow. 3/3

— Hareeb al-Saq (@Hareeb_alSaq) May 23, 2021

This announcement along with al-Saq’s published report (language warning) has not landed without controversy, with some players accusing al-Saq of making the game worse for players- both by instantly conceding so many matches, and also by exposing how the current system is highly exploitable. Conversely, al-Saq claims that they are only interested in improving how the ranking system in MTGA works, and that they hope their report may lead to improvements to the current system. Regardless of the motivations, this report makes a number of claims which seem to correspond with past sentiments from high playtime players including well-known streamers and MPL members. The full report is fascinating, but also lengthy and filled with mathematical jargon, so we have made an attempt to break it down to its primary points.

Before we proceed, it must be stated that al-Saq’s report is entirely unverified and NOT backed up by anything other than anecdotal evidence from other players. This report has not been verified or endorsed by Wizards of the Coast in any capacity, nor is it verified or endorsed by the MTGA Zone team.

A rating system exists and is used to pair players not just in Mythic, but in the lower ranks as well.

This is something that has been reported by other players who have noticed it in the past, as al-Saq themself mentions in their report. The reason that this is purported to be problematic or unfair has to do with the rank floors. If you play ranked ladder on Arena, then you already know that once you achieve a new rank (Silver, Gold, Mythic, etc), you cannot lose that rank until the season resets. However, if there is indeed a hidden rating system that is used for pairings, it exists separately from the shown ladder system. Therefore, if a player concedes or otherwise loses a large number of matches at the very bottom of a rank, their rating will continue to drop and drop while still staying at the same visible rank at the very bottom. After throwing a large number of games, the player would have a low enough rating that they will be paired against other players who are actually at a lower skill level, as opposed to starting out with a middling rating and facing equally skilled opponents. Theoretically, they would then have a much easier time climbing the ladder to the next rank since they’re still starting at the same place on the visible rank ladder even though their hidden player rating is much lower.

Interestingly, this hidden player rating is something that has been addressed by Wizard’s of the Coast in a State of the Beta post from 2018. WotC refers to the player ratings as MMR, or Matchmaking Rating. They even specifically reference the idea of players losing intentionally: “…it also creates a bit of a ‘safety net’ against players who are willing to sacrifice their Limited Ranking (or vice versa) to lower their MMR and ensure “easier” matches in their format of choice.” The report written by al-Saq directly disputes a few of the points in this official post from Wizards, especially the idea the MMR shields against intentional losses as it seems to do the opposite.

A player’s performance before entering Mythic has little impact on their placement upon entering Mythic.

In their experiment, al-Saq alleges to have found that their two burner accounts entered Mythic at roughly the same percentage level, even though the first account played matches normally, and the second “conceded hundreds of matches at Diamond 4 before trying to win.” According to al-Saq, this supposed fact, paired with a recent change to Arena that has allowed more players to get to Mythic, has caused issues that include the well-known severe rank decay at the end of the month.

Starting with the Ikoria ranked season, wins in the Gold rank award two pips on the ladder instead of one. This has caused more players to reach Mythic, and based on al-Saq’s findings, this has caused the players entering Mythic in the late season to have an over-inflated player rating when compared to the players who have already been in Mythic since they enter at roughly the same player rating. This causes the early-season Mythic players to “farm” the newcomers, who are presumably weaker players despite having similar player ratings. This then contributes to the rank decay at the end of the month, as more lower-skill players enter Mythic and are facing against the high skill, high playtime players who can achieve high winrates in those matches.

The K-value for Best-of-Three matches appears to be very high, almost the same as it was in high level tournament play of the past.

In ranking systems that are based on the well-established Elo system, the K-value refers to a mathematical constant that is applied to the ranking change for each match. It is used to essentially determine the maximum change in a player’s rank for each matchup, which helps to calibrate the effect that the most recent games played by that player have on their ranking over time. If al-Saq’s research and conclusions are correct, the K-value for a best-of-three match on Arena is somewhere around 45, while the K-value for best-of-one is only around 20. Wizards used to use a basic Elo system for ranking competitive play, where Pro Tour matches had a K-value of 48. That means that if al-Saq’s estimations are even close to correct, the effect on a player’s rank in best-of-three on Arena is almost the same as it was for tournament players at Magic’s highest levels of play.

What does it mean?

The findings of al-Saq’s report are very much based on estimations and mathematical modeling paired with observable changes in rank. There are many details and huge amounts of data that nobody has any access to outside of Wizards of the Coast. For this reason, it cannot be assumed that all of al-Saq’s findings are completely accurate- in fact, al-Saq would probably not claim that they are, and the report explains the full details of how its findings were reached if you would like to evaluate it for yourself. Instead, al-Saq was presumably trying to point out inconsistencies with the details that Wizards has provided about the ranking system, as well as to explain unexplained phenomena such as the insane rank decay at the top of the Mythic ladder.

It’s possible that more transparency from Wizards about how the ranking system works specifically would debunk some of al-Saq’s claims and prevent other players from feeling the need to do similar explorations in the future. It’s also possible that Wizards keeps most of the details about the ranking system private because they don’t want that information to be used to exploit it. Regardless of the accuracy of al-Saq’s findings, exploitation of the ranked ladder does seem to be possible at the moment. We can only hope that widely publicized reports like this encourage Wizards to make changes to the ladder in ways that will result in more fair gameplay and less opportunity for exploitation.