MTG Arena Anniversary Week Issues

The MTG Arena development team at Wizards of the Coast has provided players with an update to what’s been happening behind the scenes, with the various issues the game has had in recent times. It covers the five year celebration rewards, draft event errors, how it all happened and when it all might be fixed. Check out the full post below.

---

MTG Arena Anniversary Week Issues

Greetings MTG Arena players,

Well, this last week sure hasn’t gone as we planned. What was originally scheduled to be a celebration of five years of fun hasn’t quite lived up to that promise, so today I wanted to apologize. We’ll talk a little later about some of the “how,” but one of the things that I emphasize with the team is that players don’t care what’s happening behind the scenes when their experience is bad, they just want to know when it will get better, so let’s get to it.

The short answer is, we’re working on it but don’t have an official timeline yet. For many, the game is fine, but there are certain areas that are still experiencing issues that we’re troubleshooting. As a steward of this game that so many play each day, I know this isn’t a fulfilling answer, but it is where we are right now, and for that we’re sorry.

On the bright side, most areas of the game are currently stable and functional (and if you read the entire article, you’ll even see where we made improvements), though we’re treading with some caution as we’re not fully out of the woods yet. Here’s our current status:

• Premier Drafts are up and running after being disabled part of last week. We messaged players who may have been affected. We’ve heard some feedback that we may have missed some players and will validate that this week, but …
• Player inbox is currently disabled as we’re troubleshooting some issues that started on Friday and persisted through the weekend. This means that messaging slated for the last couple days of our anniversary is paused until we get that sorted out. Once we’re confident the system is working correctly, we’ll finish our anniversary messaging.
• The system is currently stable, but we’re aware of a brief outage that occurred early Monday morning (Pacific time), but we believe that was coincidental and not related to our core issues. Unfortunately, it has complicated our investigations.

As for next steps, we’ll continue to troubleshoot and fix the underlying issues that are creating disruptions. We’ve already made a couple fixes that are minimizing some of our symptoms, but we’ve not yet cured the illness. While we don’t expect more unplanned disruptions, our team remains on high alert in case they happen.

We’ll need to deploy additional fixes over the next week or two as we identify solutions to issues, and while we’d expect those deployments to be like our normal release process, there is a chance a full system outage may be required. If that is the case, then we’ll let players know the day prior to the outage. 

Finally, once we’re comfortable that all systems are go, we’ll turn on anything that is disabled and get back to the fun. Thanks for all your support and may all your topdecks be great.

Chris Kiritz
Executive Producer, Magic Digital

…Oh, are you still here and interested in learning more? Great, let’s pull back the curtain a bit and talk about how we got here. First, a peek at our process (queue the Schoolhouse Rock! music).

THE “HOW”

Everything we do starts on a software engineer’s machine. We have a problem or need, and they create the code that solves that problem or need. Sometimes it is one engineer, sometimes a team, but the general process is the same.

Once they’ve completed their work and run some basic tests, we identify which release that change will go into. Easy or time-sensitive things might go into a release that is only a couple weeks away. Other changes might need extensive testing, have high risk, or require other changes to be delivered first. In that world, the work is checked into a release that may be months away.

Once checked into a release, our QA team tests it using a variety of manual and automated tests. This will be both discrete tests around the change, as well as more generalized system tests to ensure the change hasn’t inadvertently affected other parts of the game. As the release gets close to the live date, we raise the bar on what can be considered for that release—it is easier to get changes in for a release that is two months away than it is to get a change in for a release that is two weeks away.

The P0 Live Team is a strike team of the people best qualified to solve P0 issues—critical issues more important than Priority 1.

A couple weeks before a release goes to the public, we lock it down so no additional changes are made as we prepare to submit it to our publishing partners, like Apple and Google, for review.

On the day of release, we deploy to live and then move players gradually over from the old version of MTG Arena to the new version. This is when you get your download notification and grab the latest version. When everything goes well, that is the entirety of the release. 

Last Tuesday, we updated some database services as part of our general ongoing maintenance. This is like keeping your phone operating system up to date to ensure you’re getting the latest features and security updates. Since we’re always cautious when dealing with major systems like our databases, this work was completed in March and then propagated to all our internal environments for feature testing, load testing, and just a general soak to see if anything organically crops up as we go about our daily work. After months of internal operation and stabilization, we felt confident in releasing to our production environment.

Unfortunately, after that release went live, we found that it didn’t behave on production the way we expected, but instead degraded the performance of various system messages. While this was unnoticeable in most areas of the game, Premier Draft felt this degradation acutely. After we discovered the issue, we disabled Premier Draft and activated our Priority Zero Live Team (otherwise known as the P0 Live Team) to troubleshoot. The P0 Live Team is a strike team of the people best qualified to solve P0 issues—critical issues more important than Priority 1.

BEN SMITH TALKS PREMIER DRAFTS

Ben Smith, a senior software engineer, was the person who built the Premier Draft system and a shoo-in member of the P0 Live Team dealing with Premier Draft problems, so I’m going to turn it over to him for the next part:

As part of the leadership team, I am often part of the P0 Live Team, and I’m always impressed with how the team rises to meet challenges. While we would obviously prefer players never experience issues, we’re aware that we must be prepared to solve problems on the fly, and having a team you can trust to engage in direct and active conversations while looking for player-facing solutions is a key component to successfully running a game like this.

If you’ve made it this far, hopefully it’s been interesting, informative, or both. Thanks again for being the best community in the world, and hopefully next time I’m writing something like this, it isn’t in response to a major issue.

Good luck and have fun,

Chris