Actually, Arrowhead architected their servers extremely well

Hi there,

This isn't your typical circlejerk post about how the game is awesome and yada yada. I wanted to genuinely congratulate Arrowhead on how well the server infrastructure is built.

Look, I know, I've also been victim to the server issues, I've had random disconnects, I've had extremely poorly timed attempts to log in, I've had times where my mission rewards and currency were delayed or skipped entirely. I'm not saying the servers are perfect, but please, indulge me for a 5-10 minute read here.

In a previous job I worked for a cloud provider, and several of our clients were actually video game companies and studios. For a number of games, we actually provided the cloud infrastructure that I would later that very evening go play video games on. You can imagine how frustrated I was knowing I was playing games that had the ability to scale but just didn't. As I worked there for 7.5 years, this was a somewhat common occurrence. Now, my job is for a different company but instead of operations, I now advise people on how to build things properly. An architect of solutions you might say.

The thing is, most games simply are not built to operate and take advantage of what makes cloud computing good. Most games are simply built the same way they were "in the old days" as if they were still on metal servers sitting in the office. Sure a lot of companies have gotten some cloud tech into their pipelines like imaging and automated setup of servers, but few have actually built their application to scale automatically on the fly without human intervention.

I have reason to believe that Arrowhead has done so. Look at it this way, we've seen countless memes in this subreddit about "how could they have expected the scale, the first peaked at 7K". Arrowhead has even stated they expected 2 to hit maybe 50K concurrent. They hit a peak of 64K on steam alone on day one. Server issues didn't really start happening until they hit 150K(ish) concurrent on steam alone (again) and didn't get really bad until they hit 200K concurrent, 4 times their wildest expectations.

Now that alone should show that the infrastructure behind the game is fairly solid. Sure there were a few stability issues but the fact that with no additional work really, the game was able to sustain 200K concurrent and not fall completely over is impressive, especially by video game industry standards I'd say. But then they actually started talking and while the updates since have been much more tight lipped, the original "server stability" updates from Arrowhead indicated rate limit issues and negotiating with their server provider. I understand both why these were said and why they probably aren't still being said, as the cloud provider doesn't wanna be plowed under the bus, and initially Arrowhead was probably in panic mode and so was maybe a little more honest than modern capitalism really likes (even though I personally think there is nothing wrong with a bit of honesty).

Only a few days later, the cap was clearly raised to 400K, and it only took the game 4 days to hit this new limit. Now a couple of days after that the cap was raised again to 450K. I think the initial doubling of the capacity was simply due to Arrowhead asking the cloud provider for more infrastructure. You see, cloud providers are also still a business and thus have some protections in place. No one gets access to the entire cloud infrastructure, because providers quickly learned that customer credentials get stolen all the time and then hackers will spin up as many VM's as possible to mine bitcoin with. Thus everyone gets a limit to how much they can buy from the provider essentially. Once you hit that limit its extremely simple generally to contact the provider and ask for more. This can also be done pre-emptively, which I think is the reason we were capped at 200K initially instead of their 50K projections.

So the original server problems, based on what I can see in steamdb stats, what was said in the initial posts about server capacity, what we're seeing now, and my own knowledge of large server infrastructure, are likely due to hitting the pre-negotiated limit with the cloud provider. Arrowhead has clearly raised this limit significantly and we doubled cap.

Now we're finally, at 450K, running into what I believe to be actual technical limitations of the way the game was architected. This scale is where certain backend services like databases, message queues, API frameworks, and so on may stop being able to horizontally or vertically scale well (adding more servers or making the server bigger are horizontal and vertical scaling for those less in on cloud lingo). I'm guessing this is what's happening now as we're seeing smaller cap increases and additional comments like "turning a vespa into a sports car" instead of limits conversations. We're at the point where something about the game's systems might need to be refactored or rebuilt with scale in mind. This takes both more time, and has less impact on total amount of allowable players before another limitation may be hit. We're now in for the long haul folks.

At 450K. Maybe 20 other games have ever gotten to the point where this sort of architecture limitation has become relevant, and I would argue the vast majority knew they were going to run into that limitation and STILL fell flat on their face far before that point because they weren't architected for cloud scaling correctly. Arrowhead had no idea they would hit this level of success and still built the game for what in their minds at the time was absolutely pie in the sky levels of players. And we still surpassed that.

So genuinely Arrowhead, from the bottom of my heart thank you for actually leveraging modern compute resources in a modern way. Also please do not take the wrong lesson from this and say "oh well we put in the effort and clouded correctly and it still didn't work", recognize that that effort was a huge factor in allowing the game's meteoric rise. Also for any other game devs out there that may read this, please, dear god, take this lesson to heart.