The (Potentially) Long Road Of Congestion in Final Fantasy XIV

Endwalker launched last week officially, you might have heard about it.

As of this last Wednesday, Yoshi P and team have a new request – unhear of it.

In the wake of the massive ongoing congestion, queues, and 2002 errors, the team is taking some good steps towards what will hopefully resolve the issue – a mix of a seemingly-accelerated server addition program with more details to come, the previously-announced February launch of the Oceanic data center for the game, the suspension of sales of the base game and lockdown of trial account sign-ups and logins even further, and two additional weeks of free game time for existing subscribers with the base game purchased.

All of these are interesting steps, and with a further step – a coded resolution to one cause of the 2002 errors that particularly stings while in line for the game – we might be getting close to a point where the game is predictably available in some way.

Today, I want to analyze this aspect of the launch a bit more closely, because with the addition of these details from the team, we have a slightly clearer indication of what went wrong, how technology debt caused the community some frustration, and we can start to predict the long tail of this launch and where we’ll land as Endwalker moves forward.

The Launch Issues

Going into the launch, congestion was inevitable, and I think that’s pretty well established by now and most people knew that and were ready to deal with what it could mean, at least on paper. Long lines aren’t unique to FFXIV but are a thing at most MMO launches – when a game is popular and has a lot of early content in a new expansion or patch that players can play once and move on from, the game won’t add new server capacity to count that as a peak but will instead manage as it can with what is available at that moment. In a world where resources are limited and where the survival of an MMO can often depend on how it manages costs, I think that players generally understand this and are accepting of it to a point – even as it can also frustrate. If I have to wait in a line to play but the experience at the end of that line is great, then I’ll do it happily.

On paper, FFXIV was ready for what I just described – wait your turn in line, get on the ride, it goes well, and that line wait is time well spent for content worth playing. If that is where the story ended, this wouldn’t be all that remarkable of a launch and I wouldn’t be here today writing this.

FFXIV has been hit particularly hard by factors both in and out of the team’s control. While all of the above applies mostly, the thing about server capacity in FFXIV is that the game in-general has needed increased server capacity for a while. Over the summer, the game was inundated with new players to such an extent that sales of the game were temporarily halted until optimizations could be made that would help keep things stable in lieu of adding new servers, as a new datacenter was already planned for the Oceanic region in Endwalker and acquiring hardware in the current chip crunch was not so easily or quickly done. That the players who tried the game over the summer would have a high-enough retention rate for problems to persist was perhaps harder to predict, but at the same time, it has been a while since Square Enix has added meaningful server capacity to the game.

The Logic of Adding Servers

This is where we get into things that Square Enix could control for, at least in theory. One of the team’s most frustrating habits from the server side is adding new datacenter capacity via new Worlds, instead of via capacity to existing worlds. They use a mix of new worlds and gameplay incentives to drive players to those new servers, but in a game with the design of FFXIV, this doesn’t always work. In FFXIV, the longer you’ve played and the more you’ve explored the game, the more likely you are to have roots on your current server. You probably have a Free Company, maybe an FC house room or a private apartment, maybe even a house! Those things don’t come with you, and the game requires you to relinquish housing prior to moving, which means that all the Gil spent on that house is gone.

By adding new worlds instead of new capacity, Square Enix instead creates a different problem – long-standing servers have an entrenched population with players who have little incentive to move, while new players have all the incentive to move for Road to 80 bonuses (now, at least) and a chance at getting those roots, creating a new set of entrenched players. The thing is, for an MMO, a lot of this is healthy and good behavior – players who feel a connection to a server probably play more, participate in more activities, and you get a vibrant server community where you run into the same folks out in the world, at Aetheryte Plazas and in dungeons and raids. However, when the core problem is too many players, just making new servers means that not everyone is going to transfer and thus the problem is not solved so easily.

My hope is that the wording in the latest congestion update signals added capacity for existing servers is a part of the agenda, or at least my parsing of it would suggest to me that all options are on the table, and not just new servers to create further entrenchment of the playerbase. Speaking of…

The Oceanic Datacenter And What It Does (And Doesn’t) Mean

The biggest server news in the halcyon days of February 2021 was that a new datacenter was being added for the Oceanic region. This was a bit of a headscratcher, but good news all the same – a set of players who get stuck with suboptimal ping due to needing to cross seas and oceans for their data to reach a server or vice versa would be in a better boat!

There’s two real problems with this, however. The first is that this addition is, again, new worlds, and thus entrenched Oceanic players are in a bind with getting a better play situation but having to start over on some aspects of their progression and community. The second is that Oceanic players have more choice in FFXIV, and a lot of data suggests that they wouldn’t alleviate much congestion by moving. Right now, the evidence available suggests that there’s a fair contingent of players from that region who play on the Japanese server Tonberry, as that server is the de facto “gaijin” server in the JP datacenters and the Japanese DCs offer better ping to Oceanic players compared to North America or Europe. There are a fair number spread out across the various datacenters from most indications, but it seems like the biggest benefit will be to a single server in Japan (which, to be fair, is one that is absolutely suffering in the Endwalker launch window!).

I’m quite on-board with new datacenter capacity in any form, even if it doesn’t benefit me or my friends directly (or at all), but I think there’s an assumption that there will be some sort of mass exodus away from other servers towards this datacenter that is difficult to predict with certainty. The same factors that apply to any other player apply here – if you have a presence and home on your current server, lower ping alone may not suffice as reason to transfer, but there will be some impact.

Long-Term Prior Needs of Capacity

I don’t play on EU datacenters and I only hear about them in passing, but for as often as I have been in queue to wait for the game, EU players apparently have it far, far worse. They’ve had queues long before Endwalker, and some players will talk about how additional capacity has been needed there for years – even before Shadowbringers, in the case of some servers. The EU region has a similar number of players to the NA and JP ones, yet has two datacenters only to the NA 3 and JP 3, and they have a smaller number of total worlds at present (EU with 12, NA with 24, and JP with 32). This also applies for NA in relation to JP, as player census data (unreliable as that can be, so grain of salt here) suggests that NA has more total players than JP with fewer worlds.

So while I am sympathetic to a point on the issues that the current supply chain woes have had on everything, it does seem to be the case that there was a trend towards being too conservative with long-term planning, as a lot of these issues have been long-term and before the mass exodus from WoW and the streamer pile-ons (one of the census datasets I looked at was from early 2020).

The Impact of Technological Debt (Error 2002)

The single most frustrating part of the launch has been queueing, not just because waiting in line sucks when you have limited time to play but because the nasty uprising of error 2002 means each wait in queue can be a crapshoot.

On a basic level theoretically, the problems we had shouldn’t have existed. Error 2002 is a lobby server error that prevents new connections from hitting the login servers when they reach capacity (17,000 total connections across a datacenter, raised to 21,000 with the development server emergency deployment). It’s designed to keep things running, so that the people who are in line can get through and get off those servers while preventing new connections to the lobby server from overloading and crashing it. The logic is decent and sound on some level, and as a layman to server tech and load balancing, I can get behind it.

However, if we extend this theory fully, it should not happen once you are in line, as you are connected and one of those connections being counted against the limit. So what was the deal?

Well, it turned out that an intrepid Reddit user found a potential root cause from the game itself – that the game would, while in line for access, disconnect from the lobby server and reconnect on a 15 minute timer. You might already see how this could be a problem, because if you connect and get in the first time, but then too many people come along after you and the server locks down new connections with an error 2002, then if the server is still rejecting new connections when the game forcibly cycles your connection…boom goes the dynamite, and you are out.

As it turns out, the team has indicated that legacy FFXIV 1.0 code is responsible for the errors when people are in line, and the raid patch this week will contain a fix. While they haven’t said out loud that the Reddit analysis specifically was accurate, they did note that player research helped them uncover the issue and work towards a resolution.

I was fairly critical of the team for their tone-deaf “it’s probably your internet, hard wire” responses to the error 2002 debacle, and I feel vindicated for that with what has come out since. I am also glad that the team listened to that analysis from players and worked to a solution, instead of just ignoring it and waiting things out. Ultimately, this is an example of how sometimes, a strong test is not enough to find everything that can break when literally hundreds of thousands of players worldwide throw themselves at the game and how technology debt from years past can come back to bite you in the ass if left unattended. The error 2002 itself is fine and a reasonable response to prevent players getting booted from the login server, but this mixed with the old login code like oil and water, and it does make me slightly curious as to how this was missed by the team with the root cause analysis they claim to be doing whenever players report issues during launch.

The Long-Tail of Logins

Ultimately, the last thing worth talking about here prior to the response from the team is how an MMO launches and what the expectation is for player count over the launch window. In most MMOs, the audience will naturally segregate into groups based on what they want from the game, but at launch, you get them all. The natural expectation in most MMOs is that there is a casual audience who is tuned in for the story, does it, finishes it, and logs off until the first full patch or even the next expansion. You have a semi-hardcore audience that has a smaller list of goals, and they taper off their play until you have a hardcore, daily-driver audience left, who logs in daily and plays the game as their primary hobby.

FFXIV compounds this by having a lot to do at launch in a relatively compressed window. You get two weeks from launch to just level, do the MSQ, gear your main job, take in the world and ambience. Then there’s a normal raid, and you get two weeks of that, where there is story content doled out in small doses via the raid, new fights, and the gear and rewards from that content. Finally, there is the Savage raid at 4 weeks in, with those fights and the basis for the story (usually just the handwave that justifies redoing the same fights in a more difficult mode) alongside the new tomestone for high-end gear, new crafting recipes, and then the treasure map dungeon. All of this has a boomerang effect, where players stop playing for short windows of time and then resume again, on top of the baseline of hardcore, daily logins and those with longer term goals (leveling all the jobs to the new level cap, gearing those jobs, meeting a gathering or crafting goal, doing all the sidequests, etc), and those who waited to do the MSQ until after the launch rush dies down.

The problem right now is that these factors all conspire against the game having a stable login situation, in the short-term at least. Right now, queues are still rather large and long – I waited for 3 hours each day this weekend to play, and while I had no 2002 errors (hooray!), it took a long time to get in, which has made every session stretch on as I work to move through my list of goals exhaustively for the time I do get. More people are finally starting to nudge over the finish line of the MSQ, but right as that starts happening and things are generally more stable, here comes the raid patch!

Further compounding all of this is the situation of new players. Right now, FFXIV has a large number of new players working through the game’s massive back-catalog of content, in addition to the veterans experiencing Endwalker and the hardcore players working on their post-MSQ Endwalker goals. This creates a layer of challenges – any predictions that could have been made from prior expansion data are basically invalid in the face of the massive paradigm shift in the MMO market, the game is likely to remain congested for at least a while longer as people work on their goals in-game, and the game’s success in the face of issues with various other MMOs (delayed content, delayed pending expansions, whatever the fuck Amazon is doing with New World, etc) only further exacerbates the conditions that we are in right now.

So I have two mindsets about this – one is sort of upset with Square Enix and the team as the longer-term server needs have been a known issue prior to the problems with supply chain and it feels like there was a better plan that could have been put into place there, and the other is sympathetic to a point – even a better-laid plan in theory could have crumbled under the burden that the game has taken on from being so many people’s next MMO stop. Commenter Cerila had some excellent points in their comments on my spoiler-laden review of Endwalker, and I won’t attempt to summarize here but rather direct you to those comments for the more knowledgeable take on datacenter ops and how such a thing comes together.

The Solutions Presented

Yoshida and team have offered what I think is a reasonable plan out of some of these issues. As mentioned above, the patching of the client-driven 2002 error cause is a big boost and one that gives me a lot of hope that the queues will be manageable such that I can log in, get in line, and walk away or do something else without babysitting the game. Given that I am eager to try the raid and to have my first expansion being able to reap the rewards of day 1 raid crafting for Savage, anything that ensures I can get into the game with greater ease makes me happier, as I have to plan less of my day around simply getting into the game.

Secondly, the team is offering an additional two weeks of free game time on top of the 7 days offered previously, for a total of 21 free days. This feels nice but also feels like a minimum-threshold of legal prevention, as the core complaint a player could perhaps rightly make (and I am not a lawyer, this does not constitute legal advice) is that they cannot access a service they are paying for, so paying for the time instead is a neat solution to nip that in the bud.

The team has also suspended sales of the base edition of the game and shut off new account signups, including the free trial. This makes sense and is a fairly smart solution, if one that draws the most oddly bewildered comments from people outside the FFXIV fandom. It allows those with active, paid accounts to log in and play (lines notwithstanding) and those players can still put in their Endwalker keys and move into expansion content, but new players are basically locked out until that restriction is removed. The game’s advertising is also ceasing, as it has been too effective and the game is now, quite literally, suffering from success (at least in part, as we discussed above).

Lastly, the team is going to present a roadmap for datacenter plans by the end of January 2022 (or next month, in more common parlance), and has indicated that things have started to come together with regards to delivery dates for new server hardware and long-term expansion plans for new datacenter capacity. The new update does not make mention of new Worlds specifically, which is a balm for me, as I think a more appropriate solution given what I put above is to make sure that players entrenched on a popular server can still get logged in and play, instead of funneling the whole server budget to new worlds that will fail to meaningfully split off the number of players needed to reduce congestion on popular servers like JP Tonberry or my own NA Gilgamesh. The wording makes clear that this won’t be a short-term bandaid they can deploy, but will instead be a long-term solution that will take a much longer stretch of time.

As a blend of solutions, I’m pretty okay with it. I already have had an account since 2014 and so the restrictions on new accounts don’t meaningfully affect me, but they do mean that some guildies who have grown in their frustration with WoW cannot move to FFXIV as new players just yet, which then creates some frustration on their part (we probably don’t help it by making them the Squidward in the dark from the Spongebob meme as we all gush about Endwalker in the FFXIV channel in Discord, haha).

Free game time is never a gift I’ll look at hypercritically, so even while I acknowledged above that I think there’s an ounce of prevention in that, as it were, I also will gladly take not paying for the game until 2022 as some measure of compensation. Even for those who can play daily, waiting through the queue is never a purely pleasant experience with the risk of error 2002 a possibility, and removing that will also go a long way towards making things feel better.

The datacenter roadmap is something I am very curious about, because I have to wonder what circumstances pulled together to go from a vague uncertainty to a roadmap over the next month. I don’t even say that to imply nefarious intent in prior statements – just that I wonder if a supplier reached out, how they came to be able to get new servers, and if Square Enix as a business pulled together and approved some measure of additional funding for the project. In the NoClip documentary about FFXIV, the state of 1.0 was mentioned as causing a company-wide “state of emergency” that led to the game’s ARR project getting additional funding and executive support, and I would be very interested to hear someone tell the tale of this launch from the inside in a few years, even if that telling is loaded with PR spin.

In closing, this launch has been pretty fascinating, because it is ultimately a tale of success. A game that no one wanted to play 11 years ago is now a dominant force in the MMO market that is so successful that no one new can buy it or play until things calm down. It is also an interesting set of smaller parables – about the importance of planning and analytics, how the best laid plans can still be unraveled by factors out of your control, and about how an unfortunate confluence of factors can take a moment from undisputed victory to feeling like defeat, even though things once you can login to the game are exceptional.

As an FFXIV fan, I am still exceptionally happy with Endwalker and looking forward to what the future holds for the game. I am glad to see the team taking the problem seriously and being willing to share more details with us than I suspect most developers would (admitting that players figured out the problem with the 2002 error is a bold step that not many teams would be willing to say out loud!), and I am at least somewhat optimistic that things will clean up in the medium-term.

However, I have mentally prepared myself for a few more weeks of congestion, of clutter, of multi-hour waits that end up being worth it.

And at the end of the day, my opinion of Endwalker distilled simply is that the waits to play have always been worth it, and that is why I am still logged in as I write this ready to level Reaper.

Advertisement