Hey again, Toons! This month, we're providing some insight under the hood of Toontown Rewritten with Backstage blog posts! If you missed our first entry, you can glean some insight into Moderation and Coordination right here! Today, Jac has provided us an overview of what goes on in Technical Operations – or, TechOps!

A Solid Foundation

TechOps plays a big role in updating Toontown! Get ready for a whole onslaught of techy terms... We maintain the testing infrastructure for other Departments who help develop the game, such as programmers, quality assurance, design, and anyone in between! With help from Technical Toons over time, the infrastructure supporting Toontown Rewritten has gotten loads more powerful. These days, Toontown is run on top of a technology known as Kubernetes, which simplifies setting up Toontown Rewritten servers.

When combined with the powerful templating engine, Helm, we can standardize the way Toontown should work and allow our behind-the-scenes development servers to work the same as the relatively glitchinitive-free Toontown you're all familiar with. The time invested into these allows our Game Programmers to test new features and fixes in a flash – updating servers in under a minute! This technology stack has also simplified the effort needed to run semi-public and public testing servers, such as the demos we offer at ToonFest, and most importantly: the Under New Management playtests from earlier this year!

A Cracked Case

In the weeks leading up to the release of Under New Management, we were hard at work trying to stabilize the game as best we could for Toons to enjoy while they waited for news about our next major updates! I investigated an issue where certain popular Districts, such as Splat Summit and Zapwood, would slow down and run into performance problems – typically following a busy weekend of Toons playing. The issues became more prevalent leading up to the big update's release.

Using various diagnostic tools we had developed, we identified that the problem was caused by an accumulation of thousands of Cogs in Dynamic Zones – the far reaches of Toontown you must visit to fight your favorite Boss Cog or Supervisor. We then cross-referenced the most active Dynamic Zones with our server logs and identified our culprit: Cogs were accumulating inside Sellbot Field Offices! The Boiler was retaliating without Defensive Strategy by making some defeated Cogs persist on the servers. What sore losers they are!

The Aftershock

Once an update goes live, TechOps is hard at work monitoring various dashboards, looking at crash reports, and communicating with Game Programmers and the rest of the team to coordinate looking into problems. We've developed some powerful tools to help us tackle issues without tackling the entire game at the same time. Client Revisions are teeny-tiny client-only updates we send that patch up crashes (and other oddities Toons may be experiencing) without restarting the game. These can be identified by a lowercase letter at the end of a version's title, like ttr-live-v4.0.3a.

In some cases, we need to make changes to Districts as well. This is where District Draining, denoted by an orange District status color, comes into play. Draining allows us to close off and move Toons out of the District – as long they’re not busy fighting Cogs or doing other activities! Once all Toons have left a District, it's then automatically restarted with the latest bug fixes available. Pretty neat, right?

Without descending into the District-Draining depths of technobabble, that's our Technical Operations in a snapshot! If you've got your own technical skills that thrill and you've got the will to show it, I encourage you to check out the myriad of skilled positions we're... 'head-hunting' (in the most non-Bossbot way possible, I assure you) for! Regardless, know that the Toontown Team welcomes all – and everyone has something great they can bring to the team!