Trouble logging in? Click Here

Create a FREE Account
More News

Technical Operations | Backstage Toontown

Posted by The Toontown Team on July 8, 2024 at 2:00 PM

Hey again, Toons! This month, we're providing some insight under the hood of Toontown Rewritten with Backstage blog posts! If you missed our first entry, you can glean some insight into Moderation and Coordination right here! Today, Jac has provided us an overview of what goes on in Technical Operations – or, TechOps!

A Solid Foundation

TechOps plays a big role in updating Toontown! Get ready for a whole onslaught of techy terms... We maintain the testing infrastructure for other Departments who help develop the game, such as programmers, quality assurance, design, and anyone in between! With help from Technical Toons over time, the infrastructure supporting Toontown Rewritten has gotten loads more powerful. These days, Toontown is run on top of a technology known as Kubernetes, which simplifies setting up Toontown Rewritten servers.

When combined with the powerful templating engine, Helm, we can standardize the way Toontown should work and allow our behind-the-scenes development servers to work the same as the relatively glitchinitive-free Toontown you're all familiar with. The time invested into these allows our Game Programmers to test new features and fixes in a flash – updating servers in under a minute! This technology stack has also simplified the effort needed to run semi-public and public testing servers, such as the demos we offer at ToonFest, and most importantly: the Under New Management playtests from earlier this year!

A Cracked Case

In the weeks leading up to the release of Under New Management, we were hard at work trying to stabilize the game as best we could for Toons to enjoy while they waited for news about our next major updates! I investigated an issue where certain popular Districts, such as Splat Summit and Zapwood, would slow down and run into performance problems – typically following a busy weekend of Toons playing. The issues became more prevalent leading up to the big update's release.

Using various diagnostic tools we had developed, we identified that the problem was caused by an accumulation of thousands of Cogs in Dynamic Zones – the far reaches of Toontown you must visit to fight your favorite Boss Cog or Supervisor. We then cross-referenced the most active Dynamic Zones with our server logs and identified our culprit: Cogs were accumulating inside Sellbot Field Offices! The Boiler was retaliating without Defensive Strategy by making some defeated Cogs persist on the servers. What sore losers they are!

The Aftershock

Once an update goes live, TechOps is hard at work monitoring various dashboards, looking at crash reports, and communicating with Game Programmers and the rest of the team to coordinate looking into problems. We've developed some powerful tools to help us tackle issues without tackling the entire game at the same time. Client Revisions are teeny-tiny client-only updates we send that patch up crashes (and other oddities Toons may be experiencing) without restarting the game. These can be identified by a lowercase letter at the end of a version's title, like ttr-live-v4.0.3a.

In some cases, we need to make changes to Districts as well. This is where District Draining, denoted by an orange District status color, comes into play. Draining allows us to close off and move Toons out of the District – as long they’re not busy fighting Cogs or doing other activities! Once all Toons have left a District, it's then automatically restarted with the latest bug fixes available. Pretty neat, right?

Without descending into the District-Draining depths of technobabble, that's our Technical Operations in a snapshot! If you've got your own technical skills that thrill and you've got the will to show it, I encourage you to check out the myriad of skilled positions we're... 'head-hunting(in the most non-Bossbot way possible, I assure you) for! Regardless, know that the Toontown Team welcomes all – and everyone has something great they can bring to the team!


Sorry! You must log in before you can post a comment.

Toonsworth said on July 8, 2024 at 5:02 PM
Well jolly good. Good to see some technicians to keep our fellow Cogs in shape. I'm sure The VP is more than relieved that The Boiler has been tinkered with to be not as on the fritz. Afterall, sending too many Cogs into one building is just a mere waste of Cogbucks. That would be very unprofessional of Sellbots, and surely double the anger from the Cashbots up in Dreamland. A good Company Budget must be kept after all for The Cogs, no acceptions. I've heard that Cogs in other towns they wipe out entire Corporate Ladder, just to put less productive ones in their place. Could you imagine? Well, I am more than happy with the 4 we are dealing with now, there's probably 5 other ladders out in the Tooniverse, though I wouldn't dare go toe to toe with those. Cough... Pressbots... Cough... Either way, good show to you, Computer Operator. Keep our Toons and Cogs balanced, though, don't be afraid to send out a Level 20 Cold Caller once in awhile in a building. It keeps the A/C going... Heheh. Good show!

Comet said on July 8, 2024 at 5:11 PM
Legend has it that if you hold up your Doodle and shout "DOODLE TRICK BOOST" three times all TechOps Toons will immediately go sad.

Big Bongo said on July 8, 2024 at 8:45 PM
this is absolutely amazing and i love it, i love the dedication, great job guys!

Japanese Blossom said on July 10, 2024 at 7:39 AM
From Cogbucks to Stock Options to Jury Notices to Merits looks they need a promotion to enter the battle. It's time to defeat the boss cogs everyone and let's help Toontown to be on the safe side and it's looks very antonshing

Darth Anon E. Mouse said on July 11, 2024 at 12:12 PM
As a tech geek myself, it was nice to read about what goes on behind the scenes for tech ops! Thank you for sharing! I would love to see more of these kind of posts.

Max The Innovator said on July 12, 2024 at 8:27 PM
Really fun to see a technical backstage post!
Back To Top

Latest News