Lionheart, Hi-Rez Studios Vice President and Paladins Strike General Manager, has written new post on Paladins Strike official site to share more information about how Paladins Strike is developed, including answering common questions about bugs and the update process. Let’s check out!
Hi everyone. I want to share more information about how Paladins Strike is developed, including answering common questions about bugs and the update process. This will allow you to be more informed as a player about our game development, and help you learn how we mold games based on your feedback.
As you may know, Paladins Strike is a collaborative effort between Goblin Network (based in China) and Hi-Rez Studios (based in the United States, United Kingdom, and China). Goblin is the developer, which means its primary responsibility is software development. Goblin creates the game client you use and the supporting technology we deploy to servers across the world. Hi-Rez is the publisher, which means its responsibility is live operations, distribution, and marketing. Hi-Rez puts the game in the hands of players, maintains the game services, and supports the community (including esports). Goblin and Hi-Rez work together on design and priority.
Understanding this relationship is key to a complete view of how we create and maintain Paladins Strike. For example, let’s assume that Ying (a playable character) is believed to be overpowered. This information will first come from the community, which goes to Hi-Rez. Then, Hi-Rez will evaluate Ying and compare her data to other Champions to see if, in fact, she is overpowered. Once confirmed, Hi-Rez will then contact Goblin and explain that Ying is performing above expectations. Goblin will verify, and then have a discussion with Hi-Rez about the estimated reasons for her over-performance.
Does she do too much damage? After a couple days, the data says yes.
Why is she doing too much damage? 250 per attack is probably too high.
But what about the fact that her attack goes through shields and walls? That’s a bug.
Will she still be too strong if we only fix the bug? We think so.
Should we reduce her damage or make a different change? Let’s reduce her damage.
I make the conversation sound more streamlined than it is. As I’m sure you can relate, not everyone agrees on balance or why something is strong/weak among players. This is true among game developers too. The conversation becomes even more interesting when you consider cultural and language barriers. For example, most of Goblin speaks only Mandarin while most of Hi-Rez speaks only English. One benefit is we have a diverse perspective represented in each discussion. But I would be lying to you if I said that made the process faster. Oh, and don’t forget that Goblin and Hi-Rez are 12-13 hours apart in timezone, which means we have limited overlap for discussion if something takes more than one meeting to resolve. None of this is a problem, but it means we can’t move as fast as a team that is all in one location, speaking the same language, and on similar living schedules.
OK, let’s go back to our example. We’ve identified that Ying is OP because of a bug and her weapon damage. Next, we must evaluate whether Ying can be fixed without exporting a new build (developer speak for an update you download from the App Store or Google Play Store). A new build will mean we have to pass certification, which can take 24-48 hours. However, one alternative to a new build is a process called hotfixing. When we deploy a hotfix, we make simple modifications to game data that doesn’t require a new build. You cannot solve all problems with a hotfix, but sometimes it’s a good solution for balance or other configuration errors.
For Ying, Goblin identifies the correct fix will require an update to her art (attack VFX doesn’t understand how to not penetrate) and weapon configuration (reduce the value for damage on attack). Because we rushed her out, the weapon configuration is done in code (a big no-no as it prevents us from hotfixing). At this point, we have two options. The first option is to delay the next update with content, features, and bug fixes, in order to export a new build that fixes Ying. The second option is to leave Ying along and let her continue in OP glory until the next scheduled update. We went with the second option. But why?
The short answer is we believed the community would benefit more from the next update releasing on time than adjustments to one Champion, which would delay the next update that contained fixes to numerous other problems. Now, I know what you’re thinking. “Hold on, Lionheart, why didn’t you create an update that fixes Ying and includes whatever other fixes were ready? Then, you can give us the rest later and everyone is happy, right?” Unfortunately, that’s not how development works. A perfect explanation would go into detail about trunk-based development, but the practical answer is that rushing a build out like that would probably lead to more problems and could completely bring down the game services for Paladins Strike. No one wants that.
Alright, Ying aside, I want you to think of the most annoying bug you dealt with today in Paladins Strike or another game. Imagine that bug going through this same process. Probably more often than not, the bug may seem simple to you and strange that the development team hasn’t fixed it yet. But simple to understand problems don’t necessarily mean simple to implement solutions.
Cool, so how do these annoying bugs get into Paladins Strike to begin with? First and foremost, we are human (like you… maybe… beep boop). This means sometimes we make simple mistakes that lead to problems. A small typo in code. Forgetting to check a box. You would be surprised how tiny mistakes can cause big problems. I put this in the growing pains category for any team that is working together for the first time. As we create checklists, documentation, and process, we also learn from our mistakes and improve. When you care for a project like we do about Paladins Strike, I promise the person who made the mistake is way more hard on themself than even you raging on social media, email, etc. Please try to remember that the next time something isn’t to your preference.
Another way bugs make their way into Paladins Strike is from resource constraints and scalability. Even with a dedicated QA team, we can’t anticipate all the ways you will interact with the game client. We also can’t test under all the network conditions you will play on, or on all the devices you may own (there are a TON of phones out there). Goblin is a new business with less than 50 employees. The Hi-Rez publishing team for Paladins Strike is 4 employees, and some of us are working on multiple projects. This means our QA resources are somewhat limited, for good or bad, and we have to pick our battles in testing. While unrealistic from a consumer-view, there are many systems that work well and are often forgotten behind the wall of problems that need fixing. And those well-functioning systems are the result of a positive development and QA process.
Lastly, there are some “bugs” that are actually incomplete development. An example of this is the problem associated with network loss compensation. There isn’t necessarily a malfunction with Paladins Strike, but rather improvements are needed to reach our goals for the game. You can think of it like a car that has 300 horsepower. There’s nothing wrong with that, but it’s probably not going to win a race against other cars with 600 horsepower at the same weight and aerodynamics. Thus, you will need more horsepower before you can win the race. Since you can’t just buy a new engine, you make upgrades to the engine you have over time. We know other games are running on 600 horsepower engines, and we have plans to improve our Paladins Strike engine to 1000 horsepower. But it’s going to take some time to complete the upgrades.
Hopefully, this is helpful information to some of our community members who are curious about the process we use for Paladins Strike. As always, if you have questions feel free to reach out on Twitter. Thanks for reading!
— Lionheart
