Hey everybody, MeatShield back at it again with some number crunching and data analysis! Brace yourself, this is going to be another LONG post.

Last Tuesday (March 2nd), I posted a survey in r/DeepRockGalactic asking people to rate all 302 weapon Mods and Overclocks in terms of how balanced they are. Not how much they like them, or which ones they think are the best, but if the option is balanced. After 7 days, a total of 296 people submitted responses that rated at least 1 Mod or Overclock. I've already published an Infographic showing the high-level results and interesting factoids about the survey results, and this is going to be the first of two blog posts that expand on the data analysis behind that Infographic. This first part is primarily going to consist of objective analysis of the data itself, and the next blog post (currently planned for this Friday) will be a more subjective interpretation of the data. Hopefully you all find this interesting!

If you don't care about how the survey was designed or my analysis of the data, you can skip all the way to the bottom to see the average balance scores of all 302 Mods and Overclocks according to the survey results.

Methodology

Goals of this project

First and foremost, I wanted to collect a moderate-size dataset about how the DRG community perceives the current weapon balance. Any time weapon balance gets discussed, much of it comes down to different opinions of people in the discussion. My hope is that by collecting enough responses to get a meaningful average rating of every Mod/OC, it will be possible to start making meaningful balance suggestions. I wanted to start at the lowest level of balance (the Mods and Overclocks inside each weapon) and identify what things need "micro" balancing. After that's done, it should be a lot easier to do "macro" balancing between the weapons and classes.

Here are some of the things I'm most interested in gathering data about from the survey:

• Directly use the results to measure the perceived balance of Mods and Overclocks
• Directly use the results to identify how the balance scores are distributed (preferably would be a Normal Distribution)
• Indirectly use the results to gauge the perceived "micro balance" of each weapon
• Indirectly use the results to gauge how much people care about each weapon's Mods/OCs

Designing the survey

Right from the start, I chose to abide by a few self-imposed rules when creating the survey:

• It must be open to all DRG players, regardless of how much they play the game. Everyone's opinion matters, not just veterans -- I refuse to be exclusionary.
• It must not require a Google account to complete. I don't want people to be turned away by having to reveal personal information about themselves (email address, potentially real name).

The first choice that I made was that I wanted this survey to be Quantitative, not Qualitative. Having hundreds of people type up mini essays with their thoughts about 302 Mods/OCs would mean a massive amount of data for me to sort through and anaylze at the end. By making all the questions multiple-choice with a uniform, predefined scale, I was able to expedite both taking the survey as well as analyzing the results afterwards. Additionally, multiple-choice questions are incredibly fast to answer, as opposed to filling out lots of short-answer text fields.

The second choice made was to exclude any questions about the weapons themselves. My reasoning was that by letting people start by rating the weapon itself, most of their ratings for the following Mods/OCs would be skewed towards the rating they gave the overall weapon (a form of confirmation bias). Not only that, but by letting them rate the weapons directly, I was worried that those questions would become "which weapon do you prefer using" instead of "how balanced is the weapon".

The third choice was to make every Mod/OC question optional. Not only was that a practical choice to get more people to actually finish and submit the survey, it was also a subtle way to identify the weapons, Mods, and Overclocks that the participants actually care about. If people only answered a dozen or so of the questions, I can infer that they care enough about those options to give them a rating. That's just as telling of a result as if they had rated all 302 options, but in a different manner -- more on this later in Meta Results.

The fourth major choice I made while creating the survey was to intentionally not tell survey-takers what each Mod/Overclock does. This was done for a few reasons:

• Sometimes the written explanation of how something works is more confusing than helpful unless the player already understands the game mechanics that are being affected (Breach Cutter's Overclock "Inferno" is a great example of this counter-intuitive behavior). I didn't want people who were newer to the game to be confused by having tons of new/hidden mechanics thrown at them each question, and rather just be able to take the survey at face value.
• This survey wasn't designed to measure how balanced the Mods/OCs actually are -- this was designed to measure the community's perceptions of balance. It is often the case that if a Mod or Overclock is moderately strong but perceived to be weak, its pickrate will still be low just from people telling each other that it's bad. Some of the top examples are GK2's T5.B "Battle Cool" and Revolver's T2.B "Floating Barrel". I was worried that giving a fully-detailed explanation of how everything works under the hood might skew perceptions and mess up the data.
• I wanted people to come to terms with their own knowledge of the game, and the limits thereof. If they don't know what a Mod/OC does off the top of their head, I would rather have them look it up themselves and be comfortable with their understanding of how it works before giving it a balance rating.

So, once the survey had been designed, I set about creating the 1-5 scale that everyone would use. I wanted a system that was able to identify things that were both too weak and too strong, so something as simple as 1 being "bad" and 5 being "good" was out. I eventually settled on this scale:

1. This mod/OC suffers from a fatal design flaw -- it should be either moved to a different tier, redesigned, or replaced with a different option.
2. The design is sound, but it needs some numerical buffs to be usable.
3. It's well-balanced in design, numerical values, and location.
4. The design is sound but it needs some numerical nerfs so that it doesn't outcompete its neighbors anymore.
5. This mod/OC is "broken/overpowered/must-pick at this tier" and needs to be either redesigned or moved to a different location.

Feedback from comments

I read every single comment on the reddit thread, including the ones criticizing the survey. Here's a high-level summary of the 196 comments at the time of writing this post:

Regarding the survey itself

• People strongly desired to have Qualitative answers somewhere. Suggestions ranged from having one long-answer field at the end of every weapon to having every mod/OC get its own short-answer field. In general, there was some dissatisfaction at being constricted to the 1-5 scale without being able to personalize or explain their answers.
• People disliked having to look up Mods/OCs on their own, and would have much rather had me tell them what each option did in the questions themselves. As I outlined above, it was an intentional choice not to add that information rather than an oversight. That said, it seems that the survey would have been received more positively if I had added the information so I'm not sure if I made the right call or not.
• The length of the survey was daunting to several people. There were concerns that people would burn out before reaching the end, which would lead to incomplete data for Scout's weapons. There was a suggestion to randomize the order of weapons each time the survey is taken so that any potential burnout wouldn't be clustered on the same questions. Another suggestion was to split this into 16 different surveys (one per weapon) spread out across a short period of time.
• Beyond the 1-5 scale, there should be an option for "I haven't tried this", and a checkbox that indicates the Mod/OC's score is affected by an issue with the weapon rather than the Mod/OC itself.
• One person asked if the survey could include Equipment mods and grenades, too.
• [Criticism]: Without the context of demographic things like players' skill, hazard levels, and what relative system they're using to make the 1-5 judgement, the results of the survey won't be meaningful.
• [Criticism]: By not adding questions about ranking the classes or ranking the weapons in each slot, the results of the survey will be an incomplete picture of the game's balance.
• [Criticism]: The survey is bloated by having a question for every Mod/OC, and should have focused on known problematic weapons/builds. Qualitative answers to identify why something is unbalanced would have been better than quantitative answers to identify what is unbalanced. As such, not only are the results bloated, they will also be diluted amongst the rest of the Mods/OCs that aren't a problem. The survey addresses neither inter-weapon balance nor inter-class balance, and as such glosses over the core issues of imbalance in the game right now. By not gathering demographic data about what Hazard Level the players play most often, or if they play DD vs EDD, the results are useless. "It completely misses the point."
• [Criticism]: Not allowing people to rank classes or rate weapons as a whole was an obvious oversight. Without demogrpahic data of players' hazard levels, the results will be muddied. Ranking everything on a 1-5 scale is wrong; it would have been better to let people rank Mod tiers and Overclock groups from worst-to-best. "It's pretty terribly made."
• [Criticism]: Rating a Mod/OC in a vacuum is worthless; the value of a Mod can change wildly depending on which Overclock you equip. "The survey is worthless, unfortunately."

Unrelated to the survey

• The most upvoted comment on the thread was from someone who started taking the survey but then realized that they hadn't experimented with other builds in a long time. As a result of this survey, they decided to start experimenting again to try to get a feel for the weapons' balance. I'm happy that this survey had a positive effect on at least one person (and potentailly a few dozen of the people that upvoted that comment).
• This website is starting to become recognized as a major source of information for this type of information, alongside the Wiki.
• A few people thought I was a developer at GSG -- sorry to disappoint. I'm just a player who's very passionate about this game.
• On the other hand, a few people had no clue that I even existed before this survey was posted.
• One person said that they follow this blog -- hi u/idiotwitch!
• There were a lot of comments on what balance changes people want implemented in U34 & U35.

Potential future surveys

I'm on the fence about adding Mod/OC descriptions to each question in the future. On one hand, it was the second-most requested addition, and one of the top things pointed out as a failing of the survey. On the other hand, I'm still worried about confusing people with over-complicated explanations of some under-the-hood mechanics that they may not even be aware of, or influencing people's perceptions of Mods/OCs by listing every single effect. At the time of writing this blog post, I'm leaning towards adding the descriptions to future surveys.

I am definitely planning on adding one long-answer text field at the end of every weapon page so that people have a much-requested outlet for their explanations and justifications. In addition, I'll probably add the "I haven't used this" option to the 1-5 multiple choice and "problem exists outside this mod" checkbox.

At the time of writing, I don't plan to add questions about inter-class or inter-weapon balance to future surveys. There's already an abundance of ways to collect that data, between posts on the subreddit, loadouts submitted to this website, word-of-mouth from other players, and my own observations. This survey was designed to gather information on the Mods and Overclocks beyond just pickrate, and that's exactly what it did.

I don't think people would be happy if I were to post 16+ survey links at once, and it would make it harder for me to aggregate the data at the end. As such I'm currently planning to keep it as one long survey for all weapons. I'm open to the idea of randomzing weapon order; the only thing giving me pause is that if people want to give specific feedback on a specific weapon, having them all in order is extremely useful.

The last thing I'm going to address is the request to collect demographic data about players' preferred Hazard level, hours played, DD vs EDD, player rank, promotions for each class, etc: I'm going to stick to my guns and say "no". I firmly believe that the point of doing community surveys like this is to get everyone's opinion and value it equally. I don't care if you've just finished the "Conquer Hoxxes" Assignment or if you have more hours played than ArcticEcho himself, you're part of the DRG community and as such I want to hear your opinion on these matters. If I wanted to only get data from veteran players or people who play Haz5+, I would just ask people on my friends list or in any of the Discord servers that I frequent. Frankly, I'm a little disappointed that some people wanted this type of thing to be elitist. That's not the spirit of DRG, and not very Rock and Stone.

Meta Results

This section is a meta-analysis about how the questions were answered, rather than the answers themselves. Like I alluded to earlier, by making every question optional I can see which Mods/OCs people care enough to provide ratings for. If someone only provides ratings for a primary weapon and secondary weapon of the same class, I can infer that those two weapons are probably the only ones that the player uses and that they play that class significantly more than the others. Similarly if they rate only a dozen Mods/OCs, it can be inferred that those are the Mods/OCs that the person cares most about balancing. Alternatively, not submitting a rating on a Mod/OC could indicate that they haven't tried it enough to get a feel for how balanced it is.

From an orthogonal perspective, I can see how many people voted for a Mod/OC and infer how strongly people care about that option in particular. If something is voted on by 85% of responses, it's safe to conclude that people care more about that option than something that was only voted on by 67% of the responses.

With all of that in mind, here's a breakdown of how people answered the questions:

Average number of questions answered per person

Average number of responses per weapon

While it's pretty obvious that most survey responses included the Flamethrower because it's the first page, it's somewhat gratifying to see that there was only a nominal amount of burnout. In fact: EPC, BRT, and Zhukovs are all around the same number of responses, whereas GK2, M1000, and Boomstick all had comparable numbers of responses to Cryo Cannon, SMG, Breach Cutter, Minigun, Autocannon, and Revolver.

Most and least voted-on Mods/OCs

For this section, I took the count of how many people submitted a rating for each Mod/OC, and divided it by the average number of ratings for that weapon (see the table right above this section). By doing so, this lets me easily find any Mod/OC that received a significantly higher amount of ratings than its weapon's average.

Statistical Analysis

Normal Distribution: achieved!

I am quite pleased to report that using the 1-5 scale as outlined in the Methodology section provided results that were easily turned into a Normal Distribution. For anyone that hasn't taken a Statistics and Probability class, that means that there's a mountain of data points all grouped together in the middle, and there are fewer and fewer data points the further away you get from the middle. This is exactly the type of data I was hoping this survey would generate! For anyone who cares, the Mean of the Mods/OCs' scores is 2.863581535 and the Standard Deviation is 0.39831432. Here's a plot that overlays a Histogram of the 302 Mods/OCs with the Normal Distribution bell curve generated by the data:

Outliers (beyond the 2nd Standard Deviation)

For the non-statisticians reading this blog post, there's a bit of "rule" regarding Normal Distributions: 68.27% of the data should be within one Standard Deviation of the Mean, 27.18% should be between one and two Standard Deviations away from Mean, and the last 4.55% of the data should be outside two Standard Deviations. Using the Mean and Standard Deviation listed above, 69.205% of the 302 Mods/OCs live within 1 Std. Dev., 25.828% between 1-2 Std. Devs., and the last 4.967% are outside 2 Std. Devs. In this case, I'm choosing to highlight the 15 Mods/OCs that fall outside of two Standard Deviations from Mean as the "outliers" most desperately in need of a balance change in Update 34, according to the survey results.

Ranking the Weapons, Mods, and OCs in terms of how balanced they are

Weapons

By taking the absoloute value of [3 - Rating] for every Mod/OC a weapon has, summing up all of those differences, and then dividing that sum by how many Mods/OCs were counted, it's possible to reduce the "micro balance" of each weapon to just one number. According the survey, here are the 16 weapons ranked in order of having the fewest "micro balance" issues at the top to the ones most in need a balance pass at the bottom:

To be clear: this ordering does not represent how balanced each weapon is on a "macro" scale (weapon-to-weapon, or class-to-class). The third column can be used to indicate the general power level indicated by the results, but that one number isn't as meaningful as seeing the individual scores of every Mod/OC in a weapon. That data is available at the very bottom of this post.

Mod Tiers

Another way to look at the data is by subtracting the lowest score in each Mod tier (or Overclock group) from the highest score, and seeing what the difference is. The closer to 0 this number is, the more balanced that group is against each other. Here's the 16 weapons ranked by the sum of all 5 Mod tiers' [high - low] scores in ascending order (top means they're well-balanced, bottom means they're not):

1. SMG
2. Zhukovs
3. Subata
4. M1000
5. Revolver
6. GK2
7. Shotgun
8. Boomstick
9. Cryo Cannon
10. Flamethrower
11. Autocannon
12. BRT
13. Breach Cutter
14. PGL
15. Minigun
16. EPC

Overclocks

Similarly, here's the 16 weapons ranked by their Overclocks' [high - low] scores in ascending order:

1. Shotgun
2. Zhukovs
3. Subata
4. Minigun
5. M1000
6. Cryo Cannon
7. Flamethrower
8. PGL
9. BRT
10. Autocannon
11. EPC
12. Revolver
13. Boomstick
14. GK2
15. SMG
16. Breach Cutter

Highest and Lowest Rated Mods/OCs

Here's the section that some of you are probably reading this blog post for: the 30 highest-rated and 30 lowest-rated Mods and Overclocks according the survey:

Conclusions and Observations

Because this post is already excessively long, I'm choosing to confine my comments in this section to be primarily objective assessments of the data itself. I'm already planning to write a second post with my subjective interpretation of the results, and I hope to get that published sometime on Friday.

Take the weapons to the gym, because players want them to get buff(ed)

One of the goals I have for DRG's weapon balance is to get most Mods/OCs to have scores that fall within the 2.7 - 3.3 range, and nothing would be less than 2.4 or higher than 3.6. Using those arbitrary ranges:

• 44 Mods/OCs have a score less than 2.4
• 50 have a score between 2.4 - 2.7
• 177 are well-balanced are have a score between 2.7 - 3.3
• 27 have a score between 3.3 - 3.6
• 4 have a score greater than 3.6

It's pretty easy to see that right now, the DRG community wants a LOT of buffs in U34. Almost one third of all 302 Mods/OCs fall below 2.7, yet only one tenth are above 3.3. The fact that the average of all 302 Mods/OCs is only 2.86 (instead of the desired 3) is pretty telling on its own.

Expanding the scope from the granular Mods/OCs level and looking at the weapons overall: GK2, M1000, Zhukovs, Subata, and Breach Cutter are the 5 weapons with the most number of Mods and OCs that people want buffed.

This data mirrors a lot of the opinions already held by the community

As a scientist, it's always exciting to get data from an experiment that subverts my expectations -- but that wasn't the case for this survey. The top 30 and bottom 30 are veritable "who's who" lists of Mods and Overclocks that people have already identified as unbalanced. So, although this survey didn't provide any "new" information, at the very least it provided concrete numbers to assign to each of the problematic upgrades. Most importantly: we can use these results formed by a consensus of almost 300 players as the foundation upon which we form our weapon balance discussions until Update 34 arrives.

I'm hopeful that this data can be used to definitively answer "what needs to be changed", and let us move onto "how do we change it?"

Misc. Results

This section is dedicated to the first page of the survey, where I gathered a little information unrelated to the Mods/OCs. I asked people to enter their usernames so I would be able to identify if anyone submitted the survey twice, evaluate their own skill levels on each class, and at the end of the page there was an optional short answer field at the bottom with the prompt "Rock and Stone!".

Plot of responses received vs time

Self-reported skill levels of each class

Distribution of responses to "Rock and Stone!" optional question

Raw Data

This section will list the 302 average "balance ratings" for every weapon Mod and Overclock. I'm choosing not to disclose the number of votes for each mod/OC in this blog post, but if you're interested in seeing that data feel free to get in contact with me on the DRG Community Tools Discord server.