An Attempt of an Objective Look at Scenario Balance in RoR

[MasterYoda]

Hi everyone!


as most everyone who spends nowadays any amount of time in RoR knows is that there is a lot of bitterness in both realms about the balance of the classes in the global chat channels. That bitterness ranges from frustration with scenario win rates to a frustration with the generally overpowered classes and abilities of the opposing realm to the frustration of the overpoweredness of classes of their own realm that one happens to currently not play. In my experience as someone who likes to play both sides (and stick to the side I chose for the day!) is it universal to both realms and can best be summarized as the 'the grass is always greener on the other side' attitude.

However when people are vocally complaining about class balance on both sides - may it be on the global chat or in the forums - is it always done through the lens of their own experiences and skills. This is to say it is highly subjective and not an objective metric based on which fair decisions can be made - and therefore a metric the developers can orient themselves on and act upon.

Finding such an objective metric that testifies that something should be changed however is very hard; and this forum post most certainly won't provide an answer on how to find this. Instead what I am trying to intend with this post is to deliver an attempt on how to look at the balance of the game more objectively by using the data publicly available from https://killboard.returnofreckoning.com/

The RoR killboard has a section listing details about all the past scenarios https://killboard.returnofreckoning.com/scenarios going into the same detail available at the end of scenario overview in-game, including player names, level, renown rank, total damage done, damage leading to kill, deathblows, healing, protection, objective score, etc. The killboard scenario information furthermore entails the total points achieved by both realms and therefore who won the scenario. The amount of scenario information goes back to the 1st of april and entails a differentiation between pug scenarios, regular standard scenarios, city scenarios and group-ranked scenarios all for the tiers T1, mid-tier and T4. However no solo-ranked scenario information is present in the dataset because either no solo-ranked scenario took place or the API blocks access for some reason.

I will try to take a look at this data in the following sections.

DISCLAIMER: I am not a professional data scientist though I am currently working in related trade. Due to the fact that I am not doing this professionally, have only a limited timeframe of about 1-2 slow work days to write this, may not have the best angles to view the data, am working with a limited amount of information of only the past 3 weeks and am working with scenario only data to make a statement about the balance of the game: PLEASE KEEP IN MIND THAT THE STATEMENTS ABOUT CLASS BALANCE ARE NOT PERFECT AND BY NO MEANS ABSOLUTE. Consider this: Maybe I have bugs in the code and all data presented is wrong. Take it with a grain of salt. As I said prior it's only an attempt on how I think class balance should be discussed in RoR.

All sections below refer to the following code repository: https://github.com/PaulPauls/ror-killboard-analysis which is intentionally kept in pure python to allow for easy reproducability of the results listed below. The repository also contains all json data that was used for the creation of the results. Installation is easily done by executing pip -r requirements.txt to install all required library dependencies. The code is only tested on Linux.



Data Collection and Preprocessing

The data collection process is relatively simple, though technical. Run download_killboard_scenario_listings.py to create a list of all the unique ids for each scenario and use download_killboard_scenario_statistics.py to download the detailed stats of those scenaios, which are retrieved based on their prior obtained unique id. Both scenario listings and statistics are saved locally as json. Both scripts access the GraphQL API at https://production-api.waremu.com/graphql, which supplies an exemplary instruction on how to use the API. From there on it's reasonably simple by sending the correctly formatted json requests to the API and saving the json answer as demonstrated in both download scripts.

I was only able to scrape 14.586 scenario listings even though at the time of me scraping the database (central EU noon of 21. April 2022) there were 14.822 scenarios listed. There seems to be an issue with the scraping in that it skips over certain database entries. Particularly noticeable is the skipping over city and group-ranked listings, so I requested data on them seperately to ensure I have all 36 city instances and 50 group ranked instances that happened since the 1st of April until the time of me scraping the database available for analysis.

All scenario statistics are then seperated into multiple json files using subdivide_scenario_statistics.py with the intention to analyse different type of scenarios in-game separately. This results in the available datasets:

Code: Select all

scenario_statistics_t1_pug.json
scenario_statistics_t1_standard.json
scenario_statistics_mid-tier_pug.json
scenario_statistics_mid-tier_standard.json
scenario_statistics_t4_pug.json
scenario_statistics_t4_standard.json
scenario_statistics_t4_city.json
scenario_statistics_t4_group-ranked.json

The names of the datasets are self-explanatory for the most part, though a word to 'pug' scenarios: The pug scenario is the scenario in game that can only be joined solo or as a group of 2 in game. Which scenario is the current pug scenario is checked with the .pug command in chat, though it is usually Nordenwatch. PUG scenarios allow people to queue for scenarios if they have the feeling that premades of the opposing realm are dominating regular scenarios too much, so it is an interesting separate statistic to look at as it controls for premades.



Looking at Realm Win Rates

The first and most trivial statistic to look at are the realm win rates. Let's take a look at a couple and see if there are some outliers:

Code: Select all

$ python3 data_analysis_realm_win_rate.py

Code: Select all

Source: ./subdivided_scenario_statistics/scenario_statistics_t4_standard.json

Total Scenarios:    5835
Order Wins:         2855 (48.9%) [total points: 1985633]
Destro Wins:        2978 (51.0%) [total points: 1970324]
Draws:                 2 (0.0%)

Code: Select all

Source: ./subdivided_scenario_statistics/scenario_statistics_t4_pug.json

Total Scenarios:     525
Order Wins:          211 (40.2%) [total points: 152780]
Destro Wins:         314 (59.8%) [total points: 180265]
Draws:                 0 (0.0%)

Code: Select all

Source: ./subdivided_scenario_statistics/scenario_statistics_t4_city.json

Total Scenarios:      34
Order Wins:           18 (52.9%) [total points: 9030]
Destro Wins:          10 (29.4%) [total points: 7280]
Draws:                 6 (17.6%)

Code: Select all

Source: ./subdivided_scenario_statistics/scenario_statistics_mid-tier_standard.json

Total Scenarios:    5174
Order Wins:         3060 (59.1%) [total points: 1954039]
Destro Wins:        2113 (40.8%) [total points: 1588487]
Draws:                 1 (0.0%)

Code: Select all

Source: ./subdivided_scenario_statistics/scenario_statistics_mid-tier_pug.json

Total Scenarios:     387
Order Wins:          262 (67.7%) [total points: 152381]
Destro Wins:         125 (32.3%) [total points: 104366]
Draws:                 0 (0.0%)

Code: Select all

Source: ./subdivided_scenario_statistics/scenario_statistics_t1_standard.json

Total Scenarios:    2283
Order Wins:         1325 (58.0%) [total points: 858741]
Destro Wins:         958 (42.0%) [total points: 722871]
Draws:                 0 (0.0%)

Delightfully seems the most played scenario type - the standard t4 scenario - very balanced, which is great to hear and a testament to the efforts to balance the game by the devs, as this type of scenario surely is an aspect they consider. The T4 pug scenario however is considerably in the favour of destro whereas the T4 city scenarios are clearly dominated by order. I wouldn't consider either to be an indicator of imbalance yet though as one has to remember that the especially for city the sample size is very low and timeframe of the samples is also short within 3 weeks. Though an interesting observation nonetheless.

Mid-tier and T1 on the other hand seem considerably favoured toward order. A 60/40 win rate difference isn't all too critical however the sample size is high enough that a considerable trend can be spotter. However whereas the sample size is high enough is the sample timeframe not large enough and it very well just might be that in those 3 weeks very talented order players created twinks and were putting the odds in their favour during this time. It might change throughout a longer period of time where players have enough time to level out of mid-tier, so keeping that mid-tier win rate in mind should be interesting.

Whereas 3 weeks is a sufficient amount of time to easily level out of T1 and the sample size for T1 with 2283 scenarios is high enough to make a meaningful statement. If the higher win-rate for order in T1 scenarios is there owed to the fact that order classes seem to have an edge in T1 or if simply more skilled veterans were creating twinks on order during this time is an interesting aspect to consider, though impossible to check as the data does not allow me to check if any character belongs to an account with higher level characters.



Looking at Career Win Rates

Moving on to career win rates, a similar metric to realm win rates but breaking down the win rates by class. This is calculated as per player in scenario, meanining if 2 Swordmasters were in a scenario that was lost then that class was assigned 2 losses. If 5 Swordmasters were in a winning scenario then the class was assigned 5 wins. Let's look at some results:

Code: Select all

$ python3 data_analysis_career_win_rate.py

Code: Select all

Source: ./subdivided_scenario_statistics/scenario_statistics_t4_standard.json

Order Win-Rate: 48.9%
Order Careers Win-Rate:
SWORDMASTER:    48.9% (0.0%)
IRONBREAKER:    48.8% (-0.1%)
KNIGHT:         53.4% (+4.5%)
WHITE_LION:     48.7% (-0.2%)
SLAYER:         51.3% (+2.4%)
WITCH_HUNTER:   42.1% (-6.8%)
ENGINEER:       42.4% (-6.5%)
SHADOW_WARRIOR: 42.9% (-6.0%)
BRIGHT_WIZARD:  45.6% (-3.3%)
ARCHMAGE:       47.0% (-1.9%)
RUNE_PRIEST:    51.6% (+2.7%)
WARRIOR_PRIEST: 51.7% (+2.8%)

Destro Win-Rate: 51.0%
Destro Careers Win-Rate:
BLACK_ORC:      53.7% (+2.7%)
BLACKGUARD:     55.0% (+4.0%)
CHOSEN:         52.9% (+1.9%)
MARAUDER:       47.5% (-3.5%)
CHOPPA:         49.4% (-1.6%)
WITCH_ELF:      47.4% (-3.6%)
MAGUS:          45.0% (-6.0%)
SQUIG_HERDER:   48.0% (-3.0%)
SORCERER:       46.9% (-4.1%)
SHAMAN:         51.7% (+0.7%)
ZEALOT:         53.6% (+2.6%)
DISCIPLE:       53.0% (+2.0%)

Code: Select all

Source: ./subdivided_scenario_statistics/scenario_statistics_t4_city.json

Order Win-Rate: 52.9%
Order Careers Win-Rate:
SWORDMASTER:    54.2% (+1.3%)
IRONBREAKER:    67.7% (+14.8%)
KNIGHT:         71.4% (+18.5%)
WHITE_LION:     72.7% (+19.8%)
SLAYER:         65.6% (+12.7%)
WITCH_HUNTER:   48.6% (-4.3%)
ENGINEER:       75.0% (+22.1%)
SHADOW_WARRIOR: 50.0% (-2.9%)
BRIGHT_WIZARD:  69.0% (+16.1%)
ARCHMAGE:       72.1% (+19.2%)
RUNE_PRIEST:    66.7% (+13.8%)
WARRIOR_PRIEST: 61.1% (+8.2%)

Destro Win-Rate: 29.4%
Destro Careers Win-Rate:
BLACK_ORC:      31.0% (+1.6%)
BLACKGUARD:     44.4% (+15.0%)
CHOSEN:         38.6% (+9.2%)
MARAUDER:       38.6% (+9.2%)
CHOPPA:         36.0% (+6.6%)
WITCH_ELF:      26.5% (-2.9%)
MAGUS:          25.0% (-4.4%)
SQUIG_HERDER:   30.0% (+0.6%)
SORCERER:       34.4% (+5.0%)
SHAMAN:         38.2% (+8.8%)
ZEALOT:         33.3% (+3.9%)
DISCIPLE:       37.5% (+8.1%)

Code: Select all

Source: ./subdivided_scenario_statistics/scenario_statistics_mid-tier_standard.json

Order Win-Rate: 59.1%
Order Careers Win-Rate:
SWORDMASTER:    62.6% (+3.5%)
IRONBREAKER:    61.2% (+2.1%)
KNIGHT:         64.0% (+4.9%)
WHITE_LION:     57.9% (-1.2%)
SLAYER:         57.7% (-1.4%)
WITCH_HUNTER:   56.3% (-2.8%)
ENGINEER:       55.0% (-4.1%)
SHADOW_WARRIOR: 53.3% (-5.8%)
BRIGHT_WIZARD:  56.8% (-2.3%)
ARCHMAGE:       61.8% (+2.7%)
RUNE_PRIEST:    63.5% (+4.4%)
WARRIOR_PRIEST: 62.2% (+3.1%)

Destro Win-Rate: 40.8%
Destro Careers Win-Rate:
BLACK_ORC:      40.4% (-0.4%)
BLACKGUARD:     42.2% (+1.4%)
CHOSEN:         41.2% (+0.4%)
MARAUDER:       38.3% (-2.5%)
CHOPPA:         37.2% (-3.6%)
WITCH_ELF:      35.5% (-5.3%)
MAGUS:          40.2% (-0.6%)
SQUIG_HERDER:   36.6% (-4.2%)
SORCERER:       36.4% (-4.4%)
SHAMAN:         40.6% (-0.2%)
ZEALOT:         46.9% (+6.1%)
DISCIPLE:       42.4% (+1.6%)

Looking at career win rates is important as it encapsulates the utility benefit of a class, something the pure relative DD/Tanking/Healing comparison performed below does not take into consideration. And even though the T4 city career win rates are neglible because the sample size is simply too low did I still include it as I knew people were certainly interested in seeing city performance of certain classes.

Those career win rates reveal that there are no extreme outliers that are very considerable, though in the end every player and therefore class is only 1 piece in a group of 5-23 others - so extreme outliers on a group rating aren't to be expected. However to determine a slight trend that is valid for both realms one can say that in order to win T4 scenarios you best hope for Tanks > Healers > melee DDs > ranged DDs, whereas in mid-tier on order side you best hope for Tanks = Healers > melee DDs > ranged DDs and on destro mid-tier side you best hope for Healers > Tanks > ranged DDs > melee DDs.

Either way it seems that the often repeated advice to play a tank or healer in global chat when someone is asking what classes are needed seems to be a reasonable one.

However to mitigate the potential bias towards certain classes that might stem from such a career win rate analysis is it important to remember that these statistics only show the win-rate and skill of the average player and also penalize a utility classes twice if only 1 of those classes is needed to apply their utility. E.g. utility classes that apply the 'Pierce Defenses' tactic debuff of -15% to block/dodge/parry don't stack if both are applying them as AoE, reducing their utility. That doesn't mean that a class that is specced to apply such debuffs isn't highly useful in an organized setting - however in a random standard scenario 2 of those classes might be penalized double when losing even though their utility could only be applied once.

As a general rule of thumb and also from my personal experience can all classes be highly useful if played right - no matter the setting. However some are harder to play right than others, a fact that is not showing when looking at the average. So please take note of my efforts to get you to take these statistics - while interesting - with a grain of salt.



Comparing Damage Dealers Intrarealm Relative to one Another

The next section of the analysis is a bit more difficult to perform as the damage, damage leading to kill (aka kill-damage) and the deathblows are very hard to compare to one another across scenarios and realms. E.g. a simple analysis of the mean damage, kill-damage and deathblows of every class in all available scenarios would not take into consideration if a scenario was long and drawn-out (leading to more total damage, kill-damage and deathblows for everyone) or if it was a short stomp of either side. Or maybe it is a total engagement domination of one side and in order to maximize their kills they disregard objective to draw the fight out longer.

The point is in order to compare class performances across scenarios one has to control for the 'engagement-intensity' of the scenarios. Therefore in order to determine average damage of a class in the scenarios I will determine the average damage/kill-damage/deathblows share of the total damage/kill-damage/deathblows done in the respective scenario and later extrapolate this average stat share in scenario to the average absolute stat across all scenarios. This leads to the effect that if one side is dominated and their DDs are only able to do A:5_000 and B:3_000 in one scenario but A:35_000 and B:40_000 in the next scenario, their average damage is calculated as follows:

Code: Select all

total_damage_sc1 = 5_000 + 3_000              # 8_000
total_damage_sc2 = 35_000 + 40_000            # 75_000

dmg_share_A_sc1 = 5_000 / total_damage_sc1    # 0.625
dmg_share_B_sc1 = 3_000 / total_damage_sc1    # 0.375

dmg_share_A_sc2 = 35_000 / total_damage_sc2   # 0.47
dmg_share_B_sc2 = 40_000 / total_damage_sc2   # 0.53

avg_dmg_share_A = (dmg_share_A_sc1 + dmg_share_A_sc2) / 2   # 0.55
avg_dmg_share_B = (dmg_share_B_sc1 + dmg_share_B_sc2) / 2   # 0.45

total_damage = total_damage_sc1 + total_damage_sc2   # 83_000

avg_dmg_A = avg_dmg_share_A * total_damage           # 45_650
avg_dmg_B = avg_dmg_share_B * total_damage           # 37_350

This IMO leads to better controlled scenario statistic values as it controls for one-sided dominations as well as long drawn out scenarios.

Furthermore, setting aside the prospect of comparing DD stats across realms (which may be somewhat possible in a limited sense when controlling for win/loss ratio, but more on that later) let's stay at the topic of comparing damage dealers intrarealm. If class A is very lucky and only every joins scenarios that are going very favourable for his realm then he will be able to achieve very high average damage values as he is playing to his full potential with a good team supporting him. However if class B is very unlucky and this class only every joins scenarios that are going very badly he will never play to his full potential and will only be able to achieve very low damage stats. Comparing the stats of both classes is accordingly unfair if one class has had the luck to join more scenarios that allowed for favourable conditions for damage dealers. Therefore will I also control for each class present in every scenario by only comparing the scenario damage stats of a class to those classes that were also present in that scenario.

The damage stats of a class are therefore only compared to another class if that class was also present in the same scenario to allow for a fair comparison.

But we have to control for more things than that to try to be as fair as possible in the comparison. For example what about classes that joined the scenario late and were not able to unfold their full potential? It is hard to control for that since the scraped scenario data does not contain a point in time when a player joined the scenario, so for now all character stats are disregarded that did less than 10% of the total damage of the top player. This value can certainly be tweaked, though for now and given the amount of information I would have to hope that a large sample size allows for each class to join somewhat equally late.

At last do we also control for tanks and healers that don't intend to play in an offensive spec but instead focus on healing and protection. It would be unfair to consider them in the average of offensively specced tanks/healers as they would water down the average without even intending to excel at damage, kill-damage and deathblows. Therefore all classes with more healing + protection than total damage are also disregarded for the damage dealer comparison.

With all those factors being controlled, let's have a look at some DD comparison tables:

How to read these tables, using the first table below as an example:
Each cell specifies on the left of the top line the damage done by the row class and on the right of the top line the damage done by the column class. The middle line specifies the relation between both and a relation > 1 means that the row class on average does more damage in scenarios than the column class. A relation < 1 means the opposite. The bottom line specifies on how many datapoints the average has been determined. The right of a row consists of the average damage and relation done throughout the whole row.
Taking for example upper-left most cell as an example (row: Ironbreaker, column: Swordmaster) does this mean that in 390 scenarios there were both a Swordmaster and an Ironbreaker present and they were both offensively specced in that they did more damage than healing+protection. On average does an Ironbreaker in those 390 scenarios 65_317 damage, whereas a Swordmaster is doing 79_268 damage. The offensively specced Ironbreaker therefore did only 82% (=0.82) of the damage an offensively specced Swordmaster did in those 390 scenarios. The row average tells us that throughout all relative comparisons the ironbreaker does on average 65_896 damage whereas the relative other class is doing 108_799 damage.

Code: Select all

$ python3 data_analysis_dd_comparison.py

Now, I am aware that this is a huge information dump and any extensive interpretation would exceed the frame of such a forum post and since I explained above on how to read and hopefully understand these tables do I hope that everyone will be able to get some insights from these tables themselves.

Though what I do want to shortly point out is the order of tables shown. The first pair of tables shows the mean damage of each class in relation to other classes if they were in the same scenario (and offensively specced) in regular T4 standard scenarios for order and destro respectively. The second pair likewise shows stats about the T4 standard scenarios but in this case it's damage leading to kill (kill-damage). The third pair shows the deathblows achieved by each class in relation also in T4 standard scenarios. The last pair shows the mean kill damage in mid-tier scenarios. Now as you see yourself this is only a fraction of the information tables that are possible to be shown. Feel free to download the repository, install the dependencies and execute the data analysis on different scenarios like T4 pugs, T1 standard, T4 city, etc.

However, while I won't go into an extensive interpretation of each table do I want to give at least one example of how to use them to gain possible insights.

Let's for example look at Engineers and Magus in T4 standard scenarios mean damage and mean kill-damage comparison (tables 1-4). Engineers are superior in terms of doing raw damage to all classes except for the Slayer and the Runepriest, having a mostly positive damage relation of 1.6, 1.87, 2.12, 1.08, 0.83, 1.54, 1.14, 1.02, 1.14, 0.99 and 1.33. The same thing holds true for Magus being the overall most superior damage dealer on average in T4 scenarios for destro having a consistently positive relation to other classes of 1.49, 1.58, 1.63, 1.2, 1.04, 1.51, 1.2, 1.08, 1.16, 1.24 and 1.35. Looking at the mean total damage therefore implies both being excellent DDs and therefore being prone to be classes that carry their respective realms. However when looking at the win rate of the Engineer and Magus careers does that impression change.

Engineers only have a win rate on average in T4 standard scenarios of 42.4%, 6.6% lower than the average for order. Same goes for Magus having only an average win rate of 45% for destro, 6% lower than the average. This does not compute with them being the allegedly best order and destro overall damage dealers.

So let's take a look at how much of the damage Engineers and Magus do actually leads to a kill and is therefore kill-damage. Let's look at tables 3-4. On order side does every other damage dealer, including offensively specced Archmages and Warrior Priests exceed the Engineer in terms of damage actually leading to a kill, only leaving offensive Tanks and the Runepriest as inferior damage dealers in terms of doing damage that actually leads to a kill. The same thing holds true for Magus on destro side, being exceeded in terms of damage actually leading to a kill by all damage dealers except for offensive Tanks and Zealots.

Now such an insight probably coincides with the generally held opinion that dot damage is often only fluff damage that does not actually lead to the kill and this is an especially striking issue for Engineers and Magus as they are usually played as ranged AoE dot damage dealers. All I have done now is somewhat substantiate the often held subjective opinion that the ranged AoE dotter classes are actually underperforming despite their impressive total damage numbers with data and thereby supply a somewhat objective metric as to why the consideration of them being buffed is valid.

"But oh wait MasterYoda!", you say, "Engineers and Magus are only underperforming according to your data because people don't know how to play the class properly and it's actually a learn-to-play issue!". Well darn, you might be right dear reader. It may help if we only look at the top 50% of Engineer and Magus players...
Well consider yourself lucky because I have some more data for you [people climaxing in the background out of joy]!

Alright, let's look at the T4 standard scenarios kill damage but this time let's cut off the 50% of the worst performances for each class, meaning only the 50% of the best Engineer performances are compared with the 50% of the best White Lion performances, etc. Let's also due an additional little gimmick where we normalize for rr. This time a class having only rr60 has it's damage stats boosted by 33% (80/60 = 1.33) whereas an rr80 (or higher) player has his stats recorded 1:1. It is important to mention that players higher than rr80 are not penalized. Personally I consider such a normalization reasonable as I can very much imagine that a full sov at rr80 has 33% more damage potential than a full vanq at rr60 or more than twice the damage potential of a full Annihilator at rr40 (more so even IMO). So this normalization is a reasonable first approach IMO, but as usual feel free to change those numbers and algorithms yourself in the code. Let's have a look at this l2p-controlled and rr-normalized data:






Looking at the Engineer and Magus data however we can still see that while situation certainly did improve and they are now able to outperform all offensively specced tanks and healers (except for the Disciple on destro) are they still not able to out kill-damage any other damage dealer class of their respective realm.

Now, after thoroughly badmouthing Engineers and Magus, should they be considered a bad class? No, certainly not. Please always take these analysis insights with a grain of salt. After all they are working with limited data and are only looking at scenarios - just one aspect of RoR, a game that is focused on RvR after all. Personally for example I couldn't want enought Engineers or Magus at keep defenses, a static setting where that class excels. But more on that topic in the closing thoughts. Moving on to tanks now:



Comparing Tanks Intrarealm Relative to one Another

Let's look at some tables for tank protection stats only and protection + heal stats seperately, as heal auras by Swordmasters and Black orcs genuinely contribute to the protection and support of the group:

Code: Select all

$ python3 data_analysis_tank_comparison.py

I won't go into too much detail on those tables as by now the way to read them should be understood and since the tables are relatively small do I think that the information contained is self-explanatory. The first 2 tables show protection scores only of the tanks, while the third and fourth table show the healing and protection score of each tank and the closing pair of tables shows all relevant stats of a defensive tank condensed, which is the average damage done (not kill-damge) + the healing done + the protection. All data inspected stems from T4 standard scenarios.

Now, more so than with any other category is it important to keep the purpose of tanks in mind when looking at those tables and inspecting them critically. Swordmasters and Black Orcs seem to excel in T4 scenarios according to this data not least because of their great group healing support using the blade enchantment or war bellow respectively. However tanks are about much more than stats as it their foremost priority to crowd control the enemy, debuff them and buff their own group - all things not reflected in those stat tables. The excellent group buffs of the Ironbreaker are for example not reflected in this table and neither are the excellent crowd control punts of the Blackguard. This becomes evident as you can see that even though the Swordmaster and Black Orcs are dominated those tables total stat wise does the average Knight have a 4.5% higher percentage point chance of winning a scenario than the average Swordmaster and the Blackguard has a 1.3% higher percentage point chance of winning a scenario than the Black Orc. So as with any data analysis section, remember that each stat is only an indicator and does not necessarily tell the full story.

Furthermore, please keep in mind that this time unlike with the DD comparison offensively specced tanks are excluded as they would water down the comparison, same as defensively specced tanks were excluded in the damage dealer comparison. Tanks that achieved less than 10% of the total healing+protection of the top player were furthermore presumed to have joined late and were disregarded.



Comparing Healers Intrarealm Relative to one Another

Let's have a look at the same data comparison we just performed for tanks but looking at healers this time instead. Same as with the data analysis for tanks were healers disregarded that were offensively specced or achieved less than 10% of the total healing+protection of the top player as they were presumed to have joined late.

Code: Select all

$ python3 data_analysis_healer_comparison.py

Again I won't go into much details about those tables as I consider the information contained to be self-explanatory by now. All tables summarize data from the T4 standard scenarios and are ordered in a similar way that the tables were ordered in the tank section - the first pair looking at total average healing, the second pair looking at the total average healing + protection and third pair furthermore including damage into the average total stat comparison.

As with the tanks does the total stats not necessarily paint a full picture, as utility is not considered in those stats. I again seems that one class archetype (Runepriest and Zealot) seem to dominate the healer stats though looking at the average career win rate for healers are Runepriest and Warriorpriest as well as Zealot and Disciple on pretty the same level in terms of the average player enjoying a win at about 51.7% chance of winning for order and 53.3% of winning for destro. The Archmage and Shaman are very close behind in terms of career win rate though they are statwise at about the same level as the Warrior priest and Disciple respectively, though both classes excel in damage dealing potential compared to their 2 respective counterparts as can be seen in the damage dealer section.

This consideration of Archmage and Shaman lacking slightly behind healing and protection potential behind the other classes though excelling at damage dealing potential in comparison bridges over nicely to my closing thoughts section, in which I firmly intend to negate every statement I have done prior as quintessentially looking at class balance in scenarios does not give insight about their performance in RvR wbs, 6mans or when soloing - and therefore should never be the sole metric based upon which a class should be balanced.



Closing Thoughts

As said when I just bridged over to this closing section, would I like to put the insights gained through the analysis of those scenario statistics into perspective. Taking concrete decisions to adjust the balance of the game based on scenarios alone would be a disservice to the game as RoR entails much more than just scenarios. I think it is even reasonable to say that scenarios are only a minor - though important - aspect of RoR as the unique selling point of this game is it's great RvR after all. The RvR in RoR is unmatched by any other game and may RoR be lacking in many areas it is still the best RvR game out there IMO and the primary reason most of us play it after all.

Therefore to balance the game fairly one has to look at the performance of each class and the respective realms in general for each aspect of the game. The balance of the classes in 24man wbs in RvR, in 6man groups roaming the lakes or when classes go soloing either because their archetype is excellently suited for soloing - thinking for example about Witchhunter and Witchelves or Archmages and Shamans. But also of course when classes play in 12-48 player scenarios.

Since I only had scenario data available for my data analysis do I consider some insights one might extract from these data tables to be incomplete as those scenarios on average occured with 11.8 order players and 11.9 destro players (for T4 standard scenarios). This means that classes that excel in smaller groups or even at soloing like the aforementioned Witchhunter and Witchelf, Archmage and Shaman, Ironbreaker and Blackguard or the Shadow Warrior and Squig Hunter are not considered fairly. Same as with classes that excel in the very big fights due to their excellent survivability and utility like Knight and Chosen or their excellent AoE damage and utility like Slayer and Choppa. Or classes that excel in certain settings like sieges where Engineers and Magus are invaluable.

To analyze class balance more fairly would more data be required, particularly data about the RvR lakes in a timeframe that spans more than the measly 3 weeks that were available for this analysis. If such data can be logged and then be made available I would happily try to have another look at it and apply my novice data analytic skills to it. If one might dream we could even go full Big Brother and also log the combat logs and specs of each class so data analytics tells us exactly which skills are overperforming, which tactics are never used (looking at you +252 spiritual resistance...) and which skill progression should be adapted so a class becomes better at an earlier stage so pre-T4 stages are more balanced. All scenario controlled and l2p-controlled so it can be done as objectively as possible.

So, to close this forum post - after you have probably been numbed by my endless monologue by now - is my last request to please disregard all insights you think you might have gained from this forum post and only consider it as an idea of how I think class balance should be decided and discussed in RoR and most of our opinions on class balance is highly subjective after all. The RoR devs have been with the project for years - as volunteers not receiving any compensation for it but deserving nothing but our thanks - and are more experienced than all of us and have insights to objective data we don't.

With that said: Thanks for reading and if you have any question then I'll be reachable in game most weekend as @Yodapriest on order and @Yodadisciple on destro. =)


Cheers

[Acidic]

Wow, that’s impressive.

I am a strong believer in balance being based around facts and numbers, not on looks, feels and my experience.

[lumpi33]

Pretty cool.

Try to use the median value instead of the average value. It's more robust and a better representation of the "typical" value.

I'd like to see what the median total damage, mean damage, healing, protection, kills and deaths for every class is.

A class distribution would be also interesting. How often do you have class x in low tier, midtier and high tier SCs, as well as cities.

EDIT: For classes that can play two roles like heal or dps and tank or dps you should do a split up. If damage > 2x heal or damage > 2x protection then it is a damage dealer, when damage and heal/protection roughly the same then it is a hybrid and when protection/heal > 2x damage then it is a tank or healer.

[Minisynn]

Love your work here, great analysis - I've got one I'd love you to do though;

Compare win rates in a scenario by healer differential. For example if both teams have one healer, or +1 healers over the other team, +2, etc etc. It's no secret that the vast majority of pug scenarios when levelling and at max level *feel* lost just by looking at the rosters of both teams at the start of a scenario and seeing that the other team has 3 more healers than you, but I'd be interested to see if it reflects reality. Can't account for player skill difference of course but hopefully there's a decent enough sample size that some conclusions can be drawn

[Xergon]

A very interesting work... ngl pretty impressive data to look on.

Although despite all this, balancing on pure numbers would make sens if characters would be used by AI not by Humans. That is a big factor that makes balance very hard job to achieve. Regardless, first and one of most important is to define what actually balance means. I'm pretty sure it means very much different for a lot people.

In my opinion, achieving perfect balance between 2 realms w/o mirroring classes will never be possible. That being said i would not want to ever go into that road because it would made gameplay eremy flat and boring. Maybe if you would be able to lets say duplicate each class on opposite realm, with ofc different animation and skill names and so far. This could be interesting experiment but i'm not sure if work put into it would be worth.

Nonetheless, all this desant mean that you should not try to look for balance solutions. Basing balance ideas on numbers can be good base but should not be the only factor. Human error is not something you can easily put into numbers.

[Koteskoo]

Great work and data to look on. Thanks

[Phantasm]

When topics like that pop up out of nowhere I wish there was a up button to just simply appreciate the work someone put into it. Despite i dont fully understand meaning of your post I hope devs which have better idea what that numbers mean can use it to make already great game an even better one.

[BluIzLucky]

...

Woah.. what a monologue, was a fun read, well done you, appreciate your efforts!
You could get a data analysis job if you wanted

While I get that using purely Python was probably more fun as an excerise, you might have been better off (analytically) by just dumping data to Google Sheets and doing boring bar-charts instead of matrix diagrams

Try to use the median value instead of the average value. It's more robust and a better representation of the "typical" value.

Not that I think it matters much but using averages for this kind of analysis is better than using median (because we don't just care about the "typical" user, but the entire spectrum of class performance).

[Omegus]

Apologies if I missed this in your post, but do you have - or can you provide - any insight on the overall role composition of winning scenarios? E.g. how many tanks, how many DPS, and how many healers (in heal spec) were on the team, just to see if having a 2-2-2 setup for each group (would need to assume 4-4-4 was spread equally over 2 groups) greatly increases the chances of winning? The combat in this game is heavily balanced around 2 tank for guard/challenges, 2 DPS, and 2 heals per group.

A comparison of the win rates when just one side is 2-2-2 and the other side is not 2-2-2 would be valuable too.

One absolutely vital piece missing from the stats (and the killboard), is distinguising between premades and solo queue in a scenario. From 100% anecotal evidence, I have seen far more complaints about premades (order or destro) dominating mid-tier scenarios as people form groups to level 6 characters up together as quick as possible, which would greatly distort the win rate of whichever classes/factions are currently being levelled.

Having access to group composition stats would also show insight in the greatest stat of all - premade vs solo Q win ratios. It could also be used to look at class popularity in premades (the "meta") compared to that classes's popularity in scenarios overall.

There's a lot more that can be done with having access to the information about who is in a premade (including examining how frequently premades are paired off against each other for scenarios running at a similar time), but that's on the wishlist for now.

Very nice post .

[Martok]

Although despite all this, balancing on pure numbers would make sense if characters would be used by AI not by Humans. That is a big factor that makes balance very hard job to achieve.

This is true. Playing a class to its strengths as opposed to its weaknesses while in an SC group is a more difficult factor to quantify, but no less important.

The OP work is impressive and I applaud him for both his effort and quality of work. The server needs more people such as he, people willing to dedicate their time to positively contribute. Truly good work.

Admittedly I am still working through the entire post however this section drew my attention:

The point is in order to compare class performances across scenarios one has to control for the 'engagement-intensity' of the scenarios. Therefore in order to determine average damage of a class in the scenarios I will determine the average damage/kill-damage/deathblows share of the total damage/kill-damage/deathblows done in the respective scenario and later extrapolate this average stat share in scenario to the average absolute stat across all scenarios. This leads to the effect that if one side is dominated and their DDs are only able to do A:5_000 and B:3_000 in one scenario but A:35_000 and B:40_000 in the next scenario, their average damage is calculated as follows:

The 'engagement-intensity' term is both an appropriate description for where the most relevant data would manifest itself and valid factor to consider. Again, considering the provision stated I have yet to work through your entire dissertation, can and/or did you consider the effects of a DPS class dealing damage over time while benefiting from a dedicated guard as opposed to not benefiting from such?

Specifically, any DPS class can have their damage contribution numbers affected adversely simply when accounting for the time they may be out of an SC due to respawn and return to the action timers. Given the (most often) thirty second respawn timer combined with the time it may take to return to the battle area, a time which can often exceed one minute of SC game time, this time factor can certainly impact the overall numbers.

One impetus for this question stems from SC play on the night of the 23rd, wherein during two distinct scenarios there were three tanks in the SC group none of which were using guard. Perhaps specific numbers concerning such situations would help promote such base-line gameplay.

Thanks again for your efforts here.