A Call to Arms - MoP Mechanics Testing

Warning: Theorycraft inside.

Moderators: Fridmarr, Worldie, Aergis, theckhd

Re: A Call to Arms - MoP Mechanics Testing

Postby theckhd » Tue Sep 11, 2012 6:31 pm

That addon is dynamite. I just generated a quick data set using my prot and ret gear, and got the following results:
Code: Select all
General model:
     f(x,y) = 3+164./243.6+1/(1/C+0.885/(y+(x-164)./243.6))
Coefficients (with 95% confidence bounds):
       C =       235.1  (235, 235.2)

Goodness of fit:
  SSE: 0.0006873
  R-square: 1
  Adjusted R-square: 1
  RMSE: 0.00431


Curiously, I get a slightly better fit if I allow the str->parry conversion factor to vary:
Code: Select all
General model:
     f(x,y) = 3+164./a+1/(1/C+0.885/(y+(x-164)./a))
Coefficients (with 95% confidence bounds):
       C =       237.5  (237, 238)
       a =       243.9  (243.9, 244)

Goodness of fit:
  SSE: 0.0001621
  R-square: 1
  Adjusted R-square: 1
  RMSE: 0.002122


Now, it's not curious that the fit gets better - more degrees of freedom will do that. But I noticed this gives me a C that matches the warrior version. How sure are we that a is identically 243.6?

Also, I've written a quick MATLAB function that extracts the data from a chat log produced by the addon. If I could convince a bunch of people to install the addon, follow the instructions to generate a chat log, and send me their text files, we can generate a truly massive amount of data and nail these values down very accurately.
"Theck, Bringer of Numbers and Pounding Headaches," courtesy of Grehn|Skipjack.
Simcraft 6.x, Call to Arms 6.0, Talent Spec & Glyph Guide 6.x, Blog: Sacred Duty
User avatar
theckhd
Moderator
 
Posts: 6211
Joined: Thu Jul 31, 2008 3:06 pm
Location: Harrisburg, PA

Re: A Call to Arms - MoP Mechanics Testing

Postby mythor » Wed Sep 12, 2012 5:02 am

I should have made it at the start of beta, took 30 minutes perhaps, and a breeze instead of a bother now. The added significant figures and reduced chance of typo's is a benefit as well.

More data would be lovely, the addon does not check for duplicate entries, as multiple datasets still might give you those, so this has to be filtered out post process.

I still want to believe that the Cp's to be the same :)

On a side note, I remember in early beta that Paladin and Warrior values where matching, does anybody remember when they changed? My guess would be when Blizzard decided that Shield Block would block everything for 6 seconds. I would just love to know their motivation behind the numbers.
mythor
 
Posts: 24
Joined: Sat Aug 25, 2012 4:21 pm

Re: A Call to Arms - MoP Mechanics Testing

Postby mythor » Wed Sep 12, 2012 6:06 am

I updated statslog to also include player class and level to make it easier for mass processing.
mythor
 
Posts: 24
Joined: Sat Aug 25, 2012 4:21 pm

Re: A Call to Arms - MoP Mechanics Testing

Postby NinJOu » Wed Sep 12, 2012 6:48 am

Does your addon detects when stats are modified by a buff ?

Thanks for your work
NinJOu
 
Posts: 3
Joined: Wed Feb 03, 2010 3:26 am

Re: A Call to Arms - MoP Mechanics Testing

Postby mythor » Wed Sep 12, 2012 7:01 am

No it does not. In that case you can manually call /printstats.

I might add it in later, it requires keeping track of all your combat stats.
mythor
 
Posts: 24
Joined: Sat Aug 25, 2012 4:21 pm

Re: A Call to Arms - MoP Mechanics Testing

Postby theckhd » Wed Sep 12, 2012 10:10 am

mythor wrote:I still want to believe that the Cp's to be the same :)

On a side note, I remember in early beta that Paladin and Warrior values where matching, does anybody remember when they changed? My guess would be when Blizzard decided that Shield Block would block everything for 6 seconds. I would just love to know their motivation behind the numbers.

I'll take a look at some more paladin data and see if I can convince myself that I'm under-estimating, but I doubt it.

I forget exactly when the equations changed, but I think it was around the same time as the Vengeance changes were implemented. It could have been earlier, though, and we just didn't notice it until then.
"Theck, Bringer of Numbers and Pounding Headaches," courtesy of Grehn|Skipjack.
Simcraft 6.x, Call to Arms 6.0, Talent Spec & Glyph Guide 6.x, Blog: Sacred Duty
User avatar
theckhd
Moderator
 
Posts: 6211
Joined: Thu Jul 31, 2008 3:06 pm
Location: Harrisburg, PA

Re: A Call to Arms - MoP Mechanics Testing

Postby theckhd » Wed Sep 12, 2012 10:48 am

Taking this data set provided by Pauladin, I think I can make fairly solid statements about our DR equations. Mythor will be pleased. :)

Parry fit (using x=str-baseStr, y=preParry):
Code: Select all
General model:
     f(x,y) = 3+164./a+1./(1/C+k./(y+x./a))
Coefficients (with 95% confidence bounds):
       C =       237.2  (237.2, 237.2)
       a =       243.6  (243.6, 243.6)
       k =       0.886  (0.886, 0.886)

Goodness of fit:
  SSE: 7.574e-011
  R-square: 1
  Adjusted R-square: 1
  RMSE: 7.13e-007


Dodge fit (using x=preDodge):
Code: Select all
General model:
     f(x) = 2+3.01+1/(1/C+k/x)
Coefficients (with 95% confidence bounds):
       C =       66.55  (66.41, 66.69)
       k =       0.886  (0.8856, 0.8863)

Goodness of fit:
  SSE: 0.003583
  R-square: 1
  Adjusted R-square: 1
  RMSE: 0.004888


Block fit, excluding two obviously errant data points caused by ret paladins (low by exactly 10%):
Code: Select all
General model:
     f(x) = 13+1/(1/C+k/x)
Coefficients (with 95% confidence bounds):
       C =       150.4  (150.3, 150.5)
       k =       0.886  (0.8859, 0.8861)

Goodness of fit:
  SSE: 0.0005365
  R-square: 1
  Adjusted R-square: 1
  RMSE: 0.001904


Comments:
1) Our estimate of k=0.885 was slightly off, probably due to a lack of solid data (all previous data sets used rounded values, which limited our accuracy). The consistency of k between all three fits is just too hard to ignore. Most importantly, the parry fit is extremely good; none of the residuals are larger than 0.000005, or 5E-6. The parry fit alone is enough to convince me of k and Cp, both because of the sheer quality of the fit and because it predicts a correctly.
2) The fitted Cp value is fairly consistent with the warrior one. Given how good a fit we have, I suspect that Cp=237.2 for both warriors and paladins.
3) The dodge cap isn't exactly the same as Cataclysm. If I fix the doge cap to the Cata value, I get this fit:
Code: Select all
General model:
     f(x) = 2+3.01+1/(1/65.631440+k/x)
Coefficients (with 95% confidence bounds):
       k =      0.8839  (0.8837, 0.8841)

Goodness of fit:
  SSE: 0.007511
  R-square: 1
  Adjusted R-square: 1
  RMSE: 0.007053

While that's good, it's not as good as the one where we allow C to vary, nor is the k value consistent with the incredible parry fit we have. If I force k=0.884 in the parry fit, I get this:
Code: Select all
General model:
     f(x,y) = 3+164./a+1./(1/C+0.884./(y+x./a))
Coefficients (with 95% confidence bounds):
       C =       235.8  (235.6, 236)
       a =       244.1  (244.1, 244.2)

Goodness of fit:
  SSE: 0.0007984
  R-square: 1
  Adjusted R-square: 1
  RMSE: 0.002307

We already know that this value of a is blatantly wrong, and the RMSE gets almost 4 orders of magnitude worse. Furthermore, the residual errors now include values as high as 0.006, or 6E-3, large enough to cause demonstrable rounding errors on the tooltip.

If we instead assume that k=0.886, I get this fit for dodge:
Code: Select all
General model:
     f(x) = 2+3.01+1/(1/C+0.886/x)
Coefficients (with 95% confidence bounds):
       C =       66.56  (66.5, 66.61)

Goodness of fit:
  SSE: 0.003584
  R-square: 1
  Adjusted R-square: 1
  RMSE: 0.004872

As much as I really, really think it makes sense logically that the dodge cap is the same as the Cata version, I feel like the data just doesn't support it anymore. I can't fathom a guess at why it changed by ~0.9, but it seems to have.
4) Block's cap is slightly altered based on the new value of k; this is also more consistent with the estimates I've seen from warrior data. Fixing k gives us a very clear value:
Code: Select all
General model:
     f(x) = 13+1/(1/C+0.886/x)
Coefficients (with 95% confidence bounds):
       C =       150.4  (150.4, 150.4)

Goodness of fit:
  SSE: 0.0005367
  R-square: 1
  Adjusted R-square: 1
  RMSE: 0.001898



Conclusions:
I think our new best estimates for the following constants are:
k=0.886
Cp=237.20
Cd=66.56
Cb=150.40
"Theck, Bringer of Numbers and Pounding Headaches," courtesy of Grehn|Skipjack.
Simcraft 6.x, Call to Arms 6.0, Talent Spec & Glyph Guide 6.x, Blog: Sacred Duty
User avatar
theckhd
Moderator
 
Posts: 6211
Joined: Thu Jul 31, 2008 3:06 pm
Location: Harrisburg, PA

Re: A Call to Arms - MoP Mechanics Testing

Postby xstratax » Wed Sep 12, 2012 11:35 am

You guys rule, My addon is getting more and more accurate by the day. :D

This is what I have currently for level 90:

Paladin:

Code: Select all
k = .886
c_D = 66.56
c_P = 237.2
c_B = 150.4
Strength Scalar = 951.16
Mastery Scalar = 600
Block Scalar = 1

Warrior:

Code: Select all
k = .956
c_D = 91.03
c_P = 237.2
c_B = 150.4
Strength Scalar = 951.16
Mastery Scalar = 272.6
Block Scalar = (.5/2.2) = .227272


Did I get the wrong values for anything?

<Edit>

Also are base stats (Str & Stam in this case) independent of Class? If not then Ill have to come up with a better solution for my tables.
xstratax
Maintankadonor
 
Posts: 94
Joined: Fri Aug 14, 2009 1:18 pm

Re: A Call to Arms - MoP Mechanics Testing

Postby theckhd » Wed Sep 12, 2012 12:26 pm

I'm pretty sure base stats vary by class. My notes have 164 str for a L85 paladin and 189 str for a L85 warrior.
"Theck, Bringer of Numbers and Pounding Headaches," courtesy of Grehn|Skipjack.
Simcraft 6.x, Call to Arms 6.0, Talent Spec & Glyph Guide 6.x, Blog: Sacred Duty
User avatar
theckhd
Moderator
 
Posts: 6211
Joined: Thu Jul 31, 2008 3:06 pm
Location: Harrisburg, PA

Re: A Call to Arms - MoP Mechanics Testing

Postby xstratax » Wed Sep 12, 2012 12:44 pm

Well crud...so much for the simple solution. Now I have to find a list of values for the various races per Warrior, thankfully your spreadsheet covers Paladin values already.
xstratax
Maintankadonor
 
Posts: 94
Joined: Fri Aug 14, 2009 1:18 pm

Re: A Call to Arms - MoP Mechanics Testing

Postby mythor » Wed Sep 12, 2012 1:28 pm

Early Christmas!

I have to agree that it does not sit well with me that Cp_pala changed, but the data can't be lying.
One guess is that they redid some of their root calculations to make it work for all classes including monks, and just took the nearest cata value for paladins. (isn't guessing fun).

5E-6 I can live with, nice work :)

xstratax the mastery scalar still needs a checkup, i'll try to do that soonish.
mythor
 
Posts: 24
Joined: Sat Aug 25, 2012 4:21 pm

Re: A Call to Arms - MoP Mechanics Testing

Postby mythor » Wed Sep 12, 2012 4:40 pm

Changed statslog to now include the total mastery value instead of the bonus effect (apparently mastery had it own special function for that). And put on some PvP gear to get nice mastery values.

The result:
272.7272 mastery rating is 1% mastery.
Base mastery = 4800 (The 17.6 you see on your char pane when naked)

Data: https://docs.google.com/spreadsheet/ccc ... tcWc#gid=2
mythor
 
Posts: 24
Joined: Sat Aug 25, 2012 4:21 pm

Re: A Call to Arms - MoP Mechanics Testing

Postby mythor » Wed Sep 12, 2012 5:19 pm

Ignore this post.
Pro tip: Don't do stuff while sleeping.
Last edited by mythor on Thu Sep 13, 2012 5:28 am, edited 1 time in total.
mythor
 
Posts: 24
Joined: Sat Aug 25, 2012 4:21 pm

Re: A Call to Arms - MoP Mechanics Testing

Postby xstratax » Wed Sep 12, 2012 8:03 pm

mythor wrote:The result:
272.7272 mastery rating is 1% mastery.



I hope I am not the only one who finds it...humorous(Lack of a better word) that this would be the current mastery per 1% for Crit Block when (.5/2.2) = 0.227272 is the Block per 1% mastery...sorry I find the repeating 2s and 7s oddly amusing for some strange reason
xstratax
Maintankadonor
 
Posts: 94
Joined: Fri Aug 14, 2009 1:18 pm

Re: A Call to Arms - MoP Mechanics Testing

Postby mythor » Thu Sep 13, 2012 6:28 am

Statslog contained a serious error, wow API does not update values it has to calculate it self on all triggers.
This means that when you changed a piece of gear, asking the API for a block value or mastery effect value, it could still return the old value. Even after a delay it sometimes did this. I am not sure if this is calculated in another thread but I do know that the current approach can not be trusted. I disabled the automatic reporting on item change and you now have to call it manual. This seems to return correct values all the time.

This also means my previous dataset for mastery testing was wrong and I have updated those with new correct values.

Using these new values for warrior block scaling with the comments of the blog in mind and the new found C.

Code: Select all
General model:
  mast_fit(x) = 13 + 1 / (1 / 150.4 + 0.956 / (x * a))
  Coefficients (with 95% confidence bounds):
    a =       0.227  (0.2263, 0.2276)

mast_fit_GoF =
           sse: 0.1106
       rsquare: 0.9988
           dfe: 33
    adjrsquare: 0.9988
          rmse: 0.0579


Plugging in the plausible scaling factor.

Code: Select all
General model:
  mast_fit(x) = 13 + 1 / (1 / 150.4 + 0.956 / (x * 0.5 / 2.2)) + b
  Coefficients (with 95% confidence bounds):
    b =   6.625e-07  (-0.02053, 0.02053)

mast_fit_GoF =
           sse: 0.1142
       rsquare: 0.9987
           dfe: 33
    adjrsquare: 0.9987
          rmse: 0.0588
mythor
 
Posts: 24
Joined: Sat Aug 25, 2012 4:21 pm

Re: A Call to Arms - MoP Mechanics Testing

Postby mythor » Thu Sep 13, 2012 7:47 am

Seeing the ~ for warrior dodge in Thecks new blog post made me sad. So I set out to get more and better dodge data, using the DS dodge trinket to get more values, and being able to set base dodge more accurate to 5.0135002136 (for level 90) helped nicely.

Code: Select all
General model:
  dodge_fit(x) = 1 / (1 / Cd + 0.956 / x)
  Coefficients (with 95% confidence bounds):
    Cd =       90.64  (90.64, 90.64)

dodge_fit_GoF =
           sse: 8.5606e-12
       rsquare: 1.0000
           dfe: 46
    adjrsquare: 1.0000
          rmse: 4.3139e-07


Data available here: https://docs.google.com/spreadsheet/ccc ... tcWc#gid=3
mythor
 
Posts: 24
Joined: Sat Aug 25, 2012 4:21 pm

Re: A Call to Arms - MoP Mechanics Testing

Postby mythor » Thu Sep 13, 2012 9:32 am

After jumping through some hoops I got an Agility data set for prot lvl 90 warriors: https://docs.google.com/spreadsheet/ccc ... tcWc#gid=4

Plugging in the data for the entries without dodge from dodge:
Code: Select all
General model:
  agi_dodge_fit(x) = 5 + 135 / Qa + 1 / (1 / 90.64 + 0.956 / (x / Qa))
  Coefficients (with 95% confidence bounds):
    Qa =       1e+04  (1e+04, 1e+04)

agi_dodge_fit_GoF =
           sse: 4.7453e-13
       rsquare: 1.0000
           dfe: 9
    adjrsquare: 1.0000
          rmse: 2.2962e-07


The assumed Qa of 10k sounds very reasonable.

<edit> Noticed I passed wrong data, fixed </edit>
mythor
 
Posts: 24
Joined: Sat Aug 25, 2012 4:21 pm

Re: A Call to Arms - MoP Mechanics Testing

Postby theckhd » Thu Sep 13, 2012 12:06 pm

My blog post went up early, sorry about that. Pauladin and I are doing another round of analysis, because we discovered some weird bugs in the data. We've improved our estimates some since I wrote that blog post (which was supposed to be scheduled for tomorrow anyway, so I'd have time to finish analyzing stuff - damn Wordpress). I took the blog post down so that I can work on it this afternoon and evening.

Pauladin showed me that his residual data had some strange curvature which suggested that our values could be improved, and after some more work in MATLAB we managed to fine tune most of the constants out to 6 or 7 decimal places.


Our updating fitting has the agility conversion factor at exactly 10k (not 10025 as earlier guessed). So it's good to see your scame out to that as well.
"Theck, Bringer of Numbers and Pounding Headaches," courtesy of Grehn|Skipjack.
Simcraft 6.x, Call to Arms 6.0, Talent Spec & Glyph Guide 6.x, Blog: Sacred Duty
User avatar
theckhd
Moderator
 
Posts: 6211
Joined: Thu Jul 31, 2008 3:06 pm
Location: Harrisburg, PA

Re: A Call to Arms - MoP Mechanics Testing

Postby KysenMurrin » Thu Sep 13, 2012 12:07 pm

(Wordpress scheduling is broken as hell. I gave up on it once it stuck a post in scheduling limbo and refused to publish it no matter what I did.)
KysenMurrin
 
Posts: 4816
Joined: Thu Jun 26, 2008 6:37 am
Location: UK

Re: A Call to Arms - MoP Mechanics Testing

Postby theckhd » Thu Sep 13, 2012 1:38 pm

Mythor: Can you give more sig figs than that? I fit your first data set for dodge and got:
Code: Select all
General model:
     f(x) = 5+135/1e4+1/(1/C+0.956/x)
Coefficients (with 95% confidence bounds):
       C =       90.64  (90.64, 90.64)

Goodness of fit:
  SSE: 8.514e-012
  R-square: 1
  Adjusted R-square: 1
  RMSE: 4.302e-007

And then extracted more precision with:
Code: Select all
>> num2str(coeffvalues(warrDodgeModel)','%10.10f')

90.6425052550


However, your RMSE is about half of what mine is, for some reason. Not that it's a big deal, since we're talking errors well below the rounding limit, but still. I'd like to have the most accurate value I can.
"Theck, Bringer of Numbers and Pounding Headaches," courtesy of Grehn|Skipjack.
Simcraft 6.x, Call to Arms 6.0, Talent Spec & Glyph Guide 6.x, Blog: Sacred Duty
User avatar
theckhd
Moderator
 
Posts: 6211
Joined: Thu Jul 31, 2008 3:06 pm
Location: Harrisburg, PA

Re: A Call to Arms - MoP Mechanics Testing

Postby theckhd » Thu Sep 13, 2012 1:45 pm

Update: fitting all of your data (both the "dodge test" and "agi test" data sets) gives me this surface fit:


Code: Select all
General model:
     f(x,y) = 5+135/1e4+((x-135)/1e4+y)/(((x-135)/1e4+y)/C+0.956)
Coefficients (with 95% confidence bounds):
       C =       90.64  (90.64, 90.64)

Goodness of fit:
  SSE: 1.048e-011
  R-square: 1
  Adjusted R-square: 1
  RMSE: 4.179e-007




Code: Select all
>> num2str(coeffvalues(war2dDodgeModel)','%10.10f')

90.6425070888


I think that we can feel certain of at least 90.6425(1)
"Theck, Bringer of Numbers and Pounding Headaches," courtesy of Grehn|Skipjack.
Simcraft 6.x, Call to Arms 6.0, Talent Spec & Glyph Guide 6.x, Blog: Sacred Duty
User avatar
theckhd
Moderator
 
Posts: 6211
Joined: Thu Jul 31, 2008 3:06 pm
Location: Harrisburg, PA

Re: A Call to Arms - MoP Mechanics Testing

Postby mythor » Thu Sep 13, 2012 3:04 pm

Was busy gathering some more data :)

Code: Select all
dodge_fit(x) = 5 + 135 / 10000 + 1 / (1 / Cd + 0.956 / x)

           sse: 8.5182e-12
       rsquare: 1.0000
           dfe: 46
    adjrsquare: 1.0000
          rmse: 4.3032e-07

Cd = 90.6425009496


Dodge and Agi set (x = x - baseAgi):
Code: Select all
agi_dodge_fit(x,y) = 5 + 135 / 10000 + (x / 10000 + y) / (( x / 10000 + y)/ C + 0.956)

           sse: 9.6901e-12
       rsquare: 1.0000
           dfe: 60
    adjrsquare: 1.0000
          rmse: 4.0187e-07

Cd = 90.6424918568


Dodge + Agi + Parry set (some gear still had some agi on it) (x is again pre-processed)

Code: Select all
agi_dodge_fit(x,y) = 5 + 135 / 10000 + (x / 10000 + y) / ((x / 10000 + y) / C + 0.956)

           sse: 1.1450e-11
       rsquare: 1.0000
           dfe: 93
    adjrsquare: 1.0000
          rmse: 3.5088e-07

Cd = 90.6424868793


It's moving more then I would like, I'll recheck data sets so see if I made a mistake.
mythor
 
Posts: 24
Joined: Sat Aug 25, 2012 4:21 pm

Re: A Call to Arms - MoP Mechanics Testing

Postby theckhd » Fri Sep 14, 2012 6:21 am

Can you also give confidence intervals?

I.e.
Code: Select all
>> num2str(coeffvalues(fittedmodel1),'%10.15f')

ans =

150.375950272320640


Code: Select all
>> num2str(coeffvalues(fittedmodel1)-confint(fittedmodel1),'%10.15f')

ans =

0.000009426271134
-0.000009426271134


That's the block cap, obviously. In the blog post going live in 40 minutes, I have most of the paladin ones out to nearly that accuracy.
"Theck, Bringer of Numbers and Pounding Headaches," courtesy of Grehn|Skipjack.
Simcraft 6.x, Call to Arms 6.0, Talent Spec & Glyph Guide 6.x, Blog: Sacred Duty
User avatar
theckhd
Moderator
 
Posts: 6211
Joined: Thu Jul 31, 2008 3:06 pm
Location: Harrisburg, PA

Re: A Call to Arms - MoP Mechanics Testing

Postby `Davitz » Fri Sep 14, 2012 7:54 am

Hey Theck, is there any eta on when all the info for MoP will be compiled in the Matlab thread or into a kind of "okay here's how you do it right" guide?
`Davitz
 
Posts: 1
Joined: Sun Sep 27, 2009 8:17 am

Re: A Call to Arms - MoP Mechanics Testing

Postby theckhd » Fri Sep 14, 2012 8:01 am

`Davitz wrote:Hey Theck, is there any eta on when all the info for MoP will be compiled in the Matlab thread or into a kind of "okay here's how you do it right" guide?

Most of the preliminary data is already posted in the thread further on. I just haven't gotten around to doing some finishing touches and writing everything up. I am hoping to get that all done next week though. Possibly this weekend if I get more time than I expect, but we have family visiting, so I'm not optimistic.
"Theck, Bringer of Numbers and Pounding Headaches," courtesy of Grehn|Skipjack.
Simcraft 6.x, Call to Arms 6.0, Talent Spec & Glyph Guide 6.x, Blog: Sacred Duty
User avatar
theckhd
Moderator
 
Posts: 6211
Joined: Thu Jul 31, 2008 3:06 pm
Location: Harrisburg, PA

PreviousNext

Return to Advanced Theorycraft and Calculations

Who is online

Users browsing this forum: Yahoo [Bot] and 1 guest

Who is online

In total there are 2 users online :: 1 registered, 0 hidden and 1 guest (based on users active over the past 5 minutes)
Most users ever online was 380 on Tue Oct 14, 2008 6:28 pm

Users browsing this forum: Yahoo [Bot] and 1 guest