Effectiveness Ratings: Week 7

Hadrian Mertins-Kirkwood's picture
Submitted by Hadrian Mertins... on Tue, 2018-12-18 14:34

Forums:

By request, I've made the full effectiveness ratings publicly available here: http://bit.ly/HMKParityRatingSummary

Stats are updated through Week 7 (December 17).

Note that this week's ratings are calculated using a slightly different formula than the version discussed on Parity Podcast episode #306. The basic approach is unchanged and all tweaks are documented in the methodology section of the leaderboard, which is still available here: http://bit.ly/HMKParityRatingLeaderboard

As always, it's worth reiterating that this particular rating system is as arbitrary as any other. Given the many limitations of the stats, this is my best judgement of the optimal way to evaluate players. Definitely open to criticism and suggestions for how to refine the methodology (or discussion of why this kind of analysis is not valid in the first place).

John Haig

Tue, 2018-12-18 15:12

Where are you grabbing data from?  I had been messing around using the spread sheets posted on the parity stats page, but it doesn't look like those have week 7 on them.

The only observation I had on the ratings was whether you were concerned that the second and third offensive metrics (possession and scoring) tend to relfect favourably for the same type of player?  Both I assume would be much higher for receivers who prefer to score goals or reset the disc rather than making in-cuts and move the disc around.  I'm thinking of the difference between a mid-cutter and a striker.  The touches metric seems to clearly favour contributions made by handlers and mid-cutters while I assume the two others favour strikers, and handlers who like to chuck.

Jim Robinson

Tue, 2018-12-18 15:41

I haven't dug into this in depth, but I feel like you need to consider the number of touches alongside the 'performance' per touch.  If you don't, a player who makes 1 catch for a goal and never touches the disc otherwise would move to the top of the list, despite being mostly useless. Capturing touches is the way to say you do well, and you do it often v. You do well, but rarely. 

John Haig

Tue, 2018-12-18 18:03

One of the factors is touches per point. That’s probably the only factor keeping Hadrian’s numbers thinking I’m even more over rated. 

Christopher Keates

Tue, 2018-12-18 22:50

The top three in catches per point are:

Hadrian

Haig

Kelsey

Wallace

Keates

The top 5 in throws per point are:

Keates

Hadrian

Money

Regular Brian

Haig

I haven't aggregated these into "touches" yet, but touches per point will be similar to the above lists. These are either measures of high utility, high leverage, or selfishness. By this I mean, "you're so good you get the disc a lot!", "you're on a relatively weak team and are used a lot", or "you want the disc and hog it a lot." Some mixture of the three is probably true.

Good news John, you pick it up half as much as me and catch it 12% more than me per point, but you also turn it over about 95% as often as I do, per interaction with the disc! This puts you in rarified air, no matter how you look at it.

 

 

 

Christopher Keates

Tue, 2018-12-18 16:24

It's taken care of now though.

Matthew Cole

Tue, 2018-12-18 21:21

Having noticed that while I had a decent number of Ds, it didn't seem to actually help my team score while I was on D, I tried to come up with a metric to measure this futility.

I call it the "Do nothing defense" metric.  Which could indicate that:

  • you poach which gets a lot of D's but often results in easy scores for the other team
  • after you get a D, in your excitement you often immediately throw it away (quite likely)
  • you played on teams that can't score no matter what you do

Since I am lazy I took Hadrians (Team Defense% - (D's per possession % /1.5)) for this.  The division by 1.5 was to account for the fact that some D's occured on O points and roughly gives the number of D's while playing a D point (maybe Hadrian can actually calculate this value).  The lower the number the more your D's don't actually seem to help your team to score.

Here are the top/bottom five:

Worst:

  • Luca Lafontaine: 18%
  • Kelsey Charie: 19%
  • Mark Donahue: 19%
  • Thomas Sattolo: 19%
  • Neena Sidhu: 21%

Best:

  • Ashlin Kelly: 55%
  • Genevieve Labelle: 53%
  • Kristyn Berquist: 52%
  • Jon Rowe: 50%
  • Emily Kavanagh: 49%
Hadrian Mertins-Kirkwood's picture

Hadrian Mertins...

Tue, 2018-12-18 22:50

You've got to be careful with suggestions like this, Matt, or you'll start getting Parity things named after you (RIP Simon Berry).

It should be possible to calculate what you've suggested using the full game stats (not the publicly-posted spreadsheets). You could basically calculate how often after a player gets a D their team scores on the ensuing possession.

You'd have to do another layer of analysis to determine how much each of your possible explanations mattered. You could specifically look at the completion percentage of a player when they take immediate possession of the disc after a turn (either a catch D or an immediate pickup). For the bad team effect, you could control for team performance in a few different ways. I actually think the poaching hypothesis is a good one, since it explains high-risk, high-reward defensive play.

Christopher Keates

Tue, 2018-12-18 22:54

This is part of why most turnovers are weighted as -5k but most scoring events are ~10k. Partly because a goal is twice as valuable as a D, and partly because negative salaries suck for low event players who may not have as many chances to correct for mistakes they make.

This value will shift each year given quality of the league (and will be different for any league), but it's hovered around every D being worth about 0.45 G for your team for a few years.

Hadrian Mertins-Kirkwood's picture

Hadrian Mertins...

Tue, 2018-12-18 22:43

You're definitely onto something here, John. There are inherent biases in these categories (as there would be with any metrics we could've chosen) that favour certain types of players. Low-touch, high-scoring receivers do especially well, followed by high-touch handlers, compared to unders-cutters who move the disc mostly in the midfield.

But I don't think possession percentage necessarily favours receivers (primarily goal scorers) over passers (primarly disc movers). Yes, handlers and unders-cutters are required to take more risks with the disc, but the volume of passes they make tends to compensate for higher throwaway numbers. Based on my analysis, touches per point played correlates weakly (but positively) with possession percentage. So it doesn't seem like the possession metric favours strikers as you suggest (and the opposite may be true).

Here are two charts to illustrate:

https://docs.google.com/spreadsheets/d/e/2PACX-1vT945F-OwAJY8yE-74yKcLh3...

https://docs.google.com/spreadsheets/d/e/2PACX-1vT945F-OwAJY8yE-74yKcLh3...

Justine Price

Tue, 2018-12-18 22:32

This thread should be titled: An Ode to Alex Bush, though I feel he may be posting a rebuttal soon. Beware the Oncoming Storm. 

Jon Rowe

Wed, 2018-12-19 00:02

That Alex Bush is still alive. Last we heard, he was lost in the Minnesota wilderness in search of a pickup game.

Alex Bush

Fri, 2018-12-21 09:44

I really like the discussion in this thread. Please keep it going.

 

Best line: "Does it even matter?" Answer: No, but neither does frisbee and it's still fun. 

Geofford Seaborn

Fri, 2018-12-21 10:38

:doritos:

Hadrian Mertins-Kirkwood's picture

Hadrian Mertins...

Wed, 2018-12-19 10:16

As Seb, Laura, Keates and others have pointed out, one of the biggest confounding factors in these ratings is the effect of player matchups. Basically, if you're (perceived as) the best player on your team, you will attract the best defender, which limits your effectiveness relative to your linemates. You can see this in the ratings of players like Justine, Proulx and Heather, who are typically forced to take the toughest matchup on the other team. Each of them is rated below several other women on their teams who face weaker matchups, even though Justine, Proulx and Heather are likely the strongest women on the field.

The matchup effect is especially pronounced when a team has 2 strong players on offense but there's only 1 strong defender on the field, which results in the second-best offensive player being disproportionately effective. A good example is Jim Robinson, who played with Martin and Travis for the first 6 weeks. Defences typically prioritize Travis/Martin with their biggest/best male defender, which has created a ton of space for Jim to rack up goals. That's not to discredit Jim at all, because he's an excellent receiver. But he probably wouldn't have 31 goals if he was facing the toughest matchup every week.

What do we do about it? The matchup effect is baked right into the stats so it's hard to correct for. We can't really predict how a player's stats would change if they faced an easier/tougher matchup on average. I suppose you could do a deep dive into the stats to try to adjust for the strength of linemates/defenders on any given point, but that wouldn't necessarily reflect the on-field reality. Another option would be to assign each player a "reputation" score based on their perceived strength in the eyes of defenders, but that's pretty subjective and messy territory.

A better question might be: does it matter? One of the main goals of the ratings is to determine the relative value of players. Even if Jim's "objective" effectiveness is inflated by his weaker matchups, it's still an accurate reflection of his value, since he's going to continue to get those matchups. Is Jim as effective as Matt Cole with an almost identical rating? Maybe not head-to-head, but relative to the matchups they're getting, he probably is.

John Haig

Wed, 2018-12-19 13:32

A better question might be: does it matter? 

I think it matters a lot.  In terms of offensive contributions the league tends to be quite top heavy.  Each team will have one or two men and women who are very good and contribute a lot to their team's success.  Those players tend to be higher in salary.  Teams get in trouble when their high salaried players can't defend the other team's high salaried players.  As your spreadsheet has pointed out, I am a high usage, high turnover cutter who can't generate blocks or convert D points in to scores.  This results in our team having a fairly high salaried player who tends to get lit up by the other team's high salaried players night in and night out.  This causes a big problem for roster building for my GM (sorry Boss!).   

To compensate we've recently had to rely a lot on Nick Amlin to cover the other team's top guy so that we can hide my slow ass on someone else.  Nick then has to fight to get open against the Patrick Mapp's of the world after he's chased them all over on defence.

If you're trying to figure out who contribute the most compared to their salary, I feel like there is a lot of value in having players with mid-level salaries who can defend, or at least keep up with the top players in the league.

Mehmet Karman

Thu, 2018-12-20 10:49

When I was a GM in the past I frequently employed a certain Mr. Close for his valuable contributions on these issues.