Ron Delorme, scouting woes, and process over results

Recently Canucks Army and Pass it to Bulis have each looked at just how dreadful the Canucks have been at drafting and developing their prospects in recent years. You can read their takes on that here and here. What each of them found is that using very simple methodologies and data available to the public, you can easily (EASILY) do better at the draft than the Canucks have in the past decade.

Ron Delorme is a former Canucks player who retired in 1985. By 1988 he had a job with the Canucks as a scout, and in 2000 he took over as the Canucks’ Chief Amateur Scout. Graphics like the one below (courtesy of That’s Offside! at Canucks Army) show just how ineffective the Canucks have been at drafting since he took over in 2000, up until 2010. Drafts after 2010 are hard to judge as many or most of the players selected in those drafts are still prospects. The graphic measures Games Played per draft pick as measured against the league average.
Image

Of note to Canucks fans is that both Boston and Buffalo have had excellent drafting records during Jim Benning’s time at each.

My one concern with this type of analysis is that it judges results, not processes. Of course this is by necessity, as we have no way of knowing how Delorme and his scouts work so there’s no method to judge their process. Nonetheless, any time you are judging results instead of processes you have to be very careful to consider things like sample size, randomness, and luck or else you risk sounding like Steve Simmons on the Corsi Hockey League. In other words, what I’m suggesting is that careful consideration needs to be given as to how much of Delorme’s failure and Benning’s success actually comes down to luck and factors outside of their control.

The Canucks have picked 79 players in the 11 drafts of the Delorme era. Those 79 players can be broken down as follows:

  • 9 first round picks (averaging out to the 19th pick in the draft)
  • 7 second round picks
  • 9 third round picks
  • 8 fourth round picks
  • 11 fifth round picks
  • 12 sixth round picks
  • 9 seventh round picks

Since 2005 there have been 7 rounds in the draft, instead of 9 as it was previously. From 2000-2004 the Canucks also picked:

  • 6 eighth round picks
  • 5 ninth round picks (one of whom turned out to be Jannik Hansen)

In every round of every draft there will be players who don’t turn into effective NHL players. Even in the first round, there will be good hockey players picked who won’t make it as NHLers for reasons that are usually totally unforeseeable. On every team’s draft board there will be players who will end up being bad picks. There’s a large element of luck involved in whether or not each team actually ENDS UP with those players. A good example of this is former Vancouver Giant Gilbert Brulé, picked in the 2005 draft at 6th overall. He had a great junior career and was ranked 5th by Central Scouting leading up to the draft. You can bet that the Canucks would have been thrilled to take him at their number 10 slot had he not already been taken. In this case the process of the Blue Jackets picking him at number 6 was probably sound, but the result doesn’t necessarily show that. Furthermore, as you get further and further down the draft order, the randomness involved in each pick increases. The odds of drafting a useful NHL player out of a 5th round pick are slim. I’m not suggesting that the entire business of the draft is random, just that it’s closer than we think. The most important work scouts do is identifying what traits lead to successful NHL players and ones that are valuable to their team (more on this later), ranking all the available prospects by those standards, and picking the best available. After they do that it is largely luck that determines whether draft picks turn into useful contributors or not.

I would be very, very surprised if the Canucks scouting department is substantially different from many others across the league. The job of scouts, above all, should be to identify the traits that will make 18 year old hockey players become successful NHLers and to draft for those qualities. There are teams across the league who have long been drafting for things that aren’t necessarily likely to lead to success at the NHL level – grit, size, and character, among others – like Toronto, Boston, Philadelphia and others that are all above the league average for GP per draft pick, unlike Vancouver. What that suggests to me is that randomness, not scouting effectiveness, is dictating much of these results. The Canucks have drafted 79 players in the 11 drafts under Delorme, and with those players ranging across 9 rounds I think it’s very likely that isn’t a sufficient sample size to draw the conclusions being drawn from it.

Finding the qualities that lead prospects to become successful NHL players is perhaps better done with data analysis than by scouts. For example, we know that drafting defencemen who don’t put points on the board in junior (often picked for their strong defence, size, or grit) is unlikely to produce NHL players. Every single NHL defencemen has strong puck skills, and at the junior level that shows through with substantial point totals. What I’m going to do is examine the 9 first round picks that the Canucks have made in the Ron Delorme era and compare them to the Central Scouting rankings on those players to see if there are any obvious differences between what the Canucks are doing as compared to other scouts.

2000: Nathan Smith, 23rd overall, ranked 13th among North American skaters by Central Scouting
2001: R.J. Umberger, 16th overall, ranked 5th among North American skaters
2003: Ryan Kesler, 23rd overall, ranked 16th among North American skaters
2004: Corey Schneider, 24th overall, ranked 7th among North American goalies
2005: Luc Bourdon, 10th overall, ranked 6th among North American skaters
2006: Michael Grabner, 14th overall, ranked 23rd among North American skaters
2007: Patrick White, 25th overall, ranked 23rd among North American skaters
2008: Cody Hodgson, 10th overall, ranked 9th among North American skaters
2009: Jordan Schroeder, 22nd overall, ranked 5th among North American skaters

What I’m trying to show with these rankings is that other scouts and other teams thought these were good players. They were not bad prospects that the Canucks, because of their uniquely mistaken draft philosophy, chose. They were good prospects chosen by the Canucks because they were available at that time. I doubt that the Canucks draft board often differs all that much from the Central Scouting rankings – and that’s okay. To try to test my intuition about the draft being largely random, I computed the GP/Pick of just first round draft picks for each team and compared them to the GP/Pick of all picks used in the graph above.

Image

What we get here is Vancouver now in the 22nd slot, fairly in line with their average selection of the 19th overall pick during that time period. This is obviously a good sign of process for the Canucks scouting and drafting methods. The first round should be the least random (while still affected significantly by chance), so this suggests that the Canucks are actually doing a better job drafting than what their GP/Pick ranking (29th) suggested.

There is not a particularly high correlation between the rankings of each team in the two lists. Tampa Bay, it seems, has been genuinely awful at the draft in the past decade or so (hello Jonathan Drouin), and Ottawa and Pittsburgh both have top 5 spots on each list, but on the whole drafting well in the first round does not seem to indicate that a team will draft well in the later rounds, as it gets more random.

I think there have been some promising tendencies showing up within the Canucks drafting record in recent years. Picking Jordan Subban (who should not have fallen to the third round) showed a willingness to avoid the kneejerk reaction we have against small defencemen. His puckhandling, skating and vision all make him far more likely to make the NHL than most third round picks. Further, the Canucks’ draft strategy with picks in late rounds showed a willingness to try different things as opposed to sticking with the tried and true scouting methods used as long as the NHL has been around. There are definitely ways to improve the way teams handle their scouting and player selection, and while that particular method was not effective for the Canucks, it shows a good attitude towards trying new things with drafting.

Yes, something as simple as drafting the highest ranked player by Central Scouting has outperformed the Canucks for the last decade. I’m not arguing that Ron Delorme should keep his job with the Canucks. His record is such that there’s no compelling reason that he should be kept on as Chief Amateur Scout or even be kept on as a Canucks employee in any capacity. My only defence of him is that I think his process is probably fine and he’s just been unlucky. I think odds are that he’s probably a reasonably competent scout who for the most part does things the right way, but has had some tough luck. Some draft picks that were risky, but worth making didn’t turn up for him (Jordan Schroeder, as the 5th ranked skater but standing at just 5’9), some went wrong for things totally unforeseeable (Luc Bourdon), and some picks just didn’t work out. That’s the nature of the game and the nature of the draft. I think Delorme’s processes are in all likelihood very similar to most other teams out there, and I’m sure he would find a new job without too much trouble if he were let go. Hence I’m not actually too pessimistic about Delorme staying on with the Canucks, as seems likely from what Benning and Linden have said so far. I think what the Canucks (and other NHL teams) need more than anything is better analysis of the types of players and types of skills they should target when scouting. Nonetheless, 26 years is a long time to have a job with any organization, and there is no compelling reason why Ron Delorme should be kept on by the Canucks.

The Canucks have an astonishingly low drafting PDO. Let’s all hope really really hard for some #regression

Note: I don’t think the GP/pick model is an effective way of judging drafting without a bigger sample size. The most effective (albeit labour intensive) method I can think of to judge drafting effectiveness is to come up with an expected GP number for each pick: for 1st overall, 2nd, 3rd, all the way down to 210th. This would account for selection advantage in the draft, and from there one could see how close to that expected value for GP teams are able to get, and whether that difference is within what can be attributed to variance or if certain teams really are underperforming at the draft.

Note on note: this http://www.puckwatch.com/2014/05/vancouver-canucks-draft-record.html analysis came out not long after I wrote this post and is, I think, the best I’ve seen on the topic.

About these ads

7 thoughts on “Ron Delorme, scouting woes, and process over results

  1. Three massive problems with this analysis:

    1) When “luck” is a factor all one can do is do their best to mitigate it’s effect. The data points in a long term down trend and if “luck” was such a large factor as you’re suggesting, odds would have more variance from year to year. There’s a consistency of bad picks throughout his tenure and I believe the two analysis’ made by CanucksArmy are accurate and illuminating. Even choosing with a random, intentionally flawed system, better drafting was done. They even committed to “having bad luck” by excluding international players and still did better. Over such a long period of time; competency should beat luck. I play a hell of a lot of poker and understand the theories of luck – this is not applicable for this duration or depth if the “right” system is in place.

    2) You make the point that other scouting also had a similar rankings as the Canuck’s. Why didn’t another team take them then? There’s a reason for other teams selecting other players in lieu of who the Canucks picked.

    3) You only analyze the first round over 9 seasons. Choose a bigger sample size. The first round could be chosen by a 7 year old. (Who again, would most likely do better than Delorme!)

    • I don’t understand most of what you’re saying in your first point. Contrary to what you said the variance from year to year in GP/pick is actually pretty large, and “over such a long period of time; competency should beat luck” doesn’t really mean much to me. Maybe I’m misunderstanding what you’re saying but competency and luck are totally unrelated. Over a long enough period of time, results will regress to the mean absolutely. If that’s what you’re saying then I totally agree with you. 11 years just isn’t a large enough sample for something with this much variance (especially at the lower rounds) to regress. The Detroit Red Wings are so often lauded for making great late round picks, but so much of that rests on the Henrik Zetterberg pick. Picks that late aren’t quite a lottery, but they’re pretty damn close. If they had felt good about him as a prospect they would have drafted him earlier than 210th. They got lucky with that pick. What you’re addressing here – the role of randomness and how prone draft success is to luck – is really the crux of my whole argument so I’m sorry if I didn’t get it across well enough.

      2) That’s a reasonable point. What I really mean by that is that if the Canucks pick moves + or – one way, we likely end up with a different player and another team with ours. San Jose reportedly wanted Patrick White in 2007. They later traded for him (we got Erhoff). A large part of how we end up with the players we do is the interplay between each teams individual draft boards and the draft order. Chance has a huge impact on who is still on the table.

      3) The whole idea with that was to limit the sample. Canucks Army and PITB have done all-round analyses. Part of the reason why I thought it was valuable to do just the first round is because as the rounds get later, the variance gets much higher. First round picks are closer to ‘sure things’ than later round picks, and so you get a much better look at what the team is trying to do with their player selection. Increasing the sample size by including later rounds doesn’t actually necessarily make for a more precise test, since randomness plays so much of a greater role in the later round picks.

      Hope I made things a little more clear! Thanks for the comment

  2. At the top of the article you compare Central Scouting Ranking to the Canucks selection, which I find interesting but I think it can be read two ways.

    If a player was ranked 3rd but the Canucks drafted him 23rd, it’s a ‘common sense’ pick because he’s the ‘best available player’, so you can’t really blame scouting. However, 22 teams passed on him. So what did other teams ‘know’ or think that the Canucks didn’t? I’m curious if that’s a good sign or a bad sign. In the case of Michael Grabner, he was ranked so low but drafted higher by the Canucks, it leads you to believe the Canucks were very high on him, and rightly so. I recall Kesler in 2003 was only supposed to be a third liner, and the Canucks were criticized for not ‘trying to hit a home run’ with a first round pick. “Why draft a 3rd liner in the 1st round?” Thankfully he turned out better.

    2) I agree with you on Detroit and Zetterberg for the most part. A late round pick becoming a full-time NHLer is one thing, but finding an all-star talent that low is astonishing, and creates a reputation for the club. To your point, if the Wings ‘knew’ through superior drafting that he’d be THAT good, you’d think they wouldn’t risk drafting him so low, on the off-chance another team wanted him.

    3) Thanks for the shoutout to my post!

    • That’s a good point about other teams passing on Canucks players for a reason – I tend to think it’s just because they were higher on other players, or were picking players outside the scope of the rankings I used (eg goalies, European players). With some picks, like Jordan Schroeder, everybody seemed to agree he was a player with a high ceiling and it was pretty clear he was dropping in the draft because of his size. That, in my books, is the right type of risk to take late in the first round. Skill trumps size and although Schroeder hasn’t panned out as hoped I think that shows a really good process.

      and hey no worries, your post was something I was hoping someone would put together and you did a great job of it!

      • I liked the Schroeder pick too…worth a shot. I was most disappointed at Bourdon over Kopitar though in 2005. I remember multiple teams passing on him because he was from a ‘non-traditional hockey market’ as if that’s a bad thing, and doesn’t indicate his passion for the game.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s