Recently Canucks Army and Pass it to Bulis have each looked at just how dreadful the Canucks have been at drafting and developing their prospects in recent years. You can read their takes on that here and here. What each of them found is that using very simple methodologies and data available to the public, you can easily (EASILY) do better at the draft than the Canucks have in the past decade.
Ron Delorme is a former Canucks player who retired in 1985. By 1988 he had a job with the Canucks as a scout, and in 2000 he took over as the Canucks’ Chief Amateur Scout. Graphics like the one below (courtesy of That’s Offside! at Canucks Army) show just how ineffective the Canucks have been at drafting since he took over in 2000, up until 2010. Drafts after 2010 are hard to judge as many or most of the players selected in those drafts are still prospects. The graphic measures Games Played per draft pick as measured against the league average.
Of note to Canucks fans is that both Boston and Buffalo have had excellent drafting records during Jim Benning’s time at each.
My one concern with this type of analysis is that it judges results, not processes. Of course this is by necessity, as we have no way of knowing how Delorme and his scouts work so there’s no method to judge their process. Nonetheless, any time you are judging results instead of processes you have to be very careful to consider things like sample size, randomness, and luck or else you risk sounding like Steve Simmons on the Corsi Hockey League. In other words, what I’m suggesting is that careful consideration needs to be given as to how much of Delorme’s failure and Benning’s success actually comes down to luck and factors outside of their control.
The Canucks have picked 79 players in the 11 drafts of the Delorme era. Those 79 players can be broken down as follows:
- 9 first round picks (averaging out to the 19th pick in the draft)
- 7 second round picks
- 9 third round picks
- 8 fourth round picks
- 11 fifth round picks
- 12 sixth round picks
- 9 seventh round picks
Since 2005 there have been 7 rounds in the draft, instead of 9 as it was previously. From 2000-2004 the Canucks also picked:
- 6 eighth round picks
- 5 ninth round picks (one of whom turned out to be Jannik Hansen)
In every round of every draft there will be players who don’t turn into effective NHL players. Even in the first round, there will be good hockey players picked who won’t make it as NHLers for reasons that are usually totally unforeseeable. On every team’s draft board there will be players who will end up being bad picks. There’s a large element of luck involved in whether or not each team actually ENDS UP with those players. A good example of this is former Vancouver Giant Gilbert Brulé, picked in the 2005 draft at 6th overall. He had a great junior career and was ranked 5th by Central Scouting leading up to the draft. You can bet that the Canucks would have been thrilled to take him at their number 10 slot had he not already been taken. In this case the process of the Blue Jackets picking him at number 6 was probably sound, but the result doesn’t necessarily show that. Furthermore, as you get further and further down the draft order, the randomness involved in each pick increases. The odds of drafting a useful NHL player out of a 5th round pick are slim. I’m not suggesting that the entire business of the draft is random, just that it’s closer than we think. The most important work scouts do is identifying what traits lead to successful NHL players and ones that are valuable to their team (more on this later), ranking all the available prospects by those standards, and picking the best available. After they do that it is largely luck that determines whether draft picks turn into useful contributors or not.
I would be very, very surprised if the Canucks scouting department is substantially different from many others across the league. The job of scouts, above all, should be to identify the traits that will make 18 year old hockey players become successful NHLers and to draft for those qualities. There are teams across the league who have long been drafting for things that aren’t necessarily likely to lead to success at the NHL level – grit, size, and character, among others – like Toronto, Boston, Philadelphia and others that are all above the league average for GP per draft pick, unlike Vancouver. What that suggests to me is that randomness, not scouting effectiveness, is dictating much of these results. The Canucks have drafted 79 players in the 11 drafts under Delorme, and with those players ranging across 9 rounds I think it’s very likely that isn’t a sufficient sample size to draw the conclusions being drawn from it.
Finding the qualities that lead prospects to become successful NHL players is perhaps better done with data analysis than by scouts. For example, we know that drafting defencemen who don’t put points on the board in junior (often picked for their strong defence, size, or grit) is unlikely to produce NHL players. Every single NHL defencemen has strong puck skills, and at the junior level that shows through with substantial point totals. What I’m going to do is examine the 9 first round picks that the Canucks have made in the Ron Delorme era and compare them to the Central Scouting rankings on those players to see if there are any obvious differences between what the Canucks are doing as compared to other scouts.
2000: Nathan Smith, 23rd overall, ranked 13th among North American skaters by Central Scouting
2001: R.J. Umberger, 16th overall, ranked 5th among North American skaters
2003: Ryan Kesler, 23rd overall, ranked 16th among North American skaters
2004: Corey Schneider, 24th overall, ranked 7th among North American goalies
2005: Luc Bourdon, 10th overall, ranked 6th among North American skaters
2006: Michael Grabner, 14th overall, ranked 23rd among North American skaters
2007: Patrick White, 25th overall, ranked 23rd among North American skaters
2008: Cody Hodgson, 10th overall, ranked 9th among North American skaters
2009: Jordan Schroeder, 22nd overall, ranked 5th among North American skaters
What I’m trying to show with these rankings is that other scouts and other teams thought these were good players. They were not bad prospects that the Canucks, because of their uniquely mistaken draft philosophy, chose. They were good prospects chosen by the Canucks because they were available at that time. I doubt that the Canucks draft board often differs all that much from the Central Scouting rankings – and that’s okay. To try to test my intuition about the draft being largely random, I computed the GP/Pick of just first round draft picks for each team and compared them to the GP/Pick of all picks used in the graph above.
What we get here is Vancouver now in the 22nd slot, fairly in line with their average selection of the 19th overall pick during that time period. This is obviously a good sign of process for the Canucks scouting and drafting methods. The first round should be the least random (while still affected significantly by chance), so this suggests that the Canucks are actually doing a better job drafting than what their GP/Pick ranking (29th) suggested.
There is not a particularly high correlation between the rankings of each team in the two lists. Tampa Bay, it seems, has been genuinely awful at the draft in the past decade or so (hello Jonathan Drouin), and Ottawa and Pittsburgh both have top 5 spots on each list, but on the whole drafting well in the first round does not seem to indicate that a team will draft well in the later rounds, as it gets more random.
I think there have been some promising tendencies showing up within the Canucks drafting record in recent years. Picking Jordan Subban (who should not have fallen to the third round) showed a willingness to avoid the kneejerk reaction we have against small defencemen. His puckhandling, skating and vision all make him far more likely to make the NHL than most third round picks. Further, the Canucks’ draft strategy with picks in late rounds showed a willingness to try different things as opposed to sticking with the tried and true scouting methods used as long as the NHL has been around. There are definitely ways to improve the way teams handle their scouting and player selection, and while that particular method was not effective for the Canucks, it shows a good attitude towards trying new things with drafting.
Yes, something as simple as drafting the highest ranked player by Central Scouting has outperformed the Canucks for the last decade. I’m not arguing that Ron Delorme should keep his job with the Canucks. His record is such that there’s no compelling reason that he should be kept on as Chief Amateur Scout or even be kept on as a Canucks employee in any capacity. My only defence of him is that I think his process is probably fine and he’s just been unlucky. I think odds are that he’s probably a reasonably competent scout who for the most part does things the right way, but has had some tough luck. Some draft picks that were risky, but worth making didn’t turn up for him (Jordan Schroeder, as the 5th ranked skater but standing at just 5’9), some went wrong for things totally unforeseeable (Luc Bourdon), and some picks just didn’t work out. That’s the nature of the game and the nature of the draft. I think Delorme’s processes are in all likelihood very similar to most other teams out there, and I’m sure he would find a new job without too much trouble if he were let go. Hence I’m not actually too pessimistic about Delorme staying on with the Canucks, as seems likely from what Benning and Linden have said so far. I think what the Canucks (and other NHL teams) need more than anything is better analysis of the types of players and types of skills they should target when scouting. Nonetheless, 26 years is a long time to have a job with any organization, and there is no compelling reason why Ron Delorme should be kept on by the Canucks.
The Canucks have an astonishingly low drafting PDO. Let’s all hope really really hard for some #regression
Note: I don’t think the GP/pick model is an effective way of judging drafting without a bigger sample size. The most effective (albeit labour intensive) method I can think of to judge drafting effectiveness is to come up with an expected GP number for each pick: for 1st overall, 2nd, 3rd, all the way down to 210th. This would account for selection advantage in the draft, and from there one could see how close to that expected value for GP teams are able to get, and whether that difference is within what can be attributed to variance or if certain teams really are underperforming at the draft.
Note on note: this http://www.puckwatch.com/2014/05/vancouver-canucks-draft-record.html analysis came out not long after I wrote this post and is, I think, the best I’ve seen on the topic.