RE: Prediction . . . and charting defect rates

["Alan Meekings" <alan@landmarkconsulting.co.uk>]

Bob,

In your recent posting, you said, "I think it would help our readers if Alan
gave examples of how he would monitor raw data on process behaviour."

The point I was trying to make is that, when looking at process-based data,
the 'default setting' ought to be to chart absolute measures of performance,
rather than proxy measures of performance (such as defect levels).

To define a 'defect', there has to be a specification. The specification is
either met (i.e. this is not a 'defect') or it is not met (i.e. this is a
'defect'). 

This approach is problematic even in manufacturing industries, where defects
are relatively easy to spot. You'll doubtless be familiar with Don Wheeler's
definition of "world class quality" as "on target with minimum variation",
and the profound implications of this approach in practice.

In the context of services, especially where customers are intimately
involved in the service process itself (such as in health care, travel,
local government services, etc) the problem is even more pressing. In such
circumstances, customers bring to bear their own point of view on what being
'on target' actually means to them personally.

Let me be bold and say, "Meeting specifications is never the best way to
facilitate fundamental improvement." Feel free to challenge me on this
assertion, Bob.

However, returning to the question of whether to chart defect rates or
absolute measures of performance, let me cite one classic example.

Consider rail travel in Great Britain. 

Prior to Privatisation in 1995, British Rail conducted customer research
that seemed to show passengers were, broadly speaking, happy if their
commuter trains arrived within 5 minutes of published arrival times and
their inter-city trains within 10. 

This seems plausible, doesn't it? And it could be quite a useful finding in
several respects.

Unfortunately, it led British Rail to define punctuality in terms of
'defects', rather than in terms of actual punctuality (in other words, how
close to published arrival times did the train actually arrive). 

A 'defect' in this context (a measure called Non-Arrival in Right Time or
NART) meant a commuter train arriving 5 or more minutes late or an
inter-city train arriving 10 or more minutes late. 

So, a 'punctuality' figure of 80% meant that 80% of trains arrived outside
the (seemingly customer-focused) specification for punctuality of 5 or 10
minutes late respectively.

Well, this all seems pretty sensible, doesn't it? Sadly not.

A two-hour delay, for instance, makes essentially no difference to
punctuality figures under this regime (since it gets lost in comparison with
the mass of trains only slightly late), whereas a two-hour delay certainly
makes a big impact on the passengers involved - let alone on the
consequential impacts to other service, planned maintenance, crew
positioning, etc.

What's worse, though, is that it fosters a culture that says 5 or 10 minutes
late is kind-of OK. Well, it jolly well isn't. Taguchi would certainly have
had something to say about this.

For example, this approach significantly reduces capacity across bottlenecks
in the rail network. Queuing Theory proves that, if trains arrive
spot-on-time across choke points in the network (such as the two-track
viaduct north of Welwyn Garden City on the four-track East Coast mainline),
capacity is much higher than if they arrive with random variations of up to
5 or 10 minutes (or, frequently, longer).

And the picture gets worse. Contracts between the train operators and the
infrastructure owner, Network Rail, are 'incentivised' around 'impact
minutes' (i.e. minutes of delay for which blame has to be attributed to one
side or the other). This compounds the focus on lots of small delays (the
common causes), rather than the systemic causes of big delays (the special
causes).

How do operators avoid delays in this sort of regime? 

This is not a difficult question. They simply add 'recovery time' (that's to
say, they add extra time beyond what it should predictably take to get from,
say, Peterborough to London), in order to avoid being penalised for 'late
running'.

What does this do to do to capacity? Yes, it reduces it. 

What did the Strategic Rail Authority do, as a result, in order to improve
punctuality? Yes, you guessed it: they authorised significant reductions in
services.

Has this improved the customer proposition, or even the customer experience?
Absolutely not. In fact, given that all the train operators have gone down
the route of building-in additional 'recovery time', in the context of a
collective culture of sloppy punctuality, customers have suffered massively.
The experience of the past decade is a matter of record.

I could go on . . . but I find it far too depressing.

The answer is simple: measure punctuality, not some arbitrary definition of
a 'defect' in punctuality. 

Then the whole system could start to look at punctuality through a
completely different lens. 

This point applies almost universally. 

Take the National Health Service in the UK, for instance. Why count the
number of breaches (i.e. 'defects') in the current 17-week target time from
General Practitioner referral to being seen by a hospital Consultant? Why
not look at actual waiting times too? It's very revealing.

Again, I could go on. However, I hope I've made my point.

I grant there's a place for tracking 'defects' in certain circumstances, but
I think tracking defect rates needs to be set in the context of tracking
absolute process performance too.

I hope this message provides the examples you had in mind, Bob.

Regards as ever,

Alan