Grading for system impovement

[Bob Clements <clementz@kos.net>]

DENizens

Forgive me for opening this thread again. I was reminded recently of an
exchange between Anton Tolman and John Purchase, The impact of grades:
(http://deming.ces.clemson.edu/pub/den/archive/99.10/msg00055.html)


Dr. Tolman's closing words were:

 Isn't our goal to improve the system,
 to transform the system?  We can't get there,
 folks, by continuing to support and emphasize
 grades......

 I fully agree with Wayne that we need to spend more
 time on improving learning than on arguing about
 grades.

I have a short story followed by a question.

Where I live (Ontario, Canada) we're hot on k-12 assessment, big time.
The data is public, and rich in depth. And it isn't going away any time
soon. It easily converts to a GPA by school, board, and the province
itself. As a system performance measure, it proves its utility. It is a
robust indicator of extreme outliers.

I publish annually a ranking of the board-by-board GPAs. The public love
the ranking. Directors of Education hate it. I judge this a positive
effect--for a statistical byproduct of my ANOVA.

During the month preceding and following the annual release of the data,
I field a lot of questions from the press and delight in putting hard
questions to Directors and other public education PTBs.

The publisher allows me a small column for my comment on the assessment
data. The first year I focused attention on the extreme outliers, both
by name and by concept. They should have management's attention I
argued. The education set thought me off the wall.

While waiting for the second year's data, I spent a good deal of time
dancing with a Ph.D. (Epidemiology) consigned to a Faculty of Education,
the head designer of the assessment regime. She's since gone very quiet,
at least publicly.

With the second year's ranking I dwelt again on extreme outliers,
particularly where there were two in a row. And while waiting for this
year's data I built a parent satisfaction index, by school, board, and
province. (Yes, the assessment agency collects data on individual
parents' satisfaction which is tagged to individual schools.)

Only last month I published the board rankings for the third year. My
editor is fabulous. She helped me introduce the concepts of accuracy and
precision as I talked about the spread of the data and it's march toward
improvement. On her own she introduced the metaphor of a rank of archers
and the assessment of their collective groupings, year over year.
Scholtes picked a winner.

Only two weeks ago I had a conversation with a high ranking ministry
officer in the assessment policy division. I asked her to imagine the
curve generated by repeated measures of her mpg. We hypothesized a
distribution with a mean of 25 mpg and a range of 22 to 27 mpg. What
would you make of a measure of 35 mpg or 11 mpg? I asked her. She
grasped the management import immediately, and offered to make it policy
that boards perform an outlier analyses and make an exploratory site
visit to each. I was dumbstruck with the offer, the ease of
enlightenment. She fessed up that her Ph.D. studies in statistics were
long behind her.

Next year I'm going to fight for two pages. I like to think I'm making
some headway.

And while I wait for next year's data I'll be hustling my satisfaction
index. It's an excellent locator of dissatisfied customers.

At the time of the last grade rage on the DEN someone asked what we
would use for system performance data if we did away with assessment. My
observation was, it isn't going away any time soon. My question is why
are we not using it for system improvement, to the extent that we can.

Bob Clements.
--
C. Robert Clements
226 Dundas Street West
Belleville, Ontario
CANADA  K8P 1A8
Tel: 613-966-8157
Fax: 520-223-3027
======================================================================