RE: R-Bar/d2: Baysianism and Analytic Inference

["Jonathan Siegel" <jmsiegel@yahoo.com>]

Dr. Tribus has raised a debate that comes over statisticians from time
to time, about what "probability" means. There have been two traditional
schools of thought: Frequentists, who define it as the ratio of events
to trials repeated "a large number of times", and Bayesians, who define
it as a way of quantifying ones subjective degree of belief in the
likelihood of an event. (There is a third approach, advocated by Sir
Ronald Fisher, called Fiducial) The two principle ways of thinking
usually agree in practice, but there are situations where ones approach
affects how one behaves as well as how one thinks about it. 

I'm going to sidestep the debate on whether the Frequentist or Bayesian
approach is preferable, and instead point out why I believe analytic
problems, of which a control chart is an example, present difficulties
to both ways of thinking. These difficulties arise from the fact that
the possible arrangements of events (the frame) that could arise during
the course of observation are only a subset, often a small subset, of
the total possibilities that one must consider, and a substantial
portion of the total set of possibilities is often not only unknown but
unknowable. That is, not only can't one observe what will be selected
from portions of the frame, one can't see portions of the frame itself
or know what possibilities will be available to select from. This
situation suggests a certain caution in drawing inferences in analytic
situations, such as an attempt to predict the future based on the past.
We are always in the situation of being one of the three blind men
examining the legs, trunk, and ears of the elephant, except that we
can't communicate with the others, or even know if they exist. 

An example of an analytic problem that comes up in my current work is
how to set the maximum safe dose in a cancer drug. This is generally
done in the earliest human trials of the drug. Cancer is a very broad
and ill-defined term, covering a variety of illnesses in a variety of
bodily functions with a variety of symptoms; we know so little about it
that we often can't even tell if two people have "the same" cancer or
not for purposes of predicting a drug's therapeutic effect. The science
of genetic biomarkers and other theories as to why some people respond
to certain drugs and others do not is in its infancy. Moreover, the
people involved in the earliest trials can be quite different from a
cancer drug's eventual audience. They are people for whom other therapy
has failed. This tends to make them survivors, with a potentially
different response profile than a more general population. The trial
itself often involves only one or two clinics which may attract a
specialized clientele. A consequence of this situation is that results
are rarely if ever in control in the sense meant by a control chart.
Outliers -- particularly hardy or susceptible individuals -- can easily
skew them if one is not careful, but these same outliers can also
provide clues about the drug's potential benefits or harms. The earliest
trials are best used, not to calculate final answers or reach permanent
conclusions, but to open questions, and help guide the future course of
the investigation. Any attempt to apply results requires a significant
element of judgment based on expertise.

It is for reasons like this that Fisher advocated using confirmatory
trials, multiple trials with somewhat different research approaches on
somewhat different approaches, with the idea that different
investigations, each feeling about in the dark, might hope to reach
portions of legs, ears, and trunk, and if these all agreed, it might
then be reasonable to infer something about elephants.

A difficulty with Bayesian approaches as used in practice -- not
necessarily an inherent difficulty, but one which strikes me as
suggested if not required by the philosophy -- is that simply
quantifying what one does not know through a probability distribution
may possibly do justice to the unknown, but it doesn't do justice to the
UNKNOWABLE -- the part of the frame one can't see. Deming criticized the
Frequentist approach for this reason. I don't see that the Bayesian
approach, whether or not otherwise preferable, cures the problem Deming
was concerned with.

In an analytic problem of any depth, part of the essential purpose of
any investigation is to help us conceptualize -- help us imagine what
might be possible. An approach that merely helps us pick one among
pre-existing possibilities cannot help us here.  When we first expose
patients to a drug,  we do not know whether their experiences will
represent the general population or whether some or all of them will
prove outliers. We can't be sure what those experiences will be. We
don't know, for example, whether those "outliers" will represent mere
nuisance factors or noise to be ignored, or important clues that might
help in determining the best use of the drug or assist in researching a
new one. 

Academic Bayesian approaches have tended to work by putting
preconceptions into priors and plugging them into formulas. The more one
is willing to preconceive, the smaller the sample size one needs to be
able to draw an inference. Efficiency, which academic statisticians
prize, generally means the greatest number of preconceptions in order to
produce the smallest sample size. One might call the result Occam's Hair
Growth Tonic, the opposite of Occam's Razor. Such preconceptions, and
the hairy situations they have led to, have more than once led
practitioners astray. While articulated preconceptions are much to be
preferred to unarticulated ones, approaches requiring fewer
preconceptions are often of more use in the early stages of analytic
work. 

There is a process to the development of any new technology: one begins
with an exploratory, opening process, raising possibilities, which
changes as ones process takes on greater and greater definiteness as one
comes to know more and more about the potential technology, market,
customers and needs, etc. Each new development is itself an iteration in
an overall process of development and learning. A fractal process of
repeated Shewart cycles and cycles-within-cycles guides the broad
outlines of development strategy that takes place over years, just as it
guides short-term problem-solving over hours.

Traditional statisticians -- both of the Frequentist and the Bayesian
kind -- have tended to focus on the last stages of the process, on
definitive trials and final decisions to be made after all data is
gathered and all possibilities explored. Indeed, exploration of
possibilities is at best a poor cousin in many academic statistics
departments, in extreme cases so ill-regarded as to be concentrated
mainly in the general or remedial courses given to people who are not
expected to continue on as serious students. 

Deming thought this a waste of statisticians' time. Statisticians could
be far more helpful if they were more accepting of uncertainty. Time is
usually better spent helping people upstream, in the exploratory stages,
helping people open, consider, and cull possibilities. If this is done
well, the downstream work can be much simpler, as well as reached more
quickly. Early-stage investment tends to pay off exponentially. 

I'll point out that the control chart has been most accepted by the
general statistics community in part because it is one of the simplest
possible kinds of analytic problems. It involves the variability of a
single quantity, making its unknowns much easier to characterize than
most. It should not be regarded as typical of analytic problems
generally. 


Sincerely,
 
 Jonathan Siegel
40714 Reisa Lane
Apt 204
Canton, MI, 48188
(734) 657-1900
jmsiegel@yahoo.com