I am using the ROC curve analysis software available from U. Chicago
("LABROC1" as part of the "ROCKIT" package). It's really
straightforward, and I've been able to get the main results that I
want.

The output gives the parameters of the fit (a,b), the area under the
curve, and the false positive fraction (FPF) and true positive
fraction (TPF) for a selection of critical test values.

I would like to calculate the FPF and TPF for an *arbitrary* critical
test value, based on the fit. How does one do that, given the output
of the regression (i.e., the a and b parameters)?

I've been reading up on ROC curves and curve fitting, and it seems
like this should be a simple thing, but I'm really stuck!

Thanks,
-N

BTW, our LABROC1 algoritm is now obsolete.  ROCKIT has always employed
the LABROC5 algorithm, which handles data from high ROC curves better
and is described in:

  Metz CE, Herman BA, Shen J-H.  Maximum-likelihood estimation of ROC
curves from continuously-distributed data.  Statistics in Medicine 1998;
17: 1033.

Hoping this helps,

  Charles Metz

--------------

Lenin

Thank you for replying to my question.

What I was wondering is if it would be straightforward to calculate
the FPF and TPF for critical test values not included in the output
file. For example, if my output were what is given in item 22a in the
Rockit manual, could I calculate FPF and TPF for a critical test value
of -1.0000 given the fit parameters a and b?

-N

 > Thank you for replying to my question.
 >
 > What I was wondering is if it would be straightforward
 > to calculate the FPF and TPF for critical test values
 > not included in the output file. For example, if my
 > output were what is given in item 22a in the
 > Rockit manual, could I calculate FPF and TPF for a
 > critical test value of -1.0000 given the fit parameters
 > a and b?

I'm not aware of any generally-useful way to do this except to
interpolate the relationship between critical test-result value and FPF
that's given in the output.  The corresponding value of TPF is then
given by PHI(a+b*PHI^-1(FPF)), in which PHI(z) represents the standard
normal distribution function (i.e., the integral of the standard normal
density from negative infinity to z) and PHI^-1(FPF) represents the
normal deviate that corresponds to the probability FPF (i.e., the upper
limit of integration that causes PHI(z) to equal FPF).

  Charles Metz

 > Since I did not know what ROC analysis was, I looked
 > around in the Web and started reading about it.  I work
 > in the manufacturing world, and what I use is the usual
 > t-tests, ANOVA, regression, etc.  I plan to
 > read more on ROC analysis, but from the little that I
 > read, I believe this can also be used for industrial
 > statistics.  It has always being difficult to explain
 > statistics to engineers but some of the things that
 > I read on comparing populations seems straightforward.
 > The question is, is the method adequate for
 > applications other than medicine?  Could you give me
 > the benefit of using this over the usual tests of
 > hypothesis?

Most fundamentally, receiver operating characteristic (ROC) analysis
quantifies accuracy in two-group classification tasks in terms of the
relationship, as a critical value is manipulated, between two
conditional probabilities, each of which is conditional upon actual
membership in one or the other of the two groups -- e.g., Prob(classify
as group 2 | actually a member of group 2) and Prob(classify as group 2
| actually a member of group 1).  A graphical display of this
relationship constitutes an ROC curve.  ROC analysis isn't a way of
testing hypothesies; however, hypothesis-testing methods have been
developed to assess the statistical significance of differences between
estimates of ROC curves or summary indices thereof.

I hope that the reading list appended below may be helpful.

  Charles Metz

----------------------------------------

Readings in ROC Analysis, with Emphasis on Medical Applications

Prepared by Charles E. Metz
Department of Radiology
The University of Chicago

BACKGROUND:

Egan JP.  Signal detection theory and ROC analysis.  New York: Academic
Press, 1975.  

Fryback DG, Thornbury JR.  The efficacy of diagnostic imaging.  Med
Decis Making 1991; 11: 88.

Griner PF, Mayewski RJ, Mushlin AI, Greenland P.  Selection and
interpretation of diagnostic tests and procedures: principles and
applications.  Annals Int Med 1981; 94: 553.

International Commission on Radiation Units and Measurements.  Medical
imaging: the assessment of image quality (ICRU Report 54).  Bethesda,MD:
ICRU, 1996.

Lusted LB.  Signal detectability and medical decision-making.  Science
1971; 171: 1217.

McNeil BJ, Adelstein SJ.  Determining the value of diagnostic and
screening tests.   J Nucl Med 1976; 17: 439.

McNeil BJ, Keeler E, Adelstein SJ.  Primer on certain elements of
medical decision making.  New Engl J Med 1975; 293: 211.

Metz CE, Wagner RF, Doi K, Brown DG, Nishikawa RN, Myers KJ.  Toward
consensus on quantitative assessment of medical imaging systems.  Med
Phys 22: 1057-1061, 1995.

National Council on Radiation Protection and Measurements.  An
introduction to efficacy in diagnostic radiology and nuclear medicine
(NCRP Commentary 13).  Bethesda, MD: NCRP, 1995.

Robertson EA, Zweig MH, Van Steirtghem AC.  Evaluating the clinical
efficacy of laboratory tests.  Am J Clin Path 1983; 79: 78.

Zweig MH, Campbell G.  Receiver-operating characteristic (ROC) plots: a
fundamental evaluation tool in clinical medicine.  Clinical Chemistry
1993; 39: 561.  [Erratum published in Clinical Chemistry 1993; 39: 1589.]

GENERAL:

Hanley JA.  Alternative approaches to receiver operating characteristic
analysis.  Radiology 1988; 168: 568.

Hanley JA.  Receiver operating characteristic (ROC) methodology: the
state of the art.  Critical Reviews in Diagnostic Imaging 1989; 29: 307.

King JL, Britton CA, Gur D, Rockette HE, Davis PL.  On the validity of
the continuous and discrete confidence rating scales in receiver
operating characteristic studies.  Invest Radiol 1993; 28: 962.

Metz CE.  Basic principles of ROC analysis.  Seminars in Nucl Med 1978;
8: 283.

Metz CE.  ROC methodology in radiologic imaging.  Invest Radiol 1986;
21: 720.

Metz CE.  Some practical issues of experimental design and data analysis
in radiological ROC studies.  Invest Radiol 1989; 24: 234.

Metz CE.  Evaluation of CAD methods.  In Computer-Aided Diagnosis in
Medical Imaging (K Doi, H MacMahon, ML Giger and KR Hoffmann, eds.).
Amsterdam: Elsevier Science (Excerpta Medica International Congress
Series, Vol. 1182), pp. 543-554, 1999.

Metz CE.  Fundamental ROC analysis.  In: Handbook of Medical Imaging,
Vol. 1: Physics and Psychophysics (J Beutel, H Kundel and R Van Metter,
eds.).  Bellingham, WA; SPIE Press, 2000, pp. 751-769.

Metz CE, Shen J-H.  Gains in accuracy from replicated readings of
diagnostic images: prediction and assessment in terms of ROC analysis.
Med Decis Making 1992; 12: 60.

Rockette HE, Gur D, Metz CE.  The use of continuous and discrete
confidence judgments in receiver operating characteristic studies of
diagnostic imaging techniques.  Invest Radiol 1992; 27: 169.

Swets JA.  ROC analysis applied to the evaluation of medical imaging
techniques.  Invest Radiol 1979; 14: 109.

Swets JA.  Indices of discrimination or diagnostic accuracy: their ROCs
and implied models.  Psychol Bull 1986; 99: 100.

Swets JA.  Measuring the accuracy of diagnostic systems.  Science 1988;
240: 1285.  

Swets JA.  Signal detection theory and ROC analysis in psychology and
diagnostics: collected papers.  Mahwah, NJ; Lawrence Erlbaum Associates, 1996.

Swets JA, Pickett RM.  Evaluation of diagnostic systems: methods from
signal detection theory.  New York: Academic Press, 1982.

Wagner RF, Beiden SV, Metz CE.  Continuous vs. categorical data for ROC
analysis: Some quantitative considerations.  Academic Radiol 2001, 8:
328, 2001.

BIAS:

Begg CB, Greenes RA.  Assessment of diagnostic tests when disease
verification is subject to selection bias.  Biometrics 1983; 39: 207.

Begg CB, McNeil BJ.  Assessment of radiologic tests: control of bias and
other design considerations.  Radiology 1988; 167: 565.

Gray R, Begg CB, Greenes RA.  Construction of receiver operating
characteristic curves when disease verification is subject to selection
bias.  Med Decis Making 1984; 4: 151.

Ransohoff DF, Feinstein AR.  Problems of spectrum and bias in evaluating
the efficacy of diagnostic tests.  New Engl J Med 1978; 299: 926.

CURVE FITTING:

Dorfman DD, Alf E.  Maximum likelihood estimation of parameters of
signal detection theory and determination of confidence intervals
rating method data.  J Math Psych 1969; 6: 487.

Dorfman DD, Berbaum KS, Metz CE, Lenth RV, Hanley JA, Dagga HA.  Proper
ROC analysis: the bigamma model.  Academic Radiol 1997; 4: 138.

Grey DR, Morgan BJT.  Some aspects of ROC curve-fitting: normal and
logistic models.  J Math Psych 1972; 9: 128.  

Hanley JA.  The robustness of the "binormal" assumptions used in fitting
ROC curves.  Med Decis Making 1988; 8: 197.

Metz CE, Herman BA, Shen J-H.  Maximum-likelihood estimation of ROC
curves from continuously-distributed data.  Stat Med 1998; 17: 1033.

Metz CE, Pan X.  "Proper" binormal ROC curves: theory and
maximum-likelihood estimation.  J Math Psych 1999; 43: 1.

Pan X, Metz CE.  The "proper" binormal model: parametric ROC curve
estimation with degenerate data.  Academic Radiol 1997; 4: 380.

Swensson RG.  Unified measurement of observer performance in detecting
and localizing target objects on images.  Med Phys 1996; 23: 1709.

Swets JA.  Form of empirical ROCs in discrimination and diagnostic
tasks: implications for theory and measurement of performance.  Psychol
Bull 1986; 99: 181.

STATISTICS:

Agresti A.  A survey of models for repeated ordered categorical response
data.  Statistics in Medicine 1989; 8; 1209.

Bamber D.  The area above the ordinal {*filter*} graph and the area below
the receiver operating graph.  J Math Psych 1975; 12: 387.

Beiden SV, Wagner RF, Campbell G.  Components-of-variance models and
multiple-bootstrap experiments: and alternative method for
random-effects, receiver operating characteristic analysis.  Academic
Radiol. 2000; 7: 341.

Beiden SV, Wagner RF, Campbell G, Metz CE, Jiang Y.
Components-of-variance models for random-effects ROC analysis: The case
of unequal variance structures across modalities.  Academic Radiol.
2001; 8: 605.

Beiden SV, Wagner RF, Campbell G, Chan H-P.  Analysis of uncertainties
in estimates of components of variance in multivariate ROC analysis.
Academic Radiol. 2001; 8: 616.

DeLong ER, DeLong DM, Clarke-Pearson DL.  Comparing the areas under two
or more correlated receiver operating characteristic curves: a
nonparametric approach.  Biometrics 1988; 44: 837.

Dorfman DD, Berbaum KS, Metz CE.  ROC rating analysis: generalization to
the population of readers and cases with the jackknife method.  Invest
Radiol 1992; 27: 723.

Dorfman DD, Berbaum KS, Lenth RV, Chen Y-F, Donaghy BA.  Monte Carlo
validation of a multireader method for receiver operating characteristic
discrtet rating data: factorial experimental design.  Academic Radiol
1998; 5: 591.

Dorfman DD, Metz CE.  Multi-reader multi-case ROC analysis: comments on
Beggs commentary.  Academic Radiol 1995; 2 (Supplement 1): S76.

Hanley JA, McNeil BJ.  The meaning and use of the area under a receiver
operating characteristic (ROC) curve.  Radiology 1982; 143: 29.

Hanley JA, McNeil BJ.  A method of comparing the areas under receiver
operating characteristic curves derived from the same cases.  Radiology
1983; 148: 839.

Jiang Y, Metz CE, Nishikawa RM.  A receiver operating characterisitc
partial area index for highly sensitive diagnostic tests.  Radiology
1996; 201: 745.

Ma G, Hall WJ.  Confidence bands for receiver operating characteristic
curves.  Med Decis Making 1993; 13: 191.

McClish DK.  Analyzing a portion of the ROC curve.  Med Decis Making
1989; 9: 190.

McClish DK.  Determining a range of false-positive rates for which ROC
curves differ.  Med Decis Making 1990; 10: 283.

McNeil BJ, Hanley JA.  Statistical approaches to the analysis of
receiver operating characteristic (ROC) curves.  Med Decis Making 1984;
4: 137.

Metz CE.  Statistical analysis of ROC data in evaluating
...

read more »

Looking at plots of my results, I see that interpolating like you
suggest should be accurate enough for my purposes (I want to find the
critical value that minimizes FPF and maximizes TPF).

One last question: Can I assume that the functional relationship
between the the 95% confidence intervals of FPF vs TPF found by your
software is the same as for FPF and TPF themselves?
 -N