ROC curve analysis questions
Author Message
ROC curve analysis questions

Hello,

I am using the ROC curve analysis software available from U. Chicago
("LABROC1" as part of the "ROCKIT" package). It's really
straightforward, and I've been able to get the main results that I
want.

The output gives the parameters of the fit (a,b), the area under the
curve, and the false positive fraction (FPF) and true positive
fraction (TPF) for a selection of critical test values.

I would like to calculate the FPF and TPF for an *arbitrary* critical
test value, based on the fit. How does one do that, given the output
of the regression (i.e., the a and b parameters)?

I've been reading up on ROC curves and curve fitting, and it seems
like this should be a simple thing, but I'm really stuck!

Thanks,
-N

Sat, 16 Jul 2005 07:14:00 GMT
ROC curve analysis questions
I'm glad to know that you found our software useful.  The answer to your
question can be found in item 22a on page 37 of the ROCKIT user's guide.

BTW, our LABROC1 algoritm is now obsolete.  ROCKIT has always employed
the LABROC5 algorithm, which handles data from high ROC curves better
and is described in:

Metz CE, Herman BA, Shen J-H.  Maximum-likelihood estimation of ROC
curves from continuously-distributed data.  Statistics in Medicine 1998;
17: 1033.

Hoping this helps,

Charles Metz

--------------

Quote:

> Hello,

> I am using the ROC curve analysis software available from U. Chicago
> ("LABROC1" as part of the "ROCKIT" package). It's really
> straightforward, and I've been able to get the main results that I
> want.

> The output gives the parameters of the fit (a,b), the area under the
> curve, and the false positive fraction (FPF) and true positive
> fraction (TPF) for a selection of critical test values.

> I would like to calculate the FPF and TPF for an *arbitrary* critical
> test value, based on the fit. How does one do that, given the output
> of the regression (i.e., the a and b parameters)?

> I've been reading up on ROC curves and curve fitting, and it seems
> like this should be a simple thing, but I'm really stuck!

> Thanks,
> -N

Sat, 16 Jul 2005 09:58:57 GMT
ROC curve analysis questions
Since I did not know what ROC analysis was, I looked around in the Web
and started reading about it.  I work in the manufacturing world, and
what I use is the usual t-tests, ANOVA, regression, etc.  I plan to
read more on ROC analysis, but from the little that I read, I believe
this can also be used for industrial statistics.  It has always being
difficult to explain statistics to engineers but some of the things
that I read on comparing populations seems straightforward.  The
question is, is the method adequate for applications other than
medicine?  Could you give me the benefit of using this over the usual
tests of hypothesis?

Lenin

Quote:

> I'm glad to know that you found our software useful.  The answer to your
> question can be found in item 22a on page 37 of the ROCKIT user's guide.

> BTW, our LABROC1 algoritm is now obsolete.  ROCKIT has always employed
> the LABROC5 algorithm, which handles data from high ROC curves better
> and is described in:

>   Metz CE, Herman BA, Shen J-H.  Maximum-likelihood estimation of ROC
> curves from continuously-distributed data.  Statistics in Medicine 1998;
> 17: 1033.

> Hoping this helps,

>   Charles Metz

> --------------

> > Hello,

> > I am using the ROC curve analysis software available from U. Chicago
> > ("LABROC1" as part of the "ROCKIT" package). It's really
> > straightforward, and I've been able to get the main results that I
> > want.

> > The output gives the parameters of the fit (a,b), the area under the
> > curve, and the false positive fraction (FPF) and true positive
> > fraction (TPF) for a selection of critical test values.

> > I would like to calculate the FPF and TPF for an *arbitrary* critical
> > test value, based on the fit. How does one do that, given the output
> > of the regression (i.e., the a and b parameters)?

> > I've been reading up on ROC curves and curve fitting, and it seems
> > like this should be a simple thing, but I'm really stuck!

> > Thanks,
> > -N

Sat, 16 Jul 2005 21:52:51 GMT
ROC curve analysis questions
Dr. Metz,

Thank you for replying to my question.

What I was wondering is if it would be straightforward to calculate
the FPF and TPF for critical test values not included in the output
file. For example, if my output were what is given in item 22a in the
Rockit manual, could I calculate FPF and TPF for a critical test value
of -1.0000 given the fit parameters a and b?

-N

Quote:

> I'm glad to know that you found our software useful.  The answer to your
> question can be found in item 22a on page 37 of the ROCKIT user's guide.

> BTW, our LABROC1 algoritm is now obsolete.  ROCKIT has always employed
> the LABROC5 algorithm, which handles data from high ROC curves better
> and is described in:

>   Metz CE, Herman BA, Shen J-H.  Maximum-likelihood estimation of ROC
> curves from continuously-distributed data.  Statistics in Medicine 1998;
> 17: 1033.

> Hoping this helps,

>   Charles Metz

> --------------

> > Hello,

> > I am using the ROC curve analysis software available from U. Chicago
> > ("LABROC1" as part of the "ROCKIT" package). It's really
> > straightforward, and I've been able to get the main results that I
> > want.

> > The output gives the parameters of the fit (a,b), the area under the
> > curve, and the false positive fraction (FPF) and true positive
> > fraction (TPF) for a selection of critical test values.

> > I would like to calculate the FPF and TPF for an *arbitrary* critical
> > test value, based on the fit. How does one do that, given the output
> > of the regression (i.e., the a and b parameters)?

> > I've been reading up on ROC curves and curve fitting, and it seems
> > like this should be a simple thing, but I'm really stuck!

> > Thanks,
> > -N

Sat, 16 Jul 2005 22:43:12 GMT
ROC curve analysis questions

> Thank you for replying to my question.
>
> What I was wondering is if it would be straightforward
> to calculate the FPF and TPF for critical test values
> not included in the output file. For example, if my
> output were what is given in item 22a in the
> Rockit manual, could I calculate FPF and TPF for a
> critical test value of -1.0000 given the fit parameters
> a and b?

I'm not aware of any generally-useful way to do this except to
interpolate the relationship between critical test-result value and FPF
that's given in the output.  The corresponding value of TPF is then
given by PHI(a+b*PHI^-1(FPF)), in which PHI(z) represents the standard
normal distribution function (i.e., the integral of the standard normal
density from negative infinity to z) and PHI^-1(FPF) represents the
normal deviate that corresponds to the probability FPF (i.e., the upper
limit of integration that causes PHI(z) to equal FPF).

Charles Metz

Sun, 17 Jul 2005 02:20:14 GMT
ROC curve analysis questions

> Since I did not know what ROC analysis was, I looked
> around in the Web and started reading about it.  I work
> in the manufacturing world, and what I use is the usual
> t-tests, ANOVA, regression, etc.  I plan to
> read more on ROC analysis, but from the little that I
> read, I believe this can also be used for industrial
> statistics.  It has always being difficult to explain
> statistics to engineers but some of the things that
> I read on comparing populations seems straightforward.
> The question is, is the method adequate for
> applications other than medicine?  Could you give me
> the benefit of using this over the usual tests of
> hypothesis?

Most fundamentally, receiver operating characteristic (ROC) analysis
quantifies accuracy in two-group classification tasks in terms of the
relationship, as a critical value is manipulated, between two
conditional probabilities, each of which is conditional upon actual
membership in one or the other of the two groups -- e.g., Prob(classify
as group 2 | actually a member of group 2) and Prob(classify as group 2
| actually a member of group 1).  A graphical display of this
relationship constitutes an ROC curve.  ROC analysis isn't a way of
testing hypothesies; however, hypothesis-testing methods have been
developed to assess the statistical significance of differences between
estimates of ROC curves or summary indices thereof.

Charles Metz

----------------------------------------

Readings in ROC Analysis, with Emphasis on Medical Applications

Prepared by Charles E. Metz
The University of Chicago

BACKGROUND:

Egan JP.  Signal detection theory and ROC analysis.  New York: Academic
Press, 1975.

Fryback DG, Thornbury JR.  The efficacy of diagnostic imaging.  Med
Decis Making 1991; 11: 88.

Griner PF, Mayewski RJ, Mushlin AI, Greenland P.  Selection and
interpretation of diagnostic tests and procedures: principles and
applications.  Annals Int Med 1981; 94: 553.

International Commission on Radiation Units and Measurements.  Medical
imaging: the assessment of image quality (ICRU Report 54).  Bethesda,MD:
ICRU, 1996.

Lusted LB.  Signal detectability and medical decision-making.  Science
1971; 171: 1217.

McNeil BJ, Adelstein SJ.  Determining the value of diagnostic and
screening tests.   J Nucl Med 1976; 17: 439.

McNeil BJ, Keeler E, Adelstein SJ.  Primer on certain elements of
medical decision making.  New Engl J Med 1975; 293: 211.

Metz CE, Wagner RF, Doi K, Brown DG, Nishikawa RN, Myers KJ.  Toward
consensus on quantitative assessment of medical imaging systems.  Med
Phys 22: 1057-1061, 1995.

National Council on Radiation Protection and Measurements.  An
introduction to efficacy in diagnostic radiology and nuclear medicine
(NCRP Commentary 13).  Bethesda, MD: NCRP, 1995.

Robertson EA, Zweig MH, Van Steirtghem AC.  Evaluating the clinical
efficacy of laboratory tests.  Am J Clin Path 1983; 79: 78.

Zweig MH, Campbell G.  Receiver-operating characteristic (ROC) plots: a
fundamental evaluation tool in clinical medicine.  Clinical Chemistry
1993; 39: 561.  [Erratum published in Clinical Chemistry 1993; 39: 1589.]

GENERAL:

Hanley JA.  Alternative approaches to receiver operating characteristic

Hanley JA.  Receiver operating characteristic (ROC) methodology: the
state of the art.  Critical Reviews in Diagnostic Imaging 1989; 29: 307.

King JL, Britton CA, Gur D, Rockette HE, Davis PL.  On the validity of
the continuous and discrete confidence rating scales in receiver
operating characteristic studies.  Invest Radiol 1993; 28: 962.

Metz CE.  Basic principles of ROC analysis.  Seminars in Nucl Med 1978;
8: 283.

21: 720.

Metz CE.  Some practical issues of experimental design and data analysis

Metz CE.  Evaluation of CAD methods.  In Computer-Aided Diagnosis in
Medical Imaging (K Doi, H MacMahon, ML Giger and KR Hoffmann, eds.).
Amsterdam: Elsevier Science (Excerpta Medica International Congress
Series, Vol. 1182), pp. 543-554, 1999.

Metz CE.  Fundamental ROC analysis.  In: Handbook of Medical Imaging,
Vol. 1: Physics and Psychophysics (J Beutel, H Kundel and R Van Metter,
eds.).  Bellingham, WA; SPIE Press, 2000, pp. 751-769.

Metz CE, Shen J-H.  Gains in accuracy from replicated readings of
diagnostic images: prediction and assessment in terms of ROC analysis.
Med Decis Making 1992; 12: 60.

Rockette HE, Gur D, Metz CE.  The use of continuous and discrete
confidence judgments in receiver operating characteristic studies of
diagnostic imaging techniques.  Invest Radiol 1992; 27: 169.

Swets JA.  ROC analysis applied to the evaluation of medical imaging
techniques.  Invest Radiol 1979; 14: 109.

Swets JA.  Indices of discrimination or diagnostic accuracy: their ROCs
and implied models.  Psychol Bull 1986; 99: 100.

Swets JA.  Measuring the accuracy of diagnostic systems.  Science 1988;
240: 1285.

Swets JA.  Signal detection theory and ROC analysis in psychology and
diagnostics: collected papers.  Mahwah, NJ; Lawrence Erlbaum Associates, 1996.

Swets JA, Pickett RM.  Evaluation of diagnostic systems: methods from
signal detection theory.  New York: Academic Press, 1982.

Wagner RF, Beiden SV, Metz CE.  Continuous vs. categorical data for ROC
328, 2001.

BIAS:

Begg CB, Greenes RA.  Assessment of diagnostic tests when disease
verification is subject to selection bias.  Biometrics 1983; 39: 207.

Begg CB, McNeil BJ.  Assessment of radiologic tests: control of bias and
other design considerations.  Radiology 1988; 167: 565.

Gray R, Begg CB, Greenes RA.  Construction of receiver operating
characteristic curves when disease verification is subject to selection
bias.  Med Decis Making 1984; 4: 151.

Ransohoff DF, Feinstein AR.  Problems of spectrum and bias in evaluating
the efficacy of diagnostic tests.  New Engl J Med 1978; 299: 926.

CURVE FITTING:

Dorfman DD, Alf E.  Maximum likelihood estimation of parameters of
signal detection theory and determination of confidence intervals
rating method data.  J Math Psych 1969; 6: 487.

Dorfman DD, Berbaum KS, Metz CE, Lenth RV, Hanley JA, Dagga HA.  Proper

Grey DR, Morgan BJT.  Some aspects of ROC curve-fitting: normal and
logistic models.  J Math Psych 1972; 9: 128.

Hanley JA.  The robustness of the "binormal" assumptions used in fitting
ROC curves.  Med Decis Making 1988; 8: 197.

Metz CE, Herman BA, Shen J-H.  Maximum-likelihood estimation of ROC
curves from continuously-distributed data.  Stat Med 1998; 17: 1033.

Metz CE, Pan X.  "Proper" binormal ROC curves: theory and
maximum-likelihood estimation.  J Math Psych 1999; 43: 1.

Pan X, Metz CE.  The "proper" binormal model: parametric ROC curve

Swensson RG.  Unified measurement of observer performance in detecting
and localizing target objects on images.  Med Phys 1996; 23: 1709.

Swets JA.  Form of empirical ROCs in discrimination and diagnostic
tasks: implications for theory and measurement of performance.  Psychol
Bull 1986; 99: 181.

STATISTICS:

Agresti A.  A survey of models for repeated ordered categorical response
data.  Statistics in Medicine 1989; 8; 1209.

Bamber D.  The area above the ordinal {*filter*} graph and the area below
the receiver operating graph.  J Math Psych 1975; 12: 387.

Beiden SV, Wagner RF, Campbell G.  Components-of-variance models and
multiple-bootstrap experiments: and alternative method for

Beiden SV, Wagner RF, Campbell G, Metz CE, Jiang Y.
Components-of-variance models for random-effects ROC analysis: The case
2001; 8: 605.

Beiden SV, Wagner RF, Campbell G, Chan H-P.  Analysis of uncertainties
in estimates of components of variance in multivariate ROC analysis.

DeLong ER, DeLong DM, Clarke-Pearson DL.  Comparing the areas under two
or more correlated receiver operating characteristic curves: a
nonparametric approach.  Biometrics 1988; 44: 837.

Dorfman DD, Berbaum KS, Metz CE.  ROC rating analysis: generalization to
the population of readers and cases with the jackknife method.  Invest

Dorfman DD, Berbaum KS, Lenth RV, Chen Y-F, Donaghy BA.  Monte Carlo
1998; 5: 591.

Hanley JA, McNeil BJ.  The meaning and use of the area under a receiver
operating characteristic (ROC) curve.  Radiology 1982; 143: 29.

Hanley JA, McNeil BJ.  A method of comparing the areas under receiver
operating characteristic curves derived from the same cases.  Radiology
1983; 148: 839.

Jiang Y, Metz CE, Nishikawa RM.  A receiver operating characterisitc
partial area index for highly sensitive diagnostic tests.  Radiology
1996; 201: 745.

Ma G, Hall WJ.  Confidence bands for receiver operating characteristic
curves.  Med Decis Making 1993; 13: 191.

McClish DK.  Analyzing a portion of the ROC curve.  Med Decis Making
1989; 9: 190.

McClish DK.  Determining a range of false-positive rates for which ROC
curves differ.  Med Decis Making 1990; 10: 283.

McNeil BJ, Hanley JA.  Statistical approaches to the analysis of
receiver operating characteristic (ROC) curves.  Med Decis Making 1984;
4: 137.

Metz CE.  Statistical analysis of ROC data in evaluating
...

Sun, 17 Jul 2005 05:37:45 GMT
ROC curve analysis questions

Quote:

>  > Thank you for replying to my question.

>  > What I was wondering is if it would be straightforward
>  > to calculate the FPF and TPF for critical test values
>  > not included in the output file. For example, if my
>  > output were what is given in item 22a in the
>  > Rockit manual, could I calculate FPF and TPF for a
>  > critical test value of -1.0000 given the fit parameters
>  > a and b?

> I'm not aware of any generally-useful way to do this except to
> interpolate the relationship between critical test-result value and FPF
> that's given in the output.  The corresponding value of TPF is then
> given by PHI(a+b*PHI^-1(FPF)), in which PHI(z) represents the standard
> normal distribution function (i.e., the integral of the standard normal
> density from negative infinity to z) and PHI^-1(FPF) represents the
> normal deviate that corresponds to the probability FPF (i.e., the upper
> limit of integration that causes PHI(z) to equal FPF).

>   Charles Metz

Dr. Metz,

Looking at plots of my results, I see that interpolating like you
suggest should be accurate enough for my purposes (I want to find the
critical value that minimizes FPF and maximizes TPF).

One last question: Can I assume that the functional relationship
between the the 95% confidence intervals of FPF vs TPF found by your
software is the same as for FPF and TPF themselves?
-N

Mon, 18 Jul 2005 23:53:14 GMT
ROC curve analysis questions

Quote:

>  > Thank you for replying to my question.

>  > What I was wondering is if it would be straightforward
>  > to calculate the FPF and TPF for critical test values
>  > not included in the output file. For example, if my
>  > output were what is given in item 22a in the
>  > Rockit manual, could I calculate FPF and TPF for a
>  > critical test value of -1.0000 given the fit parameters
>  > a and b?

> I'm not aware of any generally-useful way to do this except to
> interpolate the relationship between critical test-result value and FPF
> that's given in the output.  The corresponding value of TPF is then
> given by PHI(a+b*PHI^-1(FPF)), in which PHI(z) represents the standard
> normal distribution function (i.e., the integral of the standard normal
> density from negative infinity to z) and PHI^-1(FPF) represents the
> normal deviate that corresponds to the probability FPF (i.e., the upper
> limit of integration that causes PHI(z) to equal FPF).

>   Charles Metz

Please disregard my last message. I see that the 95% confidence
intervals do not follow the same functional form. Sorry.
-N

Tue, 19 Jul 2005 01:00:03 GMT

 Page 1 of 1 [ 8 post ]

Relevant Pages