>           Nonparametric Receiver Operating Characteristic Analysis
>                      Stephen Vida, M.D., F.R.C.P.(C)
>                         Department of Psychiatry
>                         Montreal General Hospital
>                            McGill University

>      Receiver operating characteristic analysis is a leading tool for
> the analysis of the sensitivity and specificity of

>         - signal detection devices, e.g. radar receivers.

>         - diagnostic tests, e.g. laboratory tests.

>         - diagnostic instruments, e.g. rating scales.

>         - radiologic instruments, e.g. x-ray machines, CT scanners,
>           and MRI scanners.

>         - diagnosticians, e.g. clinicians.

>      Sensitivity and specificity are key measures of test performance.
> For tests yielding numerical scores, sensitivity and specificity
> usually vary inversely over the range of theoretically possible cutoff
> scores, complicating the task of quantifying and comparing the
> diagnostic accuracy of tests.

>      Receiver Operating Characteristic analysis (ROC) approaches this
> problem by plotting the curve of sensitivity, or true positive rate,
> versus 1 - specificity, or false positive rate, for all possible cutoff
> scores of the test.  The area under the ROC curve (AUC) can be used to
> describe the diagnostic accuracy of the test.  Parametric and
> nonparametric methods exist that allow the calculation of the AUC and
> the comparison of tests.  A disadvantage of parametric formulations is
> the assumption of a normal or Gaussian distribution of test scores.

>      Parametric ROC formulations, based on normal or Gaussian
> distributional assumptions, have existed for some time.  While to some
> degree they are robust to non-normality of underlying data, some
> authors have reported non-normal datasets for which parametric
> formulations either failed to compute solutions or computed erroneous
> solutions.

>      Nonparametric ROC formulations have more recently evolved that are
> distribution-free, in that they do not depend on the distribution of the
> test scores being analyzed but rather on their ranking.  Although
> nonparametric, statistical efficiency comparable to parametric approaches
> has been reported.

>      Nonparametric ROC may be useful when

>      1.   a distribution-free method is preferred because of
>           non-normality of data.

>      2.   parametric methods have failed to compute or have yielded
>           erroneous or implausible results.

>      3.   corroboration or comparison of parametric and nonparametric
>           methods is desired.

>      A Nonparametric Receiver Operating Characteristic Program is
> available for IBM-compatible computers that

>      -     plots from one to three ROC curves on the same graph.

>      -     calculates the area under the ROC curve, its standard error,
>            and its 95% confidence limits.

>      -     calculates the sensitivity and specificity, with 95%
>            confidence limits, at all possible cutoff scores of the
>            instrument being analyzed.

>      -     statistically compares two ROC curves from independent
>            samples.

>      -     statistically compares two or three ROC curves simultaneously
>            from correlated samples.

>      -     exports ROC curve coordinates to Harvard graphics templates,
>            included.

>      -     performs statistical power analysis to estimate sample sizes
>            required for ROC studies.

>      -     is accompanied by a comprehensive user's manual that includes
>            a review of the theory of ROC analysis and detailed
>            instructions for the use of the program.

>      For further information about ROC or the program described above,