Professor David Heeger
The starting point for signal detection theory is that nearly all reasoning and decision making takes place in the presence of some uncertainty. Signal detection theory provides a precise language and graphic notation for analyzing decision making in the presence of uncertainty. The general approach of signaldetection theory has direct application for us in terms of sensory experiments.But it also offers a way to analyze many different kinds of decision problems.What you should know from this lecture
- Information acquisition
- Criterion
- Internal response and external noise
- Probability of occurrence curves
- Receiver operating characteristic (ROC curve)
- Discriminability index (d')
Information and Criterion
I begin here with medical scenario. Imagine that a radiologist is examining a CT scan, looking for evidence of a tumor. Interpreting CT images is hard and it takes a lot of training. Because the task is so hard, there is always some uncertainty as to what is there or not. Either there is a tumor (signal present) or there is not (signal absent). Either the doctor sees a tumor (theyrespond "yes'') or does not (they respond "no''). There are four possibleoutcomes: hit (tumor present and doctor says "yes''), miss (tumor presentand doctor says "no''), false alarm (tumor absent and doctor says "yes"),and correct rejection (tumor absent and doctor says "no"). Hits and correctrejections are good. False alarms and misses are bad.There are two main components to the decision-making process: information acquisition and criterion.
Information acquisition: First, there is information in the CT scan.For example, healthy lungs have a characteristic shape. The presence of atumor might distort that shape. Tumors may have different image characteristics: brighter or darker, different texture, etc. With proper training a doctor learns what kinds of things to look for, so with more practice/training theywill be able to acquire more (and more reliable) information. Running anothertest (e.g., MRI) can also be used to acquire more information. Regardless,acquiring more information is good. The effect of information is to increasethe likelihood of getting either a hit or a correct rejection, while reducingthe likelihood of an outcome in the two error boxes.
Criterion: The second component of the decision process is quite different. For, in addition to relying on technology/testing to provide information,the medical profession allows doctors to use their own judgement. Differentdoctors may feel that the different types of errors are not equal. For example,some doctors may feel that missing an opportunity for early diagnosis maymean the difference between life and death. A false alarm, on the other hand,may result only in a routine biopsy operation. They may choose to err toward"yes" (tumor present) decisions. Other doctors, however, may feel that unnecessarysurgeries (even routine ones) are very bad (expensive, stress, etc.). Theymay chose to be more conservative and say "no" (no turmor) more often. Theywill miss more tumors, but they will be doing their part to reduce unnecessarysurgeries. And they may feel that a tumor, if there really is one, will bepicked up at the next check-up. These arguments are not about information.Two doctors, with equally good training, looking at the same CT scan, willhave the same information. But they may have a different bias/criterion.
Internal Response and Internal Noise
Detecting a tumor is hard and there will always be some amount of uncertainty. There are two kinds of noise factors that contribute to the uncertainty: internal noise and external noise.External noise: There are many possible sources of external noise. There can be noise factors that are part of the photographic process, a smudge,or a bad spot on the film. Or something in the person's lung that is finebut just looks a bit like a tumor. All of these are to be examples of externalnoise. While the doctor makes every effort possible to reduce the externalnoise, there is little or nothing that they can do to reduce internal noise.
Internal noise: Internal noise refers to the fact that neural responsesare noisy. To make this example really concrete, let's suppose that our doctorhas a set of tumor detector neurons and that they monitor the response ofone of these neurons to determine the likelihood that there is a tumor inthe image (if we could find these neurons then perhaps we could publish andarticle entitled ``What the radiologist's eye tells the radiologist's brain'').These hypothetical tumor detectors will give noisy and variable responses.After one glance at a scan of a healthy lung, our hypothetical tumor detectorsmight fire 10 spikes per second. After a different glance at the same scanand under the same conditions, these neurons might fire 40 spikes per second.
Internal response: Now I do not really believe that there are tumordetector neurons in a radiologist's brain. But there is some internal state,reflected by neural activity somewhere in the brain, that determines thedoctor's impression about whether or not a tumor is present. This is a fundamentalissue; the state of your mind is reflected by neural activity somewhere inyour brain. This neural activity might be concentrated in just a few neuronsor it might be distributed across a large number of neurons. Since we donot know much about where/when this neural activity is, let's simply referto it as the doctor's internal response.
This internal response is inherently noisy. Even when there is no tumor present (no-signal trials) there will be some internal response (sometimes more, sometimes less) in the doctor's sensory system.
Probability of Occurrence Curves
Figure 1 shows a graph of two hypothetical internal response curves. The curve on the left is for the noise-alone (healthy lung) trials, andthe curve on the right is for the signal-plus-noise (tumor present) trials. The horizontal axis is labeled internal response and the verticalaxis is labeled probability. The height of each curve represents howoften that level of internal response will occur. For example, on noise-alone trials, there will generally be about 10 units of internal response: very little. However, there will be some trials with more (or less) internal responsebecause of the internal noise.Notice that we never lose the noise. The internal response for the signal-plus-noise case is generally greater but there is still a distribution (a spread) ofpossible responses. Notice also that the curves overlap, that is, the internalresponse for a noise-alone trial may exceed the internal response for a signal-plus-noisetrial.
Figure 1: Internal response probability of occurrence curves for noise-alone and for signal-plus-noise trials.
Just to be really concrete, we could mark the horizontal axis in units of firing rate (10, 20, 30,..., etc. spikes per second). This would mean thaton a noise-alone (no tumor) trial, it is most likely that the internal responsewould be 10 spikes per second. It is also rather likely that the internalresponse would be 5 or 15 spikes per second. But it is very unlikely thatthe internal response would be 25 spikes per second when no tumor is present.Because I want to remain noncommittal about what and where in the brain theinternal response is, I did not label the horizontal axis in terms of firingrates. The internal response is in some unknown, but quantifiable, units.
The role of the criterion: Perhaps the simplest strategy that thedoctor can adopt is to pick a criterion location along the internal response axis. Whenever the internal response is greater than this criterion they respond "yes". Whenever the internal response is less than this criterion they respond "no".
An example criterion is indicated by the vertical lines in Figure 2. Thecriterion line divides the graph into four sections that correspond to: hits,misses, false alarms, and correct rejections. On both hits and false alarms,the internal response is greater than the criterion, because the doctor isresponding "yes''. Hits correspond to signal-plus-noise trials when the internalresponse is greater than criterion, as indicated in the figure. False alarmscorrespond to noise-alone trials when the internal response is greater thancriterion, as indicated in the figure.
Figure 2: Internal response probability of occurrence curves for noise-alone and signal-plus-noise trials. Since the curves overlap, the internal response for a noise-alone trial may exceed the internal response for a signal-plus-noise trial. Vertical lines correspond to the criterion response.
Suppose that the doctor chooses a low criterion (Figure 3, top), so thatthey respond "yes'' to almost everything. Then they will never miss a tumorwhen it is present and they will therefore have a very high hit rate. Onthe other hand, saying "yes'' to almost everything will greatly increasethe number of false alarms (potentially leading to unnecessary surgeries).Thus, there is a clear cost to increasing the number of hits, and that costis paid in terms of false alarms. If the doctor chooses a high criterion(Figure 3, bottom) then they respond "no'' to almost everything. They willrarely make a false alarm, but they will also miss many real tumors.
Figure 3: Effect of shifting the criterion
Notice that there is no way that the doctor can set their criterion toachieve only hits and no false alarms. The message that you should be takinghome from this is that it is inevitable that some mistakes will bemade. Because of the noise it is simply a true, undeniable, fact that theinternal responses on noise-alone trials may exceed the internal responseson signal-plus-noise trials, in some instances. Thus a doctor cannot alwaysbe right. They can adjust the kind of errors that they make by manipulatingtheir criterion, the one part of this diagram that is under their control.
The Receiver Operating Characteristic
We can describe the full range of the doctor's options in a single curve, called an ROC curve, which stands for receiver-operating characteristic. The ROC curve captures, in a single graph, the various alternatives that areavailable to the doctor as they move their criterion to higher and lowerlevels.ROC curves (Figure 4) are plotted with the false alarm rate on the horizontal axis and the hit rate on the vertical axis. The figure shows several different ROC curves, each corresponding to a different signal strengths. Just pay attentionto one of them (the curve labeled d'=1) for the time being. We already knowthat if the criterion is very high, then both the false alarm rate and thehit rate will be very low, putting you somewhere near the lower left cornerof the ROC graph. If the criterion is very low, then both thehit rate and the false alarm rate will be very high, putting you somewherenear the upper right corner of the graph. For an intermediate choiceof criterion, the hit rate and false alarm rate will take on intermediatevalues. The ROC curve characterizes the choices available to the doctor.They may set the criterion anywhere, but any choice that they make will landthem with a hit and false alarm rate somewhere on the ROC curve. Notice alsothat for any reasonable choice of criterion, the hit rate is always largerthan the false alarm rate, so the ROC curve is bowed upward.
Figure 4: Internal response probability of occurrence curves and ROC curvesfor different signal strengths. When the signal is stronger there is lessoverlap in the probability of occurrence curves, and the ROC curve becomesmore bowed.
The role of information: Acquiring more information makes the decisioneasier. Running another test (e.g., MRI) can be used to acquire more informationabout the presence or absence of a tumor. Unfortunately, the radiologistdoes not have much control over how much information is available.
In a controlled perception experiment the experimenter has complete controlover how much information is provided. Having this control allows for quitea different sort of outcome. If the experimenter chooses to present a strongerstimulus, then the subject's internal response strength will, on the average,be stronger. Pictorially, this will have the effect of shifting the probabilityof occurrence curve for signal-plus-noise trials to the right, a bit furtheraway from the noise-alone probability of occurrence curve.
Figure 4 shows two sets of probability of occurrence curves. When the signalis stronger there is more separation between the two probability of occurrencecurves. When this happens the subject's choices are not so difficult as before.They can pick a criterion to get nearly a perfect hit rate with almost nofalse alarms. ROC curves for stronger signals bow out further than ROC curvesfor weaker signals. Ultimately, if the signal is really strong (lots of information),then the ROC curve goes all the way up to the upper left corner (all hitsand no false alarms).
Varying the noise: There is another aspect of the probability ofoccurrence curves that also determines detectability: the amount of noise.Less noise reduces the spread of the curves. For example, considerthe two probability of occurrence curves in Figure 5. The separation betweenthe peaks is the same but the second set of curves are much skinnier. Clearly,the signal is much more discriminable when there is less spread (less noise)in the probability of occurrence curves. So the subject would have an easiertime setting their criterion in order to be right nearly all the time.
Figure 5: Internal response probability of occurrence curves for two differentnoise levels. When the noise is greater, the curves are wider (more spread)and there is more overlap.
In reality, we have no control over the amount of internal noise. But itis important to realize that decreasing the noise has the same effect asincreasing the signal strength. Both reduce the overlap between the probabilityof occurrence curves.
Discriminability index (d'): Thus, the discriminability ofa signal depends both on the separation and the spread of the noise-alone and signal-plus-noise curves. Discriminability is made easier either by increasingthe separation (stronger signal) or by decreasing the spread (less noise).In either case, there is less overlap between the probability of occurrencecurves. To write down a complete description of how discriminable the signalis from no-signal, we want a formula that captures both the separation andthe spread. The most widely used measure is called d-prime (d' ),and its formula is simply:
d' = separation / spread
This number, d', is an estimate of the strength of the signal. Itsprimary virtue, and the reason that it is so widely used, is that its valuedoes not depend upon the criterion the subject is adopting, but instead itis a true measure of the internal response.
Estimating d': To recap... Increasing the stimulus strength separates the two (noise-alone versus signal-plus-noise) probability of occurrencecurves. This has the effect of increasing the hit and correct rejection rates.Shifting to a high criterion leads to fewer false alarms, fewer hits, andfewer surgical procedures. Shifting to a low criterion leads to more hits(lots of worthwhile surgeries), but many false alarms (unnecessary surgeries)as well. The discriminability index, d', is a measure of the strengthof the internal response that is independent of the criterion.
But how do we measure d'? The trick is that we have to measure boththe hit rate and the false alarm rate, then we can read-off d' froman ROC curve. Figure 4 shows a family of ROC curves. Each of these curvescorresponds to a different d-prime value; d'=0, d'=1, etc.As the signal strength increases, the internal response increases, the ROCcurve bows out more, and d' increases.
So let's say that we do a detection experiment; we ask our doctor to detecttumors in 1000 CT scans. Some of these patients truly had tumors and someof them didn't. We only use patients who have already had surgery (biopsies)so we know which of them truly had tumors. We count up the number of hitsand false alarms. And that drops us somewhere on this plot, on one of theROC curves. Then we simply read off the d' value corresponding tothat ROC curve. Notice that we need to know both the hit rate and thefalse alarm rate to get the discriminability index, d'.
Medical Malpractice Example: A study of doctors' performance wasperformed in Boston. 10,000 cases were analyzed by a special commission. Thecommission decided which were handled negligently and which well. They foundthat 100 were handled very badly and there is good cause for a malpractice suit. Of these 100, only 20 cases were pursued. What should we conclude?
Ralph Nader and others concluded that doctors are not being sued enough. But this conclusion was based on only partial information (hits and misses). I did not tell you what happened in the other 9900 cases. How many law suitswere there in those cases? What if there were many (e.g., 9000 out of 9900)false alarms? The AMA concluded that doctors are being sued too much.
For a more advanced treatment of signal detection theory, download my Signal Detection Theory handout (75 KB, pdf).
Copyright © 2003-2007, Department of Psychology, New York UniversityDavid Heeger