Logistic Regression is a statistical method that we use to fit a regression model when the response variable is binary. To assess how well a logistic regression model fits a dataset, we can look at the following two metrics:
- Sensitivity: The probability that the model predicts a positive outcome for an observation when indeed the outcome is positive.
- Specificity: The probability that the model predicts a negative outcome for an observation when indeed the outcome is negative.
One easy way to visualize these two metrics is by creating a ROC curve, which is a plot that displays the sensitivity and specificity of a logistic regression model.
This tutorial explains how to create and interpret a ROC curve in SPSS.
Example: ROC Curve in SPSS
Suppose we have the following dataset that shows whether or not a basketball player got drafted into the NBA (0 = no, 1 = yes) along with their average points per game in college:
To create an ROC curve for this dataset, click the Analyze tab, then Classify, then ROC Curve:
In the new window that pops up, drag the variable draft into the box labelled State Variable. Define the Value of the State Variable to be 1. (This is the value that indicates a player got drafted). Drag the variable points into the box labelled Test Variable.
Check the boxes next to With diagonal reference line and Coordinate points of the ROC Curve. Then click OK.
Here is how to interpret the output:
Case Processing Summary:
This table displays the total number of positive and negative cases in the dataset. In this example 8 players got drafted (positive result) and 6 players did not get drafted (negative result):
ROC Curve:
The ROC (Receiver Operating Characteristic) curve is a plot of the values of sensitivity vs. 1-specificity as the value of the cut-off point moves from 0 to 1:
A model with high sensitivity and high specificity will have a ROC curve that hugs the top left corner of the plot. A model with low sensitivity and low specificity will have a curve that is close to the 45-degree diagonal line.
We can see that the ROC curve (the blue line) in this example hugs the top left corner of the plot, which indicates that the model does a good job of predicting whether or not players will get drafted, based on their average points per game.
Area Under the Curve:
The Area Under the Curve gives us an idea of how well the model is able to distinguish between positive and negative outcomes. The AUC can range from 0 to 1. The higher the AUC, the better the model is at correctly classifying outcomes.
We can see that the AUC for this particular logistic regression model is .948, which is extremely high. This indicates that the model does a good job of predicting whether or not a player will get drafted.
Coordinates of the Curve:
This last table displays the sensitivity and 1 – specificity of the ROC curve for various cut-off points.
For example:
If we allow the cut-off point to be 8.50, this means we predict that any player who scores less than 8.50 points per game to not get drafted, and any player who scores greater than 8.50 points per game to get drafted.
Using this as a cut off point, our sensitivity would be 100% (since each player that scored less than 8.50 points per game indeed did not get drafted) and our 1 – specificity would be 66.7% (since 8 out of 12 players who scored more than 8.50 points per game actually did get drafted).
The table above allows us to see the sensitivity and 1-specificity for every potential cut-off point.
Great job explaining by use of a simple illustration!
Reading once was enough to fully understand the entire text 🙂
Thank you so much!!! It is very descriptive.
what does Diagonal segments are produced by ties mean?
Fantastic and clear explanation
nice tips
What if the state variable have three or more classes for the outcome or “draft” ?
eg: class 1 -vehicle is ok
class 2- vehicle needs repair
class 3- vehicle to be scrapped
Hello,
Why did you choose 8.5 as cut-off and not 10.5, since that was the point after which sensitivity went below 1.00?
What is to be considered in choosing the cut-offs looking at the Coordinates of the Curve Table?
Thanks.
Thanks for this great article! Do you know if there is a way to change the cut-off scores which are automatically generated? For example, I have just created a ROC curve and in the coordinates table, the cut-offs are all decimals (2.5, 3.5, 4.5 etc). Is it possible to change this table so that the cut-offs for whole numbers are shown (2, 3, 4, 5 etc)?
Good morning.
Thank you very much for this educative write up.
Sir, I am trying to plot a ROC curve for AFP performance in HCV, but only diagonal line shown. ROC curve did not show (i. e. That blue line
Excellent and concise explanation of how to create and interpret ROC curves. Well done!
Hi Joseph…Thank you for feedback!
Why did you choose 8.5 as cut-off and not 10.5, since that was the point after which sensitivity went below 1.00?
What is to be considered in choosing the cut-offs looking at the Coordinates of the Curve Table?
Hi V Jayanthi…When interpreting a Receiver Operating Characteristic (ROC) curve and choosing cut-off points using the Coordinates of the Curve table in SPSS, there are several key factors to consider:
### Key Factors in Choosing Cut-Off Points
1. **Sensitivity (True Positive Rate)**:
– This is the proportion of actual positives correctly identified by the test.
– High sensitivity means the test is good at identifying positive cases.
2. **Specificity (True Negative Rate)**:
– This is the proportion of actual negatives correctly identified by the test.
– High specificity means the test is good at identifying negative cases.
3. **Balance Between Sensitivity and Specificity**:
– The choice of cut-off depends on the trade-off between sensitivity and specificity.
– Higher sensitivity usually means lower specificity and vice versa.
4. **Context of the Problem**:
– Consider the consequences of false positives and false negatives.
– In medical diagnostics, a high sensitivity may be more important to ensure that no cases are missed, even at the expense of more false positives.
5. **Youden’s Index (J)**:
– This is calculated as \( J = \text{Sensitivity} + \text{Specificity} – 1 \).
– The cut-off point that maximizes Youden’s Index is often used as it provides the best balance between sensitivity and specificity.
### Steps to Choose Cut-Off Points Using SPSS
1. **Generate ROC Curve**:
– Analyze your data and create the ROC curve using SPSS.
– Go to `Analyze > ROC Curve`.
2. **Coordinates of the Curve Table**:
– After running the ROC analysis, view the Coordinates of the Curve table, which lists various cut-off points along with their associated sensitivity and specificity values.
3. **Examine Youden’s Index**:
– Add a new column to the Coordinates of the Curve table to calculate Youden’s Index for each cut-off.
– Choose the cut-off point with the highest Youden’s Index value.
4. **Evaluate Sensitivity and Specificity**:
– Review the sensitivity and specificity values for each cut-off point.
– Depending on the context of your analysis, decide on an acceptable trade-off between sensitivity and specificity.
### Example Interpretation of Coordinates of the Curve Table
Suppose the table provides the following values for different cut-off points:
| Cut-off | Sensitivity | Specificity | Youden’s Index |
|———|————-|————-|—————-|
| 0.1 | 0.95 | 0.20 | 0.15 |
| 0.2 | 0.85 | 0.50 | 0.35 |
| 0.3 | 0.75 | 0.70 | 0.45 |
| 0.4 | 0.65 | 0.85 | 0.50 |
| 0.5 | 0.55 | 0.95 | 0.50 |
In this example, cut-off points of 0.4 and 0.5 have the highest Youden’s Index (0.50). You can then consider which cut-off to use based on your desired balance between sensitivity and specificity. If missing positive cases is more critical, you might choose 0.4 for higher sensitivity. If avoiding false positives is more critical, you might choose 0.5 for higher specificity.
### Practical Steps in SPSS
1. **Run ROC Analysis**:
– Go to `Analyze > ROC Curve`.
– Select your test variable and state variable.
– Specify the positive state value.
2. **Obtain Coordinates of the Curve Table**:
– After running the analysis, click on the Coordinates of the Curve table.
3. **Calculate Youden’s Index**:
– Export the table to Excel or manually add a new column in SPSS to calculate Youden’s Index.
4. **Select the Optimal Cut-Off**:
– Choose the cut-off point with the highest Youden’s Index or based on your specific sensitivity and specificity requirements.
By following these steps and considering the key factors, you can choose an appropriate cut-off point for your ROC curve analysis in SPSS.