AUC and AUROC: Classification Thresholds Intuitive Analogy


#1

Here is an intuitive analogy on “classification thresholds”, using the boy who cried wolf story as an example. To recap:
04%20pm

In regards to “classification thresholds”, imagine the shepherd boy has different levels of paranoia every night. On some nights when he is super paranoid, hearing footsteps approaching (which might be another animal’s) would cause him to cry “wolf”. When he is less paranoid, he would wait until he sees a huge shadow, or even wait until he sees a huge animal lurking nearby, before he cries “wolf”. Lowering the “classification thresholds” could be analogous to when he is more paranoid, i.e when he requires less convincing conditions to cry “wolf”. As you can imagine, this would mean he will be crying “wolf” more often. As a result:

  • Being super paranoid, he would be able to identify every appearance of a wolf.
  • This also means, there will be less situations where there was a wolf, but he didn’t spot it, i.e less false negative.
  • However, there will be more situations when the boy thinks there is a wolf, but there isn’t one, i.e more false positives.

As a result, the true positive rate and false positive rate will increase when he is more paranoid (“classification thresholds” is lower). This is as if moving to the right of the ROC curve:

Discussion points:

  • What happens to the curve when the boy becomes less and less paranoid?
  • Here is another thing about ROC curves: If true positive rate = false positive rate, this means an input has a 50% chance of being classified as positive or negative. In other words, the model is useless. Can you see why?

Additional resource on ROC.