Summarizes how well a fitted model's predicted probabilities reproduce the observed categories: the classification (confusion) table of observed versus modal predicted categories, the share of correct classifications, the Brier score and the ranked probability score (both strictly proper scoring rules, smaller is better, 0 for perfect probabilistic prediction), and for every category the precision, the recall (hit rate), and the adjusted noise-to-signal ratio of Kaminsky and Reinhart (1999). The same measures are used by Dale and Sirchenko (2021) to compare ordered and inflated ordered fits; because they are computed from predicted probabilities they apply identically to every model in the package and to new data.
Value
An object of class "iop_classification": a list with table
(observed in rows, predicted in columns; with weights, sums of weights
per cell, so that it agrees with the weighted scores), correct (share correctly
classified), brier, rps (ranked probability score), by_category
(precision, recall, noise-to-signal, and the observed share per
category), n, and loglik (the per-observation mean log score,
another proper scoring rule).
Details
The modal-class accuracy measures (share correctly classified, precision, recall) can look poor for a minority category even when the model is well specified: the modal category of a probability vector is rarely a minority category, so its recall is often zero; the proper scoring rules (Brier, ranked probability, log score) are the better summaries of fit. The Brier score is \(\frac{1}{n}\sum_i \sum_j (P_{ij} - I_{ij})^2\) and the ranked probability score \(\frac{1}{n}\sum_i \sum_j (Q_{ij} - D_{ij})^2\) with \(Q\) and \(D\) the cumulative predicted probabilities and the cumulative indicator; both are weighted by the prior weights when present. Precision is TP/(TP + FP), recall TP/(TP + FN), and the adjusted noise-to-signal ratio {FP/(FP + TN)}/{TP/(TP + FN)}, all from the modal-category classification.
References
Brier, G.W. (1950). Verification of forecasts expressed in terms of probability. Monthly Weather Review, 78, 1-3. Epstein, E.S. (1969). A scoring system for probability forecasts of ranked categories. Journal of Applied Meteorology, 8, 985-987. Kaminsky, G.L. and Reinhart, C.M. (1999). The twin crises: The causes of banking and balance-of-payments problems. American Economic Review, 89, 473-500. Dale, D. and Sirchenko, A. (2021). Estimation of nested and zero-inflated ordered probit models. Stata Journal, 21, 3-38.
See also
predict.iord(), compare_models(), vuong()
Other model comparison:
compare_models(),
inflation_test(),
lr_test(),
parallel_test(),
split_test(),
vuong()
Examples
data(bp)
m_op <- oprobit(violence ~ loggdppc + parliament + disaster, data = bp)
m_zi <- iop(violence ~ loggdppc + parliament + disaster | loggdppc + parliament + disaster,
data = bp, inflate = "bottom")
#> The inflation equation contains no covariate that is excluded from the outcome equation; the split is then identified by functional form alone. An exclusion restriction is advisable.
classification(m_op)
#> Classification and accuracy: Ordered probit
#>
#> predicted
#> observed none repression civil war
#> none 1414 0 7
#> repression 380 0 7
#> civil war 174 0 2
#>
#> Correctly classified: 71.4% Brier score: 0.417 Ranked probability score: 0.265 Mean log score: -0.723
#>
#> By category (modal-category classification):
#> share predicted_share precision recall noise_to_signal
#> none 0.716 0.992 0.718 0.995 0.989
#> repression 0.195 0.000 NA 0.000 NA
#> civil war 0.089 0.008 0.125 0.011 0.681
classification(m_zi)
#> Classification and accuracy: Inflated ordered probit (inflated category: none)
#>
#> predicted
#> observed none repression civil war
#> none 1413 0 8
#> repression 376 0 11
#> civil war 165 0 11
#>
#> Correctly classified: 71.8% Brier score: 0.404 Ranked probability score: 0.254 Mean log score: -0.702
#>
#> By category (modal-category classification):
#> share predicted_share precision recall noise_to_signal
#> none 0.716 0.985 0.723 0.994 0.966
#> repression 0.195 0.000 NA 0.000 NA
#> civil war 0.089 0.015 0.367 0.062 0.168
c(op = classification(m_op)$brier, ziop = classification(m_zi)$brier)
#> op ziop
#> 0.4165235 0.4037648