Proper scoring rules for a fitted count model

Computes the mean logarithmic score and the ranked probability score (RPS) of a fitted model's predicted distribution against observed counts. Lower is better for both, and both are proper, so they are the natural way to compare the calibration of competing count models.

Usage

score(fit, newdata = NULL, kmax = NULL, weights = NULL, offset = NULL)

Arguments

fit

A fitted underdisp count model (cpb, cpb_fe, hurdle_cpb, zi_cpb, count_reg, hurdle_count, zi_count, gec, gec_fe, hurdle_gec, or zi_gec).

newdata

Optional data frame of held-out observations (including the response). When supplied, the fitted model's predicted distribution is evaluated on these rows. A fixed-effects unit the fit has not seen, or a level of any factor its training rows lack (a held-out block's own units or years), enters at the reference unit: the effect, or the dummies' average, of the units the data identify, the reference that first_difference() uses (on the floor support, the participation and zero equations of hurdle_cpb() and zi_cpb() enter such a unit at the mean of all their unit dummies). A unit the hurdles or mixtures hold at a limit is scored at that limit: a positive count of a unit with no positive count in the fitted data, and a hurdle's zero of a unit whose every count was positive, are impossible (see compare_models(), Degrees of freedom). A fit made with an offset reads the rows' offset from its own offset column in newdata, or from offset.

kmax

Highest count to evaluate; defaults to the maximum observed count in the fitting data, raised to the maximum held-out count when newdata contains larger values. Predicted probabilities are floored at 1e-12 in the log score, so an observation outside a hard-ceiling model's support contributes 27.6 to it.

weights

Frequency weights for the rows of newdata: a column name of newdata or a numeric vector with one value per row. The held-out scores are then weighted means, as the in-sample scores of a weighted fit are.

offset

Optional offset (log scale) for the rows of newdata: a column name of newdata or a numeric vector with one value per row. A fit whose offset was given as a vector needs it here; one given as a column name reads that column of newdata by default.

Value

A named numeric vector c(logscore, rps).

Details

By default the scores are computed in sample (against the data the model was fit to, as means weighted by its frequency weights). Supply newdata to score a fitted model on held-out observations, or use cv_score() for a cross-validated score; an in-sample log score equals -logLik/n and does not penalize model complexity, so models of different size are better compared on the held-out or cross-validated score.

See also

cv_score()

Examples

set.seed(1)
y <- rcpb(400, lambda = 3, alpha = 0.5)
fit <- cpb(y ~ 1, data = data.frame(y = y), truncated = FALSE, se = "none")
score(fit)
#>  logscore       rps 
#> 1.6090142 0.6676622