Skip to contents

Create tidymodels tuning metrics for concentration-index model selection

Source: R/ci_parsnip_formula.R
ci_tuning_metric_set.Rd

Builds a yardstick yardstick::metric_set() for use in tune::tune_grid() with ineqTrees parsnip engines. The returned metrics recover the socioeconomic rank and optional case weights from the original analysis data using the .row column supplied by tune_grid().

Validation gain uses the actual terminal-node IDs carried by the engine's numeric predictions. Forest gain averages the member-tree partition gains; it does not group observations by their averaged predicted risks. Training gain is computed from the fitted analysis sample and carried separately in prediction metadata. It is descriptive training performance, and should not be used to select parameters in place of validation gain. Relative gains divide by the corresponding analysis or assessment root impurity, returning NA when that root is numerically zero.

Usage

ci_tuning_metric_set(
  data,
  rank_name,
  case_weight_name = NULL,
  type = "CIg",
  metrics = c("validation_gain", "relative_validation_gain"),
  node_name = NULL
)

Arguments

data

Original analysis data used to create the resamples.

rank_name

Name of the socioeconomic rank column in data.

case_weight_name

Optional name of the case-weight column in data.

type

One of "CI", "CIg", "CIc", or "L".

metrics

Character vector of CI metrics to include. Supported values are "validation_gain", "relative_validation_gain", "training_gain", and "relative_training_gain".

node_name

Optional name of a node or partition column. Defaults to the engine's .ci_node column. Supply an explicit column when scoring manually prepared predictions. Missing partition metadata is an error; equal predicted risks do not establish equal terminal-node membership.

Value

A yardstick metric set.

Examples

if (requireNamespace("yardstick", quietly = TRUE)) {
  toy_data <- data.frame(
    rank = c(10, 20, 30, 40),
    outcome = c(1, 0, 1, 0),
    weight = c(1, 1, 2, 2)
  )

  ci_tuning_metric_set(
    toy_data,
    rank_name = "rank",
    case_weight_name = "weight",
    type = "CI",
    metrics = c("validation_gain", "relative_validation_gain")
  )
}
#> A metric set, consisting of:
#> - `validation_gain()`, a numeric metric          | direction: maximize
#> - `relative_validation_gain()`, a numeric metric | direction: maximize