V03 Check reference data

Use your current course session. Preparation and course runner · R script ↓

Optional deep dive. Enable only the options named in each block, after preparing its packages or services.

V03a Warn instead of blocking

Goal: Distinguish quality actions.

Prerequisites: H06; extras.

A warning allows data to continue. This is a business decision, not a repair.

training_step(
  "V03a",
  {
    warning_rule <- dr_quality_rule(
      "nonnegative",
      ~ premium >= 0,
      action = "warn",
      threshold = 0
    )
    warning_product <- dr_product(
      "policies.warning",
      bad_policies,
      contract = policy_contract
    ) |>
      dr_add_quality(warning_rule)
    warning_run <- dr_run(warning_product, write = FALSE)
    print(dr_quality_report(warning_run))
  },
  needs = c("H06"),
  enabled = training_options$extras,
  optional = TRUE
)

Expected result: A warning status and unchanged data.

Task: Has the value -150 disappeared?

V03b Quarantine failing rows

Goal: Retain rejected rows separately.

Prerequisites: H06; extras.

A row-level formula can exclude failing rows from the accepted candidate.

training_step(
  "V03b",
  {
    quarantine_rule <- dr_quality_rule(
      "nonnegative",
      ~ premium >= 0,
      action = "quarantine",
      threshold = 0
    )
    quarantine_product <- dr_product(
      "policies.quarantine",
      bad_policies,
      contract = policy_contract
    ) |>
      dr_add_quality(quarantine_rule)
    quarantine_run <- dr_run(quarantine_product, write = FALSE)
    print(dr_quarantine_rows(quarantine_run))
    stopifnot(nrow(dr_collect(quarantine_run)) == 5L)
  },
  needs = c("H06"),
  enabled = training_options$extras,
  optional = TRUE
)

Expected result: Five accepted rows; the negative premium remains visible in quarantine.

Task: Why can this be risky for a portfolio rate?

V03 Check reference data

Goal: Validate allowed customer IDs.

Prerequisites: H16a; extras.

A reference rule checks membership without enriching columns through a lookup.

training_step(
  "V03",
  {
    reference_rule <- dr_quality_reference(
      customers,
      by = "customer_id",
      name = "known_customer"
    )
    reference_product <- dr_product("policies.reference", linked_policies) |>
      dr_add_quality(reference_rule)
    reference_run <- dr_run(reference_product, write = FALSE)
    print(dr_quality_report(reference_run))
  },
  needs = c("H16a"),
  enabled = training_options$extras,
  optional = TRUE
)

Expected result: All customer IDs are found.

Task: How does this differ from lookup?