H31 Check raw data at ingestion

Use your current course session. Preparation and course runner · R script ↓

H31 Check raw data at ingestion

Goal: Distinguish received input from accepted RAW data.

Prerequisites: H26 and H06.

dr_ingest archives the incoming delivery and checks before RAW publication. Archiving does not mean acceptance.

training_step(
  "H31",
  {
    raw_accepted <- dataraft.lake::dr_ingest(
      policies,
      to = training_lake,
      name = "policies.raw",
      contract = policy_contract,
      quality = ~ premium >= 0
    )
    raw_rejected <- dataraft.lake::dr_ingest(
      bad_policies,
      to = training_lake,
      name = "policies.raw",
      contract = policy_contract,
      quality = ~ premium >= 0,
      stop_on_failure = FALSE
    )
    print(dr_quality_report(raw_rejected))
    stopifnot(
      raw_accepted$status == "published",
      raw_rejected$status == "blocked"
    )
    stopifnot(sum(dr_collect(raw_accepted)$premium) == 740)
  },
  needs = c("H26", "H06"),
  enabled = training_options$lake,
  optional = TRUE
)

Expected result: Only the valid delivery is accepted.

Task: How does this differ from H09?