V11 Retry catalog delivery

Use your current course session. Preparation and course runner · R script ↓

Optional deep dive. Enable only the options named in each block, after preparing its packages or services.

V11a Export a catalog snapshot

Goal: Inspect metadata without the app.

Prerequisites: H26; adapters; extras.

An export is a metadata snapshot, not a live data source.

training_step(
  "V11a",
  {
    catalog_path <- file.path(training_root, "catalog.json")
    dataraft.adapters::dr_catalog_export(training_lake, catalog_path)
    print(file.info(catalog_path)[, "size", drop = FALSE])
  },
  needs = c("H26"),
  packages = c("dataraft.adapters"),
  enabled = training_options$extras,
  optional = TRUE
)

Expected result: A nonempty snapshot file.

Task: Does the file update automatically?

V11b Read freshness metadata

Goal: Interpret freshness in context.

Prerequisites: H26; adapters; extras.

Without agreed business deadlines, lateness has no defined meaning.

training_step(
  "V11b",
  {
    print(dataraft.adapters::dr_freshness(training_lake))
  },
  needs = c("H26"),
  packages = c("dataraft.adapters"),
  enabled = training_options$extras,
  optional = TRUE
)

Expected result: Available freshness metadata, possibly without a deadline.

Task: Does fresh mean correct?

V11c Connect OpenLineage

Goal: Configure a destination for lineage events.

Prerequisites: catalogs=TRUE; TRAINING_OPENLINEAGE_URL; adapters, httr2.

Use your test instance. dr_run sends metadata; write=FALSE suppresses delivery.

training_step(
  "V11c",
  {
    lineage_endpoint <- Sys.getenv("TRAINING_OPENLINEAGE_URL")
    stopifnot(nzchar(lineage_endpoint))
    lineage_catalog <- dataraft.adapters::dr_catalog_openlineage(
      lineage_endpoint,
      namespace = "dataraft-training"
    )
    lineage_product <- policy_product |> dr_add_catalog(lineage_catalog)
    lineage_run <- dr_run(
      lineage_product,
      evidence = file.path(training_root, "lineage-evidence")
    )
    print(lineage_run)
  },
  needs = c("H07"),
  packages = c("dataraft.adapters", "httr2"),
  enabled = training_options$catalogs,
  optional = TRUE
)

Expected result: Delivery or a diagnosed delivery failure.

Task: What identity should retries use for deduplication?

V11d Connect OpenMetadata

Goal: Use a separate catalog destination.

Prerequisites: catalogs=TRUE; TRAINING_OPENMETADATA_URL and TRAINING_OPENMETADATA_SCHEMA; optional TRAINING_OPENMETADATA_TOKEN.

The schema must match your prepared test instance. Credentials stay in the request and are not written into course files.

training_step(
  "V11d",
  {
    om_url <- Sys.getenv("TRAINING_OPENMETADATA_URL")
    om_schema <- Sys.getenv("TRAINING_OPENMETADATA_SCHEMA")
    stopifnot(nzchar(om_url), nzchar(om_schema))
    om_request <- httr2::request(om_url)
    if (nzchar(Sys.getenv("TRAINING_OPENMETADATA_TOKEN"))) {
      om_request <- httr2::req_auth_bearer_token(
        om_request,
        Sys.getenv("TRAINING_OPENMETADATA_TOKEN")
      )
    }
    om_catalog <- dataraft.adapters::dr_catalog_openmetadata(
      om_url,
      database_schema = om_schema,
      request = om_request
    )
    om_run <- dr_run(
      dr_add_catalog(policy_product, om_catalog),
      evidence = file.path(training_root, "om-evidence")
    )
    print(om_run)
  },
  needs = c("H07"),
  packages = c("dataraft.adapters", "httr2"),
  enabled = training_options$catalogs,
  optional = TRUE
)

Expected result: Metadata in the prepared schema, or a diagnosed delivery failure.

Task: Is catalog delivery equivalent to data publication?

V11 Retry catalog delivery

Goal: Retry pending delivery deliberately.

Prerequisites: V11c.

Already acknowledged deliveries are not sent again. Use only the evidence path created in V11c.

training_step(
  "V11",
  {
    retry_history <- dr_retry_catalogs(
      file.path(training_root, "lineage-evidence"),
      list(lineage_catalog)
    )
    print(retry_history)
  },
  needs = c("V11c"),
  enabled = training_options$catalogs,
  optional = TRUE
)

Expected result: Pending deliveries are retried; if none are pending there is nothing to resend.

Task: Must the data product be recalculated?