V22 Use an experimental Iceberg target

Use your current course session. Preparation and course runner · R script ↓

Optional deep dive. Enable only the options named in each block, after preparing its packages or services.

V22a Use an experimental Iceberg source

Goal: Understand a separate table adapter.

Prerequisites: iceberg=TRUE; adapters, duckdb; prepared connection in option dataraft.training_iceberg_connection; TRAINING_ICEBERG_PATH.

The connection must already have the Iceberg extension loaded. REST catalog configuration and authentication are installation-specific.

training_step(
  "V22a",
  {
    iceberg_con <- getOption("dataraft.training_iceberg_connection")
    iceberg_path <- Sys.getenv("TRAINING_ICEBERG_PATH")
    stopifnot(
      !is.null(iceberg_con),
      DBI::dbIsValid(iceberg_con),
      nzchar(iceberg_path)
    )
    iceberg_source <- dataraft.adapters::dr_source_iceberg(
      iceberg_con,
      iceberg_path
    )
    print(dr_collect(dr_run(
      dr_product("training.iceberg", iceberg_source),
      write = FALSE
    )))
  },
  packages = c("dataraft.adapters", "duckdb"),
  enabled = training_options$iceberg,
  optional = TRUE
)

Expected result: Data from the configured test table version.

Task: Is this a DataRaft lake release?

V22 Use an experimental Iceberg target

Goal: Create an explicit test table.

Prerequisites: V22a; TRAINING_ICEBERG_CATALOG and TRAINING_ICEBERG_SCHEMA; transactional REST catalog.

create refuses an existing table. The new table belongs only to the course. This does not promise replacement, lake registry history or atomicity across tables.

training_step(
  "V22",
  {
    iceberg_catalog <- Sys.getenv("TRAINING_ICEBERG_CATALOG")
    iceberg_schema <- Sys.getenv("TRAINING_ICEBERG_SCHEMA")
    stopifnot(nzchar(iceberg_catalog), nzchar(iceberg_schema))
    iceberg_target <- dataraft.adapters::dr_target_iceberg(
      iceberg_con,
      DBI::Id(
        catalog = iceberg_catalog,
        schema = iceberg_schema,
        table = "dataraft_training_policies"
      ),
      mode = "create"
    )
    print(dr_publish(policy_product |> dr_set_target(iceberg_target)))
  },
  needs = c("V22a", "H07"),
  enabled = training_options$iceberg,
  optional = TRUE
)

Expected result: A new test table or an explicit backend error.

Task: Why is iceberg_con not closed automatically?