V18 Publish with PostgreSQL

Use your current course session. Preparation and course runner · R script ↓

Optional deep dive. Enable only the options named in each block, after preparing its packages or services.

V18a Configure a PostgreSQL catalog

Goal: Store metadata externally.

Prerequisites: postgres=TRUE; TRAINING_PG_CONNECTION with a libpq connection to an isolated test database.

Only the environment-variable name is stored in configuration. Multiple hosts need shared access to both data files and landing storage.

training_step(
  "V18a",
  {
    stopifnot(nzchar(Sys.getenv("TRAINING_PG_CONNECTION")))
    pg_catalog <- dataraft.lake::dr_registry_postgres(
      "TRAINING_PG_CONNECTION",
      lock_timeout = 10
    )
    pg_config <- dataraft.lake::dr_lake_config(
      catalog = pg_catalog,
      storage = dataraft.lake::dr_storage_local(file.path(
        training_root,
        "pg-data"
      )),
      landing = file.path(training_root, "pg-landing"),
      backend = "ducklake",
      install_extensions = training_options$install_extensions
    )
    print(pg_config)
  },
  packages = c("dataraft.lake", "RPostgres", "duckdb", "bit64"),
  enabled = training_options$postgres,
  optional = TRUE
)

Expected result: Configuration without an embedded password.

Task: Does local file storage suffice for two machines?

V18 Publish with PostgreSQL

Goal: Use the configured backend.

Prerequisites: V18a and H07.

The publication call opens and closes its own connection, using a separate training asset ID.

training_step(
  "V18",
  {
    pg_release <- dr_publish(
      dr_product("training.pg.policies", policies, contract = policy_contract),
      to = pg_config
    )
    print(pg_release)
    stopifnot(pg_release$status == "published")
  },
  needs = c("V18a", "H07"),
  enabled = training_options$postgres,
  optional = TRUE
)

Expected result: A release in the test database.

Task: What stays the same after changing the backend?