V19 Read output from S3

Use your current course session. Preparation and course runner · R script ↓

Optional deep dive. Enable only the options named in each block, after preparing its packages or services.

V19a Configure S3 storage

Goal: Separate file storage from the metadata catalog.

Prerequisites: s3=TRUE; the environment variables checked in this block; an available credential chain.

Use a training bucket or prefix. The exercise does not delete the bucket. This variant has a local metadata catalog, so it is not a multi-host setup.

training_step(
  "V19a",
  {
    s3_bucket <- Sys.getenv("TRAINING_S3_BUCKET")
    s3_endpoint <- Sys.getenv("TRAINING_S3_ENDPOINT")
    stopifnot(nzchar(s3_bucket), nzchar(s3_endpoint))
    s3_storage <- dataraft.lake::dr_storage_s3(
      s3_bucket,
      prefix = paste0("dataraft-training/", basename(training_root)),
      endpoint = s3_endpoint,
      credential_provider = "credential_chain"
    )
    s3_config <- dataraft.lake::dr_lake_config(
      catalog = dataraft.lake::dr_registry_duckdb(file.path(
        training_root,
        "s3-metadata.ducklake"
      )),
      storage = s3_storage,
      landing = file.path(training_root, "s3-landing"),
      backend = "ducklake",
      install_extensions = training_options$install_extensions
    )
    print(s3_config)
  },
  packages = c("dataraft.lake", "duckdb", "bit64", "paws.storage"),
  enabled = training_options$s3,
  optional = TRUE
)

Expected result: A separate S3 prefix for this run.

Task: Does S3 configuration also create a PostgreSQL catalog?

V19 Read output from S3

Goal: Reuse a stored release.

Prerequisites: V19a and H07.

Credentials are resolved at execution. Only synthetic policy data is published.

training_step(
  "V19",
  {
    s3_release <- dr_publish(
      dr_product("training.s3.policies", policies, contract = policy_contract),
      to = s3_config
    )
    print(dr_collect(s3_release))
    stopifnot(sum(dr_collect(s3_release)$premium) == 740)
  },
  needs = c("V19a", "H07"),
  enabled = training_options$s3,
  optional = TRUE
)

Expected result: The familiar total of 740.

Task: What does a distributed team need?