dr_ingest()

Reads a delivery once, keeps its original file in immutable landing, and evaluates its contract and quality checks before writing any raw table. Rejected input leaves previous accepted releases available. A successful result points to an immutable physical table in the raw schema, suitable for a dbt source binding. Landing is evidence of receipt, not acceptance.

Usage

dr_ingest(
  x,
  to = NULL,
  name = NULL,
  contract = NULL,
  quality = NULL,
  reader = NULL,
  execution = NULL,
  ...
)

Arguments

x: Data frame, lazy table, local file path, zero-argument function, source adapter or product. A product must have one ordinary source and no transformations, lookups, target or catalogs. Its contract, quality rules, name and code version are retained. Use dataraft.core::dr_publish() for transformed products.
to: Local lake folder, connected lake or dr_lake_config() with a raw layer. Defaults to a local "dataraft" folder, using the same configuration and backend marker as dr_open_lake().
name: Optional asset name. Defaults to a file basename, source asset or ID, or the data variable name; expressions use "data".
contract: Optional contract, named type vector or prototype list.
quality: Optional input checks accepted by dataraft.core::dr_add_quality().
reader: Optional file reader. CSV, TSV, RDS and Excel have defaults.
execution: Optional dataraft.core::dr_execution_config() defaults, overriding defaults stored on a product. Its layer must be NULL or "raw"; an explicit to overrides its destination.
...: Named execution options: stop_on_failure (default TRUE), business_date, notify, code_version, and cache (default FALSE). Reusing a cached raw release requires an explicit code_version; live reference checks always require cache = FALSE.

Value

A dr_run_result. Accepted results contain asset, release_id, output_config, and an outputs list with exact database, schema and table identifiers. Failures can be returned with stop_on_failure = FALSE; otherwise the error contains condition$result.

Details

Without a contract, the first accepted delivery establishes a structural schema. Subsequent deliveries must match it. Required values, keys, owners and freshness deadlines are optional and are never guessed. Additional quality checks accept the same functions, formulas, named lists and pointblank adapters as dataraft.core::dr_add_quality(). Business checks run at the input gate; the stored candidate also undergoes structural validation.

Ordinary tables, dataset directories and non-file adapters are archived as RDS snapshots. Single local files retain their original bytes. A snapshot records the received R values, not an API's original transport response. Ingestion collects lazy adapters into memory for this gate and snapshot. For data exceeding available memory, use a lazy product and database target. Connections opened from a configuration are closed before returning; dataraft.core::dr_collect() can reopen the exact accepted release when needed.

Examples

\dontshow{if (requireNamespace("duckdb", quietly = TRUE)) withAutoprint(\{ # examplesIf}
root <- tempfile("raw-ingestion-")
config <- dr_lake_config(path = root)
orders <- data.frame(id = 1:2, amount = c(25, 75))
accepted <- orders |> dr_ingest(to = config, quality = ~ amount >= 0)
dataraft.core::dr_collect(accepted)
accepted$outputs
unlink(root, recursive = TRUE)
\dontshow{\}) # examplesIf}