12  Lineage: Where Did This Number Come From?

Lineage is useful only when the question being answered is clear. DataRaft distinguishes product dependency evidence, release evidence, static column dependencies, and broader business interpretation.

Four boxes for product lineage, static column lineage, backend lineage, and business lineage point to an evidence boundary labeled known, unknown, and incomplete.
Figure 12.1: Four useful lineage views converge on an evidence boundary that must preserve what is known and what remains incomplete.

12.1 Dataset lineage from an executed graph

library(dataraft)

policies <- dr_product(
  "policies",
  data.frame(
    policy_id = 1:2,
    premium = c(120, 80)
  )
)

portfolio <- dr_product("portfolio", policies) |>
  dataraft.core::dr_add_recipe(
    dataraft.core::dr_recipe() |>
      dataraft.core::dr_step_select(
        policy_id,
        premium
      )
  )

portfolio_result <- dr_run(
  portfolio,
  write = FALSE
)

dr_lineage(portfolio_result)
# A tibble: 1 × 6
  run_id                    from_id  from_version      to_id to_version relation
  <chr>                     <chr>    <chr>             <chr> <chr>      <chr>   
1 r2b49b02b47c8b10a4c5a53f8 policies r891064029422fdc… port… r2b49b02b… consume…

This records the dependency observed for that execution. It is not a claim that DataRaft reconstructed every dependency that may exist elsewhere in the organization.

12.2 Static column lineage

lineage_recipe <- dataraft.core::dr_recipe() |>
  dataraft.core::dr_step_mutate(
    net_premium = premium / 1.19
  ) |>
  dataraft.core::dr_step_select(
    policy_id,
    net_premium
  )

dataraft.core::dr_column_lineage(
  lineage_recipe,
  c("policy_id", "premium")
)
$fields
$fields$policy_id
[1] "policy_id"

$fields$net_premium
[1] "premium"


$complete
[1] TRUE

$reason
NULL

The analyzer follows a bounded set of direct field references through supported mutate, select, and rename steps.

12.3 Incomplete is a result

Now add an opaque transformation:

opaque_recipe <- dataraft.core::dr_recipe() |>
  dataraft.core::dr_step_transform(
    function(data) {
      data$net_premium <- pmax(data$premium, 0)
      data
    }
  )

dataraft.core::dr_column_lineage(
  opaque_recipe,
  c("policy_id", "premium")
)
$fields
list()

$complete
[1] FALSE

$reason
[1] "Unsupported transformation or row-level dependency."

Unsupported transformations return complete = FALSE rather than a guessed graph. Absence of a proven edge is not proof of independence.

12.4 Release and external lineage

Lake releases can retain exact product and input relationships. The experimental dbt integration can read lineage from dbt artifacts. dataraft.adapters can publish lineage events to OpenLineage (OpenLineage Project 2026) and metadata to OpenMetadata (OpenMetadata 2026).

These mechanisms see different boundaries. Do not combine them into an unsupported claim of “complete lineage.”

12.5 Business lineage

A technical dependency from premium to net_premium still does not explain why the organization calls the result net premium, which business decision uses it, or which owner approves its definition.

Business interpretation belongs in contracts, descriptions, metric definitions, governance metadata, and surrounding documentation.

ImportantLineage is evidence, not omniscience

A trustworthy lineage view preserves unknown and incomplete states instead of turning partial observation into false certainty.

12.6 Exercises

  1. Add a rename to the executable recipe and inspect the field mapping.
  2. Compare dataset lineage for the product graph with static column lineage. Which questions can each answer?
  3. Give one business-lineage question that neither technical view can answer on its own.