• Documentation
    • Search documentation
    karat

    +

    K

    API Reference ↗
    Data connectivity & integrationIceberg tablesGovernanceMarkings

    Markings on Iceberg tables

    Markings restrict access to sensitive data. Iceberg tables can inherit markings from upstream inputs, similar to Foundry datasets. However, Iceberg snapshots can cause markings to gate table access more restrictively than they do for datasets. For example, a marking added on a branch can affect access to main and other branches. Deleting the branch that added the marking is not, on its own, sufficient to remove the marking from the table.

    Markings on Iceberg are table-scoped

    While a marking is directly associated with the Iceberg snapshot that received it, its effect on read access is effectively table-wide. This is because an Iceberg metadata file encompasses snapshots from every branch and tag on the table. To read that metadata file, a user must generally satisfy the markings on every snapshot it contains, unless stop_propagating has been applied.

    In practical terms, a user must satisfy all markings on any snapshot in the table to read the table. This holds even if those markings relate to branches or snapshots the user is not directly trying to access.

    The following table compares inherited marking behavior:

    ScenarioFoundry datasetIceberg table
    Marked transaction or snapshot added to branch ADoes not restrict main branchRestricts all branches including main
    A clean, unmarked replacement is builtThe new current view can become readable while marked historical transactions remain restrictedCurrent snapshots will remain restricted until the marked snapshot is expired
    The branch containing marked history is deletedNo impact on main. It remains unrestricted.No impact on main. It remains restricted until the marked snapshot is expired.
    The marking is removed via stop_propagatingThe marking is removed.The marking is removed.

    Removing an inherited marking

    Choose a removal workflow based on whether you are intentionally removing a marking requirement or removing obsolete marked history.

    Option 1: Stop propagating markings

    When you are intentionally removing a marking requirement from the data, use the stop_propagating workflow to get approval for removing the inherited marking.

    This approach can be useful both for incremental and non-incremental pipelines. In the incremental case this allows you to unmark the table without needing a full rebuild. Note that stopping propagation should only be used when the intent is to expose the marked data.

    Option 2: Remove obsolete marked history

    A marked snapshot may remain in the table history after it becomes obsolete and is no longer part of the current lineage. To remove the marking from the table in that case, take the following steps:

    1. Ensure the marked snapshot is no longer referenced by any branch or tag.
      • If your snapshot is on a branch that can be deleted, you can do this by dropping the branch.
      • If your snapshot is on main or another branch that should not be deleted, then ensure the branch head has a current snapshot that does not depend on the marked snapshot. One way to do this is by running a table replace that fully replaces the current snapshot and does not depend on the marked snapshot as a parent.
    2. Expire the marked snapshots after they are no longer referenced, and confirm that the targeted snapshots no longer appear in the Snapshots tab.

    You must be able to access the marked table and have the operations required to run maintenance to perform this cleanup.