Lakehouse – Phase 5 Snapshots and Time Travel
Every mutation to a lakehouse catalog — INSERT, DELETE, ALTER TABLE — produces a new snapshot. A snapshot is a monotonically increasing integer ID that represents the state of all tables, columns, and data files at that instant. Nothing is overwritten; metadata rows carry visibility ranges that say when each row was “born” and when it was “retired.” This is the foundation of time travel: query any historical state by passing a different snapshot ID to the same visibility filter.
Here is the roadmap for the phases to come:
- Phase 5: Snapshots and time travel
- Phase 6: Schema evolution
- Phase 7: Deletes and updates
- Phase 8: Storage and access
Full Source Code
The code referenced in this post can be found in https://gitlab.com/kimserey.lam/lake-learn.
Why Snapshots Matter
In a traditional database, a “snapshot” usually refers to an ephemeral MVCC view. In a lakehouse, the concept is more permanent: each snapshot is a committed version of the entire catalog, persisted in metadata and queryable indefinitely.
Without snapshots, a lakehouse has no way to answer “what did this table look like yesterday?” and no way to guarantee that a long-running scan sees consistent data while new files are being added concurrently.
Iceberg solves this with a chain of manifest-list files on object storage — each manifest list is effectively a snapshot pointer. Delta Lake uses a JSON transaction log. DuckLake replaces all of that with a single SQL pattern: WHERE begin_snapshot <= N AND (end_snapshot IS NULL OR end_snapshot > N).
In DuckLake’s C++ implementation, snapshot lifecycle is managed by DuckLakeTransaction (src/storage/ducklake_transaction.cpp). The DuckLakeSnapshot struct holds snapshot_id, schema_version, next_catalog_id, and next_file_id.
The Snapshot Table
1
2
3
4
5
6
7
8
9
10
ducklake_snapshot
┌─────────────┬─────────────────────┬────────────────┬─────────────────┬──────────────┐
│ snapshot_id │ snapshot_time │ schema_version │ next_catalog_id │ next_file_id │
├─────────────┼─────────────────────┼────────────────┼─────────────────┼──────────────┤
│ 0 │ 2025-05-25 10:00:00 │ 0 │ 1 │ 1 │
│ 1 │ 2025-05-25 10:00:01 │ 1 │ 3 │ 1 │
│ 2 │ 2025-05-25 10:01:00 │ 1 │ 3 │ 2 │
│ 3 │ 2025-05-25 10:02:00 │ 1 │ 3 │ 3 │
│ 4 │ 2025-05-25 10:05:00 │ 2 │ 4 │ 3 │
└─────────────┴─────────────────────┴────────────────┴─────────────────┴──────────────┘
snapshot_id — monotonically increasing. The latest snapshot is the current state.
schema_version — incremented when any schema-altering operation occurs (ADD/DROP/RENAME COLUMN, CREATE TABLE). Data-only changes (INSERT, DELETE) do not bump this.
next_catalog_id — next available ID for catalog objects (tables, schemas, columns). Used for gap-free ID allocation.
next_file_id — next available ID for data and delete files.
Our Go struct mirrors this:
1
2
3
4
5
6
7
type Snapshot struct {
ID int64
Time time.Time
SchemaVersion int64
NextCatalogID int64
NextFileID int64
}
Visibility Ranges
Every metadata row that can change over time has two columns:
1
2
begin_snapshot BIGINT NOT NULL -- snapshot when this row became active
end_snapshot BIGINT -- snapshot when retired (NULL = still active)
A row is visible at snapshot N when:
1
begin_snapshot <= N AND (end_snapshot IS NULL OR end_snapshot > N)
This is a half-open interval: [begin_snapshot, end_snapshot).
1
2
3
4
5
6
7
8
9
10
11
Timeline: snap 0 snap 1 snap 2 snap 3 snap 4 snap 5
──────────────────────────────────────────────────────────
File A: ├─────────────────────────────────────────────────────▶ (begin=0, end=NULL)
Active from snap 0, still active
File B: ├────────────────────────────┤ (begin=1, end=4)
Active from snap 1 through snap 3
Column X: ├─────────────────────┤ (begin=0, end=3)
Active from snap 0 through snap 2 (dropped at snap 3)
Tables affected by visibility ranges: ducklake_table, ducklake_column, ducklake_data_file, ducklake_delete_file, and ducklake_schema.
In Go, visibility checking is a simple function:
1
2
3
4
5
6
7
8
9
func IsVisible(beginSnap int64, endSnap *int64, querySnap int64) bool {
if querySnap < beginSnap {
return false
}
if endSnap != nil && querySnap >= *endSnap {
return false
}
return true
}
How a Snapshot Is Created
Every mutation follows this pattern:
1
2
3
4
5
6
7
8
9
10
1. BEGIN Postgres transaction
2. Perform the operation:
- INSERT: write Parquet file, register in ducklake_data_file
- DELETE: write delete file, register in ducklake_delete_file
- ALTER: update ducklake_column rows
3. Read the current latest snapshot to get next IDs
4. Compute snapshot_id = latest_snapshot_id + 1
5. INSERT into ducklake_snapshot (new snapshot row)
6. INSERT into ducklake_snapshot_changes (audit record)
7. COMMIT Postgres transaction
The Postgres transaction guarantees atomicity. Compare this with the alternatives:
1
2
3
4
5
6
7
8
9
DuckLake (database-backed):
BEGIN; INSERT file; INSERT snapshot; COMMIT;
→ Single atomic database transaction
Iceberg (manifest-based):
Write manifest-001.avro to S3
Write manifest-list-002.avro to S3
Atomically swap the "current" pointer
→ Requires careful rename semantics, retries on conflict
Time Travel
Time travel means querying a table as it existed at a previous snapshot. The only change in the scan pipeline is which snapshot ID gets passed to the metadata queries.
1
2
3
4
5
6
7
8
Latest snapshot (5):
Files visible: [001.parquet (begin=1), 003.parquet (begin=3)]
Columns active: [id, name, total] (amount renamed to total at snap 4)
At snapshot 3:
Files visible: [001.parquet (begin=1), 002.parquet (begin=2)]
Columns active: [id, name, amount] (not yet renamed)
Delete files: none yet (delete happened at snap 5)
The scan method already supports this via ScanOptions.SnapshotID:
1
2
3
4
5
type ScanOptions struct {
Columns []string
Predicates []Predicate
SnapshotID *int64 // nil = latest snapshot
}
When SnapshotID is set, the scan uses that instead of the latest:
1
2
3
4
5
6
var snapID int64
if opts.SnapshotID != nil {
snapID = *opts.SnapshotID
} else {
snapID, _ = l.latestSnap(ctx)
}
All downstream metadata queries use this snapID in the visibility filter. Nothing else changes.
The Snapshot Manager
The Manager type provides snapshot queries:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
type Manager struct {
store *catalog.MetadataStore
}
func (m *Manager) LatestSnapshot(ctx context.Context) (*Snapshot, error) {
// SELECT ... FROM ducklake_snapshot ORDER BY snapshot_id DESC LIMIT 1
}
func (m *Manager) SnapshotAt(ctx context.Context, version int64) (*Snapshot, error) {
// SELECT ... FROM ducklake_snapshot WHERE snapshot_id = $1
}
func (m *Manager) ListSnapshots(ctx context.Context) ([]Snapshot, error) {
// SELECT ... FROM ducklake_snapshot ORDER BY snapshot_id ASC
}
ListSnapshots returns the full version history — useful for audit trails and debugging.
The Snapshot Changes Table
Each snapshot records what happened:
1
2
3
4
5
6
7
8
9
10
ducklake_snapshot_changes
┌─────────────┬────────────────────────────────────────────────────┐
│ snapshot_id │ changes_made │
├─────────────┼────────────────────────────────────────────────────┤
│ 1 │ create schema "main" │
│ 2 │ create table main.orders (3 columns) │
│ 3 │ insert 100 rows into main.orders │
│ 4 │ add column "status" to main.orders │
│ 5 │ delete 5 rows from main.orders │
└─────────────┴────────────────────────────────────────────────────┘
Every transaction calls txn.RecordChange(description) before committing, and the commit method flushes these to the changes table.
Transaction Isolation
When a reader starts a scan, it reads at the latest snapshot at that moment. Even if another writer commits a new snapshot during the scan, the reader continues to see the old snapshot:
1
2
3
4
5
6
7
8
9
10
Time →
Reader starts scan at snapshot 5
├── Queries metadata: files WHERE begin_snapshot <= 5 ...
├── Reads file A
│
│ Writer commits snapshot 6 (adds file C)
│
├── Reads file B
└── Done. File C was never visible to this reader.
This works because each metadata query includes the snapshot filter. The reader never sees snapshot 6’s files because they have begin_snapshot = 6, which fails begin_snapshot <= 5.
