Oct 21, 2026

Lakehouse – Phase 7 Deletes and Updates

Parquet files are immutable. Once written, you cannot remove row 42 without rewriting the entire file. But lakehouse tables need DELETE and UPDATE. The solution is positional delete files: small secondary files that record which rows in a data file have been logically removed. During scans, the reader loads delete positions and skips those rows. UPDATE decomposes into DELETE + INSERT within a single transaction.

Here is the roadmap for the phases to come:

Phase 7: Deletes and updates
Phase 8: Storage and access

Full Source Code

The code referenced in this post can be found in https://gitlab.com/kimserey.lam/lake-learn.

Copy-on-Write vs Merge-on-Read

There are two strategies for deletes in lakehouse formats:

Copy-on-write (COW): Rewrite the entire data file, omitting deleted rows. Simple to read, expensive to write. Delta Lake defaults to this approach.

Merge-on-read (MOR): Write a small delete file listing positions of deleted rows. Cheap to write, slightly more expensive to read (must check positions during scan). Iceberg and DuckLake use this approach.

DuckLake uses positional delete files — Parquet files with two columns (data_file_id, position) that identify deleted rows by their 0-based position within a specific data file. The core implementation lives in src/storage/ducklake_delete.cpp.

The Delete File Format

A positional delete file is a Parquet file with a specific schema:

Delete file: ducklake-{uuid}-delete.parquet

Schema:
  data_file_id  BIGINT   -- which data file
  position      BIGINT   -- 0-based row number

Example contents:
  data_file_id | position
  -------------|--------
  3            | 42
  3            | 107
  3            | 2048

Delete files are tracked in ducklake_delete_file with the same snapshot visibility as everything else:

ducklake_delete_file
┌────────────────┬──────────┬────────────────┬──────────────┬───────────────────┬──────────────┐
│ delete_file_id │ table_id │ begin_snapshot │ data_file_id │ path              │ delete_count │
├────────────────┼──────────┼────────────────┼──────────────┼───────────────────┼──────────────┤
│ 1              │ 1        │ 5              │ 3            │ del/001-del.pqt   │ 3            │
│ 2              │ 1        │ 6              │ 5            │ del/002-del.pqt   │ 12           │
└────────────────┴──────────┴────────────────┴──────────────┴───────────────────┴──────────────┘

How DELETE Works (lake/delete.go)

DELETE FROM orders WHERE customer = 'bob';

Step 1: SCAN for matching rows (with file positions)
  file: data/001.parquet  positions: [42, 107]
  file: data/003.parquet  positions: [0, 15, 88]

Step 2: WRITE DELETE FILES
  For data/001.parquet → ducklake-uuid1-delete.parquet [42, 107]
  For data/003.parquet → ducklake-uuid2-delete.parquet [0, 15, 88]

Step 3: REGISTER IN METADATA
  BEGIN transaction
    INSERT delete files into ducklake_delete_file
    INSERT new snapshot
  COMMIT

Step 4: RETURN "Deleted 5 rows"

Data files are unchanged on disk. Only new delete files are added:

Before DELETE:
  data/001.parquet: [row0, row1, ..., row42, ..., row107, ..., row999]
  data/003.parquet: [row0, row1, ..., row15, ..., row88, ...]

After DELETE (data files unchanged, delete files added):
  data/001.parquet:                     (unchanged)
  deletes/uuid1-delete.parquet:         positions [42, 107]
  data/003.parquet:                     (unchanged)
  deletes/uuid2-delete.parquet:         positions [0, 15, 88]

The Go implementation:

func (l *Lake) Delete(ctx context.Context, schemaName, tableName string, predicates []Predicate) (int64, error) {
    txn, err := l.snaps.Begin(ctx)
    if err != nil {
        return 0, fmt.Errorf("lake: begin transaction: %w", err)
    }
    defer txn.Rollback(ctx)

    readSnap := txn.ReadSnapshot()

    matches, err := l.scanWithPositions(ctx, schemaName, tableName, predicates, readSnap)
    if err != nil {
        return 0, fmt.Errorf("lake: scan for delete: %w", err)
    }

    if len(matches) == 0 {
        return 0, nil
    }

    _, table, _ := l.resolveTableInTx(ctx, txn, schemaName, tableName)

    // Group positions by source data file
    byFile := make(map[int64][]int64)
    for _, m := range matches {
        byFile[m.dataFileID] = append(byFile[m.dataFileID], m.position)
    }

    for dataFileID, positions := range byFile {
        deleteFileName := fmt.Sprintf("ducklake-%s-delete.parquet", newUUID())
        relativePath := storage.DeletePath(schemaName, tableName, deleteFileName)
        localPath := l.localPath(relativePath)

        os.MkdirAll(filepath.Dir(localPath), 0o755)
        writeDeleteFile(localPath, dataFileID, positions)
        l.uploadFile(ctx, localPath, relativePath)

        txn.AddDeleteFile(catalog.DeleteFileMeta{
            TableID:     table.TableID,
            DataFileID:  dataFileID,
            Path:        relativePath,
            DeleteCount: int64(len(positions)),
        })
    }

    txn.RecordChange(fmt.Sprintf("delete %d rows from %s.%s", len(matches), schemaName, tableName))
    _, err = txn.Commit(ctx)
    return int64(len(matches)), err
}

Scanning with Positions

The delete operation needs to know both the row data (to evaluate the WHERE clause) and each row’s coordinates (file ID + position). scanWithPositions provides this:

type rowWithPosition struct {
    row        datafile.Row
    dataFileID int64
    position   int64
}

It reads every row from every data file (respecting existing deletes), evaluates predicates, and records the position of matching rows.

Writing Delete Files

Delete files are standard Parquet files with a fixed two-column schema:

func writeDeleteFile(path string, dataFileID int64, positions []int64) error {
    deleteCols := []catalog.ColumnDef{
        {FieldID: 1, Name: "data_file_id", Type: catalog.TypeBigInt, Nullable: false},
        {FieldID: 2, Name: "position", Type: catalog.TypeBigInt, Nullable: false},
    }

    var rows []datafile.Row
    for _, pos := range positions {
        rows = append(rows, datafile.Row{
            "data_file_id": dataFileID,
            "position":     pos,
        })
    }

    writer := datafile.NewWriter(deleteCols)
    _, err := writer.WriteFile(path, rows)
    return err
}

How UPDATE Works

UPDATE is decomposed into DELETE + INSERT within a single transaction:

UPDATE orders SET amount = 200.0 WHERE id = 5;

Step 1: SCAN for matching rows
  Find: file data/001.parquet, position 42 → {id=5, name="alice", amount=100.0}

Step 2: DELETE the old row
  Write delete file for position 42

Step 3: INSERT the updated row
  Apply update: {id=5, name="alice", amount=200.0}
  Write new data file

Step 4: REGISTER in single transaction
  BEGIN
    Register delete file (for old row)
    Register new data file (with updated row)
    Create snapshot
  COMMIT

Both the delete file and the new data file share the same snapshot, so readers see the update atomically:

func (l *Lake) Update(ctx context.Context, schemaName, tableName string, predicates []Predicate, updates map[string]any) (int64, error) {
    txn, err := l.snaps.Begin(ctx)
    if err != nil {
        return 0, fmt.Errorf("lake: begin transaction: %w", err)
    }
    defer txn.Rollback(ctx)

    readSnap := txn.ReadSnapshot()
    matches, _ := l.scanWithPositions(ctx, schemaName, tableName, predicates, readSnap)

    if len(matches) == 0 {
        return 0, nil
    }

    _, table, _ := l.resolveTableInTx(ctx, txn, schemaName, tableName)

    // DELETE step: write delete files for old positions
    byFile := make(map[int64][]int64)
    for _, m := range matches {
        byFile[m.dataFileID] = append(byFile[m.dataFileID], m.position)
    }
    for dataFileID, positions := range byFile {
        // ... write and register delete file (same as Delete)
        txn.AddDeleteFile(catalog.DeleteFileMeta{...})
    }

    // INSERT step: apply updates and write new data file
    var updatedRows []datafile.Row
    for _, m := range matches {
        newRow := make(datafile.Row, len(m.row))
        for k, v := range m.row {
            newRow[k] = v
        }
        for k, v := range updates {
            newRow[k] = v
        }
        updatedRows = append(updatedRows, newRow)
    }

    writer := datafile.NewWriter(table.Columns)
    result, _ := writer.WriteFile(localPath, updatedRows)
    txn.AddDataFile(catalog.DataFileMeta{...}, stats)

    txn.RecordChange(fmt.Sprintf("update %d rows in %s.%s", len(matches), schemaName, tableName))
    _, err = txn.Commit(ctx)
    return int64(len(matches)), err
}

Delete Filtering During Scans

The scan pipeline (Phase 4) already handles delete filtering. For each data file, it queries ducklake_delete_file for associated delete files, loads the deleted positions into a map, and skips those positions:

Scan pipeline with deletes:

1. Get active data files at snapshot N
   → [file1.parquet, file2.parquet, file3.parquet]

2. For each file, query delete files:
   file1: no delete files
   file2: del/001.parquet (positions: [5, 12, 99])
   file3: del/002.parquet (positions: [0])

3. Read and filter:
   file1: read all rows
   file2: read all rows, skip positions 5, 12, 99
   file3: read all rows, skip position 0

Deleted positions are stored in a map[int64]struct{} for O(1) lookup. For files with many deletes, a bitset (1 bit per possible position) would be more memory-efficient: 10,000 rows = ~1.2 KB.

Why Not Just Rewrite the File?

Merge-on-read adds complexity to the read path but avoids the much larger cost of rewriting data files. For a table with thousands of files totaling terabytes, deleting one row via copy-on-write means rewriting a multi-gigabyte file. The delete file is typically a few kilobytes.

The tradeoff is read amplification. But in practice: