Lakehouse – Phase 8 Storage and Access

Up to this point, the lake engine reads and writes Parquet files directly on the local filesystem. This works for development, but a real lakehouse stores data on object storage (S3, GCS, Azure Blob) — that is the whole point of separating compute from storage. This final phase introduces a Storage interface that abstracts blob I/O, adds an S3 implementation alongside the local backend, and provides a CLI REPL for interacting with the lake.

This is the final phase. Here is the complete roadmap for the series:

• Phase 1: Metadata catalog
• Phase 2: Parquet data files
• Phase 3: Ingest pipeline
• Phase 4: Scanning and query
• Phase 5: Snapshots and time travel
• Phase 6: Schema evolution
• Phase 7: Deletes and updates
• Phase 8: Storage and access

Full Source Code

The code referenced in this post can be found in https://gitlab.com/kimserey.lam/lake-learn.

The Storage Interface (storage/storage.go)

type Backend interface {
    Put(ctx context.Context, path string, r io.Reader) error
    Get(ctx context.Context, path string) (io.ReadCloser, error)
    Delete(ctx context.Context, path string) error
    Exists(ctx context.Context, path string) (bool, error)
}

This interface covers every operation the lake engine needs:

Operation	When used
Put	Writing a new Parquet data file or delete file
Get	Reading a Parquet file during scan
Delete	Removing orphaned files after failed transactions
Exists	Checking if a file exists before reading

Go interfaces are satisfied implicitly — any struct that implements these four methods is a Backend. No inheritance, no registration. This makes it trivial to add new backends.

In DuckLake’s C++, the storage integration is handled by DuckLakeStorageExtension (src/storage/ducklake_storage.cpp), which hooks into DuckDB’s filesystem layer.

Separating Metadata Location from Data Location

A DuckLake catalog has two independent locations:

┌─────────────────────────────────────────────────────────┐
│                    DuckLake Catalog                       │
│                                                           │
│  Metadata location:  postgres://db.example.com/ducklake  │
│  Data location:      s3://my-bucket/warehouse/           │
│                                                           │
│  These are independent. Metadata can be in Postgres       │
│  while data is in S3.                                     │
└─────────────────────────────────────────────────────────┘

This separation means multiple compute engines can point at the same catalog:

DuckDB    → postgres://... for metadata → s3://bucket/*.parquet for data
Spark     → postgres://... (via JDBC)   → s3://bucket/*.parquet for data
pandas    → postgres://... (psycopg2)   → s3://bucket/*.parquet for data

All three see the same tables, the same snapshots, the same data.

Local Filesystem Implementation (storage/local.go)

The simplest storage backend:

type LocalBackend struct {
    basePath string
}

func NewLocalBackend(basePath string) *LocalBackend {
    return &LocalBackend{basePath: basePath}
}

func (b *LocalBackend) Put(ctx context.Context, path string, r io.Reader) error {
    fullPath := filepath.Join(b.basePath, path)
    if err := os.MkdirAll(filepath.Dir(fullPath), 0755); err != nil {
        return err
    }
    f, err := os.Create(fullPath)
    if err != nil {
        return err
    }
    defer f.Close()
    _, err = io.Copy(f, r)
    return err
}

func (b *LocalBackend) Get(ctx context.Context, path string) (io.ReadCloser, error) {
    fullPath := filepath.Join(b.basePath, path)
    return os.Open(fullPath)
}

func (b *LocalBackend) Delete(ctx context.Context, path string) error {
    fullPath := filepath.Join(b.basePath, path)
    return os.Remove(fullPath)
}

func (b *LocalBackend) Exists(ctx context.Context, path string) (bool, error) {
    fullPath := filepath.Join(b.basePath, path)
    _, err := os.Stat(fullPath)
    if os.IsNotExist(err) {
        return false, nil
    }
    return err == nil, err
}

All paths are relative to basePath. Local storage is used during development and testing.

S3 Implementation (storage/s3.go)

For production deployments, data lives on S3 (or compatible stores like MinIO, R2):

type S3Backend struct {
    client *s3.Client
    bucket string
    prefix string
}

func (b *S3Backend) Put(ctx context.Context, path string, r io.Reader) error {
    key := b.prefix + "/" + path
    _, err := b.client.PutObject(ctx, &s3.PutObjectInput{
        Bucket: &b.bucket,
        Key:    &key,
        Body:   r,
    })
    return err
}

func (b *S3Backend) Get(ctx context.Context, path string) (io.ReadCloser, error) {
    key := b.prefix + "/" + path
    output, err := b.client.GetObject(ctx, &s3.GetObjectInput{
        Bucket: &b.bucket,
        Key:    &key,
    })
    if err != nil {
        return nil, err
    }
    return output.Body, nil
}

Object keys are constructed by joining the configured prefix with the relative file path. For example, with prefix warehouse/production and file path main/orders/data/ducklake-abc123.parquet, the full S3 key is warehouse/production/main/orders/data/ducklake-abc123.parquet.

The S3 client is configured using aws-sdk-go-v2’s default credential chain, which supports environment variables, config files, IAM roles, and instance profiles.

Constructor Injection

The Lake struct does not know or care which storage backend it uses. Tests inject LocalBackend pointing at a temp directory; production injects S3Backend with real credentials:

type Lake struct {
    store    *catalog.MetadataStore
    snaps    *snapshot.Manager
    backend  storage.Backend
    dataPath string
}

func Open(ctx context.Context, pgDSN string, backend storage.Backend, dataPath string) (*Lake, error) {
    // ...
    return &Lake{
        store:    store,
        snaps:    mgr,
        backend:  backend,
        dataPath: dataPath,
    }, nil
}

The uploadFile helper handles the difference between local and remote storage:

func (l *Lake) uploadFile(ctx context.Context, localPath, storagePath string) error {
    if _, ok := l.backend.(*storage.LocalBackend); ok {
        return nil // file is already in the right place
    }

    data, err := os.ReadFile(localPath)
    if err != nil {
        return err
    }
    return l.backend.Put(ctx, storagePath, bytes.NewReader(data))
}

For the local backend, no upload is needed because the file was written directly to the data path. For S3, the file is read back and uploaded.

The CLI REPL (cmd/lake/main.go)

The CLI provides a simple REPL for interacting with the lake:

$ lake --dsn postgres://localhost/ducklake --data ./data

lake> create-schema main
Schema "main" created

lake> create-table main orders id:int name:varchar amount:double
Table "main.orders" created (3 columns)

lake> insert main orders id=1 name=alice amount=100.50
Inserted 1 row

lake> insert main orders id=2 name=bob amount=250.00
Inserted 1 row

lake> scan main orders
┌────┬───────┬────────┐
│ id │ name  │ amount │
├────┼───────┼────────┤
│  1 │ alice │ 100.50 │
│  2 │ bob   │ 250.00 │
└────┴───────┴────────┘

lake> delete main orders name = bob
Deleted 1 row

lake> scan main orders
┌────┬───────┬────────┐
│ id │ name  │ amount │
├────┼───────┼────────┤
│  1 │ alice │ 100.50 │
└────┴───────┴────────┘

lake> scan main orders at 3
┌────┬───────┬────────┐
│ id │ name  │ amount │
├────┼───────┼────────┤
│  1 │ alice │ 100.50 │
│  2 │ bob   │ 250.00 │
└────┴───────┴────────┘

lake> snapshots
┌─────────────┬─────────────────────┬────────────────┐
│ snapshot_id  │ time                │ schema_version │
├─────────────┼─────────────────────┼────────────────┤
│ 1           │ 2025-06-01 10:00:00 │ 0              │
│ 2           │ 2025-06-01 10:00:01 │ 1              │
│ 3           │ 2025-06-01 10:00:02 │ 1              │
│ 4           │ 2025-06-01 10:00:03 │ 1              │
│ 5           │ 2025-06-01 10:00:04 │ 1              │
└─────────────┴─────────────────────┴────────────────┘

Commands: create-schema, create-table, insert, scan (with optional at <snapshot_id>), delete, add-column, drop-column, rename-column, snapshots.

External Engine Access

Because DuckLake stores data as standard Parquet files and metadata in a standard database, any tool can access the lake:

DuckDB (native DuckLake support)

ATTACH 'postgres://localhost/ducklake' AS lake (TYPE ducklake);
SELECT * FROM lake.main.orders WHERE amount > 100;
SELECT * FROM lake.main.orders AT VERSION 3;

Python with pandas

import psycopg2
import pandas as pd

conn = psycopg2.connect("postgres://localhost/ducklake")
cur = conn.cursor()
cur.execute("""
    SELECT path FROM ducklake_data_file
    WHERE table_id = 1
      AND begin_snapshot <= 5
      AND (end_snapshot IS NULL OR end_snapshot > 5)
""")
files = [row[0] for row in cur.fetchall()]

dfs = [pd.read_parquet(f"s3://bucket/warehouse/{f}") for f in files]
df = pd.concat(dfs)

This requires manual snapshot filtering and does not handle delete files, but it demonstrates the openness of the format.

The “Open” in Open Table Format

The architecture has three independent, open components:

1. DATA FILES       → Standard Parquet (readable by any Parquet library)
2. METADATA STORE   → Standard SQL database (queryable by any SQL client)
3. CONVENTIONS      → How files are organized and metadata is structured

Any system that understands all three can participate in the lake. This is the same philosophy as Iceberg and Delta Lake, with one difference: Iceberg’s metadata is in Avro/JSON files (requires custom parsing), while DuckLake’s metadata is in a database (requires only SQL).

Iceberg:     Client must implement manifest parsing + stat filtering + snapshot resolution
DuckLake:    Client runs SQL queries against metadata DB → gets file list → reads Parquet

The SQL database does the heavy lifting that Iceberg clients must implement themselves.

What We Built

Across eight phases, we built a complete, minimal lakehouse engine:

Phase	What it does	Key files
1. Metadata Catalog	Postgres-backed catalog with 8 tables	catalog/
2. Parquet Data Files	Write/read Parquet with column statistics	datafile/
3. Ingest Pipeline	INSERT: rows → Parquet → metadata	lake/insert.go
4. Scanning and Query	SELECT with file pruning and column projection	lake/scan.go
5. Snapshots	Version history and time travel	snapshot/
6. Schema Evolution	ADD/DROP/RENAME without rewriting files	lake/alter.go
7. Deletes and Updates	Positional delete files, UPDATE as DELETE+INSERT	lake/delete.go
8. Storage and Access	Storage abstraction (local, S3) and CLI	storage/, cmd/lake/

At roughly 3,400 lines of Go plus tests, it is small enough to read end-to-end but complete enough to understand how production lakehouse formats work under the hood.