Go’s standard library includes a full HTTP server and client in net/http. There are no frameworks to install for basic web services — the standard library handles routing, request parsing, response writing, and even test servers.
An HTTP handler in Go is any function with the signature func(http.ResponseWriter, *http.Request). The ResponseWriter is where you write the response, and the *Request carries everything about the incoming request — method, URL, headers, body. To use a plain function as a handler, wrap it with http.HandlerFunc:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
func helloHandler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Hello, %s %s!", r.Method, r.URL.Path)
}
func main() {
ts := httptest.NewServer(http.HandlerFunc(helloHandler))
defer ts.Close()
resp, err := http.Get(ts.URL + "/world")
if err != nil {
log.Fatal(err)
}
defer resp.Body.Close()
body, _ := io.ReadAll(resp.Body)
fmt.Printf(" status: %s\n", resp.Status)
fmt.Printf(" body: %s\n", string(body))
}
httptest.NewServer starts a real HTTP server on a random port and returns a test server whose URL field gives you the base address. This is the standard way to test HTTP handlers without binding to a fixed port. The handler receives a GET request for /world and writes back Hello, GET /world!.
For services with multiple endpoints, http.NewServeMux provides a router that maps URL patterns to handlers. Go 1.22 introduced method-based patterns, so you can write "GET /health" or "POST /echo" directly in the pattern string:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
mux := http.NewServeMux()
mux.HandleFunc("GET /health", func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
fmt.Fprint(w, "ok")
})
mux.HandleFunc("GET /greet", func(w http.ResponseWriter, r *http.Request) {
name := r.URL.Query().Get("name")
if name == "" {
name = "stranger"
}
fmt.Fprintf(w, "Hello, %s!", name)
})
mux.HandleFunc("POST /echo", func(w http.ResponseWriter, r *http.Request) {
body, _ := io.ReadAll(r.Body)
w.Header().Set("Content-Type", "text/plain")
w.Write(body)
})
ts := httptest.NewServer(mux)
defer ts.Close()
The /health endpoint returns a simple status check. The /greet endpoint reads a query parameter with r.URL.Query().Get("name"). The /echo endpoint reads the request body with io.ReadAll(r.Body) and writes it back. Each route is restricted to a specific HTTP method — a POST to /health would return a 405 Method Not Allowed.
Most services exchange JSON. The pattern is straightforward: define request and response structs with json tags, decode the request body with json.NewDecoder, and encode the response with json.NewEncoder:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
type MathRequest struct {
A float64 `json:"a"`
B float64 `json:"b"`
Op string `json:"op"`
}
type MathResponse struct {
Result float64 `json:"result"`
Error string `json:"error,omitempty"`
}
func mathHandler(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
var req MathRequest
if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
w.WriteHeader(http.StatusBadRequest)
json.NewEncoder(w).Encode(MathResponse{Error: "invalid JSON"})
return
}
var result float64
switch req.Op {
case "add":
result = req.A + req.B
case "sub":
result = req.A - req.B
case "mul":
result = req.A * req.B
case "div":
if req.B == 0 {
w.WriteHeader(http.StatusBadRequest)
json.NewEncoder(w).Encode(MathResponse{Error: "division by zero"})
return
}
result = req.A / req.B
default:
w.WriteHeader(http.StatusBadRequest)
json.NewEncoder(w).Encode(MathResponse{Error: "unknown op: " + req.Op})
return
}
json.NewEncoder(w).Encode(MathResponse{Result: result})
}
The handler sets Content-Type: application/json first, then decodes the request body into a MathRequest. If decoding fails or the operation is invalid, it writes an error response with an appropriate status code and returns early. On success, it encodes the result. The omitempty tag on the Error field means it is omitted from the JSON output when empty, so successful responses only contain the result field.
Middleware is a function that wraps an http.Handler and adds behavior — logging, authentication, rate limiting — without changing the handler itself. The pattern is: take a handler in, return a handler out. To capture the response status code, you need a wrapper around ResponseWriter that intercepts the WriteHeader call:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
type statusRecorder struct {
http.ResponseWriter
status int
}
func (sr *statusRecorder) WriteHeader(code int) {
sr.status = code
sr.ResponseWriter.WriteHeader(code)
}
type logEntry struct {
method string
path string
status int
}
func loggingMiddleware(logs *[]logEntry, next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
rec := &statusRecorder{ResponseWriter: w, status: http.StatusOK}
next.ServeHTTP(rec, r)
*logs = append(*logs, logEntry{
method: r.Method,
path: r.URL.Path,
status: rec.status,
})
})
}
statusRecorder embeds http.ResponseWriter so it satisfies the interface automatically. It overrides only WriteHeader to capture the status code before forwarding the call. The loggingMiddleware function wraps any handler: it creates a recorder, passes it to the inner handler via ServeHTTP, and after the handler returns, records the method, path, and status. Because the signature is func(http.Handler) http.Handler, middleware composes naturally — you can stack multiple layers by wrapping one around another.
Go’s net/http package also provides a client. For simple requests, http.Get and http.Post work directly. When you need custom headers or methods, build an *http.Request with http.NewRequest and execute it with a client.Do call:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
// Simple GET
resp, _ := http.Get(ts.URL + "/simple")
body, _ := io.ReadAll(resp.Body)
resp.Body.Close()
// Simple POST
resp, _ = http.Post(ts.URL+"/data", "text/plain", bytes.NewBufferString("payload"))
body, _ = io.ReadAll(resp.Body)
resp.Body.Close()
// Custom headers with NewRequest + client.Do
req, _ := http.NewRequest("GET", ts.URL+"/custom", nil)
req.Header.Set("User-Agent", "GoLearn/1.0")
req.Header.Set("X-Custom", "my-value")
client := &http.Client{}
resp, _ = client.Do(req)
body, _ = io.ReadAll(resp.Body)
resp.Body.Close()
http.Get and http.Post are convenience wrappers that use the default client. For anything beyond trivial requests — setting headers, configuring timeouts, controlling redirects — create an *http.Request explicitly and pass it to client.Do. Always close resp.Body when you are done reading, typically with defer resp.Body.Close(), to avoid leaking connections.
When a service receives a termination signal (Ctrl+C sends SIGINT, container orchestrators send SIGTERM), it should not just crash. It should stop accepting new work, finish in-flight operations, flush buffers, close connections, and then exit. Go makes this possible with os/signal, channels, and context cancellation.
The standard approach is: create a cancellable context, listen for signals on a buffered channel, cancel the context when a signal arrives, and use a WaitGroup to wait for goroutines to finish their cleanup.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
func main() {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
sigCh := make(chan os.Signal, 1)
signal.Notify(sigCh, os.Interrupt, syscall.SIGTERM)
go func() {
sig := <-sigCh
fmt.Printf("\n [signal] received %v, initiating shutdown...\n", sig)
cancel()
}()
proc := NewProcessor()
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
defer func() {
proc.Flush()
fmt.Println(" [worker] cleanup complete")
}()
ticker := time.NewTicker(500 * time.Millisecond)
defer ticker.Stop()
flushTicker := time.NewTicker(2 * time.Second)
defer flushTicker.Stop()
items := []string{
"INSERT users alice",
"UPDATE users bob",
"INSERT orders ord-1",
"DELETE sessions expired",
"INSERT events login",
}
idx := 0
for {
select {
case <-ctx.Done():
return
case <-ticker.C:
if idx < len(items) {
proc.Process(items[idx])
idx++
} else {
idx = 0
}
case <-flushTicker.C:
proc.Flush()
}
}
}()
wg.Wait()
fmt.Println(" [main] all goroutines stopped, exiting cleanly")
}
There are several details worth noting here. The signal channel has a buffer of 1 — make(chan os.Signal, 1). This is important because the signal delivery is non-blocking: if the channel is full, the signal is dropped. A buffer of 1 ensures the signal is captured even if the goroutine is momentarily busy. signal.Notify registers the channel to receive os.Interrupt (Ctrl+C) and syscall.SIGTERM (the standard termination signal from process managers).
The signal-handling goroutine blocks on <-sigCh until a signal arrives, then calls cancel(). This cancels the context, which causes ctx.Done() to close in every goroutine that is selecting on it. The worker goroutine sees the cancellation, returns from the loop, and its deferred functions run in LIFO order — first flushTicker.Stop(), then ticker.Stop(), then proc.Flush(), and finally wg.Done().
The Processor type buffers work items and flushes them either periodically or during shutdown:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
type Processor struct {
buffer []string
}
func NewProcessor() *Processor {
return &Processor{buffer: make([]string, 0, 100)}
}
func (p *Processor) Process(item string) {
p.buffer = append(p.buffer, item)
}
func (p *Processor) Flush() {
if len(p.buffer) == 0 {
return
}
fmt.Printf(" [processor] flushing %d items\n", len(p.buffer))
for _, item := range p.buffer {
fmt.Printf(" -> %s\n", item)
}
p.buffer = p.buffer[:0]
}
The Flush method resets the buffer with p.buffer[:0], which keeps the underlying array allocated but sets the length to zero. This pattern avoids allocating a new slice on every flush. The key guarantee of this architecture is that no data is silently lost — the deferred Flush call in the worker ensures that any items buffered between the last periodic flush and the shutdown signal are still written out before the process exits.
Production services need to evolve their database schema over time — adding tables, altering columns, creating indexes. Migration tools track which changes have been applied and run only the new ones. In Go, the go:embed directive lets you compile migration files directly into the binary, and goose provides the migration engine.
The go:embed directive is a compiler instruction that bakes file contents into the binary at build time. The variable it annotates becomes an embed.FS — a read-only filesystem that you can read from at runtime without any external files:
1
2
3
4
import "embed"
//go:embed migrations/*.sql
var migrationFS embed.FS
The comment //go:embed migrations/*.sql must appear directly above the variable declaration with no blank line in between. It tells the compiler to find every .sql file in the migrations/ directory and embed their contents. At runtime, migrationFS is a fully functional read-only filesystem — you can list directories, read files, and pass it to any library that accepts an fs.FS:
1
2
3
4
5
6
7
func part1_whatIsEmbed() {
entries, _ := migrationFS.ReadDir("migrations")
for _, e := range entries {
data, _ := migrationFS.ReadFile("migrations/" + e.Name())
fmt.Printf(" %-30s (%d bytes)\n", e.Name(), len(data))
}
}
This prints the name and size of each embedded SQL file. The files exist inside the compiled binary — if you delete the migrations/ directory after building, the binary still works. This is particularly useful for deployment: your migration files ship with the binary rather than needing to be copied alongside it.
Each migration file contains a -- +goose Up section with the forward change and a -- +goose Down section with the rollback. Goose uses these annotations to know which SQL to run in each direction:
1
2
3
4
5
6
7
8
9
10
-- +goose Up
CREATE TABLE IF NOT EXISTS users (
id SERIAL PRIMARY KEY,
name TEXT NOT NULL,
email TEXT NOT NULL UNIQUE,
created_at TIMESTAMPTZ DEFAULT now()
);
-- +goose Down
DROP TABLE IF EXISTS users;
A second migration adds a column to the existing table:
1
2
3
4
5
-- +goose Up
ALTER TABLE users ADD COLUMN IF NOT EXISTS role TEXT NOT NULL DEFAULT 'member';
-- +goose Down
ALTER TABLE users DROP COLUMN IF EXISTS role;
Migrations are numbered sequentially. Goose runs them in order and records the current version in a goose_db_version table so it knows which migrations have already been applied.
The workflow has four steps: open a database connection, point goose at the embedded filesystem, set the SQL dialect, and call Up to apply pending migrations:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
import (
"database/sql"
"embed"
_ "github.com/jackc/pgx/v5/stdlib"
"github.com/pressly/goose/v3"
)
//go:embed migrations/*.sql
var migrationFS embed.FS
func part2_runMigrations() {
db, err := sql.Open("pgx", dbDSN)
if err != nil {
log.Fatalf("open db: %v", err)
}
defer db.Close()
goose.SetBaseFS(migrationFS)
if err := goose.SetDialect("postgres"); err != nil {
log.Fatalf("set dialect: %v", err)
}
if err := goose.Up(db, "migrations"); err != nil {
log.Fatalf("migration failed: %v", err)
}
}
goose.SetBaseFS(migrationFS) tells goose to read migration files from the embedded filesystem instead of disk. goose.SetDialect("postgres") configures the SQL generation for PostgreSQL. goose.Up(db, "migrations") scans the migrations directory, compares the file versions against the goose_db_version table, and runs any that have not been applied yet. If all migrations are already applied, it does nothing.
The blank import _ "github.com/jackc/pgx/v5/stdlib" registers the pgx driver with database/sql so that sql.Open("pgx", dsn) works. This is a common Go pattern: importing a package solely for its init function’s side effects.
To check the current version after migrations have run:
1
2
3
4
5
6
7
8
9
10
func part3_checkVersion() {
db, _ := sql.Open("pgx", dbDSN)
defer db.Close()
version, err := goose.GetDBVersion(db)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Current version: %d\n", version)
}
goose.GetDBVersion returns the highest migration version that has been applied. After running both migration files, this returns 2. To add a new migration, you create 003_something.sql in the migrations/ directory and rebuild. The next call to goose.Up applies only the new file.
Go is a statically typed language, but sometimes you need to inspect or manipulate types at runtime. The reflect package provides this capability. Libraries like encoding/json, database/sql, and fmt all use reflection internally to work with arbitrary types. Understanding reflection helps you read those libraries and write generic utilities of your own.
The two entry points to reflection are reflect.TypeOf, which returns the type of a value, and reflect.ValueOf, which returns a reflect.Value wrapping the actual data. Every reflected value has a Kind (the broad category: int, string, struct, slice, ptr) and a Type (the specific type: main.Employee, []int, *main.Employee):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
i := 42
s := "hello"
f := 3.14
sl := []int{1, 2, 3}
m := map[string]int{"a": 1, "b": 2}
e := Employee{Name: "Alice", Age: 30, Department: "Engineering", Salary: 120000}
p := &e
values := []interface{}{i, s, f, sl, m, e, p}
labels := []string{"int", "string", "float64", "slice", "map", "struct", "pointer"}
for idx, v := range values {
t := reflect.TypeOf(v)
rv := reflect.ValueOf(v)
fmt.Printf(" %-8s Kind=%-10s Type=%-28s Value=%v\n",
labels[idx], t.Kind(), t, rv)
}
The distinction between Kind and Type matters. A Kind of struct tells you the value is some struct, while the Type tells you it is specifically main.Employee. A pointer to that struct has Kind of ptr and Type of *main.Employee. This distinction is how generic code decides what operations are valid — you check Kind to branch on the category, then use Type for specifics.
Reflection can iterate over a struct’s fields, read their types, check whether they are exported, and extract struct tags. This is exactly how encoding/json decides which fields to include and what JSON key names to use:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
type Employee struct {
Name string `json:"name"`
Age int `json:"age,omitempty"`
Department string `json:"department"`
Salary float64 `json:"salary,omitempty"`
active bool
}
e := Employee{
Name: "Bob", Age: 25, Department: "Marketing", Salary: 85000, active: true,
}
t := reflect.TypeOf(e)
v := reflect.ValueOf(e)
for i := 0; i < t.NumField(); i++ {
field := t.Field(i)
val := v.Field(i)
exported := field.IsExported()
jsonTag := field.Tag.Get("json")
valStr := ""
if exported {
valStr = fmt.Sprintf("%v", val.Interface())
} else {
valStr = "(unexported)"
}
fmt.Printf(" %-12s %-10s exported=%-5t tag=%q value=%s\n",
field.Name, field.Type, exported, jsonTag, valStr)
}
t.NumField() returns the number of fields. t.Field(i) returns a reflect.StructField with the field’s name, type, tag string, and export status. field.Tag.Get("json") extracts the value of the json key from the struct tag. The unexported active field is visible to reflection — you can see its name and type — but calling Interface() on its value will panic. The CanInterface() check (or IsExported() on the field descriptor) tells you whether it is safe to read the value.
Reflection can also modify values, but only if you pass a pointer. reflect.ValueOf(x) gives you a read-only copy. To get a settable value, you need reflect.ValueOf(&x).Elem() — the Elem() call dereferences the pointer, giving reflection access to the original variable:
1
2
3
4
5
6
7
8
9
x := 10
rv := reflect.ValueOf(&x).Elem()
fmt.Printf(" CanSet(): %t\n", rv.CanSet()) // true
rv.SetInt(42)
fmt.Printf(" x after SetInt(42): %d\n", x) // 42
// Without pointer — CanSet() returns false
rvNoPtr := reflect.ValueOf(x)
fmt.Printf(" CanSet(): %t\n", rvNoPtr.CanSet()) // false
The same principle applies to struct fields. Pass a pointer to the struct, call Elem(), then use FieldByName to find and modify individual fields:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
e := Employee{Name: "Charlie", Age: 28, Department: "Sales", Salary: 70000}
rv := reflect.ValueOf(&e).Elem()
nameField := rv.FieldByName("Name")
if nameField.IsValid() && nameField.CanSet() {
nameField.SetString("Diana")
}
salaryField := rv.FieldByName("Salary")
if salaryField.IsValid() && salaryField.CanSet() {
salaryField.SetFloat(95000)
}
fmt.Printf(" after: %+v\n", e) // {Name:Diana Age:28 Department:Sales Salary:95000 active:false}
Unexported fields cannot be set even with a pointer — CanSet() returns false for them. Go enforces its visibility rules through reflection, not just at compile time. Always check IsValid() (the field exists) and CanSet() (the field is settable) before calling a setter.
Reflection can inspect function signatures and call functions dynamically. reflect.ValueOf(fn) wraps the function, and Call invokes it with a slice of reflect.Value arguments:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
func add(a, b int) int { return a + b }
func greet(name string, excited bool) string {
if excited {
return "Hello, " + name + "!"
}
return "Hello, " + name + "."
}
addVal := reflect.ValueOf(add)
addType := addVal.Type()
fmt.Printf(" NumIn: %d, NumOut: %d\n", addType.NumIn(), addType.NumOut())
for i := 0; i < addType.NumIn(); i++ {
fmt.Printf(" Arg[%d]: %s\n", i, addType.In(i))
}
args := []reflect.Value{reflect.ValueOf(3), reflect.ValueOf(4)}
results := addVal.Call(args)
fmt.Printf(" add(3, 4) = %v\n", results[0].Interface()) // 7
greetArgs := []reflect.Value{reflect.ValueOf("Go"), reflect.ValueOf(true)}
greetResults := reflect.ValueOf(greet).Call(greetArgs)
fmt.Printf(" greet(\"Go\", true) = %q\n", greetResults[0].Interface()) // "Hello, Go!"
NumIn() and NumOut() tell you how many parameters and return values the function has. In(i) gives the type of each parameter. Call takes a []reflect.Value and returns a []reflect.Value. Each argument must match the expected type exactly — passing a float64 where an int is expected will panic. This is the mechanism that RPC frameworks and dependency injection containers use to call functions they only know about at runtime.
To see reflection applied to a real problem, here is PrintTable — a function that takes any slice of structs and prints it as a formatted table. It does not know the struct type at compile time; it discovers the field names and values at runtime:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
func PrintTable(slice interface{}) {
v := reflect.ValueOf(slice)
if v.Kind() != reflect.Slice {
return
}
if v.Len() == 0 {
return
}
elemType := v.Type().Elem()
if elemType.Kind() != reflect.Struct {
return
}
numFields := elemType.NumField()
// Collect headers from field names
headers := make([]string, numFields)
for i := 0; i < numFields; i++ {
headers[i] = elemType.Field(i).Name
}
// Collect row values as strings
rows := make([][]string, v.Len())
for i := 0; i < v.Len(); i++ {
row := make([]string, numFields)
for j := 0; j < numFields; j++ {
field := v.Index(i).Field(j)
if field.CanInterface() {
row[j] = fmt.Sprintf("%v", field.Interface())
} else {
row[j] = "(unexported)"
}
}
rows[i] = row
}
// Compute column widths
widths := make([]int, numFields)
for i, h := range headers {
widths[i] = len(h)
}
for _, row := range rows {
for i, cell := range row {
if len(cell) > widths[i] {
widths[i] = len(cell)
}
}
}
// Print header
for i, h := range headers {
fmt.Printf(" %-*s", widths[i], h)
if i < numFields-1 {
fmt.Print(" | ")
}
}
fmt.Println()
// Print separator
for i, w := range widths {
fmt.Print(" " + strings.Repeat("-", w))
if i < numFields-1 {
fmt.Print("-+-")
}
}
fmt.Println()
// Print rows
for _, row := range rows {
for i, cell := range row {
fmt.Printf(" %-*s", widths[i], cell)
if i < numFields-1 {
fmt.Print(" | ")
}
}
fmt.Println()
}
}
The function starts by validating that the input is a non-empty slice of structs. It reads the element type with v.Type().Elem() to discover the struct’s fields. Headers come from the field names, row values from formatting each field with fmt.Sprintf("%v", field.Interface()). The column width calculation passes over headers and all rows to find the widest string in each column, then uses %-*s format padding to align everything.
You can call it with any struct type:
1
2
3
4
5
6
7
8
9
10
11
12
13
type Book struct {
Title string
Author string
Pages int
}
books := []Book{
{"The Go Programming Language", "Donovan & Kernighan", 380},
{"Concurrency in Go", "Katherine Cox-Buday", 238},
{"Go in Action", "Kennedy, Ketelsen, St. Martin", 264},
}
PrintTable(books)
This prints a neatly aligned table with Title, Author, and Pages columns — without PrintTable knowing anything about the Book type at compile time. This is the same approach that encoding/json uses internally: it inspects struct fields via reflection to decide what to encode, reads struct tags to determine JSON key names, and handles exported and unexported fields differently. The difference is that encoding/json does this at scale with caching and optimization, but the underlying mechanism is the same reflect API shown here.
io.Reader and io.Writer interfaces, serializing and deserializing JSON, and writing tests. These are the building blocks you will reach for in nearly every Go project — packages keep code organized, the I/O interfaces let you compose data pipelines, JSON handles serialization, and the testing package gives you a lightweight framework for verifying everything works.
Go code is organized into packages. Every .go file starts with a package declaration, and all files in the same directory must use the same package name. A module is a collection of packages versioned together, defined by a go.mod file at the root. When you import a package, Go resolves it through the module system.
Go’s standard library provides a rich set of packages out of the box. You import them by path, and each import gives you access to the package’s exported identifiers. Here is a program that uses several standard library packages to demonstrate the breadth of what is available without any third-party dependencies:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
package main
import (
"fmt"
"math"
"sort"
"strconv"
"strings"
)
func main() {
fmt.Println("=== Standard Library Packages ===")
fmt.Println(strings.ToUpper("hello"))
fmt.Println(strings.Contains("go is great", "great"))
fmt.Println(strings.Join([]string{"a", "b", "c"}, "-"))
fmt.Println(math.Sqrt(144))
fmt.Println(math.Pi)
nums := []int{5, 2, 8, 1, 9}
sort.Ints(nums)
fmt.Println(nums)
n, _ := strconv.Atoi("42")
fmt.Printf("converted string to int: %d\n", n)
}
Each import path corresponds to a directory in the standard library or in your module. The strings package provides string manipulation, math provides mathematical constants and functions, sort provides sorting algorithms for slices, and strconv handles conversions between strings and other types. You only pay for what you import — the Go compiler rejects unused imports.
Go uses a simple naming convention for visibility. Any identifier (variable, function, type, field) that starts with an uppercase letter is exported — accessible from other packages. Anything starting with a lowercase letter is unexported — visible only within its own package.
Consider a calculator package:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
package calculator
import (
"errors"
"fmt"
)
var Version = "1.0.0" // exported — other packages can read this
var maxValue = 1_000_000 // unexported — only visible inside calculator
type Result struct {
Value int // exported field
Label string // exported field
logNote string // unexported field — hidden from other packages
}
func Add(a, b int) int { return a + b } // exported
func Multiply(a, b int) int { return a * b } // exported
func validate(n int) error { // unexported
if n < 0 {
return errors.New("negative numbers not allowed")
}
if n > maxValue {
return fmt.Errorf("value %d exceeds max %d", n, maxValue)
}
return nil
}
func SafeAdd(a, b int) (int, error) {
if err := validate(a); err != nil {
return 0, fmt.Errorf("invalid first argument: %w", err)
}
if err := validate(b); err != nil {
return 0, fmt.Errorf("invalid second argument: %w", err)
}
return a + b, nil
}
From another package, you can call calculator.Add, calculator.SafeAdd, and read calculator.Version. You cannot call calculator.validate or access calculator.maxValue — the compiler will reject it. Likewise, if you create a calculator.Result from outside the package, you can set Value and Label but not logNote. This visibility rule is Go’s entire encapsulation mechanism — there are no public/private keywords.
A constructor function is the idiomatic way to initialize structs with unexported fields:
1
2
3
4
5
6
7
8
9
10
11
12
func NewResult(value int, label string) Result {
return Result{
Value: value,
Label: label,
logNote: fmt.Sprintf("computed %s = %d", label, value),
}
}
func (r Result) String() string {
return fmt.Sprintf("Result{Value: %d, Label: %q, logNote: %q}",
r.Value, r.Label, r.logNote)
}
Calling calculator.NewResult(42, "sum") returns a Result with all fields properly set, including the unexported logNote. The String() method can read logNote because it is defined within the same package.
Each package can define one or more init() functions. These run automatically when the package is loaded — after all package-level variables are initialized, but before main() starts. You do not call init() yourself; Go handles the timing.
1
2
3
4
5
package calculator
func init() {
fmt.Println(" [calculator.init] calculator package initialized!")
}
When any other package imports calculator, Go runs this init() function as a side effect. The order follows the import dependency graph: if package A imports package B, then B’s init() runs before A’s. Within a single package, init() functions run in the order they appear in source files (sorted by filename).
The init() function is useful for registering drivers, validating configuration, or setting up package-level state. But use it sparingly — implicit initialization can make code harder to reason about.
As a project grows, you split code into multiple packages. A common pattern is a registry package that other packages register with during init(). This decouples the registry from knowing about specific implementations.
Here is a registry package that stores encoders by name:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
package registry
import "fmt"
var drivers = map[string]Encoder{}
type Encoder interface {
Encode(v any) ([]byte, error)
}
func Register(name string, enc Encoder) {
drivers[name] = enc
fmt.Printf(" [registry] registered %q driver\n", name)
}
func List() []string {
names := make([]string, 0, len(drivers))
for name := range drivers {
names = append(names, name)
}
return names
}
func Get(name string) (Encoder, bool) {
enc, ok := drivers[name]
return enc, ok
}
Individual driver packages implement the Encoder interface and register themselves via init():
1
2
3
4
5
6
7
8
9
10
11
12
package json
import (
"encoding/json"
"go-learn/ex15_packages/registry"
)
type encoder struct{}
func (encoder) Encode(v any) ([]byte, error) { return json.Marshal(v) }
func init() { registry.Register("json", encoder{}) }
1
2
3
4
5
6
7
8
9
10
11
12
package xml
import (
"encoding/xml"
"go-learn/ex15_packages/registry"
)
type encoder struct{}
func (encoder) Encode(v any) ([]byte, error) { return xml.Marshal(v) }
func init() { registry.Register("xml", encoder{}) }
Each driver package has zero coupling to the other drivers. The registry does not import the driver packages. The only thing that ties them together is the Encoder interface.
The driver packages above register themselves through init(), but nothing in the main program directly calls any function from them. Normally, Go would reject an unused import. The blank import syntax — _ "package/path" — tells the compiler “import this package for its side effects only”:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
package main
import (
"fmt"
"go-learn/ex15_packages/calculator"
"go-learn/ex15_packages/registry"
_ "go-learn/ex15_packages/drivers/json"
_ "go-learn/ex15_packages/drivers/xml"
)
func main() {
fmt.Println("Version:", calculator.Version)
fmt.Println("2 + 3 =", calculator.Add(2, 3))
result := calculator.NewResult(42, "answer")
fmt.Println(result)
drivers := registry.List()
fmt.Println("registered drivers:", drivers)
if enc, ok := registry.Get("json"); ok {
data, _ := enc.Encode(map[string]int{"x": 1, "y": 2})
fmt.Println("json output:", string(data))
}
}
When Go loads this program, the import graph triggers init() functions in dependency order: first the registry package, then the json and xml driver packages (which call registry.Register), then the calculator package, and finally main. By the time main() runs, both drivers are registered and ready to use.
This pattern — a registry interface plus blank-imported driver packages — is used throughout Go’s standard library. The database/sql package works exactly this way: you import a driver like _ "github.com/lib/pq" and the driver registers itself with sql.Register during init(). The image package uses the same pattern for image format decoders.
Go’s io package defines two small interfaces that underpin almost all I/O in the language. io.Reader reads bytes from a source; io.Writer writes bytes to a destination. Because they are interfaces, any type that implements the right method signature qualifies — files, network connections, in-memory buffers, compressors, encryptors, and anything else you can imagine. This composability is what makes Go’s I/O system so powerful.
io.Reader has a single method:
1
2
3
type Reader interface {
Read(p []byte) (n int, err error)
}
The caller provides a byte slice p, and the reader fills it with up to len(p) bytes. It returns the number of bytes read and an error. When the data source is exhausted, it returns io.EOF. The key insight is that Read may return fewer bytes than requested — you typically call it in a loop.
Here is how to read from a strings.Reader in small chunks:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
reader := strings.NewReader("Hello, Go io.Reader!")
buf := make([]byte, 5)
for {
n, err := reader.Read(buf)
if n > 0 {
fmt.Printf(" read %d bytes: %q\n", n, buf[:n])
}
if err == io.EOF {
fmt.Println(" reached EOF")
break
}
if err != nil {
fmt.Println(" error:", err)
break
}
}
The 5-byte buffer means each Read call gets at most 5 bytes. The string “Hello, Go io.Reader!” is 20 bytes, so this loop will make four full reads and one final read that returns io.EOF. Notice that we check n > 0 before checking err — a Read can return both data and io.EOF on the same call.
io.Writer is the mirror of io.Reader:
1
2
3
type Writer interface {
Write(p []byte) (n int, err error)
}
You pass in a byte slice, and the writer consumes it. Two of the most common writers are os.Stdout (standard output) and bytes.Buffer (an in-memory buffer that grows as needed):
1
2
3
4
5
6
7
var buf bytes.Buffer
buf.Write([]byte("Hello "))
buf.Write([]byte("Buffer!"))
fmt.Println("buffer contents:", buf.String())
fmt.Println("buffer length:", buf.Len())
fmt.Fprintln(os.Stdout, "this goes to stdout via io.Writer")
bytes.Buffer implements io.Writer, so you can pass it anywhere a writer is expected. fmt.Fprintln takes any io.Writer as its first argument — passing os.Stdout writes to the terminal, but you could just as easily pass a file, a network connection, or a buffer.
io.Copy connects a reader to a writer. It reads from the source and writes to the destination until it hits io.EOF or an error:
1
2
3
4
src := strings.NewReader("streaming data from reader to writer")
dst := &bytes.Buffer{}
n, err := io.Copy(dst, src)
fmt.Printf("copied %d bytes: %q\n", n, dst.String())
There is also io.CopyN for copying a specific number of bytes:
1
2
3
4
src2 := strings.NewReader("only first 10 bytes please")
dst2 := &bytes.Buffer{}
n2, _ := io.CopyN(dst2, src2, 10)
fmt.Printf("copied %d bytes: %q\n", n2, dst2.String())
io.Copy is the idiomatic way to transfer data between any reader and writer. It handles the read loop, buffering, and EOF detection for you.
The io package provides several functions that wrap readers and writers to add behavior. This is composition over inheritance — you build complex I/O pipelines by layering simple wrappers.
io.TeeReader creates a reader that writes everything it reads to a writer, like the Unix tee command:
1
2
3
4
5
6
7
original := strings.NewReader("data to tee")
var log bytes.Buffer
tee := io.TeeReader(original, &log)
result, _ := io.ReadAll(tee)
fmt.Printf("read: %q\n", string(result))
fmt.Printf("log captured: %q\n", log.String())
io.MultiReader concatenates multiple readers into one. Reads drain each reader in order:
1
2
3
4
5
6
7
r1 := strings.NewReader("Hello ")
r2 := strings.NewReader("from ")
r3 := strings.NewReader("multiple readers!")
multi := io.MultiReader(r1, r2, r3)
combined, _ := io.ReadAll(multi)
fmt.Printf("combined: %q\n", string(combined))
io.LimitReader wraps a reader and stops after a fixed number of bytes:
1
2
3
4
unlimited := strings.NewReader("this is a very long string that we want to limit")
limited := io.LimitReader(unlimited, 20)
data, _ := io.ReadAll(limited)
fmt.Printf("limited read: %q\n", string(data))
These building blocks compose freely. You could create a TeeReader that wraps a LimitReader that wraps a MultiReader — each layer adds one piece of behavior, and the io.Reader interface is the glue that holds them together.
Because io.Reader is just an interface with one method, you can create your own readers. Here is a CountingReader that wraps any reader and tracks how many bytes have been read:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
type CountingReader struct {
reader io.Reader
BytesRead int
}
func NewCountingReader(r io.Reader) *CountingReader {
return &CountingReader{reader: r}
}
func (cr *CountingReader) Read(p []byte) (int, error) {
n, err := cr.reader.Read(p)
cr.BytesRead += n
return n, err
}
The implementation delegates to the wrapped reader and adds up the byte count. Because CountingReader satisfies io.Reader, it can be used anywhere a reader is expected:
1
2
3
4
source := strings.NewReader("count these bytes")
counter := NewCountingReader(source)
io.ReadAll(counter)
fmt.Printf("total bytes read: %d\n", counter.BytesRead)
Here is a more involved example — a RepeatReader that produces the same text a given number of times:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
type RepeatReader struct {
text string
remaining int
current *strings.Reader
}
func NewRepeatReader(text string, times int) *RepeatReader {
return &RepeatReader{text: text, remaining: times}
}
func (rr *RepeatReader) Read(p []byte) (int, error) {
for {
if rr.current != nil {
n, err := rr.current.Read(p)
if n > 0 {
return n, nil
}
if err == io.EOF {
rr.current = nil
continue
}
return n, err
}
if rr.remaining <= 0 {
return 0, io.EOF
}
rr.remaining--
rr.current = strings.NewReader(rr.text)
}
}
Each time the current strings.Reader is exhausted, RepeatReader creates a fresh one from the same text — until the repeat count reaches zero, at which point it returns io.EOF. This pattern of lazily creating underlying readers is common in Go I/O code.
1
2
3
4
repeater := NewRepeatReader("Go! ", 3)
data, _ := io.ReadAll(repeater)
fmt.Printf("repeated: %q\n", string(data))
// Output: "Go! Go! Go! "
Go’s encoding/json package handles serialization (Go values to JSON) and deserialization (JSON to Go values). It uses struct tags to control how struct fields map to JSON keys, and it provides both high-level functions (Marshal/Unmarshal) and a streaming decoder for working with JSON data.
Struct tags are metadata strings attached to struct fields. The json tag tells the JSON encoder and decoder what key name to use for each field. Without tags, the JSON key matches the Go field name exactly. Unexported fields (lowercase) are always ignored by the JSON package.
1
2
3
4
5
6
7
type Person struct {
FirstName string `json:"first_name"`
LastName string `json:"last_name"`
Age int `json:"age"`
Email string `json:"email"`
password string // unexported — json package cannot see this
}
When you marshal a Person, the JSON keys will be first_name, last_name, age, and email. The password field is invisible to the JSON encoder because it starts with a lowercase letter:
1
2
3
4
5
6
7
8
9
10
p := Person{
FirstName: "Alice",
LastName: "Smith",
Age: 30,
Email: "alice@example.com",
password: "secret123",
}
data, _ := json.Marshal(p)
fmt.Println(string(data))
// {"first_name":"Alice","last_name":"Smith","age":30,"email":"alice@example.com"}
The password field is excluded entirely. This is a natural consequence of Go’s export rules — the JSON package is in a different package from your struct, so it can only see exported fields.
json.Marshal converts any Go value to a JSON byte slice. It works with structs, slices, maps, and primitive types:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
type Point struct {
X int `json:"x"`
Y int `json:"y"`
}
// Struct
p := Point{X: 1, Y: 2}
data, _ := json.Marshal(p)
fmt.Println(string(data))
// {"x":1,"y":2}
// Slice
points := []Point{{X: 1, Y: 2}, {X: 3, Y: 4}}
data, _ = json.Marshal(points)
fmt.Println(string(data))
// [{"x":1,"y":2},{"x":3,"y":4}]
// Map
m := map[string]int{"width": 100, "height": 200}
data, _ = json.Marshal(m)
fmt.Println(string(data))
// {"height":200,"width":100}
For human-readable output, use json.MarshalIndent:
1
2
3
4
5
6
data, _ = json.MarshalIndent(p, "", " ")
fmt.Println(string(data))
// {
// "x": 1,
// "y": 2
// }
The second argument is a prefix for each line, and the third is the indent string. Using an empty prefix and two spaces is the most common convention.
json.Unmarshal takes a JSON byte slice and populates a Go value. You pass a pointer to the target value so the function can modify it:
1
2
3
4
5
6
7
8
9
jsonStr := `{"first_name":"Bob","last_name":"Jones","age":25,"email":"bob@example.com"}`
var person Person
err := json.Unmarshal([]byte(jsonStr), &person)
if err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%+v\n", person)
// {FirstName:Bob LastName:Jones Age:25 Email:bob@example.com password:}
Two important behaviors of Unmarshal: unknown fields in the JSON are silently ignored, and missing fields get Go’s zero values. If the JSON contains a key phone that has no matching struct field, nothing happens. If the JSON is missing the age key, the struct’s Age field stays at 0.
1
2
3
4
5
partial := `{"first_name":"Charlie"}`
var p2 Person
json.Unmarshal([]byte(partial), &p2)
fmt.Printf("%+v\n", p2)
// {FirstName:Charlie LastName: Age:0 Email: password:}
Struct tags support several options that control marshaling behavior. The most useful ones are omitempty, -, and string.
1
2
3
4
5
6
7
8
9
type Config struct {
Name string `json:"name"`
Debug bool `json:"debug,omitempty"`
Verbose bool `json:"verbose,omitempty"`
Secret string `json:"-"`
Count int `json:"count,string"`
Timeout *int `json:"timeout,omitempty"`
MaxRetry *int `json:"max_retry,omitempty"`
}
The omitempty option tells the encoder to skip the field if it has its zero value (empty string, 0, false, nil pointer, empty slice, or empty map). This keeps the JSON output compact by omitting fields that carry no information.
The "-" tag tells the encoder to always skip this field, regardless of its value. Use this for sensitive data or internal state that should never appear in JSON output.
The "string" option encodes a numeric or boolean field as a JSON string. This is useful when interacting with APIs that represent numbers as strings.
Pointer fields interact interestingly with omitempty. A nil pointer is omitted, but a pointer to a zero value is included:
1
2
3
4
5
6
7
8
9
10
11
12
13
timeout := 0
maxRetry := 0
c1 := Config{
Name: "app",
Secret: "password",
Count: 42,
Timeout: &timeout,
MaxRetry: nil,
}
data, _ := json.MarshalIndent(c1, "", " ")
fmt.Println(string(data))
In this output, Debug and Verbose are omitted (zero booleans with omitempty), Secret is omitted (tag is "-"), Count appears as "42" (a string, not a number), Timeout appears as 0 (pointer to zero is not nil, so it is included), and MaxRetry is omitted (nil pointer with omitempty). This distinction between nil and zero is the main reason to use pointer fields with omitempty.
Sometimes you do not know the JSON structure at compile time. Go provides several tools for working with dynamic JSON.
The first is map[string]any. Unmarshaling into a map gives you a dynamic key-value structure where the JSON decoder chooses Go types for you (strings become string, numbers become float64, booleans become bool, objects become map[string]any, and arrays become []any):
1
2
3
4
5
6
7
dynamic := `{"name":"test","count":42,"active":true,"tags":["a","b"]}`
var m map[string]any
json.Unmarshal([]byte(dynamic), &m)
for key, val := range m {
fmt.Printf(" %s: %v (%T)\n", key, val, val)
}
The second tool is json.RawMessage. It lets you defer parsing part of a JSON document. The raw bytes are stored as-is, and you decode them later once you know what type to expect. This is ideal for envelope patterns where a type field determines the schema of a payload field:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
type Envelope struct {
Type string `json:"type"`
Payload json.RawMessage `json:"payload"`
}
type TextMsg struct {
Body string `json:"body"`
}
type ImageMsg struct {
URL string `json:"url"`
Width int `json:"width"`
Height int `json:"height"`
}
messages := []string{
`{"type":"text","payload":{"body":"Hello!"}}`,
`{"type":"image","payload":{"url":"pic.png","width":800,"height":600}}`,
}
for _, raw := range messages {
var env Envelope
json.Unmarshal([]byte(raw), &env)
switch env.Type {
case "text":
var msg TextMsg
json.Unmarshal(env.Payload, &msg)
fmt.Printf(" text message: %s\n", msg.Body)
case "image":
var msg ImageMsg
json.Unmarshal(env.Payload, &msg)
fmt.Printf(" image: %s (%dx%d)\n", msg.URL, msg.Width, msg.Height)
}
}
The Payload field stays as raw JSON bytes until we inspect Type and know which struct to decode into. Without json.RawMessage, you would need to unmarshal twice or use a map.
The third tool is json.Decoder, which reads JSON values from an io.Reader stream. This is useful for processing multiple JSON objects from a single source (a file, a network connection, or any reader) without loading everything into memory at once:
1
2
3
4
5
6
7
8
9
10
stream := `{"name":"Alice"}{"name":"Bob"}{"name":"Charlie"}`
decoder := json.NewDecoder(strings.NewReader(stream))
for decoder.More() {
var obj map[string]string
if err := decoder.Decode(&obj); err != nil {
break
}
fmt.Printf(" decoded: %v\n", obj)
}
decoder.More() returns true as long as there is another JSON value in the stream. Each call to Decode reads exactly one value, advances the stream position, and populates the target. This is more memory-efficient than reading the entire input and splitting it yourself.
Go has a built-in testing framework in the testing package. Test files live next to the code they test, with a _test.go suffix. Test functions start with Test and take a single *testing.T argument. You run them with go test.
The simplest test calls a function and checks the result. If the result is wrong, you call t.Errorf with a descriptive message. There is no assertion library — just conditional checks and error reporting:
1
2
3
4
5
6
7
8
9
10
11
12
// math_test.go
package main
import "testing"
func TestAdd(t *testing.T) {
got := Add(2, 3)
want := 5
if got != want {
t.Errorf("Add(2, 3) = %d, want %d", got, want)
}
}
The got/want naming convention is idiomatic Go. If got does not equal want, t.Errorf logs the failure and the test continues (unlike t.Fatalf, which stops the test immediately). A test with no errors passes.
When you need to test a function with many inputs, Go developers use table-driven tests. You define a slice of test cases — each with a name, inputs, and expected outputs — and loop over them with t.Run. This pattern is clean, easy to extend, and produces well-labeled output when a test fails:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
func TestDivide(t *testing.T) {
tests := []struct {
name string
a, b float64
want float64
wantError bool
}{
{name: "positive", a: 10, b: 2, want: 5, wantError: false},
{name: "negative result", a: -10, b: 2, want: -5, wantError: false},
{name: "divide by zero", a: 5, b: 0, want: 0, wantError: true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
got, err := Divide(tt.a, tt.b)
if tt.wantError {
if err == nil {
t.Errorf("expected error")
}
return
}
if err != nil {
t.Errorf("unexpected error: %v", err)
return
}
if got != tt.want {
t.Errorf("got %v, want %v", got, tt.want)
}
})
}
}
Each call to t.Run creates a subtest with the given name. When you run go test -v, you see output like TestDivide/positive, TestDivide/negative_result, and TestDivide/divide_by_zero. If the “divide by zero” case fails, the name tells you exactly which case went wrong without you having to count indices.
Table-driven tests make it natural to include edge cases alongside the happy path. Here is a primality check test that covers zero, one, negative numbers, and a large prime:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
func TestIsPrime(t *testing.T) {
tests := []struct {
name string
n int
want bool
}{
{"zero", 0, false},
{"one", 1, false},
{"two", 2, true},
{"four", 4, false},
{"negative", -7, false},
{"large prime", 97, true},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := IsPrime(tt.n); got != tt.want {
t.Errorf("IsPrime(%d) = %v, want %v", tt.n, got, tt.want)
}
})
}
}
The test covers the boundary conditions that are most likely to expose bugs: numbers below 2, the smallest prime, a composite number, and a negative number. Adding a new edge case is just adding one more entry to the slice.
When you find yourself repeating the same assertion logic across multiple tests, extract it into a helper function. Calling t.Helper() at the top of the helper tells the testing framework to report failures at the caller’s line number, not inside the helper. Without t.Helper(), every failure would point to the same line inside the helper function, making it hard to tell which test case failed:
1
2
3
4
5
6
7
8
9
10
11
12
13
func assertEqual(t *testing.T, got, want string) {
t.Helper()
if got != want {
t.Errorf("got %q, want %q", got, want)
}
}
func TestFizzBuzz(t *testing.T) {
assertEqual(t, FizzBuzz(1), "1")
assertEqual(t, FizzBuzz(3), "Fizz")
assertEqual(t, FizzBuzz(5), "Buzz")
assertEqual(t, FizzBuzz(15), "FizzBuzz")
}
If FizzBuzz(5) returned the wrong value, the failure message would point to the assertEqual(t, FizzBuzz(5), "Buzz") line in TestFizzBuzz, not to the t.Errorf line inside assertEqual. This is the difference t.Helper() makes.
Go strings are UTF-8 encoded, and operations that work on bytes can break multi-byte characters. Testing with unicode input catches these issues. Here is a Reverse function tested with ASCII, unicode, and emoji:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
func TestReverse(t *testing.T) {
tests := []struct {
name, input, want string
}{
{"ascii", "hello", "olleh"},
{"empty", "", ""},
{"unicode", "Hello, 世界", "界世 ,olleH"},
{"emoji", "Go🚀", "🚀oG"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := Reverse(tt.input); got != tt.want {
t.Errorf("Reverse(%q) = %q, want %q", tt.input, got, tt.want)
}
})
}
}
The Reverse function must work on runes (Unicode code points), not bytes. If it reversed bytes instead of runes, the multi-byte characters in “Hello, 世界” would be corrupted. Including unicode test cases in the table is a simple way to verify correct behavior:
1
2
3
4
5
6
7
func Reverse(s string) string {
runes := []rune(s)
for i, j := 0, len(runes)-1; i < j; i, j = i+1, j-1 {
runes[i], runes[j] = runes[j], runes[i]
}
return string(runes)
}
Converting to []rune ensures each element is a full Unicode code point. The swap loop works on runes, not bytes, so multi-byte characters like 世, 界, and 🚀 are reversed correctly. The test table makes it trivial to add more unicode edge cases as you discover them.
select statement for multiplexing channel operations. The standard library rounds this out with a context package for cancellation and deadlines, and a sync package for traditional mutual exclusion when channels are not the right fit. This tutorial walks through all five building blocks, starting from basic goroutines and working up to a concurrent cache protected by read-write locks and atomics.
A goroutine is a function that runs concurrently with other goroutines in the same address space. You start one with the go keyword followed by a function call. Goroutines are multiplexed onto a small number of OS threads by the Go runtime scheduler, so spawning thousands of them is cheap — each one starts with a stack of only a few kilobytes.
The simplest way to launch a goroutine is with an anonymous function. Here, the main goroutine creates a channel, launches a background goroutine that sends a value, and then blocks on a receive:
1
2
3
4
5
6
7
8
func part1_basicGoroutine() {
done := make(chan bool)
go func() {
fmt.Println(" hello from a goroutine!")
done <- true
}()
<-done
}
The go func() { ... }() syntax launches the anonymous function as a new goroutine. The channel receive <-done blocks main until the goroutine sends a value, ensuring we see the printed message before the program exits.
If you launch goroutines and do not wait for them, the main function may exit before they finish. This is a common mistake:
1
2
3
4
5
6
7
8
9
func part2_brokenNoSync() {
for i := 0; i < 3; i++ {
go func(id int) {
fmt.Printf(" worker %d: processing (might not print!)\n", id)
}(i)
}
time.Sleep(10 * time.Millisecond)
fmt.Println(" main continued immediately — goroutines may or may not have finished")
}
The time.Sleep gives the goroutines a brief window to run, but there is no guarantee. On a fast machine they might finish; on a loaded machine they might not. Sleeping is never a correct synchronization mechanism.
A sync.WaitGroup is the standard tool for waiting on a set of goroutines to complete. You call Add to register work, Done to signal completion, and Wait to block until all work is done:
1
2
3
4
5
6
7
8
9
10
11
12
13
func part3_waitGroup() {
var wg sync.WaitGroup
for i := 0; i < 3; i++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
time.Sleep(time.Duration(id*50) * time.Millisecond)
fmt.Printf(" worker %d: done\n", id)
}(i)
}
wg.Wait()
fmt.Println(" all workers finished!")
}
wg.Add(1) is called before launching each goroutine, not inside it — this avoids a race where Wait could return before all goroutines have registered. defer wg.Done() ensures the counter is decremented even if the goroutine panics. After wg.Wait() returns, all three goroutines have completed.
When launching goroutines in a loop, the loop variable must be passed explicitly as a function argument. Otherwise, all goroutines share the same variable and may all see the final value:
1
2
3
4
5
6
7
8
9
10
11
func part4_closureCapture() {
var wg sync.WaitGroup
for i := 0; i < 3; i++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
fmt.Printf(" worker id=%d\n", id)
}(i)
}
wg.Wait()
}
The expression go func(id int) { ... }(i) copies the current value of i into the parameter id. Each goroutine gets its own copy. If you wrote go func() { fmt.Println(i) }() instead, all three goroutines would close over the same i variable, and you would likely see 3, 3, 3 instead of 0, 1, 2.
A common pattern is to fan out work across multiple goroutines and collect results. Since each goroutine writes to a distinct index in the results slice, no lock is needed:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
func part5_concurrentWorkers() {
batches := [][]string{
{"INSERT users alice", "INSERT users bob"},
{"UPDATE orders ord-1", "DELETE orders ord-2"},
{"INSERT events evt-1", "INSERT events evt-2", "INSERT events evt-3"},
}
var wg sync.WaitGroup
results := make([]int, len(batches))
for i, batch := range batches {
wg.Add(1)
go func(id int, items []string) {
defer wg.Done()
for _, item := range items {
fmt.Printf(" worker %d: %s\n", id, item)
}
results[id] = len(items)
}(i, batch)
}
wg.Wait()
total := 0
for _, count := range results { total += count }
fmt.Printf(" all workers done. total items processed: %d\n", total)
}
Each goroutine receives its own id and items slice. The results slice is pre-allocated with one slot per worker, so concurrent writes to different indices are safe without synchronization. After wg.Wait(), the main goroutine aggregates the counts.
Channels are Go’s primary mechanism for communication between goroutines. A channel is a typed conduit — you send values into it and receive values out of it. Channels enforce synchronization: a send on an unbuffered channel blocks until another goroutine receives, and vice versa. This “communication by sharing” model is the foundation of Go’s concurrency philosophy.
An unbuffered channel has no internal storage. A send blocks until a receiver is ready, and a receive blocks until a sender is ready. This provides a synchronization point between two goroutines:
1
2
3
4
5
6
7
8
9
func part1_unbuffered() {
ch := make(chan string)
go func() {
ch <- "hello from goroutine"
}()
time.Sleep(50 * time.Millisecond)
msg := <-ch
fmt.Printf(" receiver: got %q\n", msg)
}
The goroutine’s ch <- "hello from goroutine" blocks until main executes msg := <-ch. The sleep in main demonstrates that the sender will patiently wait — there is no data loss.
A buffered channel has internal capacity. Sends do not block until the buffer is full, and receives do not block as long as the buffer is non-empty:
1
2
3
4
5
6
7
8
9
10
func part2_buffered() {
ch := make(chan int, 3)
ch <- 10
ch <- 20
ch <- 30
fmt.Printf(" sent 3 values without blocking (len=%d, cap=%d)\n", len(ch), cap(ch))
fmt.Printf(" received: %d\n", <-ch)
fmt.Printf(" received: %d\n", <-ch)
fmt.Printf(" received: %d\n", <-ch)
}
make(chan int, 3) creates a channel with a buffer of three. All three sends succeed without blocking because the buffer has room. The len function returns how many values are currently in the buffer, and cap returns the total capacity.
Channel parameters can be restricted to send-only or receive-only. This makes the intended data flow explicit and catches mistakes at compile time:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
func produce(out chan<- string) {
changes := []string{"INSERT alice", "UPDATE bob", "DELETE charlie"}
for _, c := range changes {
out <- c
}
close(out)
}
func consume(in <-chan string) {
for msg := range in {
fmt.Printf(" consumer received: %s\n", msg)
}
}
func part3_directional() {
ch := make(chan string, 5)
go produce(ch)
consume(ch)
}
The chan<- type means the function can only send to the channel. The <-chan type means the function can only receive. A bidirectional chan string is automatically convertible to either direction when passed as an argument.
Closing a channel signals that no more values will be sent. A for range loop over a channel receives values until the channel is closed:
1
2
3
4
5
6
7
8
9
10
func part4_closeAndRange() {
ch := make(chan int, 5)
for i := 1; i <= 5; i++ { ch <- i * 10 }
close(ch)
for val := range ch {
fmt.Printf(" got: %d\n", val)
}
val, ok := <-ch
fmt.Printf(" receive after close: val=%d, ok=%t <- zero value, ok=false\n", val, ok)
}
After close(ch), the for range loop drains all remaining values and then exits. A receive on a closed, empty channel returns the zero value immediately with ok=false. Only the sender should close a channel — closing a channel that has already been closed, or sending on a closed channel, will panic.
Combining goroutines, channels, and close gives you a clean producer-consumer pattern. The producer sends structured data, closes the channel when done, and the consumer processes everything until the channel is drained:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
type TxBatch struct {
Table string
Changes []string
}
func part5_producerConsumer() {
ch := make(chan TxBatch, 5)
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
batches := []TxBatch{
{Table: "users", Changes: []string{"INSERT alice", "INSERT bob"}},
{Table: "orders", Changes: []string{"INSERT ord-1"}},
{Table: "users", Changes: []string{"UPDATE alice"}},
{Table: "events", Changes: []string{"INSERT evt-1", "INSERT evt-2"}},
}
for _, b := range batches { ch <- b }
close(ch)
}()
wg.Add(1)
go func() {
defer wg.Done()
total := 0
for batch := range ch {
fmt.Printf(" consumer: processing %s batch (%d changes)\n", batch.Table, len(batch.Changes))
total += len(batch.Changes)
}
fmt.Printf(" consumer: done, processed %d total changes\n", total)
}()
wg.Wait()
}
The producer goroutine sends four batches and then closes the channel. The consumer goroutine ranges over the channel, processing each batch as it arrives. The WaitGroup ensures main waits for both goroutines to finish. This pattern scales naturally — you can add more consumers by launching additional goroutines that range over the same channel.
The select statement lets a goroutine wait on multiple channel operations simultaneously. It looks like a switch, but each case is a channel send or receive. When multiple cases are ready, Go picks one at random — this prevents starvation. Combined with tickers and timeouts, select is the building block for event loops, polling, and graceful shutdown.
A select with multiple cases executes whichever channel operation is ready. If multiple are ready, one is chosen at random:
1
2
3
4
5
6
7
8
9
10
11
func part1_basicSelect() {
ch1 := make(chan string, 1)
ch2 := make(chan string, 1)
ch2 <- "hello from ch2"
select {
case msg := <-ch1:
fmt.Printf(" received from ch1: %s\n", msg)
case msg := <-ch2:
fmt.Printf(" received from ch2: %s\n", msg)
}
}
Only ch2 has a value ready, so the second case executes. If both channels had values, either case could run — the runtime makes a pseudo-random choice.
A time.Ticker delivers ticks at regular intervals on its C channel. Combined with select, it creates a periodic loop that can also respond to other events:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
func part2_ticker() {
ticker := time.NewTicker(100 * time.Millisecond)
defer ticker.Stop()
done := time.After(350 * time.Millisecond)
count := 0
for {
select {
case t := <-ticker.C:
count++
fmt.Printf(" tick %d at %v\n", count, t.Format("15:04:05.000"))
case <-done:
fmt.Printf(" ticker stopped after %d ticks\n", count)
return
}
}
}
The ticker fires every 100ms, and the time.After channel fires once after 350ms to shut the loop down. Always call ticker.Stop() to release the ticker’s resources. Without the done channel, this loop would run forever.
time.After returns a channel that receives a single value after the specified duration. Inside a select, it acts as a deadline for any other channel operation:
1
2
3
4
5
6
7
8
9
10
11
12
13
func part3_timeout() {
slowCh := make(chan string)
go func() {
time.Sleep(500 * time.Millisecond)
slowCh <- "slow result"
}()
select {
case result := <-slowCh:
fmt.Printf(" got result: %s\n", result)
case <-time.After(100 * time.Millisecond):
fmt.Println(" timed out! (100ms elapsed before result arrived)")
}
}
The goroutine takes 500ms but the timeout is 100ms, so the timeout case wins. This pattern is useful for one-off operations, but for anything more complex, the context package (covered below) provides better cancellation semantics.
Adding a default case makes a select non-blocking. If no channel operation is immediately ready, the default case runs:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
func part4_nonBlocking() {
ch := make(chan string, 1)
select {
case msg := <-ch:
fmt.Printf(" received: %s\n", msg)
default:
fmt.Println(" no message ready (default executed)")
}
ch <- "buffered message"
select {
case msg := <-ch:
fmt.Printf(" received: %s\n", msg)
default:
fmt.Println(" no message ready")
}
}
The first select finds an empty channel, so default runs. After sending a value, the second select succeeds on the receive case. Non-blocking selects are useful for polling or try-send patterns where you want to make progress without waiting.
Combining select with multiple channels creates an event loop — a goroutine that reacts to different kinds of events as they arrive. This is the backbone of many server and pipeline designs:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
func part5_eventLoop() {
ctx, cancel := context.WithTimeout(context.Background(), 350*time.Millisecond)
defer cancel()
input := make(chan string, 10)
go func() {
changes := []string{"INSERT users alice", "UPDATE users bob", "DELETE sessions old"}
for _, c := range changes {
time.Sleep(80 * time.Millisecond)
input <- c
}
}()
ticker := time.NewTicker(150 * time.Millisecond)
defer ticker.Stop()
var buffer []string
for {
select {
case <-ctx.Done():
if len(buffer) > 0 {
fmt.Printf(" [shutdown] flushing %d remaining items\n", len(buffer))
}
return
case change := <-input:
buffer = append(buffer, change)
fmt.Printf(" [input] buffered: %s (buffer size: %d)\n", change, len(buffer))
case <-ticker.C:
if len(buffer) > 0 {
fmt.Printf(" [ticker] flushing %d items\n", len(buffer))
buffer = buffer[:0]
} else {
fmt.Println(" [ticker] nothing to flush")
}
}
}
}
This loop handles three kinds of events: incoming data (buffered for batch processing), periodic flushes (driven by the ticker), and shutdown (driven by the context timeout). Each iteration of the loop blocks in select until one of the three channels is ready. The context timeout ensures the loop eventually terminates, and the shutdown case flushes any remaining buffered items.
The context package provides a standard way to carry deadlines, cancellation signals, and request-scoped values across API boundaries and between goroutines. Almost every Go program that does I/O or spawns goroutines should use contexts. The core idea is simple: a parent creates a context, passes it to child goroutines, and can cancel it at any time — all children see the cancellation immediately.
context.WithCancel returns a new context and a cancel function. Calling cancel closes the context’s Done channel, which unblocks any goroutine waiting on it:
1
2
3
4
5
6
7
8
9
10
func part1_withCancel() {
ctx, cancel := context.WithCancel(context.Background())
go func() {
<-ctx.Done()
fmt.Printf(" goroutine: context cancelled (%v)\n", ctx.Err())
}()
time.Sleep(50 * time.Millisecond)
cancel()
time.Sleep(10 * time.Millisecond)
}
The goroutine blocks on <-ctx.Done() until the main goroutine calls cancel(). After cancellation, ctx.Err() returns context.Canceled. This is the simplest cancellation mechanism — the parent decides when to stop, and the children listen.
context.WithTimeout creates a context that cancels itself automatically after a duration. If the work finishes before the deadline, the deferred cancel call releases resources early:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
func part2_withTimeout() {
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Millisecond)
defer cancel()
select {
case <-ctx.Done():
fmt.Printf(" context done: %v\n", ctx.Err())
}
ctx2, cancel2 := context.WithTimeout(context.Background(), 200*time.Millisecond)
defer cancel2()
result := make(chan string, 1)
go func() {
time.Sleep(50 * time.Millisecond)
result <- "computation complete"
}()
select {
case r := <-result:
fmt.Printf(" got result before timeout: %s\n", r)
case <-ctx2.Done():
fmt.Printf(" timed out: %v\n", ctx2.Err())
}
}
The first context times out after 100ms because nothing else happens — ctx.Err() returns context.DeadlineExceeded. The second context has a 200ms deadline but the goroutine completes in 50ms, so the result arrives in time. Always defer cancel() even with timeouts — it frees internal timers immediately instead of waiting for the deadline.
Contexts form a tree. When a parent context is cancelled, all of its children and grandchildren are cancelled too. This is how you propagate shutdown signals through a hierarchy of goroutines:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
func part3_cascading() {
parent, parentCancel := context.WithCancel(context.Background())
child1, child1Cancel := context.WithCancel(parent)
defer child1Cancel()
child2, child2Cancel := context.WithCancel(parent)
defer child2Cancel()
grandchild, grandchildCancel := context.WithCancel(child1)
defer grandchildCancel()
parentCancel()
time.Sleep(10 * time.Millisecond)
fmt.Printf(" parent err: %v\n", parent.Err())
fmt.Printf(" child1 err: %v\n", child1.Err())
fmt.Printf(" child2 err: %v\n", child2.Err())
fmt.Printf(" grandchild err: %v\n", grandchild.Err())
}
After parentCancel(), all four contexts report context.Canceled. Cancelling child1Cancel() would only affect child1 and grandchild, leaving parent and child2 untouched. This tree structure mirrors the natural structure of request handling — an HTTP handler creates a context, passes it to a database call, which passes it to a retry loop, and cancelling the handler cancels everything downstream.
The Err() method on a context tells you why it was cancelled. There are exactly two possible non-nil values:
1
2
3
4
5
6
7
8
9
10
11
func part4_ctxErr() {
ctx1, cancel1 := context.WithCancel(context.Background())
cancel1()
fmt.Printf(" after cancel(): ctx.Err() = %v\n", ctx1.Err())
ctx2, cancel2 := context.WithTimeout(context.Background(), 1*time.Millisecond)
defer cancel2()
time.Sleep(10 * time.Millisecond)
fmt.Printf(" after timeout: ctx.Err() = %v\n", ctx2.Err())
fmt.Printf(" context.Canceled == ctx1.Err()? %t\n", ctx1.Err() == context.Canceled)
fmt.Printf(" DeadlineExceeded == ctx2.Err()? %t\n", ctx2.Err() == context.DeadlineExceeded)
}
context.Canceled means someone called the cancel function explicitly. context.DeadlineExceeded means the timeout or deadline elapsed. This distinction is important in practice — a cancellation usually means “the caller no longer cares about the result,” while a deadline exceeded usually means “the operation was too slow and should be retried or reported as an error.”
A powerful pattern combines a long-lived parent context with short-lived per-attempt contexts. Each attempt gets its own timeout, but the parent context enforces an overall deadline:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
func part5_perAttemptDeadline() {
ctx, cancel := context.WithTimeout(context.Background(), 500*time.Millisecond)
defer cancel()
for attempt := 1; ; attempt++ {
attemptCtx, attemptCancel := context.WithTimeout(ctx, 120*time.Millisecond)
result, err := simulateReceive(attemptCtx, attempt)
attemptCancel()
if err != nil {
if ctx.Err() != nil {
fmt.Printf(" attempt %d: parent context done, shutting down\n", attempt)
break
}
fmt.Printf(" attempt %d: timed out, retrying...\n", attempt)
continue
}
fmt.Printf(" attempt %d: received %q\n", attempt, result)
}
}
func simulateReceive(ctx context.Context, attempt int) (string, error) {
delay := time.Duration(attempt*80) * time.Millisecond
select {
case <-time.After(delay):
return fmt.Sprintf("WAL message #%d", attempt), nil
case <-ctx.Done():
return "", ctx.Err()
}
}
The parent context has a 500ms deadline. Each attempt gets 120ms. Early attempts complete quickly and succeed. As the simulated delay grows, individual attempts start timing out. Eventually the parent context expires, and the loop exits. The key check is ctx.Err() != nil — this distinguishes “this attempt timed out but we can retry” from “the overall operation is done.”
Channels are the preferred communication mechanism in Go, but sometimes you need shared mutable state. The sync package provides mutual exclusion locks, and the sync/atomic package provides lock-free operations on individual values. Both are lower-level than channels and should be used when the overhead of a channel is too high or when the access pattern does not fit the message-passing model.
A sync.Mutex provides mutual exclusion. Only one goroutine can hold the lock at a time. All others block on Lock() until the holder calls Unlock():
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
func part1_mutex() {
var mu sync.Mutex
counter := 0
var wg sync.WaitGroup
for i := 0; i < 100; i++ {
wg.Add(1)
go func() {
defer wg.Done()
mu.Lock()
counter++
mu.Unlock()
}()
}
wg.Wait()
fmt.Printf(" counter after 100 goroutines: %d (expected 100)\n", counter)
}
Without the mutex, concurrent counter++ operations would race — counter++ is not atomic; it reads, increments, and writes, and two goroutines could read the same value and both write the same incremented value, losing an update. The mutex serializes access so every increment is visible.
A sync.RWMutex distinguishes between readers and writers. Multiple goroutines can hold a read lock simultaneously, but a write lock is exclusive — no readers or other writers can proceed while it is held. This is ideal for data structures that are read far more often than they are written:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
type SchemaCache struct {
mu sync.RWMutex
schemas map[string][]string
ops atomic.Uint64
}
func NewSchemaCache() *SchemaCache {
return &SchemaCache{schemas: make(map[string][]string)}
}
func (c *SchemaCache) Get(table string) ([]string, bool) {
c.mu.RLock()
defer c.mu.RUnlock()
c.ops.Add(1)
cols, ok := c.schemas[table]
return cols, ok
}
func (c *SchemaCache) Set(table string, columns []string) {
c.mu.Lock()
defer c.mu.Unlock()
c.ops.Add(1)
c.schemas[table] = columns
}
Get uses RLock/RUnlock — multiple readers can call Get concurrently without blocking each other. Set uses Lock/Unlock — it waits for all readers to finish and then holds exclusive access while writing. This is significantly more efficient than a plain Mutex when reads vastly outnumber writes.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
func part2_rwMutex() {
cache := NewSchemaCache()
cache.Set("users", []string{"id", "name", "email"})
cache.Set("orders", []string{"id", "user_id", "total"})
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
if cols, ok := cache.Get("users"); ok {
fmt.Printf(" reader %d: got users schema (%d columns)\n", id, len(cols))
}
}(i)
}
wg.Add(1)
go func() {
defer wg.Done()
cache.Set("events", []string{"id", "type", "payload", "timestamp"})
}()
wg.Wait()
fmt.Printf(" total cache ops: %d\n", cache.TotalOps())
}
Ten reader goroutines and one writer goroutine run concurrently. The readers can all proceed in parallel, and the writer waits its turn. The atomic operation counter tracks total operations without needing the mutex.
The sync/atomic package provides lock-free operations on integer and pointer types. Atomics are faster than a mutex for simple counters or flags because they use hardware-level atomic instructions instead of OS-level locking:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
func part3_atomics() {
var lsn atomic.Uint64
lsn.Store(0)
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
for i := uint64(100); i <= 500; i += 100 {
lsn.Store(i)
time.Sleep(10 * time.Millisecond)
}
}()
wg.Add(1)
go func() {
defer wg.Done()
for i := 0; i < 5; i++ {
time.Sleep(15 * time.Millisecond)
fmt.Printf(" consumer: read LSN = %d\n", lsn.Load())
}
}()
wg.Wait()
fmt.Printf(" final LSN: %d\n", lsn.Load())
}
One goroutine writes increasing values with Store, and another reads them with Load. These operations are individually atomic — the reader will never see a partially written value. However, atomics only protect individual reads and writes; if you need to read-modify-write multiple fields together, use a mutex.
The Add method performs an atomic increment, which is the most common use case:
1
2
3
4
5
6
7
8
9
10
11
var counter atomic.Uint64
var wg2 sync.WaitGroup
for i := 0; i < 1000; i++ {
wg2.Add(1)
go func() {
defer wg2.Done()
counter.Add(1)
}()
}
wg2.Wait()
fmt.Printf(" atomic counter after 1000 increments: %d\n", counter.Load())
All 1000 goroutines increment the counter concurrently, and the final value is exactly 1000 — no lock needed.
Putting it all together, here is the SchemaCache under realistic concurrent load — one writer populating the cache while multiple readers query it simultaneously:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
func part4_cacheWithConcurrency() {
cache := NewSchemaCache()
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
tables := map[string][]string{
"users": {"id", "name", "email", "created_at"},
"orders": {"id", "user_id", "total", "status"},
"events": {"id", "type", "payload"},
"sessions": {"id", "user_id", "token", "expires_at"},
}
for table, cols := range tables {
cache.Set(table, cols)
time.Sleep(10 * time.Millisecond)
}
}()
for i := 0; i < 5; i++ {
wg.Add(1)
go func(id int) {
defer wg.Done()
tables := []string{"users", "orders", "events", "sessions", "missing"}
for _, t := range tables {
time.Sleep(8 * time.Millisecond)
if cols, ok := cache.Get(t); ok {
fmt.Printf(" reader %d: %s has %d columns\n", id, t, len(cols))
}
}
}(i)
}
wg.Wait()
fmt.Printf(" total ops: %d\n", cache.TotalOps())
}
The writer goroutine adds tables one at a time with a small delay between each. Five reader goroutines query for tables concurrently — some queries will find the table (if the writer has added it already), and some will miss (if the writer has not reached it yet or if the table does not exist). The RWMutex ensures that reads never see a partially written map entry, while the atomic.Uint64 counter tracks total operations without requiring the lock. This pattern — a read-heavy cache protected by an RWMutex with atomic counters for metrics — is a common building block in Go services.
error alongside its result, and the caller decides what to do with it. For cleanup, Go provides defer, which guarantees a function call runs when the enclosing function returns, regardless of how it returns. Together, explicit error returns and defer give you predictable error propagation and deterministic resource management without the complexity of exception hierarchies or finalizers. In this post we cover Go’s error handling patterns — from basic checks through wrapping, sentinel errors, and context-aware error classification — then move to defer patterns including LIFO ordering, loop pitfalls, closure capture semantics, named return manipulation, and panic recovery.
The simplest way to create an error in Go is errors.New, which returns a value that implements the error interface (any type with an Error() string method). For formatted messages, fmt.Errorf works like fmt.Sprintf but returns an error. The convention is to return a zero value alongside the error, and nil for the error on success.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
package main
import (
"errors"
"fmt"
)
func divide(a, b int) (int, error) {
if b == 0 {
return 0, errors.New("division by zero")
}
return a / b, nil
}
func main() {
err := errors.New("connection refused")
fmt.Printf("errors.New: %v\n", err)
err2 := fmt.Errorf("failed to connect to %s:%d", "localhost", 5432)
fmt.Printf("fmt.Errorf: %v\n", err2)
result, err := divide(10, 0)
if err != nil {
fmt.Printf("divide error: %v\n", err)
} else {
fmt.Printf("result: %d\n", result)
}
result, err = divide(10, 3)
if err != nil {
fmt.Printf("divide error: %v\n", err)
} else {
fmt.Printf("10 / 3 = %d\n", result)
}
}
The if err != nil pattern is the most common construct in Go code. There is no shorthand — every error is checked explicitly at the call site. This verbosity is intentional: it makes the error path visible and forces the developer to think about what should happen when something fails.
When an error passes through multiple layers of a program, you want to add context (which function failed, what it was trying to do) without losing the original error. Go 1.13 introduced the %w verb in fmt.Errorf for this. It wraps the original error so that downstream code can still match it, while prepending a descriptive message.
1
2
3
4
5
6
7
var ErrNotFound = errors.New("not found")
original := ErrNotFound
wrapped := fmt.Errorf("load user: %w", original)
fmt.Printf("wrapped: %v\n", wrapped)
fmt.Printf("errors.Is(wrapped, ErrNotFound)? %t\n",
errors.Is(wrapped, ErrNotFound)) // true — chain preserved
The critical distinction is between %w and %v. Using %v formats the error’s message into a new string but breaks the chain — the resulting error is just a string, not a wrapper around the original.
1
2
3
broken := fmt.Errorf("load user: %v", original)
fmt.Printf("errors.Is(broken, ErrNotFound)? %t\n",
errors.Is(broken, ErrNotFound)) // false — chain lost
Use %w when you want callers to be able to inspect the underlying error. Use %v only when you intentionally want to hide the original error from programmatic inspection (for example, when crossing an API boundary where you do not want to expose internal error types).
A sentinel error is a package-level variable that represents a specific, well-known error condition. Callers compare against it using errors.Is, which walks the entire wrap chain looking for a match.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
var (
ErrNotFound = errors.New("not found")
ErrInvalidFormat = errors.New("invalid format")
)
func loadConfig(path string) error {
switch path {
case "missing.yaml":
return fmt.Errorf("loadConfig(%s): %w", path, ErrNotFound)
case "bad.yaml":
return fmt.Errorf("loadConfig(%s): %w", path, ErrInvalidFormat)
default:
return nil
}
}
The caller does not need to know how many layers of wrapping sit between it and the sentinel — errors.Is unwraps automatically:
1
2
3
4
5
6
7
8
9
err := loadConfig("missing.yaml")
if errors.Is(err, ErrNotFound) {
fmt.Println("config not found — using defaults")
}
err = loadConfig("bad.yaml")
if errors.Is(err, ErrInvalidFormat) {
fmt.Println("config format invalid — check syntax")
}
Sentinel errors are best suited for conditions that callers genuinely need to branch on. Do not create sentinels for every possible failure — most errors should just be returned with context and eventually logged or displayed.
While errors.Is checks whether an error in the chain matches a specific value, errors.As checks whether any error in the chain matches a specific type. This is useful when the error carries structured data beyond a message string.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
type ValidationError struct {
Field string
Message string
}
func (e *ValidationError) Error() string {
return fmt.Sprintf("validation failed on %s: %s", e.Field, e.Message)
}
func validateChange(table, action string) error {
if table == "" {
return &ValidationError{Field: "table", Message: "cannot be empty"}
}
return nil
}
To extract the typed error from a (possibly wrapped) chain, declare a variable of the target type and pass its address to errors.As:
1
2
3
4
5
6
7
8
9
10
11
err := validateChange("", "INSERT")
var ve *ValidationError
if errors.As(err, &ve) {
fmt.Printf("validation error: field=%s, message=%s\n", ve.Field, ve.Message)
}
// Works through wrapping too
wrapped := fmt.Errorf("process change: %w", err)
if errors.As(wrapped, &ve) {
fmt.Printf("errors.As through wrap: field=%s\n", ve.Field)
}
Use errors.Is when you are checking against a known value (sentinel). Use errors.As when you need to extract a specific error type to read its fields.
When a function takes a context.Context, the operation might fail because the context was cancelled or because a deadline expired. These are not real application errors — they are intentional signals from the caller that it no longer wants the result. The pattern is to check ctx.Err() after receiving an error to determine whether the failure is genuine or just a cancellation.
1
2
3
4
5
6
7
8
9
func receive(ctx context.Context, simulateError bool) error {
if ctx.Err() != nil {
return fmt.Errorf("receive: %w", ctx.Err())
}
if simulateError {
return fmt.Errorf("receive: %w", errors.New("connection reset"))
}
return nil
}
At the call site, the distinction matters for how you respond:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
// Real error — context is fine
err := receive(ctx, true)
if err != nil && ctx.Err() == nil {
fmt.Printf("real error: %v\n", err)
}
// Context cancelled — not a real error
cancel() // cancel the context
err = receive(ctx, true)
if err != nil && ctx.Err() == nil {
fmt.Printf("real error: %v\n", err)
} else if err != nil {
fmt.Printf("context cancelled — ignoring error: %v\n", err)
}
// Timeout — deadline exceeded
timeoutCtx, timeoutCancel := context.WithTimeout(context.Background(), 1*time.Millisecond)
defer timeoutCancel()
time.Sleep(5 * time.Millisecond)
err = receive(timeoutCtx, false)
if err != nil && timeoutCtx.Err() == nil {
fmt.Printf("real error: %v\n", err)
} else if err != nil {
fmt.Printf("timed out — not a real failure: %v\n", err)
}
This pattern is essential in server code, background workers, and anything using gRPC or HTTP handlers where contexts carry deadlines. Without the ctx.Err() check, you would log cancellations as failures, cluttering your error monitoring.
In real applications, errors flow up through multiple layers — a database call fails, the repository wraps it, the service wraps it again, and the handler wraps it once more. Each layer adds context with %w, building a chain you can inspect with errors.Is, errors.As, or errors.Unwrap.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
func bottomLayer() error {
return fmt.Errorf("bottom (table lookup): %w", ErrNotFound)
}
func middleLayer() error {
err := bottomLayer()
if err != nil {
return fmt.Errorf("middle (processing batch): %w", err)
}
return nil
}
func topLevel() error {
err := middleLayer()
if err != nil {
return fmt.Errorf("top: %w", err)
}
return nil
}
The caller at the top gets a richly annotated error that still matches the original sentinel:
1
2
3
4
5
6
7
8
err := topLevel()
if err != nil {
fmt.Printf("full chain: %v\n", err)
// Output: top: middle (processing batch): bottom (table lookup): not found
fmt.Printf("is ErrNotFound? %t\n", errors.Is(err, ErrNotFound))
// Output: true
}
You can also walk the chain manually with errors.Unwrap to see each layer:
1
2
3
4
5
6
7
8
9
current := err
for current != nil {
fmt.Printf(" -> %v\n", current)
current = errors.Unwrap(current)
}
// -> top: middle (processing batch): bottom (table lookup): not found
// -> middle (processing batch): bottom (table lookup): not found
// -> bottom (table lookup): not found
// -> not found
The rule of thumb is: wrap at every layer boundary, use sentinel errors for conditions that callers branch on, and use errors.Is/errors.As to inspect the chain without breaking encapsulation.
Deferred function calls execute in last-in, first-out order when the enclosing function returns. This means the last defer registered runs first. The LIFO order is deliberate — it mirrors the typical pattern of acquiring resources in sequence and releasing them in reverse order (open A, open B, close B, close A).
1
2
3
4
5
6
7
func part1_lifoOrder() {
fmt.Println("registering defers 1, 2, 3...")
defer fmt.Println("defer 1 (registered first — runs last)")
defer fmt.Println("defer 2")
defer fmt.Println("defer 3 (registered last — runs first)")
fmt.Println("function body done — defers fire in reverse:")
}
Output:
1
2
3
4
5
registering defers 1, 2, 3...
function body done — defers fire in reverse:
defer 3 (registered last — runs first)
defer 2
defer 1 (registered first — runs last)
A common mistake is deferring a close call inside a loop. Because defer is scoped to the enclosing function (not the loop iteration), all the deferred calls pile up and only execute when the function returns. If you are opening files or database connections in a loop, they all stay open simultaneously until the function exits.
1
2
3
4
5
6
7
8
// BAD — all defers pile up until function returns
filenames := []string{"a.txt", "b.txt", "c.txt"}
for _, name := range filenames {
f, err := os.Open(name)
if err != nil { continue }
defer f.Close() // won't run until the surrounding function returns
// process f...
}
The fix is to wrap the loop body in its own function (either a named function or an anonymous closure), so defer fires at the end of each iteration:
1
2
3
4
5
6
7
8
9
// FIX — wrap in a function so defer runs each iteration
for _, name := range filenames {
func(n string) {
f, err := os.Open(n)
if err != nil { return }
defer f.Close()
// process f...
}(name)
}
Alternatively, extract the body into a named function like processFile(name). Either approach ensures resources are released promptly.
Deferred closures capture variables by reference, not by value. This means a deferred closure sees the final value of any variable from the enclosing scope, not the value at the time defer was called.
1
2
3
4
5
6
7
8
9
10
11
func part3_deferWithClosures() {
x := 0
defer func() {
fmt.Printf("deferred closure sees x = %d (final value, not 0)\n", x)
}()
x = 42
fmt.Printf("x set to %d\n", x)
}
// Output:
// x set to 42
// deferred closure sees x = 42 (final value, not 0)
If you need to capture the value at the time of the defer, pass it as a parameter. Function parameters are evaluated immediately when defer is executed, so the value is frozen:
1
2
3
4
5
6
7
8
9
y := 0
defer func(val int) {
fmt.Printf("deferred closure(val) sees val = %d (frozen at defer time)\n", val)
}(y) // y is evaluated now, val = 0
y = 99
fmt.Printf("y set to %d, but deferred param already captured 0\n", y)
// Output:
// y set to 99, but deferred param already captured 0
// deferred closure(val) sees val = 0 (frozen at defer time)
The reference-capture behaviour is what makes named-return error wrapping work (covered next), but it can also be a source of bugs when you unintentionally capture a loop variable.
One of the more powerful defer patterns uses named return values to modify the return result in a deferred function. Because the deferred closure captures the named return by reference, it can inspect and change the value that the caller ultimately receives. This is especially useful for wrapping errors with context or for flushing buffers on the way out.
1
2
3
4
5
6
7
8
9
10
11
12
func readConfig(path string) (content string, err error) {
defer func() {
if err != nil {
err = fmt.Errorf("readConfig(%s): %w", path, err)
}
}()
if path == "missing.conf" {
return "", os.ErrNotExist
}
return "key=value", nil
}
When readConfig("missing.conf") returns os.ErrNotExist, the deferred function wraps it into readConfig(missing.conf): file does not exist before the caller sees it. This avoids duplicating the wrapping logic at every return statement in the function.
The same technique works for capturing flush errors that would otherwise be silently ignored:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
func writeWithFlush() (n int, err error) {
n = 128
defer func() {
flushErr := simulateFlush()
if err == nil {
err = flushErr
}
}()
return n, nil
}
func simulateFlush() error {
return errors.New("flush: disk full")
}
Here the write itself succeeds, but the deferred flush fails. Without the named return pattern, the flush error would be lost because return n, nil has already been executed. The deferred function overwrites err before the caller receives it.
Go’s panic is not for normal error handling — it is reserved for truly unrecoverable situations (programming errors, violated invariants, corrupted state). When a panic occurs, the runtime begins unwinding the call stack, executing deferred functions in each frame. If no deferred function calls recover(), the program crashes with a stack trace.
recover() stops the panic and returns the value that was passed to panic. It only works inside a deferred function — calling recover() in normal code always returns nil.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
func safeDivide(a, b int) (result int, err error) {
defer func() {
if r := recover(); r != nil {
err = fmt.Errorf("caught panic: %v", r)
}
}()
return a / b, nil // panics if b == 0 (integer division by zero)
}
result, err := safeDivide(10, 0)
fmt.Printf("result=%d, err=%v\n", result, err)
// result=0, err=caught panic: runtime error: integer division by zero
result, err = safeDivide(10, 3)
fmt.Printf("result=%d, err=%v\n", result, err)
// result=3, err=<nil>
During a panic unwind, all deferred functions in the call stack still run in LIFO order. This is important — it means cleanup code (closing files, releasing locks) still executes even during a panic:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
func panicChain() {
defer func() {
if r := recover(); r != nil {
fmt.Printf("defer 1 (outermost): recovered panic: %v\n", r)
}
}()
defer fmt.Println("defer 2: I still run during panic unwind")
defer fmt.Println("defer 3: I run first during panic unwind")
panic("something went wrong")
}
// Output:
// defer 3: I run first during panic unwind
// defer 2: I still run during panic unwind
// defer 1 (outermost): recovered panic: something went wrong
The general guidance is: return errors, do not panic. Use panic only for programming bugs (like indexing out of bounds or an impossible state). Use recover at API boundaries (HTTP handlers, goroutine entry points) to convert panics into errors so that one bad request does not bring down the entire server.
implements declarations. Instead, Go provides three orthogonal mechanisms: interfaces for abstraction, embedding for composition, and generics for type-safe parameterization. Each one is simple on its own, and together they cover the same ground that class hierarchies cover in other languages — with less coupling. In this post, we’ll work through all three, starting with how Go interfaces are satisfied implicitly, then moving to struct and interface embedding, and finishing with generic functions and types.
In Go, a type satisfies an interface simply by implementing all of its methods. There is no implements keyword and no explicit declaration binding a type to an interface. If the methods match, the type qualifies.
Consider a Message interface with two methods:
1
2
3
4
type Message interface {
Type() string
String() string
}
Any struct that has both a Type() method and a String() method satisfies Message automatically. Here are three structs that each represent a different kind of database operation:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
type Insert struct {
Table string
Data map[string]string
}
func (i Insert) Type() string { return "INSERT" }
func (i Insert) String() string { return fmt.Sprintf("INSERT into %s: %v", i.Table, i.Data) }
type Update struct {
Table string
Key string
Data map[string]string
}
func (u Update) Type() string { return "UPDATE" }
func (u Update) String() string { return fmt.Sprintf("UPDATE %s where key=%s: %v", u.Table, u.Key, u.Data) }
type Delete struct {
Table string
Key string
}
func (d Delete) Type() string { return "DELETE" }
func (d Delete) String() string { return fmt.Sprintf("DELETE from %s where key=%s", d.Table, d.Key) }
None of these types mention Message anywhere. Yet all three satisfy it, and you can store them together in a slice:
1
2
3
4
5
6
7
8
9
10
var msg Message = Insert{Table: "users", Data: map[string]string{"name": "alice"}}
fmt.Printf("msg.Type() = %s\n", msg.Type())
fmt.Printf("msg.String() = %s\n", msg)
messages := []Message{
Insert{Table: "users", Data: map[string]string{"name": "alice"}},
Update{Table: "users", Key: "alice", Data: map[string]string{"email": "a@b.com"}},
Delete{Table: "sessions", Key: "expired-123"},
}
fmt.Printf("%d messages in slice, all satisfy Message interface\n", len(messages))
This implicit satisfaction is what makes Go interfaces so powerful for decoupling. A package can define an interface, and types from completely unrelated packages can satisfy it without importing or even knowing about the interface definition.
When you have an interface value and need to branch on the concrete type behind it, Go provides the type switch:
1
2
3
4
5
6
7
8
9
10
11
12
func dispatch(msg Message) {
switch m := msg.(type) {
case *Insert:
fmt.Printf("[DISPATCH] Insert into %s: %v\n", m.Table, m.Data)
case *Update:
fmt.Printf("[DISPATCH] Update %s (key=%s): %v\n", m.Table, m.Key, m.Data)
case *Delete:
fmt.Printf("[DISPATCH] Delete from %s (key=%s)\n", m.Table, m.Key)
default:
fmt.Printf("[DISPATCH] Unknown message type: %s\n", m.Type())
}
}
The syntax msg.(type) extracts the concrete type. In each case branch, the variable m is already narrowed to the matched type — you can access m.Table, m.Key, or m.Data directly without any additional casting. The default branch handles any Message implementation that was not explicitly listed.
1
2
3
4
5
6
7
8
messages := []Message{
&Insert{Table: "users", Data: map[string]string{"name": "bob"}},
&Delete{Table: "sessions", Key: "sess-456"},
&Update{Table: "users", Key: "bob", Data: map[string]string{"role": "admin"}},
}
for _, msg := range messages {
dispatch(msg)
}
A type switch is useful when you need to handle multiple types. When you only care about one specific type, use a type assertion with the comma-ok pattern:
1
2
3
4
5
6
7
8
func tryExtractInsert(msg Message) {
ins, ok := msg.(Insert)
if ok {
fmt.Printf("type assertion succeeded: Insert into %s\n", ins.Table)
} else {
fmt.Printf("type assertion failed: message is %s, not Insert\n", msg.Type())
}
}
The expression msg.(Insert) attempts to extract the concrete Insert value from the interface. If the underlying type matches, ok is true and ins holds the value. If it does not match, ok is false and ins is the zero value of Insert. This two-value form is safe — it never panics. If you omit the second return value (ins := msg.(Insert)), a failed assertion will panic at runtime.
1
2
3
4
5
var msg Message = Insert{Table: "orders", Data: map[string]string{"id": "100"}}
tryExtractInsert(msg) // succeeds
var msg2 Message = Delete{Table: "orders", Key: "100"}
tryExtractInsert(msg2) // fails gracefully
Go’s built-in error type is itself an interface with a single method:
1
2
3
type error interface {
Error() string
}
Any type that has an Error() string method satisfies the error interface. This means you can create custom error types that carry structured data beyond a simple message string:
1
2
3
4
5
6
7
8
type ParseError struct {
Code int
Message string
}
func (e *ParseError) Error() string {
return fmt.Sprintf("parse error %d: %s", e.Code, e.Message)
}
Because *ParseError has an Error() string method, it satisfies the error interface. You can return it wherever an error is expected, and callers can use a type assertion to extract the richer information:
1
2
3
4
5
6
var err error = &ParseError{Code: 422, Message: "invalid WAL format"}
fmt.Printf("err.Error() = %s\n", err.Error())
if pe, ok := err.(*ParseError); ok {
fmt.Printf("extracted ParseError: code=%d, message=%s\n", pe.Code, pe.Message)
}
Note the assertion is against *ParseError (a pointer), because the Error() method has a pointer receiver. The receiver type matters — ParseError (value) would not match here.
any TypeGo 1.18 introduced any as an alias for interface{}, the empty interface. Since every type satisfies an interface with zero methods, any can hold a value of any type. A type switch lets you recover the concrete type:
1
2
3
4
5
6
7
8
9
10
11
12
func processAny(val any) {
switch v := val.(type) {
case string:
fmt.Printf("any is string: %q\n", v)
case int:
fmt.Printf("any is int: %d\n", v)
case Message:
fmt.Printf("any is Message: %s\n", v)
default:
fmt.Printf("any is unknown: %T\n", v)
}
}
1
2
3
4
processAny("hello") // string
processAny(42) // int
processAny(Insert{Table: "t", Data: nil}) // Message
processAny(3.14) // unknown: float64
The any type is useful for truly generic containers and APIs. However, it erases type information at compile time. For most cases where you want type-safe parameterization, generics (covered below) are the better tool.
Go does not have inheritance. Instead, it has embedding — you place one struct type inside another without giving it a field name, and its fields and methods are “promoted” to the outer struct.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
type Base struct {
ID int
CreatedAt string
}
func (b Base) Summary() string {
return fmt.Sprintf("id=%d, created=%s", b.ID, b.CreatedAt)
}
type User struct {
Base
Name string
Email string
}
The User struct embeds Base. There is no field name — just the type. This is different from a named field like base Base; embedding promotes all of Base’s fields and methods onto User.
Because Base is embedded, you can access its fields directly on a User value:
1
2
3
4
5
6
7
8
u := User{
Base: Base{ID: 1, CreatedAt: "2025-01-15"},
Name: "Alice",
Email: "alice@example.com",
}
fmt.Printf("user.ID: %d\n", u.ID) // promoted from Base
fmt.Printf("user.Base.ID: %d\n", u.Base.ID) // explicit path also works
Both u.ID and u.Base.ID refer to the same field. The promoted path is syntactic convenience — the compiler rewrites u.ID to u.Base.ID behind the scenes.
Methods are promoted the same way. Since Base has a Summary() method, User also has Summary() without writing any additional code:
1
fmt.Printf("u.Summary() = %s\n", u.Summary()) // calls Base.Summary()
This also means User satisfies any interface that Base satisfies. If there is a Summarizer interface requiring a Summary() string method, User satisfies it through promotion:
1
2
3
4
5
6
7
8
9
type Summarizer interface {
Summary() string
}
func printSummary(s Summarizer) {
fmt.Printf("summary: %s\n", s.Summary())
}
printSummary(u) // works — User has Summary() via Base
An outer type can define its own version of a promoted method, effectively overriding it:
1
2
3
4
5
6
7
8
9
type Order struct {
Base
UserID int
Total float64
}
func (o Order) Summary() string {
return fmt.Sprintf("order %d: $%.2f (user %d)", o.ID, o.Total, o.UserID)
}
Now Order has its own Summary() that takes priority over the promoted one from Base. The original is still accessible through the explicit path:
1
2
3
4
o := Order{Base: Base{ID: 42, CreatedAt: "2025-03-01"}, UserID: 1, Total: 99.95}
fmt.Printf("o.Summary() = %s\n", o.Summary()) // Order.Summary()
fmt.Printf("o.Base.Summary() = %s\n", o.Base.Summary()) // Base.Summary()
Embedding also works with interfaces. You can compose larger interfaces from smaller ones:
1
2
3
4
5
6
7
8
9
10
11
12
type Reader interface {
Read() string
}
type Writer interface {
Write(data string)
}
type ReadWriter interface {
Reader
Writer
}
ReadWriter embeds both Reader and Writer, so any type that satisfies ReadWriter must implement both Read() and Write(). A concrete type can satisfy all three interfaces at once:
1
2
3
4
5
6
7
type File struct {
Name string
content string
}
func (f File) Read() string { return f.content }
func (f *File) Write(data string) { f.content += data }
1
2
3
4
5
6
7
8
9
10
f := &File{Name: "data.txt", content: "hello"}
var r Reader = f
fmt.Printf("Reader: %s\n", r.Read())
var w Writer = f
w.Write(" world")
var rw ReadWriter = f
fmt.Printf("ReadWriter: %s\n", rw.Read())
This is the same pattern the standard library uses. io.ReadWriter embeds io.Reader and io.Writer, and types like os.File and bytes.Buffer satisfy all three.
Embedding looks superficially like inheritance, but it has a fundamental difference: the embedded type has no knowledge of the outer type. In classical inheritance, a base class method can call an overridden method on the subclass (dynamic dispatch). With Go embedding, Base.Summary() will always run Base’s logic — it cannot “see” that it is embedded inside User or Order.
1
2
3
u := User{Base: Base{ID: 1, CreatedAt: "2025-01-15"}, Name: "Alice"}
var s Summarizer = u
fmt.Printf("Summarizer: %s\n", s.Summary())
The key differences:
Base has no idea User exists.Base methods do not dispatch to User methods.User “has a” Base, not “is a” Base.Go 1.18 introduced generics (type parameters), letting you write functions and types that work across multiple types without sacrificing compile-time type safety.
The most common use of generics is writing utility functions that operate on slices of any type. Here is a generic Filter function:
1
2
3
4
5
6
7
8
9
func Filter[T any](slice []T, predicate func(T) bool) []T {
var result []T
for _, v := range slice {
if predicate(v) {
result = append(result, v)
}
}
return result
}
The [T any] after the function name declares a type parameter T with the constraint any — meaning T can be any type. The compiler generates specialized code for each concrete type used at call sites, so there is no runtime overhead.
A Map function follows the same pattern but transforms each element:
1
2
3
4
5
6
7
func Map[T any, U any](slice []T, transform func(T) U) []U {
result := make([]U, len(slice))
for i, v := range slice {
result[i] = transform(v)
}
return result
}
Using them:
1
2
3
4
5
6
7
8
9
10
nums := []int{1, 2, 3, 4, 5, 6, 7, 8, 9, 10}
evens := Filter(nums, func(n int) bool { return n%2 == 0 })
// evens = [2, 4, 6, 8, 10]
words := []string{"hello", "world", "go"}
upper := Map(words, strings.ToUpper)
// upper = ["HELLO", "WORLD", "GO"]
lengths := Map(words, func(s string) int { return len(s) })
// lengths = [5, 5, 2]
Notice that you do not need to specify the type parameters explicitly at the call site — Go infers them from the arguments.
The any constraint is maximally permissive, but it only lets you do things that work on every type (pass values around, store them in slices). If you need to perform operations like addition, you need a narrower constraint.
A type constraint is an interface that lists the allowed types using the union syntax:
1
2
3
4
5
6
7
8
9
10
11
type Number interface {
int | float64
}
func Sum[T Number](values []T) T {
var total T
for _, v := range values {
total += v
}
return total
}
The Number constraint restricts T to int or float64. The += operator is valid because both types support it. The compiler rejects any call that passes a type not in the union:
1
2
3
fmt.Println(Sum([]int{1, 2, 3, 4, 5})) // 15
fmt.Println(Sum([]float64{1.1, 2.2, 3.3})) // 6.6
// Sum([]string{"a", "b"}) // compile error
The standard library provides common constraints in the golang.org/x/exp/constraints package (e.g., constraints.Ordered for any type that supports <, >, <=, >=).
Type parameters work on structs too. Here is a generic stack:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
type Stack[T any] struct {
items []T
}
func (s *Stack[T]) Push(v T) {
s.items = append(s.items, v)
}
func (s *Stack[T]) Pop() (T, bool) {
var zero T
if len(s.items) == 0 {
return zero, false
}
v := s.items[len(s.items)-1]
s.items = s.items[:len(s.items)-1]
return v, true
}
func (s *Stack[T]) Peek() (T, bool) {
var zero T
if len(s.items) == 0 {
return zero, false
}
return s.items[len(s.items)-1], true
}
func (s *Stack[T]) IsEmpty() bool {
return len(s.items) == 0
}
You specify the type parameter when creating an instance:
1
2
3
4
5
6
7
8
9
10
11
12
13
intStack := &Stack[int]{}
intStack.Push(10)
intStack.Push(20)
intStack.Push(30)
v, ok := intStack.Pop()
// v = 30, ok = true
strStack := &Stack[string]{}
strStack.Push("a")
strStack.Push("b")
top, _ := strStack.Peek()
// top = "b"
The var zero T pattern is worth noting — it is the idiomatic way to get the zero value of a generic type. For int it is 0, for string it is "", for pointers it is nil.
A generic type can have more than one type parameter. Here is a Pair that holds two values of different types:
1
2
3
4
5
6
7
8
type Pair[K any, V any] struct {
Key K
Value V
}
func (p Pair[K, V]) String() string {
return fmt.Sprintf("(%v, %v)", p.Key, p.Value)
}
A Zip function combines two slices into a slice of pairs:
1
2
3
4
5
6
7
8
9
10
11
func Zip[K any, V any](keys []K, values []V) []Pair[K, V] {
n := len(keys)
if len(values) < n {
n = len(values)
}
pairs := make([]Pair[K, V], n)
for i := 0; i < n; i++ {
pairs[i] = Pair[K, V]{Key: keys[i], Value: values[i]}
}
return pairs
}
And GroupBy uses a comparable constraint on the key type, since map keys must be comparable:
1
2
3
4
5
6
7
8
func GroupBy[T any, K comparable](items []T, keyFn func(T) K) map[K][]T {
groups := make(map[K][]T)
for _, item := range items {
k := keyFn(item)
groups[k] = append(groups[k], item)
}
return groups
}
1
2
3
4
5
6
pairs := Zip([]string{"a", "b", "c"}, []int{1, 2, 3})
// [(a, 1), (b, 2), (c, 3)]
words := []string{"apple", "avocado", "banana", "blueberry", "cherry"}
grouped := GroupBy(words, func(s string) string { return string(s[0]) })
// {"a": ["apple", "avocado"], "b": ["banana", "blueberry"], "c": ["cherry"]}
The comparable constraint is a built-in that permits any type supporting == and !=. It is required for map keys and is narrower than any but broader than specific type unions.
You can use any interface as a type constraint, not just unions of primitive types. This lets you write generic functions that call methods on their type parameters:
1
2
3
4
5
6
7
8
9
10
11
type Stringer interface {
String() string
}
func JoinStrings[T Stringer](items []T, sep string) string {
parts := make([]string, len(items))
for i, item := range items {
parts[i] = item.String()
}
return strings.Join(parts, sep)
}
Any type that has a String() string method can be used with JoinStrings:
1
2
3
4
5
6
7
8
9
10
11
type Color struct {
Name string
Hex string
}
func (c Color) String() string { return fmt.Sprintf("%s(%s)", c.Name, c.Hex) }
type City struct {
Name string
Country string
}
func (c City) String() string { return fmt.Sprintf("%s, %s", c.Name, c.Country) }
1
2
3
4
5
6
7
8
9
10
11
12
13
colors := []Color{
{Name: "Red", Hex: "#FF0000"},
{Name: "Green", Hex: "#00FF00"},
}
fmt.Println(JoinStrings(colors, ", "))
// Red(#FF0000), Green(#00FF00)
cities := []City{
{Name: "Tokyo", Country: "Japan"},
{Name: "Paris", Country: "France"},
}
fmt.Println(JoinStrings(cities, " | "))
// Tokyo, Japan | Paris, France
The difference between an interface constraint and any is enforcement at compile time. With any, you cannot call .String() on the type parameter because the compiler does not know it exists. With Stringer, the compiler guarantees every type passed to JoinStrings has that method, so the call is safe. This is the same implicit satisfaction mechanism from the interfaces section — the constraint is an interface, and the concrete type must satisfy it.
Go is a pass-by-value language. When you pass a variable to a function, Go copies the value. The function works on its own independent copy, and the caller never sees changes. This is safe and predictable, but sometimes you need the function to modify the original. That is what pointers are for.
A function that takes a plain int gets a copy. Incrementing the copy has no effect on the caller’s variable:
1
2
3
4
5
6
7
8
9
10
11
func incrementBroken(counter int) {
counter++
fmt.Printf(" inside incrementBroken: counter = %d\n", counter)
}
func main() {
counter := 0
fmt.Printf(" before: counter = %d\n", counter)
incrementBroken(counter)
fmt.Printf(" after: counter = %d <- still 0! the function got a copy\n", counter)
}
Inside incrementBroken, counter is 1. Back in main, it is still 0. The function modified its own copy and the original was untouched.
A pointer holds the memory address of a value. The & operator takes the address of a variable, and the * operator dereferences a pointer to access the value it points to. When you pass a pointer to a function, both the caller and the function are looking at the same memory:
1
2
3
4
5
6
7
8
9
10
11
12
func incrementFixed(counter *int) {
*counter++
fmt.Printf(" inside incrementFixed: *counter = %d\n", *counter)
}
func main() {
counter := 0
fmt.Printf(" before: counter = %d\n", counter)
fmt.Printf(" address of counter: %p\n", &counter)
incrementFixed(&counter)
fmt.Printf(" after: counter = %d <- it changed! both see the same memory\n", counter)
}
&counter produces a *int — a pointer to an int. Inside incrementFixed, *counter++ follows the pointer to the original variable and increments it. After the call, counter in main is 1.
Pointers let multiple functions share and coordinate through the same variable. Here, a producer writes values and a consumer reads the result — both operate on the same uint64 through a pointer:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
func producer(lsn *uint64) {
for i := 0; i < 5; i++ {
*lsn = uint64((i + 1) * 100)
fmt.Printf(" producer set LSN to %d\n", *lsn)
}
}
func consumer(lsn *uint64) {
fmt.Printf(" consumer reads LSN = %d <- sees the producer's last write\n", *lsn)
}
func main() {
var confirmedLSN uint64 = 0
producer(&confirmedLSN)
consumer(&confirmedLSN)
fmt.Printf(" final LSN: %d <- everyone sees the same value\n", confirmedLSN)
}
Both producer and consumer receive a *uint64 pointing to the same confirmedLSN variable. The producer’s writes are immediately visible to anyone else who holds a pointer to that address.
Pointers become essential with structs. If you pass a struct by value to a function, the function gets a full copy — modifications are lost. A method with a pointer receiver, on the other hand, mutates the original:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
type Writer struct {
Path string
BytesUsed int
}
func NewWriter(path string) *Writer {
return &Writer{Path: path, BytesUsed: 0}
}
func (w *Writer) Write(data string) {
w.BytesUsed += len(data)
fmt.Printf(" wrote %d bytes to %s (total: %d)\n", len(data), w.Path, w.BytesUsed)
}
func writeBroken(w Writer, data string) {
w.BytesUsed += len(data)
fmt.Printf(" (broken) wrote %d bytes (total in copy: %d)\n", len(data), w.BytesUsed)
}
func main() {
w := NewWriter("/data/output.parquet")
w.Write("hello")
w.Write("world")
fmt.Printf(" after writes: BytesUsed = %d <- pointer receiver mutated it\n\n", w.BytesUsed)
w2 := Writer{Path: "/data/broken.parquet", BytesUsed: 0}
writeBroken(w2, "hello")
writeBroken(w2, "world")
fmt.Printf(" after broken writes: BytesUsed = %d <- still 0, copies were modified\n", w2.BytesUsed)
}
NewWriter returns a *Writer — a pointer to a heap-allocated struct. The Write method has a pointer receiver (w *Writer), so it modifies the same struct the caller holds. By contrast, writeBroken takes a Writer by value. Each call gets a fresh copy, increments the copy’s BytesUsed, and throws it away. The caller’s w2 never changes.
A pointer that has not been assigned points to nothing — its zero value is nil. Calling a method on a nil pointer will panic at runtime. Always check before dereferencing:
1
2
3
4
5
6
7
8
9
10
11
12
func main() {
var w *Writer
fmt.Printf(" w == nil? %t\n", w == nil)
if w != nil {
w.Write("data")
} else {
fmt.Println(" skipped write: w is nil (would panic if we called w.Write)")
}
w = NewWriter("/data/safe.parquet")
fmt.Printf(" w == nil? %t <- initialized now\n", w == nil)
w.Write("safe data")
}
The var w *Writer declaration gives w a zero value of nil. The nil check prevents the panic. After calling NewWriter, w points to a valid struct and methods work normally.
Go does not have classes. Instead, you define structs to hold data and attach methods to types using receivers. The receiver type — value or pointer — determines whether the method can mutate the struct.
A struct groups fields into a single type. You can initialize it with named fields, rely on zero values, or provide only the fields you care about:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
type Duration time.Duration
type Config struct {
Host string
Port int
Interval Duration
}
func main() {
c1 := Config{Host: "localhost", Port: 5432, Interval: Duration(5 * time.Second)}
fmt.Printf(" named fields: %+v\n", c1)
var c2 Config
fmt.Printf(" zero value: %+v\n", c2)
c3 := Config{Host: "prod-db"}
fmt.Printf(" partial init: %+v\n", c3)
}
c1 sets every field explicitly. c2 uses Go’s zero values — empty string for Host, 0 for Port, 0 for Interval. c3 sets only Host; the rest default to zero values. The %+v format verb prints field names alongside values, which is useful for debugging.
Go lets you define a new named type based on an existing type. The new type is distinct — you cannot mix them without an explicit conversion — but it can carry its own methods:
1
2
3
4
5
6
7
8
9
10
11
12
13
type Duration time.Duration
func (d Duration) Std() time.Duration {
return time.Duration(d)
}
func main() {
d := Duration(10 * time.Second)
fmt.Printf(" Duration value: %v\n", d)
fmt.Printf(" As time.Duration: %v\n", d.Std())
td := time.Duration(d)
fmt.Printf(" Explicit conversion: %v\n", td)
}
Duration is a new type with the same underlying representation as time.Duration, but Go treats them as separate types. You cannot pass a Duration where a time.Duration is expected without converting. The Std() method provides a convenient way to convert back. This pattern is useful for adding domain-specific methods to standard library types.
A value receiver gets a copy of the struct. It can read fields but any changes are lost when the method returns. A pointer receiver gets the address of the original, so changes persist:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
func (d Duration) Std() time.Duration {
return time.Duration(d)
}
func (d *Duration) Set(v time.Duration) {
*d = Duration(v)
fmt.Printf(" duration set to %v\n", d.Std())
}
func main() {
d := Duration(3 * time.Second)
std := d.Std()
fmt.Printf(" d.Std() = %v (d is unchanged: %v)\n", std, time.Duration(d))
d = Duration(1 * time.Second)
fmt.Printf(" before Set: %v\n", d.Std())
d.Set(5 * time.Second)
fmt.Printf(" after Set: %v <- mutated via pointer receiver\n", d.Std())
}
Std() uses a value receiver — it reads d but does not change it. Set() uses a pointer receiver — it overwrites the value that d points to. The rule of thumb: use a pointer receiver if the method needs to modify the receiver, or if the struct is large enough that copying it would be wasteful. Use a value receiver for small, immutable reads.
Go does not have constructors, but the convention is to write a NewXxx function that returns a pointer to an initialized struct. This lets you set up internal state (like pre-allocated buffers) that callers should not need to think about:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
type Writer struct {
Path string
BatchSize int
BytesWritten int
buffer []string
}
func NewWriter(path string, batchSize int) *Writer {
return &Writer{
Path: path,
BatchSize: batchSize,
buffer: make([]string, 0, batchSize),
}
}
func (w *Writer) Write(data string) {
w.buffer = append(w.buffer, data)
w.BytesWritten += len(data)
fmt.Printf(" wrote %q (buffer: %d/%d, total bytes: %d)\n",
data, len(w.buffer), w.BatchSize, w.BytesWritten)
}
func (w *Writer) Flush() {
fmt.Printf(" flushing %d items from buffer\n", len(w.buffer))
for _, item := range w.buffer {
fmt.Printf(" -> %s\n", item)
}
w.buffer = w.buffer[:0]
fmt.Printf(" buffer cleared (len=%d, cap=%d)\n", len(w.buffer), cap(w.buffer))
}
func main() {
w := NewWriter("/data/output.parquet", 3)
fmt.Printf(" created: %+v\n", *w)
w.Write("INSERT users alice")
w.Write("INSERT users bob")
w.Write("UPDATE users alice")
w.Flush()
w.Write("DELETE users bob")
fmt.Printf(" after more writes: buffer=%d, total bytes=%d\n",
len(w.buffer), w.BytesWritten)
}
NewWriter allocates the struct on the heap (returning &Writer{...} ensures the struct escapes the function’s stack frame) and pre-allocates the internal buffer with make([]string, 0, batchSize). The lowercase buffer field is unexported — callers outside the package cannot access it directly. Both Write and Flush use pointer receivers because they mutate the struct. Notice how Flush resets the buffer with w.buffer[:0] — this keeps the allocated memory and just sets the length to zero, which we will look at more closely in the next section.
Slices and maps are Go’s two workhorse collection types. A slice is a dynamically-sized view into an underlying array. A map is a hash table. Both are reference types — they contain internal pointers, so passing them to a function does not copy the underlying data.
A slice header has three fields: a pointer to the underlying array, a length (how many elements are in use), and a capacity (how many elements the array can hold before a new allocation is needed). Understanding this distinction is key to writing efficient Go:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
func main() {
s := make([]int, 0, 5)
fmt.Printf(" make([]int, 0, 5): len=%d, cap=%d\n", len(s), cap(s))
for i := 0; i < 8; i++ {
s = append(s, i)
fmt.Printf(" append(%d): len=%d, cap=%d\n", i, len(s), cap(s))
}
var nilSlice []int
fmt.Printf("\n nil slice: len=%d, cap=%d, is nil=%t\n",
len(nilSlice), cap(nilSlice), nilSlice == nil)
nilSlice = append(nilSlice, 1)
fmt.Printf(" after append: len=%d, cap=%d, is nil=%t\n",
len(nilSlice), cap(nilSlice), nilSlice == nil)
}
make([]int, 0, 5) creates a slice with length 0 and capacity 5. The first five appends fit without allocating. On the sixth append, Go allocates a new, larger array (typically doubling the capacity), copies the existing elements, and updates the slice header. This is why append returns a new slice — the pointer inside the header may have changed.
A nil slice (var nilSlice []int) has length 0, capacity 0, and is equal to nil. But append works on nil slices — it allocates on the first call. This means you do not need to initialize a slice before appending to it.
The ... operator spreads a slice into individual arguments. This is how you merge slices:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
type Change struct {
Table string
Action string
Key string
}
func main() {
batch1 := []Change{
{Table: "users", Action: "INSERT", Key: "alice"},
{Table: "users", Action: "UPDATE", Key: "bob"},
}
batch2 := []Change{
{Table: "orders", Action: "INSERT", Key: "ord-1"},
}
var all []Change
all = append(all, batch1...)
all = append(all, batch2...)
fmt.Printf(" merged %d + %d = %d changes\n", len(batch1), len(batch2), len(all))
for _, c := range all {
fmt.Printf(" %s %s (key=%s)\n", c.Action, c.Table, c.Key)
}
}
append(all, batch1...) is equivalent to appending each element of batch1 individually. Without the ..., the compiler would complain — append expects individual elements of the slice’s element type, not another slice.
[:0]When you need to reuse a buffer without reallocating, slice it back to zero length. The capacity (and the underlying array) are preserved:
1
2
3
4
5
6
7
8
9
10
11
12
13
func main() {
buffer := make([]Change, 0, 10)
buffer = append(buffer,
Change{Table: "users", Action: "INSERT", Key: "alice"},
Change{Table: "users", Action: "INSERT", Key: "bob"},
Change{Table: "orders", Action: "INSERT", Key: "ord-1"},
)
fmt.Printf(" before reset: len=%d, cap=%d\n", len(buffer), cap(buffer))
buffer = buffer[:0]
fmt.Printf(" after [:0]: len=%d, cap=%d <- capacity retained!\n", len(buffer), cap(buffer))
buffer = append(buffer, Change{Table: "events", Action: "INSERT", Key: "evt-1"})
fmt.Printf(" after reuse: len=%d, cap=%d <- no new allocation\n", len(buffer), cap(buffer))
}
buffer[:0] creates a new slice header with length 0 but the same capacity and underlying array. The next append writes into the existing array instead of allocating a new one. This pattern is common in hot loops where you process batches repeatedly — allocate once, reset between iterations.
A map is an unordered collection of key-value pairs. Accessing a missing key returns the zero value for the value type, which can be ambiguous — is the value actually zero, or does the key not exist? The comma-ok idiom resolves this:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
func main() {
schemas := map[string]string{
"users": "id, name, email",
"orders": "id, user_id, total",
}
if cols, ok := schemas["users"]; ok {
fmt.Printf(" users columns: %s\n", cols)
}
if _, ok := schemas["missing"]; !ok {
fmt.Println(" 'missing' table not found (comma-ok prevented silent zero value)")
}
for table, cols := range schemas {
fmt.Printf(" %s -> %s\n", table, cols)
}
}
The two-value assignment cols, ok := schemas["users"] returns the value and a boolean. If the key exists, ok is true. If not, ok is false and cols is the zero value (empty string for string). Always use the comma-ok form when the distinction between “missing” and “zero” matters.
A common pattern is grouping items by some key. Since accessing a missing map key returns the zero value — and the zero value of a slice is nil — you can append directly without initializing:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
func main() {
changes := []Change{
{Table: "users", Action: "INSERT", Key: "alice"},
{Table: "orders", Action: "INSERT", Key: "ord-1"},
{Table: "users", Action: "UPDATE", Key: "bob"},
{Table: "orders", Action: "DELETE", Key: "ord-2"},
{Table: "users", Action: "INSERT", Key: "charlie"},
}
groups := make(map[string][]Change)
for _, c := range changes {
groups[c.Table] = append(groups[c.Table], c)
}
for table, tableChanges := range groups {
fmt.Printf(" %s: %d changes\n", table, len(tableChanges))
for _, c := range tableChanges {
fmt.Printf(" %s key=%s\n", c.Action, c.Key)
}
}
}
The first time groups["users"] is accessed, it returns nil. But append(nil, item) works — it allocates a new slice. On subsequent accesses, it appends to the existing slice. This means you never need to check whether a key exists before appending. The pattern works because Go’s zero values are designed to be useful, not just empty.
A map[string]bool is the standard way to track whether you have already seen a value. Combined with a slice to preserve order, it gives you a deduplicated list:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
func main() {
changes := []Change{
{Table: "users", Action: "INSERT", Key: "alice"},
{Table: "orders", Action: "INSERT", Key: "ord-1"},
{Table: "users", Action: "UPDATE", Key: "bob"},
{Table: "events", Action: "INSERT", Key: "evt-1"},
{Table: "orders", Action: "DELETE", Key: "ord-2"},
}
seen := make(map[string]bool)
var uniqueTables []string
for _, c := range changes {
if !seen[c.Table] {
seen[c.Table] = true
uniqueTables = append(uniqueTables, c.Table)
}
}
fmt.Printf(" %d changes across %d unique tables: %v\n",
len(changes), len(uniqueTables), uniqueTables)
}
seen[c.Table] returns false (the zero value for bool) if the key has never been set. The first time a table name appears, it passes the !seen check, gets added to both the map and the slice. On subsequent appearances, the map returns true and the item is skipped. The slice preserves insertion order, which the map alone does not guarantee.
This pattern generalizes to any deduplication task — swap string for whatever type you are deduplicating by, and the logic stays the same.
.ducklake file or a PostgreSQL database) tracks table schemas, snapshots, and file pointers, while Parquet files hold the actual data on a local filesystem or in S3-compatible object storage. This separation is the central design decision — the catalog knows what files exist, what columns they contain, and what value ranges they hold, but it never stores the data itself. In this post, we’ll go from running DuckLake locally to running it with PostgreSQL and MinIO, look at how partitioning organizes data in S3, and see how DuckLake’s query engine prunes files and pushes predicates down to avoid reading data it doesn’t need.
Traditional databases store metadata and data together. Lakehouse formats like Iceberg, Delta Lake, and Hudi split them apart — they use a metadata layer (manifests, commit logs) on top of Parquet files in object storage. DuckLake follows the same philosophy but uses a relational database as the catalog instead of JSON/Avro manifest files.
The catalog stores:
The data files are Parquet, stored wherever you point DATA_PATH — a local directory, an S3 bucket, or any S3-compatible object store like MinIO or SeaweedFS.
Because the catalog is a regular database, you get transactions, concurrent access (with PostgreSQL), and the ability to query metadata with SQL. And because the data files are plain Parquet, any tool that reads Parquet can access them independently of DuckLake.
The simplest setup uses a local .ducklake file for metadata and a local directory for data. No Docker, no object storage, no external dependencies.
Install and load the extension:
1
2
INSTALL ducklake;
LOAD ducklake;
DuckLake inlines small tables (10 rows or fewer) directly into the catalog to avoid creating tiny Parquet files. For testing, disable this so all data goes to Parquet:
1
SET ducklake_default_data_inlining_row_limit = 0;
Attach a DuckLake catalog. The first argument is the metadata path (prefixed with ducklake:), and DATA_PATH is where Parquet files will be written:
1
2
ATTACH 'ducklake:step1_metadata.ducklake' AS lake (DATA_PATH 'step1_data/');
USE lake;
Create a table and insert data:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
CREATE OR REPLACE TABLE sensors (
tenant_id VARCHAR,
sensor_id INTEGER,
ts TIMESTAMP,
reading DOUBLE,
location VARCHAR
);
INSERT INTO sensors
SELECT
CASE WHEN i % 2 = 0 THEN 'tenant_a' ELSE 'tenant_b' END,
i,
TIMESTAMP '2025-01-01' + INTERVAL (i) MINUTE,
round(random() * 100, 2),
CASE WHEN i % 3 = 0 THEN 'NYC' WHEN i % 3 = 1 THEN 'SF' ELSE 'CHI' END
FROM range(10000) t(i);
After the insert, step1_data/ contains the Parquet files:
1
2
3
step1_data/
main/sensors/
ducklake-<uuid>.parquet
And step1_metadata.ducklake is a DuckDB file that holds the catalog tables — table definitions, column stats, snapshot history, and pointers to the Parquet files.
You can inspect the catalog with built-in functions:
1
2
SELECT * FROM ducklake_snapshots('lake');
SELECT * FROM ducklake_table_info('lake');
This setup works for local development and single-user workflows. But it has a limitation: the .ducklake file uses an exclusive file lock. If two processes try to write at the same time, the second one gets an error immediately:
1
2
Could not set lock on file "step1_metadata.ducklake":
Conflicting lock is held in duckdb (PID 46829)
For concurrent writes, you need PostgreSQL as the catalog backend.
A production-like setup replaces the local .ducklake file with PostgreSQL (for concurrent metadata access) and the local directory with MinIO (an S3-compatible object store for data).
A docker compose file brings both services up:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
services:
minio:
image: quay.io/minio/minio:latest
command: server /data --console-address ":9001"
ports:
- "9000:9000"
- "9001:9001"
environment:
MINIO_ROOT_USER: minioadmin
MINIO_ROOT_PASSWORD: minioadmin
volumes:
- minio_data:/data
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 5s
timeout: 5s
retries: 5
postgres:
image: postgres:16
ports:
- "5432:5432"
environment:
POSTGRES_USER: ducklake
POSTGRES_PASSWORD: ducklake
POSTGRES_DB: ducklake
volumes:
- pg_data:/var/lib/postgresql/data
healthcheck:
test: ["CMD-SHELL", "pg_isready -U ducklake"]
interval: 5s
timeout: 5s
retries: 5
volumes:
minio_data:
pg_data:
Start the services and create a bucket for DuckLake data:
1
2
docker compose up -d
docker compose exec minio mc mb local/ducklake-data
Now load the required extensions — ducklake for the lakehouse logic, httpfs for S3 access, and postgres for the PostgreSQL catalog backend:
1
2
3
4
5
INSTALL ducklake; LOAD ducklake;
INSTALL httpfs; LOAD httpfs;
INSTALL postgres; LOAD postgres;
SET ducklake_default_data_inlining_row_limit = 0;
Create an S3 secret pointing at MinIO:
1
2
3
4
5
6
7
8
CREATE SECRET (
TYPE s3,
KEY_ID 'minioadmin',
SECRET 'minioadmin',
ENDPOINT 'localhost:9000',
USE_SSL false,
URL_STYLE 'path'
);
Attach the DuckLake catalog using PostgreSQL for metadata and MinIO for data:
1
2
3
ATTACH 'ducklake:postgres:dbname=ducklake user=ducklake password=ducklake host=localhost port=5432'
AS lake (DATA_PATH 's3://ducklake-data/');
USE lake;
From here, creating tables and inserting data is identical to the local setup. The difference is where things go: metadata rows land in PostgreSQL tables (ducklake_table, ducklake_column, ducklake_data_file, ducklake_snapshot), and Parquet files land in the MinIO bucket.
With PostgreSQL as the catalog, concurrent writes work. Two DuckDB processes can insert into the same table simultaneously — PostgreSQL handles the row-level locking internally. The data files in S3 are immutable (each insert creates new Parquet files), so there are no conflicts on the storage side either.
Partitioning tells DuckLake to organize Parquet files into a directory hierarchy based on column values. Instead of writing all rows into a single file, DuckLake creates separate files for each distinct combination of partition key values.
Define a table and set the partition key:
1
2
3
4
5
6
7
8
9
10
CREATE TABLE sensors (
tenant_id VARCHAR,
sensor_id INTEGER,
event_date DATE,
ts TIMESTAMP,
reading DOUBLE,
location VARCHAR
);
ALTER TABLE sensors SET PARTITIONED BY (tenant_id, event_date);
Insert 30,000 rows spanning 3 tenants and 10 dates:
1
2
3
4
5
6
7
8
9
10
11
12
13
INSERT INTO sensors
SELECT
CASE WHEN i % 3 = 0 THEN 'tenant_a'
WHEN i % 3 = 1 THEN 'tenant_b'
ELSE 'tenant_c' END AS tenant_id,
i AS sensor_id,
DATE '2025-01-01' + INTERVAL (i % 10) DAY AS event_date,
TIMESTAMP '2025-01-01' + INTERVAL (i) MINUTE AS ts,
round(random() * 100, 2) AS reading,
CASE WHEN i % 3 = 0 THEN 'NYC'
WHEN i % 3 = 1 THEN 'SF'
ELSE 'CHI' END AS location
FROM range(30000) t(i);
This produces 30 Parquet files — one for each (tenant, date) combination, each holding 1,000 rows.
The directory layout in S3 follows the Hive partitioning convention: column=value/ directories nested according to the partition key order.
For an unpartitioned table, all files sit in a flat directory:
1
2
3
4
s3://ducklake-data/
main/sensors/
ducklake-<uuid-1>.parquet
ducklake-<uuid-2>.parquet
For a table partitioned by a single column (location):
1
2
3
4
5
6
7
8
s3://ducklake-data/
main/sensors_by_location/
location=CHI/
ducklake-<uuid>.parquet
location=NYC/
ducklake-<uuid>.parquet
location=SF/
ducklake-<uuid>.parquet
For a table partitioned by two columns (tenant_id, event_date):
1
2
3
4
5
6
7
8
9
10
11
12
13
14
s3://ducklake-data/
main/sensors/
tenant_id=tenant_a/
event_date=2025-01-01/
ducklake-<uuid>.parquet
event_date=2025-01-02/
ducklake-<uuid>.parquet
...
tenant_id=tenant_b/
event_date=2025-01-01/
ducklake-<uuid>.parquet
...
tenant_id=tenant_c/
...
The path structure is {DATA_PATH}/{schema}/{table}/{partition_key=value}/.../{file}.parquet. Each Parquet file is a standard columnar file with row groups, column chunks, and a metadata footer containing min/max statistics and byte offsets.
The catalog in PostgreSQL (or the .ducklake file) tracks every one of these files: their paths, row counts, file sizes, and per-column statistics. DuckLake never needs to list the S3 bucket to find files — it queries the catalog instead. This is what makes pruning fast: the catalog is a regular database table, and filtering it is a local operation that doesn’t touch object storage at all.
Partition pruning is the first and most impactful optimization. When a query includes a WHERE clause on a partition column, DuckLake consults the catalog to determine which Parquet files could possibly match. Files that can’t match are eliminated before any S3 request is made.
Using the 30-file dataset (3 tenants x 10 dates), EXPLAIN ANALYZE shows exactly how many files DuckLake reads for each query:
Full scan (no filter) — 30 files:
1
2
3
EXPLAIN ANALYZE
SELECT COUNT(*), AVG(reading) FROM sensors;
-- Total Files Read: 30
Both partition keys — 1 file:
1
2
3
4
EXPLAIN ANALYZE
SELECT COUNT(*), AVG(reading) FROM sensors
WHERE tenant_id = 'tenant_a' AND event_date = DATE '2025-01-03';
-- Total Files Read: 1
First partition key only — 10 files:
1
2
3
4
EXPLAIN ANALYZE
SELECT COUNT(*), AVG(reading) FROM sensors
WHERE tenant_id = 'tenant_a';
-- Total Files Read: 10
Second partition key only (no tenant filter) — 3 files:
1
2
3
4
EXPLAIN ANALYZE
SELECT COUNT(*), AVG(reading) FROM sensors
WHERE event_date = DATE '2025-01-03';
-- Total Files Read: 3
This last query is significant. Filtering on event_date without specifying tenant_id still prunes down to 3 files (one per tenant for that date). DuckLake prunes on the second partition key independently of the first. It doesn’t need the leading key to be present — it evaluates each partition column separately against the catalog.
Date range — 9 files:
1
2
3
4
EXPLAIN ANALYZE
SELECT COUNT(*), AVG(reading) FROM sensors
WHERE event_date BETWEEN DATE '2025-01-02' AND DATE '2025-01-04';
-- Total Files Read: 9
Range predicates prune the same way. Three dates across three tenants: 9 files.
The pruning stack has three layers, each narrowing the data further:
In EXPLAIN ANALYZE output, the key number to look for is Total Files Read in the TABLE_SCAN node:
1
2
3
4
5
6
7
8
┌───────────────────────────┐
│ TABLE_SCAN │
│ Table: sensors │
│ Filters: │
│ event_date='2025-01-03' │
│ Total Files Read: 3 │
│ 3,000 rows │
└───────────────────────────┘
Partition pruning only helps if the filter reaches the table scan. In real applications, queries often go through layers of views. Does DuckLake still prune when the filter is applied to a view several layers above the base table?
Stack three views on top of the partitioned table:
1
2
3
4
5
6
7
8
9
10
11
CREATE OR REPLACE VIEW v_base AS
SELECT tenant_id, event_date, sensor_id, reading, location FROM sensors;
CREATE OR REPLACE VIEW v_enriched AS
SELECT *, reading * 1.1 AS adjusted_reading FROM v_base;
CREATE OR REPLACE VIEW v_summary AS
SELECT
v.tenant_id, v.event_date, v.adjusted_reading, v.location,
AVG(v.adjusted_reading) OVER (PARTITION BY v.tenant_id) AS tenant_avg
FROM v_enriched v;
Query the top-level view with a date filter:
1
2
3
EXPLAIN ANALYZE
SELECT * FROM v_summary
WHERE event_date = DATE '2025-01-03';
The filter on event_date pushes all the way through v_summary → v_enriched → v_base → sensors and triggers partition pruning at the TABLE_SCAN node. Only 3 files are read, not 30. The views add zero overhead to the pruning — DuckDB’s optimizer traces the filter down to the underlying table before execution begins.
This also works through joins. A view that joins v_base to itself still benefits from pushdown:
1
2
3
4
5
6
7
8
9
10
11
12
CREATE OR REPLACE VIEW v_with_join AS
SELECT a.tenant_id, a.event_date, a.reading, b.reading AS other_reading
FROM v_base a
JOIN v_base b
ON a.tenant_id = b.tenant_id
AND a.event_date = b.event_date
AND a.sensor_id != b.sensor_id;
EXPLAIN ANALYZE
SELECT * FROM v_with_join
WHERE event_date = DATE '2025-01-03'
LIMIT 10;
Both sides of the join prune to 3 files each. The optimizer pushes the predicate into both branches independently.
Before diving into AWS-specific concepts, a quick refresher on the addressing system everything is built on.
CIDR notation defines a range of IP addresses. The number after the slash is how many bits are fixed — more fixed bits means fewer addresses:
1
2
3
10.0.0.0/16 = 10.0.*.* = 65,536 IPs
10.0.1.0/24 = 10.0.1.* = 256 IPs
10.0.2.0/24 = 10.0.2.* = 256 IPs
/16 = first 16 bits fixed (first two octets) = 65,536 IPs/24 = first 24 bits fixed (first three octets) = 256 IPs/32 = all bits fixed = exactly 1 IPThree ranges are globally reserved for private networks (RFC 1918). Every router and OS knows these will never appear on the public internet:
| Range | Size |
|---|---|
10.0.0.0/8 |
16.7 million IPs |
172.16.0.0/12 |
~1 million IPs |
192.168.0.0/16 |
65,536 IPs |
Using a public range internally (say 142.250.80.0, which belongs to Google) would work — until you need to reach the real owner of that range. Internal traffic would route locally instead of reaching the real destination. This is called IP shadowing and it’s why you always use private ranges for internal networks.
Two terms that appear everywhere in AWS docs and Terraform configs: ingress means inbound traffic (coming in), egress means outbound traffic (going out). The plain English equivalents — inbound and outbound — mean the same thing.
A VPC is your own isolated network inside AWS. Think of it as your private data center’s network, built on the same private IP ranges described above.
When you create a VPC you assign it a CIDR block — for example, 10.0.0.0/16 gives you 65,536 addresses. AWS requires VPC CIDR blocks to be between /16 and /28.
The CIDR block is just the starting point. A VPC is the container for everything else: subnets, gateways, route tables, and security groups. The IP range defines the address space; the rest defines what can talk to what and how.
10.0.0.0/8 — most common for cloud and corporate networks. AWS defaults to this. Plenty of room to carve out many VPCs without overlapping.172.16.0.0/12 — sometimes used as a second range when 10.x is already taken.192.168.0.0/16 — home routers default to this. Rarely used in cloud because it’s small and likely to collide if you ever connect a VPC to an office network.The practical concern is overlap. If a VPC uses 10.0.0.0/16 and an office network also uses 10.0.0.0/16, they can’t be connected via VPC peering or VPN because the ranges collide. Companies often plan CIDR allocations across environments to avoid this — for example, 10.0.0.0/16 for prod, 10.1.0.0/16 for staging, 10.2.0.0/16 for dev.
A subnet is a subdivision of the VPC tied to a specific Availability Zone (AZ). A VPC spans a whole region (e.g., us-east-1), but each subnet lives in exactly one AZ (e.g., us-east-1a). AZs are physically separate data centers.
This distinction is not an inherent property — it’s a consequence of routing. A subnet is “public” because its route table has a route to an Internet Gateway. A subnet is “private” because it doesn’t. The labels are conventions based on how traffic flows.
Even in a “public” subnet, resources aren’t automatically exposed. A NAT Gateway in a public subnet has a public IP but only allows outbound traffic — nobody can initiate a connection to it from outside. A bastion host accepts inbound SSH but only from a specific IP, locked down by its security group.
AWS requires the DB subnet group to span at least two AZs. This is for high availability — if one data center goes down, the database can failover to the other AZ. Aurora takes this further: the storage layer automatically replicates data six ways across three AZs, so having subnets in multiple AZs is essential.
1
2
3
4
5
VPC: 10.0.0.0/16
├── Subnet A (private): 10.0.1.0/24 in us-east-1a
├── Subnet B (private): 10.0.2.0/24 in us-east-1b ← Aurora needs both
├── Subnet C (public): 10.0.101.0/24 in us-east-1a ← for bastion/NAT
└── Subnet D (public): 10.0.102.0/24 in us-east-1b
Internet Gateway (IGW) attaches to the VPC and allows resources in public subnets to communicate with the internet — both inbound and outbound.
NAT Gateway sits in a public subnet and allows resources in private subnets to make outbound requests (download patches, call external APIs) without being reachable from the internet. It’s a one-way door.
1
2
3
4
5
6
7
Internet
│
▼
Internet Gateway ─── Public Subnet ─── NAT Gateway
│
▼
Private Subnet ─── Aurora RDS
Aurora RDS should never be in a public subnet. If it needs to reach the internet (which is rare — AWS handles engine updates through its own management plane), it goes through the NAT Gateway.
No. The bastion-plus-public-subnet pattern is the simplest way to get started, but if you use alternatives like a VPN, AWS Systems Manager Session Manager, or an application server in the same VPC, you don’t need a bastion or any public subnets at all. The entire VPC can be private — you still need at least two private subnets in different AZs for the DB subnet group, but nothing needs to be public.
Route tables are the rules that tell traffic where to go. Each subnet is associated with exactly one route table.
Public subnet route table:
| Destination | Target | Meaning |
|---|---|---|
10.0.0.0/16 |
local |
Traffic within VPC stays local |
0.0.0.0/0 |
igw-xxxxx |
Everything else goes to the internet |
Private subnet route table:
| Destination | Target | Meaning |
|---|---|---|
10.0.0.0/16 |
local |
Traffic within VPC stays local |
0.0.0.0/0 |
nat-xxxxx |
Outbound goes through NAT (no inbound) |
The only difference is one line: igw vs nat for the default route.
A security group is a virtual firewall attached to a resource — an EC2 instance, an Aurora RDS cluster, etc. It controls which traffic is allowed in (ingress) and out (egress).
Security groups are a separate layer from routing. Routing controls whether traffic can physically reach a resource; security groups control whether the resource accepts that traffic. Both must allow the traffic for it to get through.
When the DB security group says “allow ingress from sg-app,” it means “allow ingress from any resource that has sg-app attached.” You don’t need to track IPs — security group references don’t break when IPs change:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
resource "aws_security_group" "app" {
vpc_id = aws_vpc.main.id
name = "app"
}
resource "aws_security_group" "db" {
vpc_id = aws_vpc.main.id
name = "database"
ingress {
from_port = 5432
to_port = 5432
security_groups = [aws_security_group.app.id]
}
}
The app server gets aws_security_group.app attached, the Aurora cluster gets aws_security_group.db. Only resources in the app group can reach the database on port 5432.
A DB subnet group is an RDS-specific concept — a named collection of subnets where RDS is allowed to place database instances. When you create an Aurora cluster, you don’t pick a subnet directly. You pick a DB subnet group, and Aurora chooses which subnets to use. For Multi-AZ deployments it places the writer in one subnet and readers in others.
The rule: the group must include subnets in at least two different AZs.
Here’s the complete picture for an Aurora RDS PostgreSQL deployment:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
┌─────────────────────── VPC (10.0.0.0/16) ────────────────────────┐
│ │
│ ┌── Public Subnet (10.0.101.0/24, us-east-1a) ────┐ │
│ │ Bastion Host (EC2) │ │
│ │ SG: allow SSH (22) from your IP │ │
│ └─────────────────────────────────────────────────┘ │
│ │ connects via private IP on port 5432 │
│ ▼ │
│ ┌── Private Subnet (10.0.1.0/24, us-east-1a) ─────┐ │
│ │ Aurora PostgreSQL (writer) │ │
│ │ SG: allow 5432 from bastion SG │ │
│ └─────────────────────────────────────────────────┘ │
│ │
│ ┌── Private Subnet (10.0.2.0/24, us-east-1b) ───────┐ │
│ │ Aurora PostgreSQL (reader replica) │ │
│ │ Same SG │ │
│ └───────────────────────────────────────────────────┘ │
│ │
│ DB Subnet Group = [private-subnet-1a, private-subnet-1b] │
└──────────────────────────────────────────────────────────────────┘
You SSH into the bastion, then from there psql into the Aurora endpoint. The security group on Aurora only allows connections from the bastion’s security group on port 5432. The database is never directly reachable from the internet.
Each layer has a distinct job: the VPC defines the address space, subnets partition it across AZs, route tables control where traffic flows, security groups decide what gets through, and the DB subnet group tells Aurora which subnets it can use. All of these must be in place before you create the first Aurora cluster.
]]>A replication slot is a server-side bookmark. While the connector is running, it continuously reads from the WAL and confirms its position back to PostgreSQL, which allows PostgreSQL to recycle old WAL segments. When the connector goes down, the slot stays — PostgreSQL holds all WAL from the slot’s last confirmed position forward.
Check the current slot state while the connector is running:
1
2
SELECT slot_name, active, restart_lsn, confirmed_flush_lsn
FROM pg_replication_slots;
The active column is true and the lag is small — Debezium is keeping up. Now delete the connector:
1
curl -X DELETE http://localhost:8083/connectors/learn-cdc-connector
The connector is gone, but the slot is still there:
1
SELECT slot_name, active, restart_lsn FROM pg_replication_slots;
active is now false. PostgreSQL will keep all WAL from this point forward because the slot indicates its consumer has not read past here.
Make some changes while the connector is offline:
1
2
3
4
5
6
7
INSERT INTO orders (customer, product, amount)
VALUES ('eve', 'Widget E', 59.99);
INSERT INTO orders (customer, product, amount)
VALUES ('frank', 'Widget F', 14.99);
UPDATE customers SET tier = 'premium' WHERE id = 2;
Check the lag:
1
2
3
4
SELECT slot_name,
pg_size_pretty(pg_current_wal_lsn() - restart_lsn) AS wal_retained
FROM pg_replication_slots
WHERE slot_name = 'learn_cdc_slot';
The retained WAL is growing. On a production database with high write throughput, this can fill the disk within hours.
Re-register the connector with the same slot.name. It reconnects to the existing slot, reads all the WAL that accumulated while it was down, and produces the missed events to Kafka. No data is lost.
This is the key guarantee: as long as the replication slot exists, PostgreSQL will not discard WAL that has not been consumed. The connector can go down and come back, and it picks up exactly where it left off.
A leaked slot is a replication slot whose connector has been permanently removed but the slot was never dropped. It is the single biggest operational risk when running Debezium.
Deleting a Debezium connector through the Kafka Connect REST API removes the connector process, but it does not drop the replication slot in PostgreSQL. The slot stays, active = false, telling PostgreSQL to hold WAL indefinitely.
In a lab environment this is harmless — a few extra kilobytes of WAL. In production with continuous writes, the WAL grows without bound because PostgreSQL will not recycle any segment past the slot’s LSN:
1
2
3
4
SELECT slot_name, active,
pg_size_pretty(pg_current_wal_lsn() - restart_lsn) AS wal_retained,
pg_size_pretty(pg_current_wal_lsn() - confirmed_flush_lsn) AS consumer_lag
FROM pg_replication_slots;
Alert if:
active = false for more than a few minuteswal_retained exceeds a threshold (e.g., 1 GB)consumer_lag is growing steadily (connector falling behind)1
2
3
4
SELECT pg_drop_replication_slot('learn_cdc_slot');
-- Verify
SELECT slot_name, active FROM pg_replication_slots;
Name slots explicitly (e.g., orders_cdc_slot, analytics_cdc_slot) so they are easy to identify when monitoring. The default Debezium-generated name (debezium) is ambiguous if multiple connectors run against the same database.
What happens when a table’s schema changes while Debezium is streaming?
1
2
3
4
5
6
7
ALTER TABLE orders ADD COLUMN notes TEXT;
INSERT INTO orders (customer, product, amount, notes)
VALUES ('grace', 'Widget G', 24.99, 'Rush delivery');
INSERT INTO orders (customer, product, amount)
VALUES ('henry', 'Widget H', 34.99);
Both events include the notes field — one with the value, one with null. Debezium picks up the schema change automatically. No connector restart is needed.
1
2
3
4
ALTER TABLE orders RENAME COLUMN notes TO remarks;
INSERT INTO orders (customer, product, amount, remarks)
VALUES ('iris', 'Widget I', 44.99, 'After rename');
The event has remarks instead of notes. Debezium reflects the current schema.
1
2
3
4
ALTER TABLE orders DROP COLUMN remarks;
INSERT INTO orders (customer, product, amount)
VALUES ('jack', 'Widget J', 54.99);
The event no longer has the remarks field.
Debezium does not care about schema evolution. It forwards whatever the WAL gives it as JSON. Column added, renamed, dropped — Debezium serializes the current row state and produces it to Kafka. No restart, no config change.
The schema evolution problem moves entirely downstream — the consumer of the Kafka events must handle the changing JSON shape. Common approaches:
JSONB in PostgreSQL, VARIANT in Snowflake, STRING in BigQuery) so the varying shapes do not cause failures. The JSON blob just has different keys over time.The before/after fields in the event envelope contain the raw row data. If you store those as schemaless JSON, adding or dropping columns in the source table has no effect on the landing table — the JSON blob absorbs the change. Schema evolution on the event envelope itself (e.g., a Debezium version upgrade adding a new metadata field to source) is a separate concern handled by the sink connector or schema registry.
A minimal monitoring setup for Debezium in production:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
-- Replication slot health
SELECT slot_name, active,
pg_size_pretty(pg_current_wal_lsn() - restart_lsn) AS wal_retained,
pg_size_pretty(pg_current_wal_lsn() - confirmed_flush_lsn) AS consumer_lag
FROM pg_replication_slots;
-- Publication tables
SELECT * FROM pg_publication_tables;
-- Replica identity per table
SELECT relname, relreplident
FROM pg_class
WHERE relname IN ('orders', 'customers');
-- 'f' = FULL, 'd' = DEFAULT, 'n' = NOTHING, 'i' = INDEX
1
2
3
4
5
6
7
8
# Connector status
curl http://localhost:8083/connectors/orders-cdc-connector/status | jq
# List all connectors
curl http://localhost:8083/connectors | jq
# Task-level status (a connector can have multiple tasks)
curl http://localhost:8083/connectors/orders-cdc-connector/tasks/0/status | jq
| Signal | Meaning | Action |
|---|---|---|
Slot active = false for > 5 min |
Connector is down | Investigate connector status, restart if needed |
wal_retained > threshold |
WAL accumulating | Either the connector is behind or the slot is leaked |
Connector state FAILED |
Connector crashed | Check task error in status endpoint, fix config or data issue, restart |
| Consumer lag growing | Connector falling behind | Check for large transactions, schema changes, or resource constraints |
| No new messages on CDC topic | Either no writes or connector stalled | Cross-reference with database write activity |
before values are missing on UPDATEs and DELETEs, all tables are captured including system tables, and each table gets its own Kafka topic. This post covers the configuration knobs that close those gaps: replica identity, table filtering, publication modes, and single-topic routing with the ByLogicalTableRouter transform.
With the default connector from the previous post, UPDATE and DELETE events have before: null. The previous row state is missing because PostgreSQL’s default replica identity only writes the primary key to the WAL on UPDATEs and DELETEs — not the full row. Debezium can only produce what the WAL gives it.
1
UPDATE orders SET status = 'shipped' WHERE id = 1;
The resulting event:
1
2
3
4
5
6
7
8
9
10
11
{
"before": null,
"after": {
"id": 1,
"customer": "alice",
"product": "Widget A",
"amount": 29.99,
"status": "shipped"
},
"op": "u"
}
There is no way to tell what status was before the update. For audit trails, sync pipelines, or any use case that needs to know what changed (not just what the current state is), this is a problem.
Set the table’s replica identity to FULL so PostgreSQL writes the entire old row to the WAL:
1
2
3
ALTER TABLE orders REPLICA IDENTITY FULL;
UPDATE orders SET status = 'cancelled' WHERE id = 1;
Now the event has before populated:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
{
"before": {
"id": 1,
"customer": "alice",
"product": "Widget A",
"amount": 29.99,
"status": "shipped"
},
"after": {
"id": 1,
"customer": "alice",
"product": "Widget A",
"amount": 29.99,
"status": "cancelled"
},
"op": "u"
}
DELETEs also benefit — a DELETE on a FULL table produces before with the deleted row’s data and after: null.
Instead of manually running ALTER TABLE ... REPLICA IDENTITY FULL on each table, Debezium can do it automatically:
1
"replica.identity.autoset.values": "public.*:FULL"
This tells the connector to set REPLICA IDENTITY FULL on all tables in the public schema matching the pattern. You can verify it took effect:
1
2
3
4
SELECT relname, relreplident
FROM pg_class
WHERE relname IN ('orders', 'customers');
-- 'f' = FULL, 'd' = DEFAULT
If you only care about the current state of a row — for example, replicating a table to another database where you overwrite the target row with the latest after — DEFAULT replica identity is fine. FULL adds overhead because PostgreSQL writes more data to the WAL on every UPDATE and DELETE.
The default connector captures all tables in the database. In practice, you want to exclude system tables (migration tracking, job queues, framework-internal tables) and capture only domain tables.
An allowlist of fully-qualified table names:
1
"table.include.list": "public.orders,public.customers"
Any table not in the list is ignored. With this config, inserts into other tables produce no CDC events and no Kafka topics are created for them.
A denylist — capture everything except these tables:
1
"table.exclude.list": "public.flyway_schema_history,public.schema_migrations"
You can use one or the other, not both.
Instead of filtering at the Debezium level, you can control it at the PostgreSQL level through publications. The publication.autocreate.mode setting controls how Debezium manages publications:
| Mode | What it does |
|---|---|
all_tables |
Auto-creates a publication for all tables. Simple, good for development |
filtered |
Auto-creates a publication only for tables in table.include.list |
disabled |
Expects a publication to already exist — you create it yourself via a migration |
For production, disabled with a manually-created publication gives the most explicit control. The publication is created in a database migration, not by the connector:
1
2
-- In a migration script
CREATE PUBLICATION cdc_publication FOR TABLE orders, customers, products;
Then the connector references it:
1
2
"publication.autocreate.mode": "disabled",
"publication.name": "cdc_publication"
This avoids surprises when new tables are added to the schema — they are only captured when explicitly added to the publication.
By default, Debezium creates one Kafka topic per table (learn.public.orders, learn.public.customers, etc.). The ByLogicalTableRouter transform routes events from multiple tables into a single topic.
1
2
3
4
5
"transforms": "route-to-single-topic",
"transforms.route-to-single-topic.type": "io.debezium.transforms.ByLogicalTableRouter",
"transforms.route-to-single-topic.topic.regex": ".*",
"transforms.route-to-single-topic.topic.replacement": "learn.v0.cdc",
"transforms.route-to-single-topic.key.enforce.uniqueness": "false"
The route-to-single-topic name is an alias you choose — it is just a label to reference the transform in the config. The .type is the actual Java class. Everything else under that name is config passed to that class.
After applying this, all events from all captured tables land on learn.v0.cdc. The consumer distinguishes them by the source.table field in the event envelope.
The trade-off: consumers need to handle mixed event types and filter by source.table.
You can chain multiple transforms — they run in order on each event:
1
2
3
"transforms": "filter,route",
"transforms.filter.type": "...",
"transforms.route.type": "..."
The snapshot.mode setting controls what happens when the connector starts:
| Mode | Behavior |
|---|---|
initial |
Snapshot all existing rows on first start, then stream. On subsequent starts, just stream from where the slot left off |
never |
No snapshot — only stream changes from the WAL. Existing rows are not captured |
when_needed |
Snapshot if the replication slot does not exist or the slot’s position is no longer available in the WAL |
initial_only |
Snapshot existing rows and then stop — no streaming |
initial is the most common choice. It gives you the full current state of the database as op: "r" events, followed by real-time changes as op: "c"/"u"/"d".
never is useful when you only care about changes going forward — for example, if the historical data has already been loaded by another mechanism.
1
"tombstones.on.delete": "false"
When set to true (the default), Debezium produces a second event after every DELETE with a null value (a Kafka tombstone). Tombstones tell Kafka’s log compaction to remove the key entirely. If your downstream consumers do not rely on log compaction, setting this to false removes the extra event.
Putting it all together — a connector config that filters tables, sets replica identity, routes to a single topic, and uses explicit publication management:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
{
"name": "orders-cdc-connector",
"config": {
"connector.class": "io.debezium.connector.postgresql.PostgresConnector",
"database.hostname": "postgres",
"database.port": "5432",
"database.user": "cdc_reader",
"database.password": "${file:/secrets/db-password}",
"database.dbname": "orders_db",
"topic.prefix": "orders",
"plugin.name": "pgoutput",
"slot.name": "orders_cdc_slot",
"publication.name": "orders_publication",
"publication.autocreate.mode": "disabled",
"table.include.list": "public.orders,public.customers,public.products",
"snapshot.mode": "initial",
"tombstones.on.delete": "false",
"replica.identity.autoset.values": "public.*:FULL",
"transforms": "route",
"transforms.route.type": "io.debezium.transforms.ByLogicalTableRouter",
"transforms.route.topic.regex": ".*",
"transforms.route.topic.replacement": "orders.v0.cdc",
"transforms.route.key.enforce.uniqueness": "false",
"key.converter": "org.apache.kafka.connect.storage.StringConverter",
"value.converter": "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "false",
"database.ssl.mode": "require",
"errors.retry.timeout": "600000",
"producer.override.max.request.size": "20971520"
}
}
| Config | Why |
|---|---|
publication.autocreate.mode: disabled |
Explicit control — publication created via migration, not by Debezium |
replica.identity.autoset.values |
Ensures before-values on UPDATE/DELETE for audit history |
database.ssl.mode: require |
Encrypted connections |
errors.retry.timeout: 600000 |
Retry transient errors for 10 minutes before failing |
producer.override.max.request.size: 20971520 |
Handles large payloads (big text fields, JSONB columns) up to 20 MB |
The transforms that come up most often in practice:
| Transform | Type class | What it does |
|---|---|---|
| ByLogicalTableRouter | io.debezium.transforms.ByLogicalTableRouter |
Routes multiple tables to a single topic via regex |
| EventRouter | io.debezium.transforms.outbox.EventRouter |
Routes outbox table rows to dynamic topics based on a field — used with the transactional outbox pattern |
| ExtractNewRecordState | io.debezium.transforms.ExtractNewRecordState |
Flattens the envelope — strips before/source/op, emits just the after payload |
| Filter | io.debezium.transforms.Filter |
Drops events based on scripting expressions |
| ContentBasedRouter | io.debezium.transforms.ContentBasedRouter |
Routes to different topics based on expressions evaluated against event content |
| TimezoneConverter | io.debezium.transforms.TimezoneConverter |
Converts timestamps between timezones |
ByLogicalTableRouter and EventRouter cover the majority of production use cases. ExtractNewRecordState is useful when the downstream consumer wants a flat record instead of the full envelope — for example, when sinking directly into a relational table where each column maps to a field in after.