Files, Buffers, and Paths

A teammate's log-shaping tool starts strong and dies in production. The implementation is three lines: data, err := os.ReadFile(path) followed by a split on newlines. On a 120 KB dev file it is perfect. On the 4 GB file the ops team actually needs, the process is killed by the OOM reaper before the first line is written. The bug is not in the filtering logic — it is in the decision to materialize a file that only ever needed to be traversed.

Go's answer is the io.Reader/io.Writer contract: operations that consume or produce bytes should stream them. For line-oriented data, bufio.Scanner sits on top of an *os.File and hands you one line at a time.

func shapeLog(in, out string) error {
	src, err := os.Open(in)
	if err != nil {
		return fmt.Errorf("open %s: %w", in, err)
	}
	defer src.Close()

	dst, err := os.Create(out)
	if err != nil {
		return fmt.Errorf("create %s: %w", out, err)
	}
	defer dst.Close()

	buf := bufio.NewScanner(src)
	// The default max token is 64 KB; one corrupt line would abort the whole run.
	buf.Buffer(make([]byte, 64*1024), 1024*1024)

	w := bufio.NewWriter(dst)
	defer w.Flush()

	for buf.Scan() {
		line := buf.Text()
		if strings.Contains(line, "level=error") {
			fmt.Fprintln(w, line)
		}
	}
	if err := buf.Err(); err != nil {
		return fmt.Errorf("scan %s: %w", in, err)
	}
	return nil
}

Four details decide whether this survives contact with real files.

Scanner has a hard token limit. The zero Scanner refuses tokens longer than 64 KB and stops with bufio.ErrTooLong. A single minified JSON line or a binary file misread as text ends the loop with an error that looks like corruption but is really a misconfigured buffer. Calling buf.Buffer(base, max) raises the ceiling — and it must be raised before the first Scan, because the buffer cannot grow past the max you passed.

Check buf.Err(), not just whether the loop ended. A clean EOF and a mid-file read failure both end the loop; only buf.Err() tells them apart. Skipping that check silently truncates the output — the most dangerous kind of success.

Wrap the writer too. A raw bufio.Scanner writes a syscall per line. One bufio.Writer around the destination batches them, and Flush must be checked — a deferred Flush() that discards its error loses the tail of the file. The same discipline for deferred closes and their errors appears in defer and the Deferred Call Stack.

Cost is linear, memory is flat. This version holds one line plus a few kilobytes of buffer regardless of file size, so the 4 GB file and the 120 KB file behave identically. When the whole content genuinely fits and you need random access, os.ReadFile is fine — the rule is knowing which case you are in, which is what the benchmarks in Coverage and Benchmarks exist to confirm.

The tool runs on Linux in CI and on a Windows laptop during the demo. It builds every path as dir + "/" + name. On Linux the result is /data/report.csv; on Windows it is data/report.csv — a relative path that resolves against whatever directory the shell happened to be in. Concatenation is wrong on any OS that uses a different separator, and path (the URL package) is wrong everywhere because it always uses /.

path/filepath is the filesystem's path package. Its rule: join with filepath.Join, split with filepath.Split, inspect with filepath.Ext/Base/Dir.

func outputPath(root, name string) (string, error) {
	// Join cleans the result: no doubled or trailing separators,
	// and it uses the separator of the running OS.
	p := filepath.Join(root, "out", name)

	// Refuse names that would climb out of the sandbox.
	rel, err := filepath.Rel(root, p)
	if err != nil || strings.HasPrefix(rel, "..") {
		return "", fmt.Errorf("%q escapes output root", name)
	}
	if err := os.MkdirAll(filepath.Dir(p), 0o755); err != nil {
		return "", fmt.Errorf("create dirs: %w", err)
	}
	return p, nil
}

filepath.Join("/data", "out", "report.csv") returns /data/out/report.csv on Linux and \data\out\report.csv on Windows — one call, correct on both. The filepath.Rel check is the interesting half: user-supplied names like ../../etc/passwd are a directory-traversal bug, and verifying that the joined path still sits under the intended root is the defense. This matters the moment a CLI takes a path argument or an HTTP handler takes a filename, as covered in Request Handling and Routing.

Three more rules that pay rent:

• Never build a path from strings.Split on /. Use filepath.Split, which returns the directory and file components using the local separator.
• Prefer the OS-provided roots. os.UserHomeDir(), os.UserConfigDir(), and os.UserCacheDir() return the right base on every platform; hardcoding ~/.mytool fails for users whose home is elsewhere or non-existent in a container.
• Walk with filepath.WalkDir, not filepath.Walk. WalkDir passes fs.DirEntry values that carry the type from the directory read, so skipping a subtree does not stat every file inside it — measurably faster on large trees and identical in shape to iterating an embed.FS in Embedding Assets with go:embed.

Path handling is boring until it is a security review. The habit to build: every path enters through filepath, gets cleaned, and gets checked against the root it is supposed to stay inside.

The config writer is straightforward: os.Create(path) truncates the file, then writes the new JSON. During one deploy the process is killed mid-write. The file now exists with 40 bytes of a 900-byte document, the truncation already happened, and the service refuses to boot. Atomicity was never considered because the code writes to a file that other processes read — which is exactly the situation where it must be.

The pattern that fixes it: write a sibling temp file, close it, then os.Rename it over the target. Rename within a directory is atomic on POSIX filesystems — readers see the complete old file or the complete new one, never a mixture.

func writeConfig(path string, cfg Config) (err error) {
	data, err := json.MarshalIndent(cfg, "", "  ")
	if err != nil {
		return fmt.Errorf("marshal config: %w", err)
	}

	dir := filepath.Dir(path)
	tmp, err := os.CreateTemp(dir, ".config-*.tmp")
	if err != nil {
		return fmt.Errorf("temp file: %w", err)
	}
	tmpName := tmp.Name()
	defer func() { // clean up on every failure path
		if err != nil {
			os.Remove(tmpName)
		}
	}()

	if _, err = tmp.Write(data); err != nil {
		return fmt.Errorf("write temp: %w", err)
	}
	if err = tmp.Chmod(0o600); err != nil {
		return fmt.Errorf("chmod temp: %w", err)
	}
	if err = tmp.Close(); err != nil {
		return fmt.Errorf("close temp: %w", err)
	}
	if err = os.Rename(tmpName, path); err != nil {
		return fmt.Errorf("rename into place: %w", err)
	}
	return nil
}

Walk the error paths. CreateTemp puts the temp file in the same directory as the target, which is not an aesthetic choice — a rename across filesystems (/tmp to /data) fails with EXDEV, so the temp file must share the target's mount. The deferred cleanup removes the temp file only when an error was recorded, leaving no debris after success. And the rename itself is the commit point; everything before it touches only private state.

Permissions belong in the same thought. os.Create inherits the process umask (usually 0644), which is fine for a report and wrong for a file holding a token. Secret-bearing files want 0600, directories 0700, and the choice should be explicit in the code that writes them — the concern resurfaces in Configuration and Secrets Handling.

The error-handling rule ties it together: a write failure at close time is a data-loss failure. defer f.Close() discards it, so close errors are folded into the named result — the same technique the cleanup section of The error Interface and Error Messages demonstrates. Files, unlike in-memory values, fail at moments your happy path has already passed: on flush, on close, on rename. Design for those moments once and every file writer you write afterward inherits the discipline.