countinghouse

Module sandbox, part 2 — file broker and virtual paths

The second of four sub-projects that harden how countinghouse runs module code (part 1: docs/superpowers/specs/2026-10-15-module-sandbox-foundation-design.md). Part 2 puts every file access a module makes behind the main thread, gives each module its own file namespace, and closes the routes part 1 left around the require boundary. It ships as 8.0.0, together with --workerThread becoming the default and a Node floor of 24.15 (amended 2026-11-04; was 22.16).

Designed with the user on 2026-10-17, and reviewed section by section by a second Claude Code session acting as reviewer. Every decision below was either made by the user or agreed between author and reviewer. The user delegated approval of this text to the reviewer’s agreement. Verified against master at fd6daee. The spikes behind the claims marked (spiked) ran on Node 24.21 in throwaway code; their results are summarised in Spike results.

Amended 2026-11-04 (plan 5 Task 6, approved by the user): the spec now says what plans 3-5 built. Each changed passage is marked (amended 2026-11-04).

The problem

Part 1 gave each module’s worker a lockdown, limits and a policy, but no file permission: docs/security-model.md says “a module can read and write what the process user can”. Under any mode a module can read auth.json (every tenant’s keys), other modules’ code and data, and anything else the service user can reach. It can also write over them. Part 1 also left routes through countinghouse’s own loader: module.parent.require(...), require.cache[...].require(...), Module._load(...) and the vm.Script dynamic-import() referrer reach any file even if fs were brokered.

Decisions

Made by the user during design:

  1. Sync fs calls go over a blocking bridge. The worker posts a request and waits on shared memory; the main thread does the I/O asynchronously and wakes it. The main thread never blocks.
  2. All of fs is brokered except watching. Path calls, descriptor calls and streams, in sync, callback and promise forms. fs.watch, fs.watchFile and fs.promises.watch answer ERR_FEATURE_UNAVAILABLE_ON_PLATFORM.
  3. A per-module disk quota on /data and /tmp, enforced by the broker.
  4. /app (the module’s code) is read-only under enforce, and writable but logged under audit.
  5. Architecture: patch fs in place in the worker, with two transports (sync bridge, async messages) and one broker per worker on the main thread.
  6. Modules see the same virtual paths in every mode (off, audit, enforce, and single-thread mode), with the single-thread differences listed in section 6. --moduleSandbox off turns off enforcement and logging, not the broker.
  7. Single-thread mode is debug-only; modules still see virtual paths there, through a lighter mechanism (section 6). Identity stays the host path in that mode (option “a” of the collision question).
  8. 8.0.0: Node >= 24.15 (amended 2026-11-04, user decision: Node 22 dropped; see section 5); --workerThread becomes the default (its own change); the part 1 spec is amended where it promised “off is today’s behaviour” and “no new hop”.
  9. Link rules apply in every --moduleSandbox mode (off, audit, enforce; section 3). Single-thread mode has none.
  10. A load whose request name differs from its package.json name is refused in every mode (MODULE_NAME_MISMATCH).
  11. Uninstall is a new admin route that deletes a module’s data unless told to keep it.
  12. Countinghouse installed under /app is refused under --workerThread (amendment, 2026-10-17, found while planning part 2’s plan 3). Modules see their files as /app/..., and Node keys loaded code (require.cache, the ES module map) by that path, so a countinghouse whose own code lives under /app would share entries with its modules, and a module could be handed countinghouse’s own module objects. No hook can separate them. The start-up check uses the install directory’s real path and exits 2 with advice (install elsewhere, e.g. /opt/countinghouse, or use --no-workerThread); a worker also refuses a load if any code it has loaded lives under /app. Single-thread mode, and modules or data under /app, are unaffected. The user chose refusal over renaming the virtual root.
  13. Earlier, part 1 design: out-of-namespace access is ENOENT to the module plus a host log line; outside paths are declared r/rw in countinghouse.permissions.fs and narrowed by the operator; modules stay standard npm packages.

What this can and cannot promise

Only enforce protects files. Under audit (the default) and off, access outside the namespace passes through to the host, so auth.json and other modules’ files stay readable; those modes give modules virtual paths and give operators a report, not protection.

Under enforce, module code cannot read or write a host file outside its install directory (read-only), its data and temp directories, and the paths it declared and the operator approved, by the routes this design knows of; the known exceptions are listed in Gaps. As in part 1, everything still runs in one process with countinghouse: an overlooked internal API or a V8 bug gets past it, and module code can reach countinghouse’s objects (part 1’s R3 gap: CHUtil.ci, the worker’s channel to the main thread) and countinghouse’s own code files. Network access stays open until parts 3 and 4. Host processes writing inside a data or declared path at the same moment can race the broker’s checks. This is a strong barrier, not a hard boundary against code written to defeat it.

1. The virtual namespace

What a module sees, in every mode, under --workerThread:

Virtual path Host path Access
/app the module’s install directory read-only under enforce (EROFS); writes allowed and logged under audit, allowed under off
/data <moduleDataPath>/<name>/data read-write, quota, kept across restarts
/tmp <moduleDataPath>/<name>/tmp read-write, quota, emptied when the worker starts
each declared path, e.g. /var/log/app the same host path r or rw as narrowed by the policy; no quota

2. The worker side

Patching fs first

The lockdown configuration travels in workerData at spawn. The first statements of lib/sandbox.js capture what the bridge needs (MessagePort, receiveMessageOnPort, Atomics), then replace every method of the shared fs and fs/promises objects and of FileHandle.prototype, before any countinghouse or dependency code loads. Each replacement is locked (non-writable, non-configurable); the originals live only in lockdown’s closure. Nothing in the worker then holds an unbrokered fs except Node’s internals and lockdown itself. Part 1’s remaining lockdown still installs at set-options.

Fail closed. Every key of fs, fs.promises and FileHandle.prototype is either handled by the broker or replaced by a function that throws. A test fails when the running Node has a key the list does not account for (the same pattern as part 1’s built-in allowlist). Keys that are easy to miss and are handled: realpathSync.native/realpath.native, cp/cpSync, glob/globSync, opendir/Dir, openAsBlob, statfs, readv/writev, exists/existsSync, FileHandle#readLines/readableWebStream/ createReadStream/createWriteStream, and the fs and flush options of the stream constructors. Stats, BigIntStats and Dirent results are real instances rebuilt in the worker. File streams look up fs methods at call time and so go through the patch (spiked).

The bridge

Two MessageChannels per worker: one for sync calls, one for async calls, so a sync wait never consumes an async reply. Plus a 4-byte SharedArrayBuffer holding a sequence number.

The loader

Node’s loader calls the public fs at call time (realpathSync, readFileSync, openSync/readSync). The hooks decide that traffic:

No privileged requests (A4). Module code can reach the worker’s channels (part 1, R3), so the broker gives every request the module’s rights. Countinghouse’s worker code therefore speaks virtual paths: load-module carries /app, and package.json, api.json, schema.json and the handlers are read through /app. Anything it needs from outside the namespace (device config and the like) comes from the main thread in a message. The plan begins with a survey of every post-lockdown fs, cwd and tmpdir use in lib/.

Other routes to files

Route Treatment
process.loadEnvFile(path) brokered
module.enableCompileCache(dir) locked
process.report.writeReport, report.directory/filename locked
process.setSourceMapsEnabled (the loader reads sourceMappingURL paths) locked
repl, trace_events removed from the built-in allowlist
net.Socket({fd}), net.Socket({handle}), tty.ReadStream/WriteStream(fd), server.listen({fd}) and server.listen(handle) on net, http, https and http2 servers, and tls.TLSSocket wrapping any of them the fd/handle option is refused under enforce, logged under audit. In the spike a worker adopted the main thread’s real listening socket with net.Socket({fd}), and with listen({fd}) on net and http servers took over the main server’s connections (spiked)
HTTP/2 respondWithFD(fd) (amended 2026-11-04) refused under enforce, logged under audit (it sends a real descriptor’s contents natively)
HTTP/2 respondWithFile(path) (amended 2026-11-04) not available to modules in any mode (ERR_FEATURE_UNAVAILABLE_ON_PLATFORM): it hands the brokered descriptor to native code, which cannot read it (user decision)
Unix-socket paths (net.connect(path), http.request({socketPath})) a gap until part 4

(amended 2026-11-04) Plan 3’s sweep of the allowlisted built-ins found three routes: respondWithFD and respondWithFile (above) and the loader’s own lookups (section 2, “The loader”). dns, tls, crypto, zlib, vm, dgram (whose bind({fd}) was already locked), tty, net and the rest needed nothing further.

3. The broker

One FsBroker per module worker, created at spawn. Its inputs: the verified package name, the mode, the host install root, the data directory, the effective fs declarations, and the quota.

Per request:

  1. Translate virtual to host: longest root match (/app, /data, /tmp, declared); relative paths against /app; synthesised ancestors as in section 1.
  2. Decide by mode: outside the namespace, /app writes and writes to r paths, as in section 1.
  3. Check on the host: lstat each component and keep the leaf’s device+inode. The real target must lie inside its root or a declared path, compared against roots that were realpath’d at spawn. Under enforce anything else is ENOENT; under audit it is logged.
  4. Run the operation with Node’s async fs.
  5. Verify after open: the descriptor’s fstat device+inode must equal the leaf identity from step 3; on Linux also readlink /proc/self/fd/N. For creation, re-lstat the parent chain. On a mismatch, close and answer ENOENT.

(amended 2026-11-04) Three optimisations (user decision, plan 5 Task 4b; commits b977146, 60ca7e8, 9246db1):

Per-module read/write lock. Mutating operations (write, rename, unlink, mkdir, rm, link, symlink, chmod, truncate, utimes…) take it exclusively; every path-resolving operation, reads and opens included, takes it shared. A module cannot race itself.

Link rules, in every --moduleSandbox mode: link() only within one writable root, and counted where the quota is counted; symlink() only with a relative target that resolves inside the same root; absolute targets are EPERM. (Amended 2026-11-09, FB1:) the target may hold no .. segment, and is decided by where it really leads — walked from the real path of the link’s directory, following links on the way — not by its text; text alone let x -> "." then y -> "x/.." lead to the root’s parent, and a .. that stays inside at creation can lead out after a rename. chmod and fchmod strip setuid, setgid and sticky bits. Under enforce, the spawn walk refuses to start a module whose /data or /tmp holds a link that can lead outside (amended 2026-11-04): a symlink with an absolute target, or a hard link to a file that also has a name outside /data and /tmp. A hard link the module made inside them, which the link rules allow, is fine. The log names the entries. The operator clears that by removing or replacing them under <moduleDataPath>/<name>/, then POST /restart-module.

Results back to virtual: realpath, mkdtemp, Dirent.parentPath/path, Dir.path, glob results, and err.path, err.dest and the message. Cross-root rename answers EXDEV, so copy fallbacks work.

Descriptor table: at most 256 per worker, numbered from 3 in the broker, operations on one descriptor serialised, all closed when the worker exits. (amended 2026-11-04) Module code sees 0x10000000 + n for broker descriptor n, so a brokered descriptor is never mistaken for a real one.

Quota:

Main-thread protection, in every mode:

Worker end (restart, crash, unload): operations in flight reject EIO, descriptors close, the broker is discarded. A per-name lifecycle lock makes the next spawn wait for the old broker’s operations to settle before it empties /tmp and walks, and keeps uninstall’s deletion and a load of the same name apart. (amended 2026-11-04) The wait is bounded at 35 s (the bridge’s 30 s deadline plus a margin); past it the next spawn goes ahead and logs what was still running.

4. Modes

  off audit enforce
Virtual paths, broker, bridge yes yes yes
Link rules, main-thread protection yes yes yes
Namespace refusals, /app and r writes pass through pass through, logged refused
Quota not counted logged enforced
Part 1 lockdown, limits, clean env no yes yes

The plan sweeps part 1’s options.moduleSandbox !== 'off' and sandbox != null branches for anything the broker now relies on in off (the heartbeat exemption, RATE_EXEMPT), and keeps the clean worker environment for audit and enforce only.

5. 8.0.0

  1. Node >= 24.15 (amended 2026-11-04; was 22.16). package.json engines, the CI matrix (24.15.0, the floor itself, and the newest 24.x), and CLAUDE.md’s target line. Part 1’s lib/sandbox-support.js fallback is removed. Why 24.15: on Node 22 and on 24.x before 24.15, module.registerHooks resolve hooks are not called for require.resolve or createRequire().resolve, so module code’s require.resolve reaches Node’s resolver with its virtual /app path and fails (found by the first CI run on 22.23.4; bisected on each minor’s latest patch: 24.0.0, 24.10.0, 24.12.0, 24.13.1 and 24.14.1 skip the hook, 24.15.0, 24.21.0 and 24.22.2 call it). The user chose dropping Node 22 over a Node-22-only require.resolve shim.
  2. --workerThread is the default (its own commit). --no-workerThread selects single-thread mode; --workerThread false and --workerThread=false, which minimist passes as the string "false" (spiked), are coerced to false. The worker’s first setOptions call passes {workerThread: false} explicitly, and the plan sweeps every options.workerThread read before set-options, the authoring tools’ child process, bin/countinghouse and the test helpers.
  3. The test-suite migration is its own task: each file is either pinned to single-thread semantics with --no-workerThread or moved to the default; part 1’s tests about off and the defaults (01-modes, the 07 off cases, 10-node-support) are rewritten.
  4. The CHANGELOG has an upgrade section: write under /data or /tmp; declare outside paths; __dirname is /app/... and the current directory is /app; the link rules; what off now means; the measured cost of the broker under off and enforce; and a downgrade note (7.x does not see data under --moduleDataPath). It also tells operators to check, before upgrading, for registry entries, load_module calls and composite runsModules bindings whose name differs from the module’s package.json name, which 8.0.0 refuses, and for countinghouse installed under /app, which 8.0.0 refuses under --workerThread (decision 12).
  5. The part 1 spec is amended: off runs the broker, and part 2 adds a hop to every file call; and a load whose request name differs from its package.json name is refused in every mode (the user approved all three).

6. Single-thread mode

Debug-only: starting modules without --workerThread logs a warning that this mode has no sandbox and is for debugging modules. Modules still see virtual paths:

7. Uninstall

POST /uninstall-module {name, keepData}, admin key required. It:

It never removes the module’s code under --modulePath; the docs say so.

Gaps that remain

Each goes into docs/security-model.md and the CHANGELOG:

Error codes and flags

New flag: --moduleDataPath. New limits: diskMb and diskEntries. New error codes: MODULE_NAME_MISMATCH, and for uninstall (amended 2026-11-04) MODULE_NAME_INVALID, MODULE_LOADED, MODULE_DATA_NOT_DIRECTORY and REMOVE_MODULE_INFO_FAIL. New route: POST /uninstall-module. Module-facing errors are Node’s own: ENOENT, EROFS, EPERM, ENOSPC, EMFILE, ETIMEDOUT, EIO, EXDEV, EBADF, ERR_FEATURE_UNAVAILABLE_ON_PLATFORM. Registry errors for uninstall follow the existing error-info files.

Docs

docs/security-model.md (layers, modes, gaps, the systemd example with --moduleDataPath in ReadWritePaths), docs/module-development.md (the namespace, the upgrade guide, single-thread differences), docs/cli-and-api-reference.md (flags, limits, uninstall), the CHANGELOG upgrade section, and the part 1 spec amendment.

Testing

Spike results

Throwaway code, Node 24.21, a 1-CPU VM, 2026-10-17.

Out of scope

fs.watch and friends; network brokering (parts 3, 4); the main thread’s trust in worker-stated identity (deferred by the user); the part 1 deferred items (keep the last good policy; tools briefly missing during a restart; a redirect-guard test). Part 1’s M7 (async tree walk) becomes required here for the quota walk.