Testing

Go tests guard the rules the design rests on; throwaway stacks, browser runs and real media catch what unit tests cannot. And one lesson recurs often enough to be the theme of this page: a check that cannot see the failure it exists for is worse than no check, because it is believed.

Go tests, in a container

go test ./... covers the server. On the development PC it runs in a container:

docker run --rm -v "//c/path/to/soundstorm:/src" -w /src golang:1.27-alpine go test ./...

Windows' Smart App Control refuses to run freshly built, unsigned test binaries - a different handful of packages on every rebuild - and it cannot simply be turned off: disabling it is one-way, with no exclusion list and no way back short of a reset. A Linux container runs no Windows binary at all, and is what CI does anyway. (The doubled slash stops Git Bash rewriting /src into a Windows path.)

Some tests worth knowing about, because they protect the rules rather than a function:

internal/federate and internal/httpapi: a dead or panicking backend never takes the search down, and the answer is always 200 with degraded.
TestPagesWalkTheMergedOrderExactly and each adapter's paging_test.go: walking every page reproduces the single sorted list, against fakes that order the way each backend really does - scrambled, case-blind, "The"-stripped.
libraries_test.go: every endpoint that can hand over bytes, metadata or a playable URL refuses an account restricted from that shelf.
TestOwnerSettingsAnswer: every owner setting is mounted as well as registered, after one answered 404 unnoticed.
TestShellHasNoInlineScript and pwa_test.go: the CSP and the service worker's rules hold.
internal/starter/integrity_test.go: every bundled EPUB opens and every MP3's frame chain walks end to end.
The uninstaller test: a marker file in library/music is still readable afterwards.

The project keeps meeting the same trap in different forms, and each one is recorded so the next test is written to fail first.

The check	What it could not see
PWA checks ran on `localhost` or with certificate errors ignored	Both are secure contexts whatever the certificate. A real phone could not install the app from a self-signed LAN address at all, while the check reported a registered service worker - truthfully.
`TestASlowBodyIsCutOff` built the body deadline on its own	Behind the real logging middleware, which lacked `Unwrap`, no read deadline had ever worked. The new test goes through `Routes()` and was checked to fail with `Unwrap` removed.
Resolving `<computer>.local` on the machine itself	Windows answers for its own name regardless. It proved nothing about whether a phone could resolve it, and failed on the first other PC.
Reading the TLS connection's local address in a unit test	Inside Docker it is the container's address, never the one the client dialled. Caught only by fetching the LAN address from a client that trusted nothing else.
A PowerShell syntax check that discarded the error collection	Every file "parsed clean", including one that would not parse at all.
A CI guard using `grep -q "\r"`	GNU grep read `\r` as the letter r, so it passed on any file containing an r - every file - for two months.
Counting reader `relocate` events	They fire on resize too. A tap that turned no page passed; the fraction has to change, and change back.
`cmd /c script.cmd`	Skips the shell's reputation check, so Smart App Control's block on a downloaded setup file never showed. A double-click does; `Start-Process` reproduces it.
Length and magic number of committed binaries	Carriage returns had been stripped from the starter media and screenshots; files were the right size and began correctly, and nine of ten ebooks could not be opened.

The rules that came out of it: a check that runs on the machine under test cannot answer a question about what another machine sees, and a committed binary needs a check that reads it the way its consumer will. Where it matters, a checker is itself checked against damage - the integrity test strips carriage returns from a bundled MP3 and requires the walk to object.

Throwaway stacks

Most backend behaviour is verified against a real server, not a fake: a disposable compose stack that SoundStorm provisions by itself, with generated media - a two-minute film, a two-season show, a library with ReplayGain tags of 0, -5 and +8 dB. The notes record versions: Jellyfin 12.1.0, Audiobookshelf 2.36.1, Navidrome 0.64.1, Immich 3.2.2, AudioMuse-AI 3.6.3.

A project name (-p) is not enough for a second stack. The compose file pins container_name on every service, which a project name does not namespace, so a second stack collides by name and refuses to start. An override file renaming every container is what makes an isolated stack possible.

One habit from Jellyfin applies everywhere: it ignores unknown query parameters with a 200, so test a filter by asking for the opposite and checking the count actually changes. Navidrome ignores unknown environment variables the same way.

Browser checks

Playwright drives the system Chrome (no browser download). scripts/mobile-check.js measures phone layouts and asserts document.scrollWidth <= clientWidth per screen, listing anything that sticks out - which found a grid item refusing to shrink and a 531 px page on a 390 px phone. scripts/reader-swipe-check.js proves swiping turns reader pages exactly as the arrows do.

Techniques that made otherwise invisible bugs visible:

Chrome's Emulation.setSafeAreaInsetsOverride gives a real safe-area inset; without it every inset is 0 and a close button under the camera cannot be seen.
Polling currentTime every 4 ms to measure gapless playback, because timeupdate's ~250 ms floor first reported ~270 ms gaps that were the event, not the gap (real: 11-23 ms).
Stubbing canPlayType to reject HLS, since recent Chrome plays it natively and the hls.js fallback otherwise never runs.
CPU throttled 4-6x and 250 ms on every request to measure swipe frames and heap drops; visualViewport.width to catch a page wider than the phone mid-swipe.
Driving the Android web view through its DevTools socket, because adb shell input does not reliably reach a web page.
XCUITest for the iPhone app, since simctl cannot tap or type.

A synthetic DataTransfer gets no filesystem entries and no automated drag can produce real ones, so the folder-walking code (readEntries until an empty batch) is tested with a fake entry tree.

Real media

scripts/fetch-test-library.ps1 builds about 750 MB from Project Gutenberg, LibriVox, the Internet Archive and the Blender open movies - public domain or CC, resumable, throttled. The first run against real files found three bugs that thirteen generated files could not, because every generated file has exactly the metadata somebody chose to write.

The same thinking runs through the parsers: ID3 unsynchronization, syncsafe sizes in 2.4, the full-atom meta in MP4 and XMP's rdf:Alt wrappers were all found on real files. Duplicate detection was measured on a real iTunes library before it was built (of nine repurchased pairs, none were identical files and five had identical audio).

These are donated services. Project Gutenberg states that automated access to its website earns an IP block, so the script uses its sanctioned mirror at two-second intervals. Check the equivalent policy before adding any source.

Parity and rehearsals

ACME against Pebble. scripts/acme-rehearsal.sh runs the project's own ACME client against Let's Encrypt's test authority, with pebble-challtestsrv standing in for the DNS provider, in CI. A fake authority written beside the client would share its mistakes; Pebble does not. It found that at a 50% nonce-refusal rate, five retries failed one run in six, so it is ten.
Beats parity. scripts/beats-parity.js runs the web app's real hearSong under Node on the Go tests' samples: 120 beats each, 0 ms apart.
Moving computers was rehearsed in all four directions (Linux and Windows to each), finding CRLF manifests, names Windows cannot hold and a drive-root quoting bug.

CI

Before any image is published, publish.yml requires:

gofmt clean, go vet ./... and go test ./...;
the certificate rehearsal against Pebble;
the Tailscale serve config points at the soundstorm-app alias, not soundstorm (which inside the sidecar resolves to itself);
compose valid with and without the Tailscale profile;
every committed PNG verified against its chunk CRCs (check-images.py);
SoundStorm-Setup.cmd still has CRLF on every line - counted against the line count, which also catches half a file converted.

How much to check

One real check per change, then commit. Long comparison runs and mutation testing are not done by default; instead each change says what was verified and what was not ("not yet run on a real phone", "the hold menu needs a remote"). That keeps the record honest without slowing every change down to the pace of the slowest possible proof.