Testing

Soak testing a protocol stack: what a 30-day run actually catches

2026-10-29 · 7 min read

A conformance suite runs for an hour and tells you whether behaviour is correct. It cannot tell you whether your process grows by 40 kB an hour, whether the association survives a weekend, or whether the sequence numbers do something unpleasant after four billion events. Those need time, and time needs a test that is designed rather than merely long.

What actually fails in a long run

• Memory that grows only on a slow path — an error branch, a reconnect, a file transfer — so it looks stable for days and then does not.
• Handle and descriptor leaks: sockets in a state that only shows up under repeated reconnect.
• Counter overflow at a boundary nobody tested, producing a wrong sequence number or a negative duration.
• Timing drift: an implementation that keeps time by accumulating small intervals slowly diverges from the reference clock.
• Log files and databases that grow without bound until the disk fills — a self-inflicted outage.

Design the run before you start it

A soak test that just repeats the happy path for 30 days finds almost nothing. It needs load variation and failure injection:

• A baseline load that is realistic, not maximal — say 60% of the measured capacity.
• Periodic bursts above the baseline, because queue behaviour under overload is where defects live.
• Scheduled failures: peer restarts, cable pulls, and a process kill on both ends, at increasing intervals.
• Time-source loss and recovery, since many implementations only synchronise at startup.
• A file transfer or other bulk operation mixed in, because it changes the memory profile.

What to record, and how often

• Resident memory and handle count, every five minutes, on both ends. A single number per day hides the pattern.
• All protocol counters: associations, reconnects, reports sent and received, buffer overflows, controls, errors. Export them, do not just display them.
• Clock offset against a reference, so drift is a trend line rather than an incident.
• Response latency percentiles, not averages. The average hides the tail that operators actually experience.
• A daily canary transaction — one read, one control refused, one file listing — whose result must stay identical for the whole run.

Reading the result

Our published figure for the 104 package is 7×24 operation with zero memory growth, and that claim is only credible because it is a measured slope, not an impression. When you read a vendor claim, ask for the slope: memory over time, with the workload described. Any vendor who has actually run one will have the graph.

Questions that come up

How long is long enough?

Long enough to cross every periodic boundary in the implementation — a counter wrap, a weekly timer, a scheduled task — and long enough that a slow leak becomes visible. Thirty days with injected failures finds most of what a month in the field would.

Can we shorten it?

Sometimes: increase the event rate to age the counters faster, and run several failure-injection scenarios in parallel on separate benches. What you cannot shorten is the memory trend, which needs elapsed time.

What is the single most useful metric?

Memory slope on both ends under a constant workload. It is the cheapest early warning of a defect class that is expensive to find later.

Keep reading