Security & Response

NDR Rollout

The value was never inbound attack detection. It was being able to prove whether data left.

Designed, deployed, and verifiedSuricataWazuhsystemdJA3 / SNIeBPFGLBA / state breach law

The vendor's stock severity mapping could never have paged anyone. Found by testing it before installing anything.
Four staged deployments, and the one host left out is a written decision rather than an omission.
Six verification tests, each designed so that it could fail. One did, immediately.

The problem

“The SIEM supports this sensor natively” is true and misleading. Before installing anything, I tested it.

The shipped integration rules mapped the sensor's alerts to severity levels of 0, 0, 0, 0 and 3. The alerting floor was 10.

A sensor installed with stock rules would have been ingested, indexed, rendered on the dashboard, and structurally incapable of paging, of opening a case, or of reaching containment. The severity mapping shipped before any sensor did.

What a host-local sensor actually buys, stated honestly

It is not attack detection. Inbound web traffic arrives TLS-encrypted from a CDN edge and the sensor cannot decrypt it, while the web server access logs already carry the true client IP. Roughly seventy percent of what “add an IDS” implies was already covered better elsewhere.

The value is egress, and egress did not exist anywhere on this fleet:

Stock severity mapping versus the alerting floor of 10
Alert class 1	severity 0
Alert class 2	severity 0
Alert class 3	severity 0
Alert class 4	severity 0
Alert class 5	severity 3
Alerting floor	severity 10

The integration's shipped severity mapping, measured before anything was installed. Four alert classes at zero, one at three, against a floor of ten. It would have ingested, indexed and rendered beautifully, and paged nobody, for as long as I let it.

DNS
Every host resolved over cleartext UDP with DNS-over-TLS off. Nothing could tell you what any host had ever looked up.
Outbound TLS destination and fingerprint
The payload stays encrypted, but where a host is talking becomes visible. A compromised application or database process calling home was invisible to every control in place.
Flow byte counts
Under GLBA Safeguards and state breach law, whether data left is the determination required in the first 72 hours. There was no artifact from which to make it.

Where it went, and where it deliberately did not

Four stages: the disposable host first as a proving ground, then the two regulated tenants that were the reason for the project, then the highest-volume host last, because its inbound was already better instrumented than a sensor could make it, so its marginal value was materially lower.

One host was excluded, and the exclusion is a written decision record rather than an omission: dozens of third-party accounts in custody, a policy forbidding state changes, no capture tooling on the box at all so a sensor means new packages, and zero new flows measured in thirty seconds. It would have accepted the fleet's highest policy risk in order to instrument its quietest interface. The record names the compensating controls and what would change the decision.

The control that makes this safe on a box holding PII

A hard memory ceiling and an OOM priority, as a service drop-in, not a configuration hope. Untuned with a full ruleset the sensor runs 800MB to 1.2GB resident, on a box with about 2GB available that also runs a portal and its database.

Under memory pressure the sensor must die, never the database. A sensor that can take down the thing it protects is worse than no sensor.

Two output streams, also deliberate: full metadata stays host-local as the hunting corpus and the exfiltration evidence; only alerts are shipped to the SIEM. The busiest host alone would otherwise produce around 950,000 events a day. Three and a half times the entire existing SIEM volume.

Verification. Each test designed so that it can fail

Packets as a rate, not a state
Two samples twenty seconds apart: strictly increasing, drop ratio under 0.001. “Is the service active” is explicitly not the test. An orphaned firewall chain that passed exactly that check is why.
Capture, decode, log
A lookup carrying a freshly generated random nonce must appear in the event log within seconds. A random nonce makes a stale or replayed pass impossible.
The detection engine fires, and this test failed first while the sensor was fine
The assertion looked for the nonce inside the alert record, but alerts are configured without payload or metadata, so the query name only ever appears in the protocol record. The test was structurally impossible against its own configuration. Corrected to join alert to protocol record on flow ID.
A negative test
Prove the containment lane is not wired: zero verdicts, zero cases, zero queue entries. Structural rather than lucky. The two lanes key on different field names, confirmed by reading the stored document rather than by inference.
Production unharmed
All application containers healthy throughout, sensor resident memory measured against its cap, host memory and swap sampled before and after.

Engineering decisions

The calls I made, and what each one cost.

Ship the severity mapping before the sensor.

A detection pipeline that cannot page is not a detection pipeline. Testing the vendor integration first turned a six-month silent failure into a config file written on day zero.

Write down the host you did not instrument.

A known-and-unaddressed gap reads very differently from one that was measured, compensated, and decided on the record before an incident. The record names what would change the decision.

Cap the sensor's memory as a unit drop-in, not a config value.

A configuration value is a request. A cgroup limit is a guarantee, and on a two-gigabyte box holding tax records the difference is whether Postgres survives a ruleset update.

More systems

Reliability & Automation

I'm looking for Incident Commander and SRE roles.

If your team is drowning in toil, alert noise, or incidents that never quite close. That is the work I do.

Get in touch