From Node + JSON + REST to Go + Protobuf + gRPC - the parsing pipeline rewrite

The Controller rewrite in Go last February was a single agent.
The lessons learned there - that Claude could carry a Node service across to Go with feature parity over a handful of evenings, and that the runtime gains were real - made one thing obvious: the parsing pipeline was next.

This release ships the result:

• Seven services migrated.
• Three serialization formats swapped from JSON to Protobuf.
• Three hot paths swapped from REST to gRPC.

info

We divided the CPU usage by 2, and the memory usage by 4, to achieve the same performance with half the resources.

What was rewritten

The Spider parsing pipeline is the part of the back office that ingests captured packets, reconstructs TCP sessions, and decodes them into HTTP and PostgreSQL communications.
It runs on every byte the Whisperers send back.
Its CPU and Redis footprint show up directly in cluster bills.

The pre-migration baseline was straightforward:

• Node.js Koa services exchanging JSON over REST, storing JSON payloads in Redis.
• The hot path between parsers and the packet store (Web-Write → Pack-Read) was a REST call returning a JSON array of packets per TCP session.

Seven services were rewritten in Go:

Service	Role
Pack-Write	Stores raw packets from Whisperers into Redis
Pack-Read	Returns packets to parsers, by TCP session
Tcp-Write	Stores TCP session creations / updates
Tcp-Update	Updates TCP session state and parsing logs
Web-Write	Parses HTTP communications from TCP sessions
Pg-Parser	Parses PostgreSQL communications from TCP sessions
Tls-Keys-Linker	Matches captured TLS master secrets to TCP sessions

Three serialization formats moved from JSON to Protobuf:

• Packets - the largest by volume, stored in redis-pack
• TCP sessions - the bookkeeping records that tie packets together
• HTTP communications and parsing logs - the decoded output

Two agents-service paths swapped from REST+JSON to REST+Protobuf:

• Whisperer ↔ Pack-Write - save packets for parsing
• Whisperer ↔ Tcp-Write - save TCP session state to trigger parsing

Two inter-service hot paths swapped from REST+JSON to gRPC:

• Web-Write ↔ Pack-Read - fetch packets to parse
• Web-Write / Pg-Parser ↔ Tcp-Update - fetch parsing jobs and patch session state

The legacy Node.js versions of these services are still in the repo, kept around for reference and emergency rollback, but no longer deployed.

note

On the inter-service communications:

• Go + JSON is slower than Node+JSON but uses less CPU
• Go + GRPC on one side and Node+GRPC on the other is slower than Node+JSON equivalent CPU
• Go + GRPC on both sides is fastest, with half the CPU usage

Why this combination

The Controller rewrite already validated Go as the language and Claude as the co-author.
Two questions remained:

Why Protobuf?
Packets are the densest cargo in the system. They sit in redis-pack between capture and parsing - sometimes for seconds, sometimes for minutes if a parser falls behind. Each packet stored as JSON carried tens of percent of overhead in key names, string-encoded byte arrays, and whitespace. Protobuf strips that down to the bare bytes plus a few field tags.

Why gRPC for two specific hops?
Pack-Read is called by parsers tens of thousands of times per minute. The wire savings from Protobuf compound when paired with HTTP/2 multiplexing and a connection that stays warm. The other inter-service calls (config polling, status pushes, management APIs) remain on REST + JSON: their volume does not justify the migration cost, and JSON's debuggability is genuinely useful at low volume.

Workflow

The pattern that worked on the Controller worked again here, with one refinement.

Each service rewrite followed the same loop:

1. Claude read the Node.js implementation and wrote an architecture document
2. A phased plan was produced, with explicit feature checklists
3. Implementation in Go, deployed locally, tested against the e2e suite
4. Deployed to a real cluster, monitored, and iterated on what pprof flagged

The refinement: protobuf schemas were defined once in a shared proto/ repository, generated for both Go (the new services) and JavaScript (the remaining Node consumers, and the back office tools that still need to read these records). This kept the legacy Node code on the network for the duration of the migration - one service could be swapped at a time without locking the others into a flag day.

The gRPC services were written behind the same authentication and circuit-breaker primitives the Spider stack already uses for REST: RS256 JWT, gobreaker-based per-target breakers, the same observability hooks feeding the monitoring. From the operator's point of view, a gRPC call surfaces in the dashboards exactly like a REST call.

Performance results

The reference comparison was a one-hour window on the sss-dev cluster, with spider-mon status snapshots before and after, against a Google-doc baseline captured on the pre-migration build.

Headline deltas

Load of cluster (night run)

Type	Avg/min
Packets	135,941
Tcp	15,251
Http	28,920
Psql	1,170

Metrics

Metric	Baseline (Node/JSON/REST)	After (Go/gRPC/Protobuf)	Delta
redis-pack memory	330 MB	80 MB	≈ −80%
Web-Write → Pack-Read latency (avg)	20 ms	13.4 ms	−33%
Web-Write CPU (sum of replicas)	105% (4 × 26%)	~39% (≈3 × 13%)	≈ −2.7×
Pack-Read CPU (sum)	39% (2 × 20%)	~12% (2 × 6%)	≈ −3×
Pack-Read RAM (per replica, real RSS)	~121 MB	~32 MB	≈ −74%
Pack-Write CPU (sum)	31% (2 × 16%)	~11% (2 × 5.3%)	−65%
Pack-Read Redis "get packets" latency	3 ms	2.3 ms	−23%
Inter-service / Redis call errors	~0	~0	flat
Whisperer post_packets latency	8 ms (p90)	7.2 ms (max 11)	flat

Where the wins came from

The three biggest wins came from three different parts of the change:

Protobuf in redis-pack carried the largest single number.

• The packet store dropped from ~335 MB to ~72 MB - about 78% less memory in Redis.
• The Redis host's own CPU dropped to ~7.5%, because every fetch moves fewer bytes and decodes faster.
• Garbage collection pressure on the Go services reading those packets also dropped: protobuf decodes into pre-shaped structs instead of allocating a parsed-JSON object graph.

gRPC on the Web-Write ↔ Pack-Read hop

• Replaced a REST call averaging 20 ms (34 ms p90) with a gRPC call averaging 13.4 ms client-side - and just 1.7 ms server-side.
• The remaining ~11.7 ms client-side over a ~2 ms handler is network + protobuf decode + the Go client's per-call overhead under load.
• A −33% average and roughly flat tail.

The same story applies to Tcp-Update: GetParsingJob averages 4.6 ms, PatchSessions 4.4 ms, with server-side handler times of 0.8 / 1.2 ms.

Node → Go gave the steady-state CPU drop.

• Web-Write went from 4 replicas × 26% (sum 105%) to roughly 3 × 13% (sum ~39%) - about 2.7× less CPU at similar HTTP throughput, with one fewer replica.
• Pack-Read fell ~3×,
• Pack-Write ~65%.
• The variance on the graphs also fell: Go's CPU curves are flatter than Node's, with fewer spikes.

What did not change

The Whisperer-facing endpoints (POST /pack-write/packets) stayed on REST, but moved to Protobuff. This reduced Whisperers CPU and back end services load. The latency kept stable at ~8 ms.

The error rates are essentially zero on both sides.
Circuit breakers, retries, and breakdown patterns were ported one-to-one.

The pieces of the parsing flow still on Node.js - the pollers that drain Redis into Elasticsearch, the read-side services that serve the GUI, the management services - were not migrated. They are not on the hot path, and the cost / benefit did not justify the rewrite.

What this looked like in practice

A few things stood out across the migration.

The plan-then-execute loop scaled.
What worked on the Controller worked again on a service three times smaller and on a service three times larger. The same model - one architecture document, one phased plan, one Go file per Node module - did not need to be reinvented.

Protobuf compatibility was the only sharp edge.
When a packet protobuf schema gained a field, all consumers had to regenerate.
The shared proto/ repo with a single generator command made this manageable, but it is now a real piece of the build process where there was nothing equivalent on the JSON side. A worthwhile trade for the size reduction.

Go's CGO toolchain was avoided.
All services build with CGO_ENABLED=0 and ship as static binaries on Alpine.

Upgrade notes for operators

One thing to be aware of when running this release:

Template & tag regular expressions.

• Go uses RE2, which does not support backreferences or lookaround.
• Any existing pcap / template / tag rule that relied on those constructs needs to be rewritten.
• The new Whisperer Config Playground (added in the same release) validates rules against the actual Go parser before saving, so the change is visible immediately.

The rest of the migration is transparent.

• The HTTP, PostgreSQL, and TCP indices on Elasticsearch are unchanged.
• The GUI queries are unchanged.
• The backend services are compatible with JSON and protobuf.

Closing

Three months ago, the Controller rewrite produced a single Go binary running as drop-in replacement as Node.js generation, with better scaling headroom on large clusters.
Today, seven more services have crossed the same line, and the wire formats they share have followed.

The combined effect on a representative cluster:

• ~78% less memory in redis-pack
• ~3× less CPU on Pack-Read
• ~2.7× less CPU on Web-Write
• ~33% lower latency on the hottest inter-service call

The global effect is clearly visible in the billing dashboard, that tracks global CPU usage over time.
The same setup and load that was measured and invoiced at 3.5 CPU on average is now running with 1.9 CPU, to parse 190 GB/day!

February:

May:

The same workflow - Claude as co-author, real cluster as integration environment, good monitoring as the scoreboard - is what made it tractable.
The remaining Node.js services in the pipeline will be revisited when their cost / benefit lines up; the migration is paced to value, not to ideology.