Claude fixed a polling inefficiency - across three repos, autonomously

Every few seconds, the Spider UI polls for the status of all whisperers.
For each one, the backend was returning the complete instances[] array - up to a week of per-instance snapshots. Most of that data was never used during polling.
Fixing it required changes to a backend service, a config service, a Helm chart, and the frontend.

Claude did all of it. Alone.

The problem

1. Spider's whisperer agents POST their status to Whisps-Status.
2. An aggregation service (Whisps-Status-Agg)
   • reads those per-instance reports,
   • computes aggregate metrics,
   • and stores a document per whisperer that includes a full instances[] array - individual snapshots kept for up to one week.
3. The UI polls this endpoint on every refresh cycle and at startup,
   • passing the list of active whisperer IDs
   • and getting back the full aggregated documents.

That instances[] array was being shipped to every browser on every poll, even though almost nothing in the UI actually reads it during normal operation.

An audit of what each UI component actually needed made the waste obvious:

UI component	Data needed from polling
Navigation chip ("- x3")	Count of recently-active instances
Status icon (Recording / Stopped / etc.)	Aggregate state field - already a string
Remote capture toolbar	Active instance count
Whisperer global tab	Instance count label ("3 recent, 1 old")
Whisperer Instances detail view	Full instances[] - but only on explicit user action

The first four components never need the array.
Only the last one does, and it is only opened when a user deliberately clicks through to the detail panel.

The task

I described the inefficiency and asked Claude to:

1. Plan the fix, including analysing the full data flow from backend to frontend
2. Execute the plan across all affected codebases
3. Handle the lazy-load path so that opening the detail view still retrieves the full data

No file was pointed out.
No repo was specified.
The instruction was: figure out what needs to change and change it.

What Claude analysed

Before writing a single line, Claude traced the entire data path:

• The aggregator (Whisps-Status-Agg) stores the full document including instances[] - it needs them to track per-instance state changes. That cannot be changed.
• The Whisps-Status service reads those documents from Redis/ES and returns them wholesale via a getByList POST endpoint.
• The frontend reads the endpoint URL from a config key (config.whispStatus.getByList), which is served by a dedicated Config service and also declared in the Helm chart for production deployments.
• Six frontend locations read whisperer.status. Only two of them - the Whisperer Instances detail view and the per-instance tab - ever access instances[].

From this, Claude produced a concrete plan:

• Backend: add a ?summary=true query parameter to the getByList endpoint. When set, strip instances[] and add two pre-computed fields: activeInstancesCount (instances updated within the last 2 minutes) and oldInstancesCount.
• Config: add a getByListSummary config key pointing to the same endpoint with ?summary=true appended. Keep the original getByList key for the detail view.
• Helm chart: mirror the same addition for production deployments.
• Frontend - polling and startup: use getByListSummary instead of getByList.
• Frontend - utility functions: update countActiveInstances() to use the pre-computed count when instances[] is absent.
• Frontend - Whisperer global tab: update the instance count label to fall back to the backend-provided counts.
• Frontend - Whisperer Instances detail view: always re-fetch full status (with instances[]) when the panel is opened, instead of reading from the Redux cache.

The execution

Claude made changes across four repositories in a single session.

Services/Whisps-Status - two files:

status-model.js gained a summary parameter on getStatusByIds(). When set, it maps each document through a function that filters instances[] by recency, computes the two count fields, and returns the document with the array stripped:

if (summary) {
    const twoMinutesAgo = DateTime.now().minus(Duration.fromObject({minutes: 2}));
    return items.map(i => {
        const instances = i.instances ?? [];
        const activeInstancesCount = instances.filter(inst =>
            inst.time && DateTime.fromISO(inst.time) >= twoMinutesAgo
        ).length;
        const oldInstancesCount = instances.length - activeInstancesCount;
        const {instances: _dropped, ...rest} = i;
        return {...rest, activeInstancesCount, oldInstancesCount};
    });
}

get-status-by-list-service.js reads ctx.request.query.summary and passes it through.

Services/Config - one file: network-view.cfg.json got a new getByListSummary key alongside getByList.

Infra/HelmCharts - one file: the same addition in the Helm chart template, using the existing publicPath variable so the URL resolves correctly in every environment.

GUIs/Network-View - five files: polling and startup switched to getByListSummary; countActiveInstances() gained a fallback branch; the Whisperer global tab was updated to show the count label even when instances[] is absent; loadDependencies for the Instances detail view was rewritten to always fetch the full endpoint.

A bug caught mid-execution

When the change was tested, opening the Whisperer Instances detail panel produced a blank view.

Claude diagnosed it by tracing the call path. loadDependencies - the saga responsible for loading the full instance list on demand - had a guard:

if (whisperersInToken.includes(id)) {
    // fetch full status
}

whisperersInToken is populated from tokenDecoded.whisperers - the list of whisperer IDs embedded in the user's JWT. Admin accounts do not have individual whisperers in their token; for them, whisperersInToken is an empty array. The condition was always false, the fetch was silently skipped, and instances stayed null.

Before the optimisation this was harmless: the whisperer object in the Redux cache already contained instances[] from the full polling fetch, and loadDependencies read it directly from there. With summary polling, the cache no longer holds instances[], so the guard now blocked the only remaining path to retrieve them.

The fix was to remove the guard in loadDependencies entirely. The context is different from loadItem: by the time a user opens the Instances detail view, they have already navigated to that whisperer - the backend JWT check handles access control. There is no reason to pre-filter at the frontend.

// Before (broken for admins)
if (whisperersInToken.includes(id)) {
    const responseStatus = yield getItemsByIds(getByList, token, [id]);
    instances = responseStatus?.[0]?.instances ?? null;
}

// After
const responseStatus = yield getItemsByIds(getByList, token, [id]);
const instances = responseStatus?.[0]?.instances ?? null;

What this looks like in practice

After the change:

• Every polling request returns lightweight documents - counts and state, no arrays.
• The navigation chip, status icon, and toolbar all work with the pre-computed activeInstancesCount.
• The Whisperer global tab shows "3 recent, 1 old" from the backend-provided counts.
• Clicking through to the Instances detail panel triggers a fresh fetch of the full document - a slight delay on first open, but one that only happens when the data is actually needed.

Observations

A few things worth noting about how this went.

The plan came first, and it was accurate. Before touching any file, Claude produced a written plan that correctly identified every file to change across all four repositories, described what each change would do, and called out the two options for how to handle the new config key. The execution matched the plan almost exactly.

Cross-repo reasoning without hand-holding. The change touched a Node.js service, a JSON config file, a Helm YAML template, and a React/Redux frontend - all in separate Git repositories. Claude navigated between them without being told where to look, following the data flow from backend response through config service through Redux state into components.

The bug was found by the same model that wrote the bug. The whisperersInToken guard was carried over from the original loadItem function, where it made sense. In the new loadDependencies context it was wrong. Claude identified the issue, explained why the guard was incorrect in this context, and fixed it in one step.

Four repositories, one session. The commits landed in Services/Whisps-Status, Services/Config, Infra/HelmCharts, and GUIs/Network-View - each with an accurate commit message - and the monorepo submodule pointers were updated and pushed in the same session.

The human contribution: describe the problem, verify the fix works, approve the commits.