Skip to main content

Controller's improvements

· 3 min read

Here comes a new release of Spider Controller with a bunch of improvements:

  • Lists Gociphers & Sidecar Whisperers
  • Allows getting logs from Controller, Attachments, Sidecars & Gociphers
  • Provides Sidecar Whisperers to Gociphers to allow TLS deciphering on their captured data
  • When an Ephemeral Whisperer fails twice in 5 minutes, the Controller stops restarting it to avoid overloading the POD manifest

Story​

All these improvements were triggered by the fact that, when a Gossiper configuration is wrong (ex: it has a text instead of a number in a JSON field), it fails to start... And the Controller restarts it, again, and again, and again...

A few minutes later, the target POD manifest includes hundreds of ephemeral containers...

Which makes the Kubernetes API fails because it is too big (10 MB) !
Which at the end tends to freeze the cluster... 😮

I figured out I need to implement a back-off in the Controller itself, since Kubernetes does not do it there.
But then I need to expose logs to the user to explain WHY the Whisperer was not starting.

And it pulled all the rest:

  • Inventory
  • Logs
  • And finally the ability to capture TLS secrets from sidecars Whisperers!

Controller inventory​

Controllers now list all deployed agent in the cluster they are in: Listing deployed agents.
Two tabs have been added aside Attachments one:

  • Sidecars
  • Gpciphers

Example for Sidecars:

Sidecars.png

Getting logs​

In these Whisperers or Gociphers container listing, the small black terminal icon allows you to get the last logs of the remote agent:

GocipherLogs.png

This is pretty neat when troubleshooting and debugging Spider 😁!
You may even access the complete log item if need be:

LogEvent.png

Wish you had this in all your product, don't you? 😉

TLS deciphering for Sidecars​

Previously, new TLS deciphering feature was only for Attached Whisperers.

Now, if your Sidecar Whisperer includes a WHISPERER environment variable in its manifest, it will be exposed to Gociphers as for Attached Whisperers.
And you will see them appear in the Gocipher targets.

Backoff implementation​

The final goal of all these new cool features (that are in fact quick wins side effects): Controller backoff implementation.

When spawning a new Whisperer for an Attachment, now the Controller checks if the same Whisperer was previously attached to the target POD:

  • If 2 recent attachments failed with and error code > 0 in the last 5 minutes, the Controller will delay the Whisperer attachment.

Thus letting you some time to fix the error (checking the terminated Whisperers logs)... before killing your cluster 😎

warning

It will still continue creating ephemeral containers every 5 minutes if you don't fix it!