Excel driven refactoring ! My first ever ;)

One of the oldest saga of Network UI was needing a huge refactor.
Well… not a refactor, the goal was to remove it completely.

I made it before I built my best practices with Saga. And this method was one that help me understand… not to do like this 😉
This method was called on various users and automatic actions to update various elements of the UIs:

  • Timeline
  • Map
  • Grid
  • Stats
  • Nodes names

While it was updating everything, it was quite simple. But for performance improvements, and to limit the queries on the servers, many parameters were added to limit some refresh to some situations.

However, this is not the right pattern.
It is better to have each component have its own saga watching the actions they would need to refresh on.
This is the pattern I implemented almost everywhere else, and it scales good, while limiting the responsibility in one place.

To perform this refactor was risky. As the function was called from many places with various arguments.

So I used … Excel ! 😉

1. List the calls

2. List the behaviors from the params

3. List the needs for each call

4. Find the actions behind each need, and subscribe the update sagas to those actions

5. Tests!!

All in all… 5h of preparation, 5h of refactor + fix and… it rocks 🙂 !

So much more understandable and maintenance easy.
What a relief to remove this old code.

Code are aging badly. Really 😉

Monitoring – New Performance view

I added a new view to monitoring.
And thanks to the big refactors of last year… this was bloody easy 🙂

This view adds several grids to get performance statistics over the period:

  • Services performance
    • Replicas, CPU, RAM, Errors
  • Whisperers -> Services communications
  • Services API
  • Services -> Services communications
  • Services -> Elasticsearch
  • Services -> Redis
    • For all: Load, Latency, Errors

Setup is managing Docker config upgrades

I wanted to remove coupling between Spider setup and infrastructure configuration.

There was one sticky bit still: the configuration service was using a volume with all configuration files of the applications mounted from it.

I moved it all in Docker configs, so that you may have many replicas of configuration service, and also so that High Availability is managed by Docker.
To be able to go towards this, I upgraded Spider setup script to:

  • Create Docker configs for each application configuration file
  • Inject them in Configuration service Docker stack definition
  • And also… manage updates of those configuration to transparently change the Docker configs on next deploy.

Now, more than ever, setup and upgrades of Spider are simple:

  • Setup your ES cluster
  • Setup your Docker Swarm cluster
  • Pull the Setup repo and configure setup.yml
  • To install:
    • Run make new-install config db keypair admin crons cluster
  • To update:
    • Run make update config db cluster

I could also manage Docker secret upgrades… but since only the signing key is in secret, there is not much value in it 🙂

Technical upgrades

I did some technical upgrades of Spider:

  • Traefik -> 2.4.8
  • Redis -> Back in Docker Swarm cluster for easier upgrade, and High Availability
  • Metricbeat & Filebeat -> 7.11

I also tested Redis threaded IO… but there was no gain, so I reverted back.

Upgrade to Redis 6.2

Just wanted to test 🙂
I just upgraded Redis from 5.0 to 6.2…

  • Nothing to change except systemd loader
  • Performance is as fast as before (with no change of settings): 10 500 op/s for 7% CPU
  • Works like a charm



Processing time

I’ll let it run for some time, then I’ll activate the new IO threads to check any improvement.

Later, I’ll see about using the new ACL and TLS features 😉

Team feature

I deployed the new feature pack I was working one these last weeks.
Many changes, both on backend services and front end to ease configuration sharing, access rights management and overall experience 🙂

Team feature

What’s in a team

  • A team gathers many whisperers to share them to a list of users
  • A team also provides common settings that are automatically set on the UI when selecting a team:
    • Shared saved queries
    • Shared clients and replicas patterns
    • Shared set of plugins
    • Shared set of correlation id headers, jsonld contexts, custom content types

All those settings are complementing your own settings when the team is selected.
This allows configuring Spider only once for a software you’re working at, and then sharing it to all users that won’t have to configure Spider themselves.

Configuring a team

Joining a team with a token

Teams may be joined easily using a shared token. Thus simplifying the onboarding of new team members.
1. Open the team icon in the menu
2. Click “join a team”
3. Paste the token in the input and press enter
4. That’s all!
Then, to have access to the team whisperers and settings:

Adding user manually to a team

Users may also be added manually to teams, in the Users tab.
Some team members may have extra rights to configure the team, or, in the opposite way, a limited access to only a subset of whisperers.
This simplifies management of users and rights.

Adding Whisperers

Whisperers can be added to a team by:

  • Creating them from the team

  • Or by sharing a Whisperer to the team

Configuring team settings

Some common UI settings may bet set at the team level.
Those settings like loaded plugins, or available saved queries are then available to all team members when they have selected a team.

Some settings are also available for the specific protocol analyser (HTTP here).

Using  teams

  1. Open the team selection in the menu
  2. Selection the team by clicking on the radio button.
  3. Then open whisperers selection panel, as usual:

Team whisperers are marked by a (team) tag.

Then, on you settings tab, you may see what settings are from the team or your own.
Team settings, and plugins are marked with a small team icon aside.
In the same way, team shared queries are marked:
I hope all these features will help onboarding Spider for all newcomers, and help scale the usage across many teams 🙂


Technically, when selecting a team, the UI is asking a specific team token that will merge the own user rights with the team whisperers and the users rights on the team.

Thus giving the user access to all Whisperers without many changes in the other services, and without increasing the token size for users. It’s getting even smaller for some 🙂

This feature is ‘dynamic’ token, built on the fly and reuse through other calls is smart and useful.
This idea has been reused for the impersonate feature to test the services and UIs!

Teams API documentation is available here: https://spider-analyzer.io/home/api/#/Teams, and Spider resource diagram has been updated:

Timeline upgrade ! Better zoom out UX and tooltips!

Better Zoom Out

Some weeks ago, Timeline did not allow to zoom out from start.

  • You could only zoom out to get back to zoomed in.
  • On Spider, the default Timeline zoom was displaying all data.

That was an issue for performance, as getting the min and max of data across all historical indices was long (300ms to 5s).
So I moved to a fixed zoom level based on the current time, and the ability to zoom out from start.

This was faster, and less stress for the server. Nice.

But then the Timeline UX was not good. When sliding the Timeline at first, the TimeLine was enlarging its domain instead sliding.
This was linked to previous (no zoom) fashion, but was not anymore welcome.

So I did some feature removal and code cleaning. Removing some bugs on the way, and now, TimeLine sliding and initial zoom out is much better. With proper management of the domain limits (maxDomain).

This is v3.0 of Timeline: https://www.npmjs.com/package/@spider-analyzer/timeline/v/3.0.0

The zoom out factor can be set in the props!

Timeline tooltips

This was a request from Streetsmart team:

Implementing tooltips to display metrics information over the Timeline.

At first, it seems simple, but it was not.

  • Indeed, HTML allows only one element to capture the ‘onMouseOver’ event. The first one under the mouse usually.
  • And either the Dragoverlay layer or the Cursor layer were already capturing it.
    • So, the histograms bar could not.

Then, I add to di it the old way:

  • Compute the position on the mouse in the referential of the histogram
  • Determine what bar was hovered
  • Send the information to the histogram component
  • And then display a darkening rectangle to show the hover effect.
  • I also delayed the tooltip appearance, as a whole for the histogram, with a straight setTimeout, that is cleared onMouseOut

For better UX,

  • Tooltips are hidden when sliding the Timeline or dragging/drawing the time selection.

All custom made 🙂 But it rocks!


It is deployed on Spider Network and Monitoring UIs, and will be soon on Streetsmart!

The tooltip content as well as look and feel can be easily customized with component props and CSS.

This is v3.1: https://www.npmjs.com/package/@spider-analyzer/timeline/v/3.1.0

Upgraded Impersonate feature. Such a Joy to use for testing !

When implementing the Team feature, I needed to change often users and users right to check all access rights linked to Team.

That was getting cumbersome, and I decided to improve the impersonification feature instead!

Now, a new API exists to impersonate a user, that generates a new token from your own, with inside:

  • the impersonated user identification
  • its whisperers access rights
  • its own users rights (option)

One the UI, select if you want the selected users rights, and the UI and Services will behave as if this user was calling!
However, all audit fields will still be valued with your own user. No cheating 😉

A shortcut option has also been added to get back your own rights fast when there are too many users.

It is damn helpful for testing, changing right and user within 2 mouse clicks!

Save password for Plugins

Previously, password type parameters in plugins were not saved in user settings.

Passwords were still saved in the local storage, but not sent to Spider backoffice.

If the credentials used for the plugin are not so important and of no risk, you may choose to save them in Spider user settings.
For this, switch the toggle below the password parameter in the plugins settings.
Nothing special to do in the plugin manifest !

Login UI upgraded to commons

Login UI has been upgraded to common components with Network UI, with an upgrade of Material UI and other libraries.

A new background for a better look also 🙂