I did some technical upgrades of Spider:

• Traefik -> 2.4.8
• Redis -> Back in Docker Swarm cluster for easier upgrade, and High Availability
• Metricbeat & Filebeat -> 7.11

I also tested Redis threaded IO... but there was no gain, so I reverted back.

Just wanted to test :) I just upgraded Redis from 5.0 to 6.2...

• Nothing to change except systemd loader
• Performance is as fast as before (with no change of settings): 10 500 op/s for 7% CPU
• Works like a charm

CPU

Load

Processing time

I'll let it run for some time, then I'll activate the new IO threads to check any improvement.

Later, I'll see about using the new ACL and TLS features ;)

I deployed the new feature pack I was working one these last weeks.

Many changes, both on backend services and front end to ease configuration sharing, access rights management and overall experience :)

What's in a team​

• A team gathers many whisperers to share them to a list of users
• A team also provides common settings that are automatically set on the UI when selecting a team:
  • Shared saved queries
  • Shared clients and replicas patterns
  • Shared set of plugins
  • Shared set of correlation id headers, jsonld contexts, custom content types

All those settings are complementing your own settings when the team is selected.

This allows configuring Spider only once for a software you're working at, and then sharing it to all users that won't have to configure Spider themselves.

Configuring a team

Joining a team with a token​

Teams may be joined easily using a shared token. Thus simplifying the onboarding of new team members.

1. Open the team icon in the menu

2. Click "join a team"

3. Paste the token in the input and press enter

4. That's all!

Then, to have access to the team whisperers and settings:

Adding user manually to a team​

Users may also be added manually to teams, in the Users tab.

Some team members may have extra rights to configure the team, or, in the opposite way, a limited access to only a subset of whisperers.

This simplifies management of users and rights.

Adding Whisperers​

Whisperers can be added to a team by:

• Creating them from the team

• Or by sharing a Whisperer to the team

Configuring team settings​

Some common UI settings may bet set at the team level. Those settings like loaded plugins, or available saved queries are then available to all team members when they have selected a team.

Some settings are also available for the specific protocol analyser (HTTP here).

Using  teams

1. Open the team selection in the menu
2. Selection the team by clicking on the radio button.
3. Then open whisperers selection panel, as usual:

Team whisperers are marked by a (team) tag.

Then, on you settings tab, you may see what settings are from the team or your own.

Team settings, and plugins are marked with a small team icon aside.

In the same way, team shared queries are marked:

I hope all these features will help onboarding Spider for all newcomers, and help scale the usage across many teams :)

Technically

Technically, when selecting a team, the UI is asking a specific team token that will merge the own user rights with the team whisperers and the users rights on the team.

Thus giving the user access to all Whisperers without many changes in the other services, and without increasing the token size for users. It's getting even smaller for some :)

This feature is 'dynamic' token, built on the fly and reuse through other calls is smart and useful. This idea has been reused for the impersonate feature to test the services and UIs!

Teams API documentation is available here: https://spider-analyzer.io/home/api/#/Teams, and Spider resource diagram has been updated:

To avoid sharing the source code in the Dockers, I decided to bundle the BO souce code with Webpack.

Dealing with native libraries was complexifying it much. Finally, I decided to keep the node modules outside the bundle, as well as static files like HTML or configuration files that are loaded by fs.readFile. However the Lua scripts for Redis are embedded in the bundles.

Here is the webpack config file needed:

const path = require('path');
const nodeExternals = require('webpack-node-externals');

module.exports = {
    entry: './index.js',
    mode: 'production',
    output: {
        path: path.resolve(__dirname, 'dist'),
        filename: 'index.js'
    },
    externalsPresets: { node: true }, // in order to ignore built-in modules like path, fs, etc.
    externals: [nodeExternals()], // in order to ignore all modules in node\_modules folder
    node: {
        __dirname: false,
    },
    module: {
        rules: [{
            test: /\\.lua/,
            type: 'asset/source'
        }]
    }
};

Then I'm using multi stage builds in Dockerfile to:

• Build the bundle
• Copy all necessary files
• Then create the simplest image

# Build bundle

FROM node:12-alpine AS build\_bundle

RUN apk --no-cache add --virtual build-dependencies python build-base

WORKDIR /app
COPY . /app

ARG registry
RUN npm install --quiet --no-progress --registry=${registry:-https://registry.npmjs.org} \\
    && npm run build \\
    && npm cache clean --force \\
    && npm prune --production

# Build server

FROM node:12-alpine AS build

WORKDIR /app
COPY --from=build\_bundle /app/dist /app/
COPY --from=build\_bundle /app/node\_modules /app/node\_modules
COPY docker-entrypoint.sh LICENSE package.json /app/
COPY resources /app/resources

# Final

FROM node:12-alpine

WORKDIR /app
COPY --from=build /app /app/

ARG port
EXPOSE $port
USER 1000:1000
ENTRYPOINT \["./docker-entrypoint.sh"\]
CMD \["index.js"\]

Login UI has been upgraded to common components with Network UI, with an upgrade of Material UI and other libraries.

A new background for a better look also :)

Spider Monitoring UI has been upgraded with common components from Network UI, and common application architecture :).

Now Excel export is available on all grids !

Previously, password type parameters in plugins were not saved in user settings.

Passwords were still saved in the local storage, but not sent to Spider backoffice.

If the credentials used for the plugin are not so important and of no risk, you may choose to save them in Spider user settings. For this, switch the toggle below the password parameter in the plugins settings. Nothing special to do in the plugin manifest !

Some weeks ago, Timeline did not allow to zoom out from start.

• You could only zoom out to get back to zoomed in.
• On Spider, the default Timeline zoom was displaying all data.

That was an issue for performance, as getting the min and max of data across all historical indices was long (300ms to 5s). So I moved to a fixed zoom level based on the current time, and the ability to zoom out from start.

This was faster, and less stress for the server. Nice.

But then the Timeline UX was not good. When sliding the Timeline at first, the TimeLine was enlarging its domain instead sliding. This was linked to previous (no zoom) fashion, but was not anymore welcome.

So I did some feature removal and code cleaning. Removing some bugs on the way, and now, TimeLine sliding and initial zoom out is much better. With proper management of the domain limits (maxDomain).

This is v3.0 of Timeline: https://www.npmjs.com/package/@spider-analyzer/timeline/v/3.0.0

The zoom out factor can be set in the props!

Timeline tooltips

This was a request from Streetsmart team:

Implementing tooltips to display metrics information over the Timeline.

At first, it seems simple, but it was not.

• Indeed, HTML allows only one element to capture the 'onMouseOver' event. The first one under the mouse usually.
• And either the Dragoverlay layer or the Cursor layer were already capturing it.
  • So, the histograms bar could not.

Then, I add to di it the old way:

• Compute the position on the mouse in the referential of the histogram
• Determine what bar was hovered
• Send the information to the histogram component
• And then display a darkening rectangle to show the hover effect.
• I also delayed the tooltip appearance, as a whole for the histogram, with a straight setTimeout, that is cleared onMouseOut

For better UX,

• Tooltips are hidden when sliding the Timeline or dragging/drawing the time selection.

All custom made :) But it rocks!

It is deployed on Spider Network and Monitoring UIs, and will be soon on Streetsmart!

The tooltip content as well as look and feel can be easily customized with component props and CSS.

This is v3.1: https://www.npmjs.com/package/@spider-analyzer/timeline/v/3.1.0

When implementing the Team feature, I needed to change often users and users right to check all access rights linked to Team.

That was getting cumbersome, and I decided to improve the impersonification feature instead!

Now, a new API exists to impersonate a user, that generates a new token from your own, with inside:

• the impersonated user identification
• its whisperers access rights
• its own users rights (option)

One the UI, select if you want the selected users rights, and the UI and Services will behave as if this user was calling! However, all audit fields will still be valued with your own user. No cheating ;)

A shortcut option has also been added to get back your own rights fast when there are too many users.

It is damn helpful for testing, changing right and user within 2 mouse clicks!

Rendering grid was getting slow once hundreds of rows were loaded. It affected resize of window, opening details and resizing columns, as these actions implied grid rendering.

After some thoughts on how to perform better, and some walkthrough various windowing libraries, I decided to implement windowing in the grid by myself, it would be less rework than integrating in a windowing library.

First trial: partial success after 2h30 work​

A first trial was to still render all lines, but to render empty lines for lines outside the visible range. It proved I was right, windowing was working great to improve performance, but there was one drawback:

Too many rendering of empty lines when scrolling was introducing visible lags in scrolling.

I tweak it a bit, manage to get acceptable performance by triggering rendering only after a certain range of scrolling... But lag were still noticeable.

Second trial: solution ready with 30min more work​

The night brought advice, as usual. I move the 'not rendering' one step above, to:

• Render a big empty line above the visible range with the cumulated height of all non rendered rows
• Render a big empty line below the visible range with the cumulated height of all non rendered rows
• Render only the visible range (plus a bit more) in between

And result is great :)

You may load thousands of records (if you have the stamina), the grid behavior and speed is constant.

• Resizing columns is fluid
• Opening details is almost immediate
• Resizing grid is immediate
• And scrolling is fluid. If you pay attention, you'll notice the lag. But, honestly it does not matter anymore :)

Example with a grid with 1000 rows loaded​