Existing issue​

The existing parsing engine of Spider had two major issues:

• Tcp sessions resources were including the list of packets they were built from. This was a limitation in the count of packets a tcp session could hold because the resource was ever increasing. And long persistent Tcp sessions were causing issues and thus were limited in terms of packets.
• Http parsing logs were including as well list of packets and of HTTP communications found.

I studied how to remove these limitations, and how to improve at the same time the parsing speed and its footprint. While keeping the same quality of course!

And I managed :) !!

I had to change part of architecture level 1 decisions I took at the beginning, and it had impacts on Whisperers code on 7 other micro services, but it seemed sound and the right decision!

Work​

4 weeks later, it is all done, full regression tested and deployed on Streetsmart! And the result is AWESOME :-)

Spider now parses Tcp sessions in streaming, with a minimal footprint, and a reduced CPU usage of the servers for the same load ! :)

I also took the time to improve the 'understandability' of the process and the code quality. I will document the former soon.

Results​

Users... did not see any improvements (nor any issue), except that 'it seems faster', but figures are here to tell us!

The first day, I achieved 65 parsing errors out of 43 million communications! Those 2 missed bugs were solved straight away thanks to good observability! :)

Effects on back end​

3 /v2 APIs have been added to the API. Corresponding /v1 APIs will be deprecated soon.

But Spider is compatible with both, which allowed me an easy non regression by comparing parsing results of the same network communications... by both engines ;-) !

Effects on UI​

1. Grids and details view of packets and Tcp sessions have been updated.
2. Pcap upload feature has been updated to match new APIs.
3. Downloading pcap packets has been fixed to match the new APIs.
4. All this also implied changes in Tcp sessions display:

• Details and content details pages are now with infinite scroll as there may be tens or hundreds of thousands of packets.
  • It deservers another improvement for later: to be able to select the time in the timeline!
• Getting all packets of a Tcp session is only a single filter away.

Performance​

Last but not least... performance!! Give me the figures :)

Statistics over

• 9h of run
• 123 MB /min of parsed data
  • 318 000 packets /min
  • 31 000 tcp sessions /min
• For a total of
  • 171 Million packets
  • 66,4 GB
  • 16 Million Tcp sessions
  • 0 error :-)

CPU usage dropped!

Before:After:

Redis footprint was divided by more than 2!

• From 80 000 items in working memory to 30 000
• From 500 MB to 200 MB memory footprint :)

Before:After:

Resource usage

CPU usage for parsing service dropped of 6%. But the most impressive is the CPU drop of 31% and 43% of inbound services: pack-write and tcp-write.

Whisperers --> Spider

Confirming the above figures, response time of pack-write and tcp-write as improved of 40% and 10%!

API stats

APIs statistics confirm the trend with service side improvements of up to 50%! Geez !!

Circuit breakers stats

When seen from the circuit breakers perspective, the difference is smaller, due to delay in client service internal processing.

Conclusion​

That was big work! Many changes in many places. By Spider is now faster and better than ever :)

One of the oldest saga of Network UI was needing a huge refactor. Well... not a refactor, the goal was to remove it completely.

I made it before I built my best practices with Saga. And this method was one that help me understand... not to do like this ;) This method was called on various users and automatic actions to update various elements of the UIs:

• Timeline
• Map
• Grid
• Stats
• Nodes names
• ...

While it was updating everything, it was quite simple. But for performance improvements, and to limit the queries on the servers, many parameters were added to limit some refresh to some situations.

However, this is not the right pattern. It is better to have each component have its own saga watching the actions they would need to refresh on. This is the pattern I implemented almost everywhere else, and it scales good, while limiting the responsibility in one place.

To perform this refactor was risky. As the function was called from many places with various arguments.

So I used ... Excel ! ;)

1. List the calls

2. List the behaviors from the params

3. List the needs for each call

4. Find the actions behind each need, and subscribe the update sagas to those actions

5. Tests!!

All in all... 5h of preparation, 5h of refactor + fix and... it rocks :) !

So much more understandable and maintenance easy. What a relief to remove this old code.

Code are aging badly. Really ;)

I added a new view to monitoring. And thanks to the big refactors of last year... this was bloody easy :)

This view adds several grids to get performance statistics over the period:

• Services performance
  • Replicas, CPU, RAM, Errors
• Whisperers -> Services communications
• Services API
• Services -> Services communications
• Services -> Elasticsearch
• Services -> Redis
  • For all: Load, Latency, Errors

I wanted to remove coupling between Spider setup and infrastructure configuration.

There was one sticky bit still: the configuration service was using a volume with all configuration files of the applications mounted from it.

I moved it all in Docker configs, so that you may have many replicas of configuration service, and also so that High Availability is managed by Docker. To be able to go towards this, I upgraded Spider setup script to:

• Create Docker configs for each application configuration file
• Inject them in Configuration service Docker stack definition
• And also... manage updates of those configuration to transparently change the Docker configs on next deploy.

Now, more than ever, setup and upgrades of Spider are simple:

• Setup your ES cluster
• Setup your Docker Swarm cluster
• Pull the Setup repo and configure setup.yml
• To install:
  • Run make new-install config db keypair admin crons cluster
• To update:
  • Run make update config db cluster

I could also manage Docker secret upgrades... but since only the signing key is in secret, there is not much value in it :)

I did some technical upgrades of Spider:

• Traefik -> 2.4.8
• Redis -> Back in Docker Swarm cluster for easier upgrade, and High Availability
• Metricbeat & Filebeat -> 7.11

I also tested Redis threaded IO... but there was no gain, so I reverted back.

Just wanted to test :) I just upgraded Redis from 5.0 to 6.2...

• Nothing to change except systemd loader
• Performance is as fast as before (with no change of settings): 10 500 op/s for 7% CPU
• Works like a charm

CPU

Load

Processing time

I'll let it run for some time, then I'll activate the new IO threads to check any improvement.

Later, I'll see about using the new ACL and TLS features ;)

I deployed the new feature pack I was working one these last weeks.

Many changes, both on backend services and front end to ease configuration sharing, access rights management and overall experience :)

What's in a team​

• A team gathers many whisperers to share them to a list of users
• A team also provides common settings that are automatically set on the UI when selecting a team:
  • Shared saved queries
  • Shared clients and replicas patterns
  • Shared set of plugins
  • Shared set of correlation id headers, jsonld contexts, custom content types

All those settings are complementing your own settings when the team is selected.

This allows configuring Spider only once for a software you're working at, and then sharing it to all users that won't have to configure Spider themselves.

Configuring a team

Joining a team with a token​

Teams may be joined easily using a shared token. Thus simplifying the onboarding of new team members.

1. Open the team icon in the menu

2. Click "join a team"

3. Paste the token in the input and press enter

4. That's all!

Then, to have access to the team whisperers and settings:

Adding user manually to a team​

Users may also be added manually to teams, in the Users tab.

Some team members may have extra rights to configure the team, or, in the opposite way, a limited access to only a subset of whisperers.

This simplifies management of users and rights.

Adding Whisperers​

Whisperers can be added to a team by:

• Creating them from the team

• Or by sharing a Whisperer to the team

Configuring team settings​

Some common UI settings may bet set at the team level. Those settings like loaded plugins, or available saved queries are then available to all team members when they have selected a team.

Some settings are also available for the specific protocol analyser (HTTP here).

Using  teams

1. Open the team selection in the menu
2. Selection the team by clicking on the radio button.
3. Then open whisperers selection panel, as usual:

Team whisperers are marked by a (team) tag.

Then, on you settings tab, you may see what settings are from the team or your own.

Team settings, and plugins are marked with a small team icon aside.

In the same way, team shared queries are marked:

I hope all these features will help onboarding Spider for all newcomers, and help scale the usage across many teams :)

Technically

Technically, when selecting a team, the UI is asking a specific team token that will merge the own user rights with the team whisperers and the users rights on the team.

Thus giving the user access to all Whisperers without many changes in the other services, and without increasing the token size for users. It's getting even smaller for some :)

This feature is 'dynamic' token, built on the fly and reuse through other calls is smart and useful. This idea has been reused for the impersonate feature to test the services and UIs!

Teams API documentation is available here: https://spider-analyzer.io/home/api/#/Teams, and Spider resource diagram has been updated:

To avoid sharing the source code in the Dockers, I decided to bundle the BO souce code with Webpack.

Dealing with native libraries was complexifying it much. Finally, I decided to keep the node modules outside the bundle, as well as static files like HTML or configuration files that are loaded by fs.readFile. However the Lua scripts for Redis are embedded in the bundles.

Here is the webpack config file needed:

const path = require('path');
const nodeExternals = require('webpack-node-externals');

module.exports = {
    entry: './index.js',
    mode: 'production',
    output: {
        path: path.resolve(__dirname, 'dist'),
        filename: 'index.js'
    },
    externalsPresets: { node: true }, // in order to ignore built-in modules like path, fs, etc.
    externals: [nodeExternals()], // in order to ignore all modules in node\_modules folder
    node: {
        __dirname: false,
    },
    module: {
        rules: [{
            test: /\\.lua/,
            type: 'asset/source'
        }]
    }
};

Then I'm using multi stage builds in Dockerfile to:

• Build the bundle
• Copy all necessary files
• Then create the simplest image

# Build bundle

FROM node:12-alpine AS build\_bundle

RUN apk --no-cache add --virtual build-dependencies python build-base

WORKDIR /app
COPY . /app

ARG registry
RUN npm install --quiet --no-progress --registry=${registry:-https://registry.npmjs.org} \\
    && npm run build \\
    && npm cache clean --force \\
    && npm prune --production

# Build server

FROM node:12-alpine AS build

WORKDIR /app
COPY --from=build\_bundle /app/dist /app/
COPY --from=build\_bundle /app/node\_modules /app/node\_modules
COPY docker-entrypoint.sh LICENSE package.json /app/
COPY resources /app/resources

# Final

FROM node:12-alpine

WORKDIR /app
COPY --from=build /app /app/

ARG port
EXPOSE $port
USER 1000:1000
ENTRYPOINT \["./docker-entrypoint.sh"\]
CMD \["index.js"\]

Login UI has been upgraded to common components with Network UI, with an upgrade of Material UI and other libraries.

A new background for a better look also :)

Spider Monitoring UI has been upgraded with common components from Network UI, and common application architecture :).

Now Excel export is available on all grids !