Skip to main content

System architecture

The system is built around several architecture concepts:

  • Microservices

    • The system is composed of tens of microservices, with independent storage and functional segregation.
    • They communicate together using HTTP REST APIs with JSON payload and JWT signed authentication & authorization.
  • CQRS - Command & Query Responsibility Segregation

    • Capture / Analysis and Search / Representation flows are decoupled and operated with independent services, optimized for their own processing.

High level system view

High level view


Spider is built using modern technologies and architecture patterns:

  • Microservices in Node.js
  • Open API with REST APIs
  • Security with JSON Web Tokens
  • Clustering with Kubernetes
  • Web application using React.js
  • Distributed memory with Redis
  • Search engine with Elasticsearch


Parsing process

Capture / Analysis and Search flows are decoupled and operated with independent services, optimized for their own processing.

Parsing Architecture

Data storage

Spider uses Redis as a distributed memory to have optimal processing speed.

Then Spider serializes its captured and monitoring data in Elasticsearch search engine.

  • Its open API exposes Elasticsearch query and aggregation features.
  • Elasticsearch is configured for both speed of ingestion and search optimisation, with time based rolling indices, and progressive asynchronous loading of data on the UI.
  • Spider is designed to store billions of communications efficiently.

Low level network data

  • Packets
  • Tcp sessions

Higher protocols

  • Http communications
  • More will come...

Configuration data

  • Users
  • Teams
  • Whisperers

Monitoring data

  • Whisperer status
  • Capture status …

Complete system view

Spider set of microservices and related storage is visible in real time in its monitoring UI, with network flows and quality.



Spider system architecture has been proven to scale very well, with a track record of parsing more than 1GB/min with only 7 cores of CPU used.