Skip to main content

System architecture

The system is built around several architecture concepts:

  • Microservices

    • The system is composed of tens of microservices, with independent storage and functional segregation.
    • They communicate together using HTTP REST APIs with JSON payload and JWT signed authentication & authorization.

  • CQRS - Command & Query Responsibility Segregation

    • Capture / Analysis and Search / Representation flows are decoupled and operated with independent services, optimized for their own processing.

High level system view​

High level view

Technologies​

Spider is built using modern technologies and architecture patterns:

  • Microservices in Node.js
  • Open API with REST APIs
  • Security with JSON Web Tokens
  • Clustering with Kubernetes
  • Web application using React.js
  • Distributed memory with Redis
  • Search engine with Elasticsearch

Technologies

Parsing process​

Capture / Analysis and Search flows are decoupled and operated with independent services, optimized for their own processing.

Parsing Architecture

Data storage​

Spider uses Redis as a distributed memory to have optimal processing speed.

Then Spider serializes its captured and monitoring data in Elasticsearch search engine.

  • Its open API exposes Elasticsearch query and aggregation features.
  • Elasticsearch is configured for both speed of ingestion and search optimisation, with time based rolling indices, and progressive asynchronous loading of data on the UI.
  • Spider is designed to store billions of communications efficiently.

Low level network data​

  • Packets
  • Tcp sessions

Higher protocols​

  • Http communications
  • More will come...

Configuration data​

  • Users
  • Teams
  • Whisperers

Monitoring data​

  • Whisperer status
  • Capture status …

Complete system view​

Spider set of microservices and related storage is visible in real time in its monitoring UI, with network flows and quality.

Microservices

Scalability​

Spider system architecture has been proven to scale very well, with a track record of parsing more than 1GB/min with only 7 cores of CPU used.

Perf