Ingest tool

The ingest tool is a script dedicated to duplicate network communications captured by a Whisperer to send them on another server / platform.

Use cases

Production migration tests

The tool has been used with success to validate a huge technical migration of a system in Flowbird.

We wanted to migrate a set of 100+ microservices from Docker Swarm to Kubernetes.
Because of some constraints on the domain name used in the data, the switch had to be done in one go, and could not be done progressively.

We used the Injest tool to do real life usage tests (functional and load) by injecting the Swarm production feed to the new Kubernetes production.

Having previously copied the reference data from one environment to another, and using the same security keys, the new environment was executing the same processing as production, with only 45s delay, and exactly the same load, while having the production feed transform to match the new Urls!

We were using Spider to compare the behavior of both systems with both Statistics bird eyes view, and deep down troubleshooting to understand the differences.

This was highly efficient and we saved days while conforting our confidence in the migration by fixing any difference in advance.

The inject tool injected effectively millions of requests to the system APIs, allowing us to perform the most massive regression test we could imagine!

New release regression tests with real requests

... (tell me!)

Overview

The ingest tool:

Connects to Spider API
Executes a query on HTTP communications resources
Loop on all communications
Transform communications requests based on Regular expressions rules
Send the resulting transformed request
Compare the response codes & bodies

info

Communications are sent at the speed they were captured. So that the target system is refreshed / updated in the same order as the original.

Which means that:

If you're replaying 2 hours of communications, it will take 2 hours
If the load in requests/s is too big for the Ingest tool... it might blow out of memory.

For finer comparison, best is to have Spider on the target server as well, then to compare statistically and in details for troubleshooting.

Technological stack

Node.js
Bundled with webpack in an Alpine container
Packaged as a Kubernetes Job
Published as a Helm chart

Packages

Type	Repo	Artifact
Helm	https://repository.floocus.com/helm/	floocus/spider-ingest
Docker	registry.gitlab.com/spider-analyzer/public-images/	ingest-tool

Configuration

Credentials to connect to Spider API are passed in ENVIRONMENT variables.
Script configuration is read from a config.json file, in the same folder as the script.
In Helm, this file is included as YAML in the values file, then push as a ConfigMap.

config:
  spiderBaseUri: https://spider.company.io
  startDate: 2023-08-25T00:00:00.000Z    # Start and end of the replay. If in the future, Spider will wait until the stopDate is pasted before stopping
  stopDate: 2023-08-25T12:00:00.000Z
                                         # Query to filter the communications to replay
  query: "req.body.size:>0 AND _exists_:stats.statusCode AND NOT stats.src.origin:10.0.1.0/16"
  whisperers:
    - MvvEYhXATLqBJfUPszX4tA
  pathsToIgnore: []                      # Regexp matching req.uri field to ignore communications (but you may use the query too)
  urlReplacementsPatterns:               # Regexp matched to the rebuilt URL to do replacements
    - match: legacy.company.com/prod
      replace: app.company.io
  queryReplacementsPatterns:             # Regexp matched to the rebuilt querystring+hash to do replacements
    - match: legacy.company.com/prod
      replace: app.company.io
  headersReplacementsPatterns:           # Regexp matched to the headers to do replacements
    - match: legacy.company.com/prod
      replace: app.company.io
    - match: legacy.company.com
      replace: app.company.io
  headersToIgnore:                       # Regexp matched to the headers to avoid sending those (if sensitive)
    - x-forwarded-basepath
  bodyReplacementsPatterns:              # Regexp matched to the requests bodies to do replacements
    - match: legacy.company.com/prod
      replace: app.company.io
  bodyValidationReplacementsPatterns:    # Regexp matched to the original response bodies to do replacements
                                         # prior to compare with new response
    - match: legacy.company.com/prod
      replace: app.company.io

email: yourSpiderAccount@yopmail.com     # Credentials to connect to Spider API
password: MySecuredP@ssW0rd

logLevel: info                           # Sets log level (trace / debug / info / warn / error)

acceptSelfSignedCertificates: false      # Accepts self signed certificate for Spider or Target server (true / false)

hostAliases:                             # Host aliases to set on the server
  app.company.io: 10.25.63.2
  spider.company.io: 10.25.63.4
  server1.company.io, server2.company.io: 10.25.63.10 # If many servers with same IP

note

Since the request body is reassembled (and may be modified) by the script before sending,

Transfer-Encoding header is removed,
Content-Length header is recomputed

However, Content-Encoding header is kept: the request body is compressed back after transformation

Accessing repository

The Helm repository has a public access.

$ helm repo add floocus https://repository.floocus.com/helm
$ helm search repo floocus

Execution

$ helm upgrade spider-ingest -f ./values.yaml floocus/ingest --namespace ingest --create-namespace --install

It deploys as a Job and executes only once... until completion.
You may run it as many times as you want... or even in parallel
- This could be useful if you would need scaling because one instance is not fast enough to ingest your load
- Then run several in parallel with queries splitting your load

Logs

The script reports a progress status every 10s.

$ kubectl -n ingest logs spider-ingest-job-11-pw96p -f | bunyan

[2023-09-19T21:08:30.801Z]  INFO: ingest-Tool/1 on spider-ingest-job-11-pw96p: You may find ingested data in Spider with this filter: "req.headers.x-spider-ingest-run:C9nCCIotQJuz53eYrn6i-w".
[2023-09-19T21:08:30.821Z]  INFO: ingest-Tool/1 on spider-ingest-job-11-pw96p: Will fetch data to inject from 2023-09-18T19:42:00.000+00:00 to 2023-09-18T19:44:00.000+00:00
[2023-09-19T21:08:31.141Z]  INFO: ingest-Tool/1 on spider-ingest-job-11-pw96p: Successfully connected to Spider at https://spider.hub.flowbird.cloud with thibaut.raballand+ingest@flowbird.group
[2023-09-19T21:08:31.142Z] DEBUG: ingest-Tool/1 on spider-ingest-job-11-pw96p: Token refresh job: Should refresh token in 0d 21h 35min 58s.
[2023-09-19T21:08:31.143Z]  INFO: ingest-Tool/1 on spider-ingest-job-11-pw96p: Showing status every 10s.
[2023-09-19T21:08:31.277Z] DEBUG: ingest-Tool/1 on spider-ingest-job-11-pw96p: Page 1, items: 2, total items: 2
[2023-09-19T21:08:31.282Z] DEBUG: ingest-Tool/1 on spider-ingest-job-11-pw96p: Last com to send is in 60s. Pausing until then.
[2023-09-19T21:08:41.146Z]  INFO: ingest-Tool/1 on spider-ingest-job-11-pw96p:
    Requests sent: 1 at 0.1/sec
      Matching statuses: 0%
      Similarity of response bodies: 3.9%
      Requests in pipe: 1.
      Last sent was at: 2023-09-18T19:43:00.142+00:00

...

tip

The job is named with Helm revision as a suffix.
So that you may easily find its logs : spider-ingest-job-11-pw96p

Tracking ingested requests

tip

You may filter easily ingested requests with the filter provided by the script at start.
The ingest script adds a x-spider-ingest-run header to the requests, with a unique id as a value, generated at script start.

Completion

The script completes before sending the first request after the stopDate defined in configuration.

tip

To execute ingest in near real time, set the startDate to now and a stopDate in the future 👍

On completion or interrupt, it provides a complete status.

[2023-08-25T17:57:48.624Z]  INFO: ingest-Tool/1 on spider-ingest-job-58zgr:
    Total requests sent: 48 at 1.6/sec
      Matching statuses: 99%
      Similarity of response bodies: 87.7%

Source code

Ingest Gitlab repository

History

Helm 0.5 / Docker latest
- Added queryReplacementsPatterns to allow replacing inside queryString in an independent manner or the domain / url
- Added security to have slight delay before pulling latest coms - 45s
Helm 0.4 / Docker latest
- Prevent restart of job on error
Helm 0.3 / Docker latest
- Added image pull policy
Helm 0.2 / Docker latest
- Added hostAliases and acceptSelfSignedCertificates in values.yaml
- Added x-spider-ingest-run header and Spider filter to track ingested requests
- Fixed bugs (when body was empty or without a content-encoding)
- Inverted bodyValidationReplacementsPatterns meaning (applies on original response now)
Helm 0.2 / Docker latest
- Added bodyValidationReplacementsPatterns matched prior to response validation
- Added Helm revision in Job's name to be able to launch it many times
Helm 0.1 / Docker 2023.08.26
- First release

Ingest tool

Use cases​

Production migration tests​

New release regression tests with real requests​

... (tell me!)​

Overview​

Technological stack​

Packages​

Configuration​

Accessing repository​

Execution​

Logs​

Tracking ingested requests​

Completion​

Source code​

History​