HTTP communications

Concept

A HTTP communication s is a request/response exchange between two parties.
It is a sequence of related TCP packets in a TCP session between a client and a server.

Each HTTP communication is associated with:

Source and destination IP address
Source and destination TCP port
The HTTP request method, url, headers and body
The HTTP response status, headers and body

Many more data are also computed or extracted to help identification and understanding of the communications.

The HTTP communications are extracted while parsing the TCP sessions in streaming, on Spider backend.

Global Tab

Timing

The timing diagram shows much information at a glance:

General

The General section shows identification information:

Field	Description
From	Host and port of the captured client
Origin	Host at the origin of the request, when reverse proxies are involved
Identification	Identification of the caller, if extracted from JWT token, Certificate, Basic auth etc.
To	Host and port of the captured server / service

From, Origin and Identification are showing, by order of preference:
- The short name of the Host, as extracted during integration in the backend, or as renamed by a user
- The FQDN of the Host
- The IP address, if no Host found
Identification field is extracted / computed at parsing time and is searchable

tip

Identification may be enriched / decoded by a plugin.
To translate clientIds in their clientNames for instance.

Request

The Request section shows summary information of the request part:

Field	Description
Template	Computed API name to identify the request
Date	Date of the request, user's local time
Request	`{Method} {URI Path}`
Query string	The querystring part of the request, URL decoded
Http version	The Http protocol version used
Content type	Request `Content-Type` header
Content length	Either `Content-Length` header when present, or the captured size. If both are different, it is shown.
Original URL	Reassembled URL on how the service was called, taking into account reverse proxies information.

Spider request Template is a very important feature allowing to have a unique name for all the same API calls, removing variables etc. It refers to the URL template standard that could be used to represent the APIs.

Response

The Response section shows summary information of the request part:

Field	Description
Response status	The status of the response, with meaningful color
Http version	The Http protocol version used
Duration	The different between request start and response end
Content type	Response `Content-Type` header
Content length	Either `Content-Length` header when present, or the captured size. If both are different, it is shown.

Correlation Ids

The Correlation Ids section shows the ids extracted from HTTP requests headers used to follow a stream of correlated communications.
They are usually injected by the application for Tracing feature.
Used correctly, they allow you to search for all linked communications as when understanding the fan out of a request.

They are exchanged as HTTP headers values. And their extraction here is a UI visual sugar configured in HTTP global settings tab.

Spider info

This section shows Spider own information:

Field	Description
Whisperer	Name and link to the Whisperer
Instance	Instance of the Whisperer that captured the communication
Tcp session	Tcp session id and link
Request parsing	Parsing status of the request
Response parsing	Parsing status of the response

Actions

Various actions are possible on this tab:

When packets are saved and available, you may download a .pcap file with the HTTP communication only
You may download a cURL instruction to replay (or adapt) the communication on your side

cURL export

You may, directly from the UI, generate and export cURL commands to replay a selected requests. The cURL command is generated on the UI and copied in your clipboard. 💪

You have these options:

Export as binary

Spider fetches the request body as it was transmitted over the network, encodes it as base64 and provides it as binary to cURL.

Export as text

Spider fetches the request body as text (or tries to), beautifies it if its content-type matches a known JSON or XML Mime, and provides it as text to cURL.
This mode is easier if you want to do changes 😉

Export original

If x-forwarded-* headers are present, Spider tries to rebuild the original query (that was made to the reverse proxy).

Export as captured

Spider provides the request as it has been captured by the Whisperer.

note

Please note that:

Headers are lower cased
If filters are set up on the parsing configuration, Spider will only fetch the saved headers

See this blog post for examples.

Headers tab

The Headers tab shows both request and response headers in two ways:

As a decoded list
- Sorted alphabetically
- With embedded decoding of authentication (Basic auth and JWT)
- With plugins hooks to decode the headers values
- With filter icons
As raw format, as it was transmitted over the network

Decoded fields are shown with a dark background under the field name.
Plugins injected fields are marked with an (i) icon and a tooltip.

Getting raw header

By clicking on view source, you get access to the raw headers as captured.
Clicking on view parsed gets you back.

Raw headers are available on a Whisperer captured communications:

If you save packets
Or if you save raw headers
And, if you did not filter out raw headers for this URI

Request tab

The Request tab shows the HTTP request payload, or body.

It indicates its advertised content-type in the top left.
You may download the payload with the download button on the right.
You may choose between 3 ways of seeing the payload:

Mode	Description
Interpreted	Spider interprets it and show it in a code editor, with: - Color syntaxing - Reformated code - Code folding - Search capabilities with `Ctrl+F` Best with: HTML, CSS, JS, JSON, XML
Browser	Shown as the browser interprets it. Best with images, videos, music...
Raw	The raw text version.

note

Spider sets the best option depending on the content-type header value.
When the header value is wrong, the display will be messy.

Response tab

The Response tab shows the HTTP request payload, or body.
The features and content are the same as the Request tab.

Packets tab

The Packets tab shows the network packets composing the HTTP communication.
You may open them by clicking on the rows.

Field	Description
Dir	Diraction of the packet
Time	Time the packet was captured
Payload	Sized of the TCP payload
Info	Flags and beginning of TCP payload, in ASCII

Specific settings

In the Main settings details panel, a tab is dedicated to HTTP communications display settings.

These settings may be defined at the team level or specifically for the customer.

Correlation Ids

You may define here one or many HTTP headers that contain correlation ids to link requests together.

Spider will show those headers values in the details Correlation Ids section, with a specific filter that filters on request and response at once.

Those headers are also used in the Links column of the grid to quickly filter linked communications.

JsonLd contexts

For those using JSON LD on their APIs, they may define specific contexts to be used to re-compact the payload based on the identified template.

Custom content types

When missing a Content-Type header in your request / response, Spider allows you to select its content-type manually. This forces how Spider displays the payload body in Request and Response tabs.

The selection is saved here for reuse, linked to the template of the request it cas defined on.

Duration unit

You may define in what unit to display the duration of Http Communications:

On the map
On the dashboard widgets
On the grid
On the details
In the exports
...

You may choose between milliseconds and seconds.

Actions

You may import the HTTP settings of any team you have access to by clicking on COPY TEAM SETTINGS.

Source

The Source tab shows the HTTP communication in JSON format.
Below is a sample HTTP communication as it is available through the API:

{
  "@id": "E4Fak43oR5K5hquEGBjbHA.enduser-orders-service-5dbff6d757-4fb8m.473758.192.168.11.75-37024-192.168.10.0-3001.4046159262.HTTP.0",
  "@type": "HttpCommunication",
  "version": "0.1",
  "name": "473758.192.168.11.75-37024-192.168.10.0-3001.HTTP.0",
  "whisperer": "E4Fak43oR5K5hquEGBjbHA",
  "instanceId": "enduser-orders-service-5dbff6d757-4fb8m",
  "tcpSession": "E4Fak43oR5K5hquEGBjbHA.enduser-orders-service-5dbff6d757-4fb8m.473758.192.168.11.75-37024-192.168.10.0-3001.4046159262",
  "httpPers": "E4Fak43oR5K5hquEGBjbHA.enduser-orders-service-5dbff6d757-4fb8m.473758.192.168.11.75-37024-192.168.10.0-3001.4046159262.HttpParsingLog",
  "stats": {
    "statusCode": 200,
    "statusText": "OK",
    "duration": 0.004927,
    "timespan": {
      "gte": "2024-01-17T22:07:15.992Z",
      "lte": "2024-01-17T22:07:15.997Z"
    },
    "src": {
      "ip": "192.168.11.75",
      "port": 37024,
      "origin": "192.168.14.167",
      "name": "gateway-59d55bf845-z5tvs.streetsmart-global-dev.pod.cluster.local",
      "socket": "192.168.11.75:37024",
      "identification": "pollersync_20180824"
    },
    "dst": {
      "ip": "192.168.10.0",
      "port": 3001,
      "name": "enduser-orders-service-5dbff6d757-4fb8m",
      "socket": "192.168.10.0:3001"
    },
    "tags": {
      "values": {
        "park": [
          "100000560"
        ]
      },
      "count": {
        "park": 1
      },
      "cardinality": {
        "park": 1
      }
    }
  },
  "req": {
    "start": 1705529235.992532,
    "end": 1705529235.992532,
    "startMinute": "2024-01-17T22:07:00.000Z",
    "startDate": "2024-01-17T22:07:15.992Z",
    "endDate": "2024-01-17T22:07:15.992Z",
    "method": "GET",
    "uri": "/v1/orders",
    "query": "modulo=1&remainder=0&view=content&afterUpdateMarker[]=280323&afterUpdateMarker[]=1338",
    "hash": null,
    "httpVersion": "1.1",
    "template": "POLL /orders",
    "headers": {
      "authorization": "JWT eyJhbGci...SuDUv4",
      "accept": "application/ld+json",
      "connection": "close",
      "correlation-token": "S0K_L7sidYUfQEU5fbGkjpLvkbdN5kJv",
      "host": "gateway",
      "x-forwarded-host": "gateway",
      "x-forwarded-for": "192.168.14.167",
      "x-forwarded-basepath": "/enduser/orders"
    },
    "size": 2088,
    "body": {
      "size": 0
    },
    "packets": [
      "E4Fak43oR5K5hquEGBjbHA.enduser-orders-service-5dbff6d757-4fb8m.473758.192.168.11.75-37024-192.168.10.0-3001.4046159262-3"
    ],
    "status": "COMPLETE"
  },
  "res": {
    "start": 1705529235.997459,
    "end": 1705529235.997459,
    "startDate": "2024-01-17T22:07:15.997Z",
    "endDate": "2024-01-17T22:07:15.997Z",
    "httpVersion": "1.1",
    "headers": {
      "access-control-allow-origin": "*",
      "access-control-allow-methods": "GET,HEAD,PUT,POST,DELETE",
      "date": "Wed, 17 Jan 2024 22:07:15 GMT",
      "content-length": "552",
      "correlation-token": "S0K_L7sidYUfQEU5fbGkjpLvkbdN5kJv",
      "vary": "Accept-Encoding",
      "content-type": "application/ld+json",
      "connection": "close",
      "x-response-time": "4ms"
    },
    "size": 869,
    "body": {
      "size": 552
    },
    "packets": [
      "E4Fak43oR5K5hquEGBjbHA.enduser-orders-service-5dbff6d757-4fb8m.473758.192.168.11.75-37024-192.168.10.0-3001.4046159262-5"
    ],
    "status": "COMPLETE"
  },
  "dateModified": "2024-01-17T22:07:30.783Z",
  "_update": 5,
  "_eTag": "\"96-cRQNDrTY6A12OJw7g0yxtMDs4+c\""
}

Most fields are self-explanatory. Most fields are indexed and searchable.

Some extra explanation:

Field	Description
stats	Stats object regroup most important information
stats.timespan	Indexed as range, to allow range searching and aggregations
stats.src\|dst.name	Hostname resolved on the Whisperer
stats.tags.*	Extracted tags values, count and cardinality
req\res.status	Parsing status
_update	Internal field linked to parsing concurrency control

HTTP communications

Concept​

Global Tab​

Timing​

General​

Request​

Response​

Correlation Ids​

Tags​

Spider info​

Actions​

cURL export​

Headers tab​

Getting raw header​

Request tab​

Response tab​

Packets tab​

Specific settings​

Correlation Ids​

JsonLd contexts​

Custom content types​

Duration unit​

Actions​

Source​

Concept

Global Tab

Timing

General

Request

Response

Correlation Ids

Tags

Spider info

Actions

cURL export

Headers tab

Getting raw header

Request tab

Response tab

Packets tab

Specific settings

Correlation Ids

JsonLd contexts

Custom content types

Duration unit

Actions

Source