Skip to main content

Parsing status

Description​

This dashboard provides an insight on the parsing quality and speed of Spider.

It tells you if Spider tuning is right for the quantity of data to parse. It also tells how many items have been parsed successfully to find how many communications items.

It is the 'operational' dashboard of Spider.

Screenshot​

ParsingScreen.png

Content​

Parsing queued (timed chart)​

Shows the amount of TcpSessions and TlsKeys that are waiting in the parsing queue, over time.

The amount should be roughly stable around (15s * count of Tcp Sessions sent by second).
Having an ever increasing amount in this graph would mean that the parsing is not scaled enough to cope with the input.

If so, then:

  • For TcpSessions
    • Scale web-write replicas
    • Scale pack-read replicas
  • For TlsKeys
    • Scale tls-keys-linker replicas

ParsingScreen-ParsingQueued.png

Communications created (timed chart)​

Shows the result of parsing ovr time.

How many communications and secrets were successfully created, and how many errors were raised.

ParsingScreen-WebWrite.png

Tcp parsing status (timed chart)​

Shows the count of Tcp sessions, grouped by their parsing status, overt time.

Tcp session may be in:

  • Waiting or Pending, when they are queued for parsing
  • Incomplete when parsing is in progress, and session not closed
  • Warning when parsing failed
  • Error or OK, when parsing is finished

The less red, the better ! :)
Errors could have many factors, but mainly: CPU contention on clients or servers, resulting in missing packets when parsing is done.

ParsingScreen-TcpParsingStatus.png

Packet lots parsing status (timed chart)​

Shows the count of packet lots, grouped by their parsing status.
A packet lot is a group of packets that together form a request or a response. Most of the time, there are 2 packet lot for 1 communication.

The status may be:

  • Pending - Packet lot has been created, but packets where not fetch yet
  • Fetched - Packets have been fetched
  • Parsed - Packet lot has been parsed, but is incomplete
  • Discarded - Spider could not match this packet lot to an HTTP request, or a response linked to a request.
  • Filtered - Spider has filtered the packet log base on its configuration.
  • Fetching warning - Packet lot has been created, but packets are not available yet.
  • Fetching error - Spider could not find any packet for this packet lot.
  • Complete - Successful parsing.

Having other statuses than Discarded, Filtered or Complete means that there are issues in parsing.

ParsingScreen-PacketLots.png

Max 95% of parsing delay (timed chart)​

Shows the 95 percentile of the dalay between the creation of:

  • the Tcp session in the backoffice, and its parsing.
  • the Tls key, and its parsing

For Tcp sessions:

  • Spider waits 10s - by default - before starting the parsing.
  • Spider starts removal of packets from cache after 45s - by default.

So doing, this metric should stay between 10 and 45 seconds.

For Tls keys:

  • Spider waits 3s - by default - before starting the parsing.
  • Spider starts removal of packets from cache after 45s - by default.

ParsingScreen-ParsingDelay.png

Avg parsing duration (timed chart)​

Shows the evolution of parsing duration:

  • for a page of 20 TcpSessions max., including calls to pack-read to get packets.
    • this should stay around 25 - 40ms.
  • for a page of 20 TlsKeys max., including a call to tcp-update to get related Tcp sessions, as well as key generation (from secrets).
    • this should stay around 4 - 8ms.

ParsingScreen-ParsingDuration.png