What's coming here is the original reason for the rework of Templates. Templates are enriching the tooltips displayed on Nodes and Links, with new filters, more valued added data and other add'ins coming from the work I achieved on my monitoring UI.

See the changes by yourselves, they are live :)

Server nodes​

Before

After

The tooltip include a summary of the Node activity over the selected period:

• Count of replicas
• Count and rate of requests in and out
• Errors listing of there are
  • 4xx
  • 5xx
  • No response
• Load and speed of data exchanges
• And, for the 5 more used templates:
  • Count and rate of requests
  • Avergage latency
  • And errors

For errors and template a quick filter icon allows you to filter selection on these communications, or to remove them from selection (if clicked with Ctrl).

It is pretty handy :-)

Client nodes​

Before

After

As for servers, we now see rate of requests, templates of requests called and latency, with the previous information still present.

This makes it much quicker to see what client calls what.

Links​

Before

After

The links data has already been much improved, on the same model.

New over the arrow display​

As for the monitoring dashboard, I added over the arrow display on the chart. When overing a link, you quickly see the global request rate and average latency. As seen on the above screenshot.

Pinnable tooltips​

All the above tooltips can be pinned with their top right icon. This allows to quickly compare different time window, by moving the time selection on the timeline and seeing the metrics being updated!

Those tooltips may have other quick improvements in the weeks to come, with feedback from usage and from you! :) Tell me if you find this useful! The whole goal is to speed analysis up, more and more.

Cheers

JSON-LD recompaction​

JSON-LD payloads may be very custom. Between the client context and the server context, there can be many difference that fundamentaly change the JSON representation.

In Streetsmart,

• This feature is used to uglify and condense payloads from devices.
• This feature is used by clients that may use different contexts than the server's.

Some payloads become very difficult to read:

You may now define custom contexts to recompact the JSON-LD payload into a more readable JSON:

This payload is the same in JSON-LD sense. But much better in human eyes ;)

Contexts and templates​

In order help you even more, the custom contexts that you defined are saved in your user settings, associated to the template of the request / API endpoint (if a template has been defined).

Thus, you may define different contexts... for different endpoints, and you don't have to keep then anywhere, Spider does it for you :)

Review​

Please tell me how it feels after usage. It is a quick dev. So, suggest me any improvement you need!

Cheers!

Previous state of URL templates​

Since many months, it was possible to defined URL templates in user settings to perform URL matching in filtering and stats:

Config​

Grid filtering​

Stats​

It worked, but:

• It was a pain to configure - a regular expression for each URL
• It was not very explicit - the regular expression as label
• It was not efficient - the regulare expression were used as filters in Elasticsearch queries, which made Elastic run ALL regular expression on all URIs everytime the filter was selected... :(
• Templates were associated to a user but not a Whisperer or a team

I then did a major rework of the feature to be more efficient, user friendly and... to allow more features to be built on it :)

The new Request Templates!​

First, you may notice from the name that it is not any more URL templates, but Request templates. Why? Because not only the URL can be analysed, but also the HTTP method, the headers and the body.

You define for each template:

• A name
• The fields to parse
• A regular expression

And Spider associates the HTTP communication to the first matching template while Parsing.

Pros: much faster, flexible, easy to use and extensible. Cons: one time processing only. You cannot change already parsed and templated communications. Testing patterns is complicated, for now.

Look.

Configuration​

The Request templates are part of Whisperer parsing configuration.

• The regular expression is matched over a combinaison of selected request parts.
• The template name of the first matching template is saved in req.template field of the HTTP communication
  • The name can be plain text or include values extracted from the match
  • The regular expression is used as a substitution pattern over the name

The regular expressions above are enough to match all (or I believe so) endpoints exposed by Flowbird Streetsmart.

Ex:

For Streetsmart, the configuration is maintained in this Google Sheet. And then I just copy paste to Spider form. Gorgeous :)

Data filtering​

The templates can be filtered in the grid:

Or in the menu:

Detail view​

The template associated to the HTTP communication is visible in the detail view, that encountered a small reorder of fields:

As with any other field, you may add or remove the template from the filter with the small glass nearby.

Stats​

Templates are available as a grouping field in HTTP stats. And they make stats much more readable than previous version:

What's the use of selecting Headers of Body in template configuration?​

In some cases, like XML or JSONRPC, the URI is not enough to 'name' the request. Then, parsing the headers or the body becomes useful.

Example with PRM software configuration:

Which makes PRM exchanges much more readable compared to the URI only :) !!

However, this is to use with caution:

• Parsing the headers introduce more memory usage on the server, but this is ok
• Parsing the body implies to decode the body (manipulated as a buffer otherwise) and to uncompress it when it is encoded in Gzip, Deflate or Brotli.
  • As Spider is processing around 400 communications /s right now on Streetsmart, that could mean 400 unzipping every second if we have to analyse the body.
  • So it must be limited to certain conditions.

I'm thinking of splitting body from the other inputs, to limit body parsing to certain URIs... but I have issue on how to make it still easy to configure.

We can even make stats on PRM:​

However... it is difficult to split success and errors when an API always sends 200.

That's when you need Tags in place! They come next!

What's coming next?​

Request templates are allowing us to:

• Derive the feature to offer a 'tagging' feature (next blog item in row)
• Enrich the network map display and tooltips (in coming weeks)
• Implement a regular croned request on production to get regular automated stats

Deprecation​

Existing URL template feature is deprecated and is going to be removed soon. Should you have issues with this, please tell me.

When designing Request Templates, I figured out that the system could be extended to extract Tags from the communications.

With a similar configuration than for templates, you may extract values from the request or the response and add it in indexed fields in the HTTP communication resource.

Then these fields will be available in Grid, in filtering, in the HTTP communication detailed view, and in the Stats grouping.

Configuration​

• The tag name is used as the field name when storing:
  • req.tags.[name]
  • res.tags.[name]
• If the pattern matches, the tag is associated to:
  • The extracted values from the capture groups of the regular expression
  • true value if no group is present

Grid​

Tags can be displayed in dedicated columns. The columns are listed in the column list based on the configuration of selected Whisperers.

Detailed view​

Tags are listed in HTTP communication first tab:

Filtering and search​

Tags field can be use in searches as any others:

And drop down filtering is available next to column headers:

Stats​

Tags can be selected for grouping statistics:

To continue​

PRM​

All below examples come from tags extracted from PRM communications. The configuration is available on PRM Integration whisperer.

This configuration should be completed with Error tags for instance : tag the response as an error when the body says so.

Streetsmart​

• We may test extraction of park value from response to give statistics by park.
• Other ideas are welcome

However, we must be careful. Because of the cost of body decompression.

New feature added on request by Flowbird integration team:

It is now possible to save raw headers inside the HTTP communication resource.

• This is interesting when the packets are not saved.
• Filtering on headers is not applied on raw headers
• Raw headers are filtered out (like content) when searching HttpCom resources by default, but they can be included with "withContent": true parameter in search body
• Raw headers are automatically included when exporting communications

Effect​

The view source link in Headers tab is then working even if packets have not been saved.

Configuration​

On the Parsing Config tab of Whisperer settings.

You may also specify URIs for which you want NOT to save rawHeaders (in case of privacy concerns for instance)

Resource change example​

In req and res field:

"rawHeaders": {
 "content": "GET /v1/posEventStatuses?modulo=1&remainder=0&view=content&afterUpdateMarker%5B%5D=85597012&afterUpdateMarker%5B%5D=38696&range=100 HTTP/1.1\\r\\nAuthorization: JWT eyJhbGci...\\r\\nHost: itproduction\_gateway\\r\\nX-Forwarded-Host: itproduction\_gateway\\r\\nX-Forwarded-For: 10.0.0.129\\r\\nX-Forwarded-BasePath: /terminal/devicemonitoring\\r\\nConnection: close\\r\\nCorrelation-Token: 19IRIKnRm68N0Gl1x6AqtyLtNNa-ztCI",
 "size": 2990
 }

I recently added index lifecycle management principles in Spider.

The data usage of the cluster has been reduced greatly with index shrinking and merging. Worth it :)

Whisperers have moved from Pets to Cattle :-)

Previously each Whisperer needed to have its own configuration and was launched independently. If many Whisperers had the same configuration, there would be conflicts in parsing.

Now, Whisperers can have many instances sharing the same configuration. For instance, on Streetsmart, instead of having one Whisperer for each gateway in the cluster, there is one instance of the same Whisperer attached to each gateway.

• Setup is simple
• Configuration is simpler
• Usage is simpler (only one Whisperer to select in UI / API)
• UI speed is also faster, thanks to Whisperer based data sharding, and some preaggregation
• And it is opening the usage of Whisperers inside Kubernetes pods ;-)

Whisperer selection now looks like:

To see one environment, you only need to select SIT1 Whisperer, and you will see all communications from all its instances. The instances count is displayed next to the Whisperer name.

Whisperers instances status are merged into one: Clicking on the instances link gives the status of each Whisperer instance:

The name resolution tracked by each Whisperer instance are also aggregated into one in the backoffice, in order to fasten UI operations and name resolution of nodes.

This change required a big rework of the data architecture of Spider. Everything seems to work good and safe, but if you notice any weird behavior or bug, please keep me informed :-)

I hope you'll like it!

I added a check on all services startup: when they connect to Elasticsearch, they check that the expected indices or aliases exist.

=> It avoids compromising Elasticsearch indices with automated created ones, and warns from incomplete installation.

As Whoosh is sending Tickets to PRM using HTTP Expect 100 continue pattern, I added it to Spider.

What is it?

This HTTP protocol pattern allow a client to ask the server if he is allowed to send a request body BEFORE sending it:

• The client adds the header: expect: 100-continue to the request without sending the body
• If the server answers 100 Continue, then the client can resume
• And then the client can send the body and receive the 'real' response

Example:

Now, Spider can parse this communication pattern. The result is two communications:

• First one with the request headers, and the 100 continue answer.
• Second one with the request headers and body and the final server answer.

Example:

With latest Node.js Elasticsearch client, I was able to integrate Rollup search feature to Spider:

• The parsing status is pre aggregated after 5 minutes in a rolled up index
• The monitoring UI is then using both this rolled up index and latest active temporal index of Tcp Sessions to display the status Timeline

The result is much faster: 400ms for 5 days histogram instead of more than 8s.

Tips to use Rollup feature:

• Use fixed interval in rollup job. Not '1m', but '60s'.
• Use autogenerated index from ES, not a custom one (as I tried ;) )