I did some reverse engineering over ACE autocompletion and syntax highlighting library and managed to improve Lucene autocompletion without changing the library :-) Man, I wish there were better documentation!

Improvements

• Regular expressions are recognized in syntax highlighting
• Numbers and ElasticSearch operators are recognized as such

• Autocompletion do not take anymore the '.' as an identifier delimiters
  • So, no more strange behavior when selecting a subsubproperty in the completions options

• Autocompletion avoid proposing to insert identifiers in the values anymore
  • I did not manage to handle all cases, but most are

2 new charts have been added to monitoring dashboards:

Redis load/s​

Tracks the load required on Redis instances in the cluster.

Redis is fast, we can notice up to 12k requests/s on one instance. Compared to CPU usage... it is almost 1k req/s = 1% CPU. Impressive !!

Services load/s​

Tracks the services load in the cluster.

Here there is no direct relation to CPU usage. As the process of each service can be quite different.

Thanks to the latest knowledge I got from Javascript and HTML5, I managed to remove the need for Cookies in Spider UI!!

So, Spider is low fat now, no more cookies needed, the JWT token is managed in all communications to the server.

Hi,

Over the last couple of weeks, I took over the - not so interesting - mission of updating the API specification. It is available here: https://spider-analyzer.io/home/api/

Changes:

• Specification updated to Open API specification 3.0
• New objects diagram
• All APIs are described with:
  • Structure of inputs
  • Structure of responses / resources
  • Examples
  • Quick start guide
• I improved the usability of the API with:
  • Date parameters available on top of timestamp parameters: startDate can be used instead of startTime
  • Added hostnames in TcpSessions and HttpCommunications resources for easier search on FQDN

During the API review, I found some security issues... All are fixed now ;-) So doing, I added:

• The possibility to purge on shared Whisperers for those who have the right. (Before, it was free for all ;))
• A protection against wrong login: after 5 attempts, the account is blocked for some time
• More security around customers details access
• The name of the customer owning the Whisperer in Whisperer details's view

Wilfried already tested it, and... was happy to integrate it in its test results checks ;-)

Cheers

December has been a month of migrations for Spider. And how much I'm happy to have done them! Read below.

Migration path​

• From NGINX to Traefik on 6/12
• From Node 7 to Node 8
• From Javascript generators to async/await programmation pattern
• From eTags generation on full resource to eTags based on id+lastUpdate date (CPU saving)
• From Node 8 to Node 10 (actually done in January)

The result of all this?​

1. A division by 2 to 5 of microservices CPU usage !
2. A division by 4 to 6 of response times as seen from Whisperers !!
3. A division by 5 to 12 of internal response times !!!

This was amazing!

I did not believe it, but yes that's proven. Node was saying that async await was faster than generators, then Google improved async/await processing speed by 8x in the V8 version embedded in Node 10.

Examples​

• Processing of packets improved from 484ms 90th percentile for 100 packets bulk request to ... 69 ms!!
• Patching TcpSessions improved from 266ms 90th percentile to 13 ms!!! Excuse me!
• CPU using of parsing job improved from 172% to 80% and for packets decoding, from 295% to ... 55% !! Amazing :-)
• Previously Spider needed 4 servers to process 2500 packets/s, now... i could almost do it on two, and much much faster ! :)

Conclusion​

Yes, Node.js is blazingly fast. This was right for callback mode, and now it is back for async/await ! :-)

Figures​

Source: Google spreadsheet

And Streetsmart?​

And you know what? Streetsmart is almost in a state before all my migrations. Imagine if migrations have the same effect for Streetsmart. It would be awesome ! :-)

Well, that's part of my plan indeed!!

NGINX​

Spider internal cluster gateway was until this week NGINX. However, NGINX was presenting various issues in current setup:

• In order to absorb scaling up and down of replicas, I was asking NGINX to resolve the IP of the services VIP on every call. DNS resolver had a cache of 30s.
• The main issue was that NGINX can't do persistent connections to the upstreams in this case.
• This made NGINX create a new TCP socket for every request. But soon enough when all TCP sockets were booked, it implied an increase of response time of 1 or 2s for linux to recycle sockets.

Change was needed!

Traefik​

Traefik is more and more used as a gateway for Docker clusters (and others). It is indeed designed to integrate with Docker and to keep updated with cluster state.

So I switched Spider to Traefik this week. And the results are ... astonishing !!

Although the response time from the clients have not changed much, the response time internal to the cluster have improved of 80% !!

Note: I only struggled with Traefik configuration on Path rewriting.  It has less options than NGINX on this field. I had to add some custom rerouting inside the UIs themselves.

Spider was working really fine with Docker Swarm until... Docker version 18.06.

Docker v18.06 improved load balancing scalability impact​

Docker 18.06 includes a change to improve scalability of Swarm Load Balancing: https://github.com/docker/engine/pull/16. The impact on Spider is so:

• Previously, when sniffing communications between services, the IP used were the IP of the replicas that sent/received the request (DSR mode).
• Now, the IPs used in the packets are the VIPs (NAT mode).
• The main issue is that... the VIPs have no PTR record in Swarm DNS, and so, the Whisperers cannot reverse resolve their names... And Spider is then much less usable.

Workaround with hosts preresolving in Whisperers​

To overcome the problem, I added the possibility to give a list of Hostnames to the Whisperers that are resolved regularly against the DNS and preloaded inside the DNS resolving mecanism.

This has many advantages:

• You can define a list of hosts without PTR records.
• Docker resolving works better that reverse resolving (more stable: you don't face bug: )
• The list can be given in Whisperer config (UI) or through environment variables of the Whisperer: HOSTS_TO_RESOLVE
  • Thus, you can script the Whisperer launch and get, at start, the list of services in the Swarm prior to launching.

This has a main drawback: you loose the power of service discovery... as the list is static. The other way would be to get the info by linking the Whisperer to the Docker socket... But this is a security risk, and would tie to much to Docker.

Localhost IPs preloading​

While at it, I added another environment variable : CONTAINER_NAME. When present:

• The list of local container own IPs are preloaded in the DNS resolving mecanism with as hostname value the CONTAINER_NAME value.

Docker 18.09​

Docker 18.09 includes a parameter when creating the Overlay network to deactivate this new NAT feature and be like before: --opt dsr

With this parameter active, Swarm behavior is back to before 18.06, and Spider works like a charm. But at the cost of scalability.

If scalability is a matter, while using Spider, the best is to move the cluster replica settings from VIP to DNSRR, and use a load balancer like Traefik ;) See my other post from today.

After much use and comments, the way Whisperers were selected on the UI needed improvements.

• We are always working with more than one Whisperers in our distributed systems
• Then, having to click ctrl or shift to select them were painful
• Moreover, having to get to one Whisperer only to edit its config was also painful. Too many clicks.

So I decided to change:

• Whisperers selection is now acting as a real checkbox
• You may use ctrl or shift while clicking to select only one
  • It then works better on my tablet ;-)
• And from the Whisperer selection drop down, icons are now giving direct access to its status tab and to its config tab.
• Consequently, the Whisperer config button has been removed from the main menu.

Isn't it neat? Small changes that make life easier :-)

Goto Now​

On special request from Michal D., you may now click on the icon below to move the selection and timeline to current time. It is even more easier to troubleshoot your current work :-)

Improvements on timeline​

I took the benefit, while I was working on it to improve timeline U/X:

• When shifting / dragging the timeline while you were zoomed in, the higher zoom levels are updated (shifted) as well so that you don't feel lost when zooming out.
• When moving selection outside current domain of timeline, the timeline shifts
  • Thus, the timeline will stay up-to-date with Step backward and Step forward actions.

Bugfixes​

I solved a bug on race conditions when updating timeline and grid. This should fix the few cases when the grid / timeline were not updated.

Description​

This dashboard provides statistics on Spider UI usage: Connected client over time, Usage statistics, Users statistics, Session history and Actions usage.

This dashboard has a different timescale as the others. Indeed, the UI tracking records are not purged at the same time as all other operational data. You can choose on the time scale to display:

• Actions
• Sessions
• Jobs (purges & uploads)

Screenshot​

Content​

Simultaneous sessions over time (chart)​

• Tracks number of simultaneous connected users using on Spider UI over time
• X axis: days
• Y axis: hours

Actions per session distribution (chart)​

• Show distribution of count of actions per sessions

Network view usage stats (chart)​

• Show usage time statistics of Spider UI Views (HTTP, TCP, Packet) and Mode (Stats, Sequence diagram, Stats)

Self Monitoring usage stats (chart)​

• Show usage time statistics of monitoring views.

NetworkView options stats (chart)​

• Show options usage of NetworkView

Most used actions (chart)​

• Show most used actions in the UI

Most used Whisperers groups (chart)​

• Show most used Whisperers groups (by time)
  • By grouping them on their first 4 characters

User stats (grid)​

• Lists each users usage statistics of the UI over the selected period
  • Total actions
  • Total duration of sessions
  • Total hours of usage (each hour started counts)
  • Total count of days of usage
  • % of working day usage (sundays and saturdays are removed from reference)
  • Last usage date and time

Session items (grid)​

• Lists users sessions items
• Preconfigured to display
  • Start time of session
  • Application
  • User
  • Duration of session
  • Active hours
  • Browser reloads count
  • Actions count
  • % spent on each view
  • % spent on each mode/subview
  • % spent on each options
• Common Spider features on grid:
  • Allows opening the status record in the detail panel
  • Allows comparing items
  • Full integrated search using ES querystring with autocompletion and color syntaxing
  • Many fields to display / hide
  • Sorting on columns
  • Infinite scroll

Upload and purge items (grid)​

• Lists frontend / backend jobs triggered by users
  • Purges
  • Uploads (Pcap and Json export)
• Preconfigured to display
  • Date of creation
  • Job type
  • Whisperer used
  • Status of action
  • Progress
  • Duration of job
• Common Spider features on grid:
  • Allows opening the status record in the detail panel
  • Allows comparing items
  • Full integrated search using ES querystring with autocompletion and color syntaxing
  • Many fields to display / hide
  • Sorting on columns
  • Infinite scroll