Following the work on Monitoring UI, I upgraded main UI.

• 2020-08 - Restructure of code for better maintenance.
• 2020-11 - Migration finished to latest versions of React, Redux, Material UI v4, and migration to Hooks.
• 2020-11 - Refactor of all code to have all display and processing related to a protocol in a single folder, loaded by dependency injection in the core of the application: Components, tabs, UI parts, actions, sagas...

Result:

• Components are regrouped in
  • base components, that can be shared across UIs, and agnostic to this specific UI
  • specifics components, that are coupled to the state structure or reall business specific
• All application specific work is located in screens folder
  • includes the app initialization and state recovery
  • includes the screen structure and coupling to the theme
  • includes containers to couple base components to the state
  • includes a collections folder in which each collection include the specific for a resource:
    • details view
    • diff view
    • download package builder
    • upload parser
    • filters definition
    • grid and excel columns definition
    • map nodes and links definition
    • sequence diagram links building rules
    • stats definition
    • timeline configurations
    • settings tab
    • whisperer parsing config options
    • custom sagas if needed

The structure is much better, as well as troubleshooting :) I refactored / rewrote much of the code that dated from my starting months with React / Redux. I could still be improved, but it is much better structured and architectured! :)

HOW MUCH I LEARNED building Spider !!

Next step: extract the protocols folders into plug-ins. But this is another story!

I rework Tags features to merge request and response tags into one result. Indeed, after using it, it offers better UX.

For instance, in Spider, you may defined Park id extractor:

These regular experssions will instruct Spider to extract the internal id of the parks referenced in the communications, and to save them in park Tag.

Spider will then, when parsing, unzip the communications, run the regular expressions and extract the values.

The park column can then be added to the grid and filtered. Thus adding a nice filtering criteria when working!

Last, but not least, you may even do stats! For instance, let's compare the latency of parkingRights call depending on the park configuration:

That's amazing !! :)

As requested by Bertrand (he had a lot of inspiration ;) ), I added resize handlers so that the resizing capability of panels is more visible.

I also improved drag and drop speed where I could by postponing store changes to a local state while dragging and using Memoization of components.

New independent alert service​

After much study of Prometheus alert manager and some other solutions, I decided it would be faster and cheaper in resources and time to implement a basic alerting service to start with :) Especially the security integration...

So I did it. A new service (45MB - 0% CPU) is checking various metrics from the monitoring and sending mail to administrators in case of issues. And it sends again after some time if the problem is not solved.

Complex rules can be written... as it is code ;-) Took a couple of days, and works like a charm! Of course, it is - itself - monitored in the monitoring, & integrated in the automated setup ;)

Implemented probes​

• ES healthcheck
• Redis healthcheck
• Change in infrastructure (increase or removal of nodes)
• Low ES free space
• No new status (Whisperers down)
• Too many logs over last minute

Of course, alert are sent when the probe cannot work.

I started a BIG and LONG task: removing the technical debt on the UI side:

• Libraries updates
• Moving React to function base and hooks whenever possible
• Refactoring
• Material UI upgrade + CSS-in-JS + theming approach

First application to be reworked: Monitoring UI. It allowed me to start with a full application, while doing common components with Networkview, while not struggling with the complexity of the latter one.

Timeline component was refactored too (https://www.npmjs.com/package/@spider-analyzer/timeline), which leads to a much easier maintenance :)

The result being not so much visible (apart in the code), I took the opportunity to introduce one feature while doing it: the dark mode. Often requested by users, and being facilitated by MUI theming :)

Here it is:

Dark mode can be activated in the settings.

... Now, let's get this work to NetworkView UI !

On May 15th, Elastic team released Elasticsearch 7.7 introducing asynchronous searches. A way to get partial results of a search request and to avoid waiting indefinitely for a complete result.

I saw there a way to improve User Experience while loading Spider, and to avoid the last timeouts that are still sometimes occurring when generating the Timeline or the Network map over many days.

So, here it is, 9 daysafter ES 7.7 release, the implementation of async searches in Spider is stable (I hope ;) ) and efficient!

Normal search

Async search

I stumble a bit at the beginning to find the right way to use this:

When to use partial results​

Loading partial results while data were already present meant resetting the existing map or timeline. The result was ugly and disturbing. I decided to limite partial loads to initial load, whisperer switch, view switch... In other words... when the resultset is empty before searching.

API integration​

Although ES does not require clients to send again the query parameters to get the async search followup, Spider API does.

Indeed, the async final result may present a 'next' link to get the next page. This link is built as hypermedia and includes everything necessary to proceed easily to the next page.

As Spider is stateless, the client is required to send all request parameters for all async follow up, in order to allow Spider to build this 'next' hypermedia link. Spider makes it easy to comply with, by providing another hypermedia link with all parameters to get the async call follow up.

Client usage​

I also tested several solutions to chain the calls in the client (UI) to finaly find that Elastic team made it really easy:

• You may define a timeout (wait_for_completion_timeout) to get the first partial results in the first query.
  • If the results are avaiable before, you get them straight, as a normal search.
  • In the other cases, you get a result with partial (or no) data.
  • On further call, you may get the progress of the search straight... or also provide a timeout

The beauty of this is that you don't have a drawback in using or not async. If you use timeouts, you always get results when they are available. :)

At first, I implemented it so:

1. Async search + timeout
2. Wait 250ms after partial results
3. Call follow-up
4. Wait 250ms
5. ...

But this method may lead you to get results later than they are available. Which is bad for a real time UI like Spider.

Using Timeouts in a clever way, you combine partial results, and ASAP response:

1. Async search + timeout
2. Call followup + timeout (no wait)

With this usage, as soon as the query is over, ES give you the results. And you may propose an incremental loading experience to your users.

Implementation results

I implemented async search on UI for the following 'long' queries:

• Timeline loading
• Timeline quality loading
• Network map loading
• DNS records loading

On all 3 views (HTTP, TCP, Packet), with 1s timeouts.

The effect is visible only when loading the full period with no filters. Indeed, other queries are way below 1s ;-) On automatic refresh, you won't see the change. The queries are not 'magically' faster: doing time based aggregation on 30 millions communications still takes... 4s :-O

As it is still new, I may have missed some stuff, so you may deactivate it in the Settings panel, to get back to normal search:

 

Share me your feelings ! :)

References

• ES 7.7 release
• ES async search description

I discovered yesterday on the web that Redis UNLINK allowed much faster answers than DEL.

Using Spider bulk edit tools, I updated all services using DEL in their Dao layer or in Lua in a glimpse, comitted, build and pushed images, redeployed, and immediately saw the benefits !

I had a couple of 'slow' bulk requests that were taking more than 15 ms, and now, all are below 6ms :-)

Speedy Redis!

Elastic stack 7.7 is out!

I already upgraded Spider Beats, Kibana and Elasticsearch ;)

Speed is there, as usual, it seems even faster than before, as foretold by Elastic team :). What is most interesting is the new Async Search! I'll try it ASAP and keep you posted!

My goal is to use it on the big aggregations on the UI (Timeline and map) to get faster answers and progressive loading!

I recently switched to using Docker volumes for logs., to reduce installation steps and coupling to infrastructure. I created a single shared volume by stack.

However this implied that logs from stopped containers were never removed... because logs rotation was not doing its work anymore.

I then understood better why 12 factors app practices recommends to only log to STDOUT, and let Docker handle with log storage and removal. And I decide to adapt.

• Stop JSON logging on files
• Replace human readable logging on STDOUT by json logging
• Change filebeat configuration to container mode

At the same time, I benefited from this change in many ways:

• Traefik, metricbeat and filebeat logs are also captured
• Elasticsearch and kibana logs are captured on dev
• All logs have been switched to JSON
• Filebeat allow enriching logs with Docker metadata, allowing to know easily the origin of the log
• Filebeat processors and scripting allow reshaping the logs to have a common format for all sources :-) Thanks Elastic devs!

It is all in place and deployed! More observability than ever.

I got the idea when working on U/X design for Flowbird products:

• How to be more 'user friendly' with filters and query
• How to show all active filters in a 'nice' way

I figured out that I could avoid adding all filters visible in the UI query component, and display all filters one by one as tags on the UI. This even brought the possibility to add modification options:

• Deactivate the filter
• Invert it
• And of course, edit the filter in the query

Outcome

After much refactor and UI development, here is the result:

• Filters are displayed as tags at the top left of the screen.
• Filters are displayed with as much 'functional' labels and value as possible

• Filters can be removed from the set with their close button

• Filters can be deactivated, invert (include or not), manually edited
• When saving a query, all filters and tags are saved with the query

• A loaded query can be broken down again in the previous filters for edition

The feeling is really great. It is easy to load a query as default filters, then search, drill down to the selected items. Then cancel some filters and search for something else.

Play with it, tell me what you think! :-)

Menu disposition change

Due to the lower importance of the free search now - a filter as others -, the save/load query, undo and clear filters (new icon) buttons have been moved to a dedicated tool bar.