Previously, when Spider tab was hidden in the browser, the background refresh tasks where still running, but browsers such as Chrome where blocking the network calls and slowing down Javascript timers.

This was generating a lot of error messages when the tab was made visible again.

After some documentation reading, Spider now listens to the event

visibilitychange

to know when it is visible on not, and deactivates background tasks when hidden, to restart them when made visible again.

No more pile of error messages in the morning! :)

I was wondering for some time how to improve Spider usage to add specific features for projects or system being monitored while not coupling Spider code to those systems, such as Streetsmart.

I found a solution on ideas that I pushed on Streetsmart business apps architecture. A solution to extend Spider's UI with plugins.

What are Spider plugins?​

Like nano frontends that add UI features into various parts of the application.

They are standalone Javascript files, implementing a specific signature depending on the plugin type. These plugins must come with their own manifest file providing metadata for the UI, and parameters definition.

The manifest is loaded in Spider UI, thus registering the plugin.

On the UI features accepting the plugins, they are loaded and called if activated (only then), and the results are injected in the UI.

Loading plugins​

Plugins are loaded in the Settings details:

Once loaded, they are displayed in the same tab, with various options:

You may:

• Enable or disable them
• Reload them (useful when developing them ;) )
• Remove them
• And edit parameters

Parameters​

Each plugin may require one to many parameters to work or adjust its work. Parameters are defined in the plugin manifest.

For now, accepted parameters types are:

• Text - free input
• Password - free input
• Select - list of options : labels and values

Parameters may have a default values and validation constraints, as defined in the manifest. Parameters and loaded plugins are saved in users settings, except for password values. Those are not saved on server side, only locally.

Manifest definition​

{
  "@id": identifier for the plugin
  "@version": version,
  "@type": type of the plugin, as defined in Spider
  "manifest": location of the manifest, for reload
  "dist": location of the code
  "name": name for the UI
  "description": description for the UI
  "parameters": [ list of parameters
      {
      "name": name of the parameter in the UI
      "description": description of the parameter for the tooltip
      "valueName": name of the parameter expected by the plugin
      "valueType": type of input (TEXT, PASSWORD, SELECT)
      "valueRequired": boolean
      "defaultValue": any
      "maxValue": for numbers / later
      "minValue":for numbers / later
      "multipleValues": if many values may be selected
      "stepValue": if values are generated from min to max
      "valueMaxLength": max length if string
      "valueMinLength": min length if string
      "valuePattern": regex if string
      "optionValue": [ list of possible options for selects
        {
          "label": string to display
          "value": any
        }
      ],
      "onlyOptionValuesAllowed": if select accepts unknown values
    }
  ]
}

Plugin common signature​

{
   inputs: {},
   parameters: {},
   callbacks: {setXXX, onShowInfo, onShowError, onShowWarning },
   libs: {React}
}

When called, the plugin is called with the above signature:

• inputs: specific inputs defined by the plugin type
• parameters: specific inputs defined by the plugin itself, in the manifest, the keys being the valueName
• callbacks:
  • functions to emit info, warning or error messages (toasts)
  • callback to return the result
• libs: libraries provided to the plugin
  • React is provided and may be used to format the output

Plugin injection​

async function myOwnPlugin({
   inputs: {},
   parameters: {},
   callbacks: {setXXX, onShowInfo, onShowError, onShowWarning },
   libs: {React}
}){
   ...
}

function plugin(props){
    myOwnPlugin(props).catch(e => {
        props.callbacks.onShowError('myOwnPlugin: Could not work!');
        console.log(e);
    });
}

plugin\['@id'\] = 'my-own-plugin';
plugin\['@type'\] = 'http-headers-decode-plugin';
plugin\['@version'\] = '1.0'

if(window.spiderPlugins){
    if(!window.spiderPlugins.find(p => p\['@id'\] === plugin\['@id'\])){
        window.spiderPlugins = \[...window.spiderPlugins, plugin\]
    }
}
else{
    window.spiderPlugins = \[plugin\]
}

• Plugins must load themselves inside windows.spiderPlugins array.
• They may be promises and perform API calls
• They must attach their @id, @type and @version to the function. @id and @type must be the same as in the manifest.
• The result of the plugin should be returned
  • As the promise result
  • And by the callback
• You're advise to copy the code above :)

Available plugin types​

http-headers-decode-plugin​

These plugins allow decoding HTTP headers, in request or in response, to reveal their inner data.

This was the original plugin need/idea: decoding Streetsmart rights that are compressed inside the JWT token.

Signature​

{
   inputs: {part, header, value},
   parameters: {env, login, password},
   callbacks: {setDecodedHeaders, onShowInfo, onShowError, onShowWarning },
   libs: {React}
}

The plugin is called for each header.

Inputs​

• part: 'req' for request headers or 'res' for response header
• header: name of the header, in lower case
• value: the header value

Output​

setDecodedHeaders callback expects an array of objects with this structure:

{
    key: String, label to display in the UI
    value: Any, value to display
    format: String, format of the value
    decoded: Boolean, whether or not the header is decoded (gray background in the UI)
}

• format may be one of:
  • application/json, application/xml --> displayed in ACE editor
  • text/plain or react --> displayed as such

examples​

• Decoding Streetsmart rights

• Decoding certificates

tag-enrichment-plugin​

This plugin extends the tag extraction feature by allowing to dereference / enrich a tag from data of your own system and inject these in the UI.

For instance, a tag could extract the city id of the client of the requests, and get the city name from the system under monitoring. Then filtering becomes easier, and Excel extract, statistics and grid becomes more meaningful.

Signature​

{
   inputs: {name, values, mode},
   parameters: {basePath, login, password},
   callbacks: {setDecodedTags, onShowInfo, onShowError, onShowWarning },
   libs: {React}
}

The plugin is called:

• For each row, and each displayed Tag in the Grid or Excel export
• For each group for stats grouped by Tag and Excel export
• For each Tag in the HTTP details panel
• For each Tag and all Tag values for the Filters component in the grid header

Depending on the call, the output is requested in REACT components mode (grids and details) or in TEXT mode (Excel, filters)

Input​

• name:
• values: array of tag values
• mode: 'REACT' or 'TEXT' for the output type.

Output​

Expected output expects an array of decoded values, in the same order, without holes as the values array input.

Output must be sent both:

• Using setDecodedTags callback
• With a return value (of the promise)

ExampleS​

Resolving parks names:

• In grid

• In filter

• In smart filters

• In details

• In stats

• In excel exports

Plugin example​

An http-headers-decode-plugin plugin example is available in https://gitlab.com/TincaTibo/certificate-decoder-spider-plugin/.

It may be reused and extended at will.

Before, when starting Spider, the timelline was computing the oldest and newest data of the Whisperer to set the starting range of the Timeline.

This query was costly on Elasticsearch side, as it required to load and search on all indices. Which, in the end, generated some timeouts the first times.

I changed the process, and now, the default timeline range is set to current day for realtime whisperers, and computed to min and max only for upload type ones.

To accomodate this new process, you may now zoom out on the timeline from the start, until you reach some defined limit or until you're tired ;)

Less timeouts, faster startup, better UX. Happy zooming and sliding !!

When opening a JSON-LD communication, you are allowed to choose another view to see it.

Thanks to the UI refactoring, it costed me almost nothing to provide an edit feature to these settings in the HTTP settings panel (and I had to find stuff to put there ;) ).

Do you fancy a quicker overview of items content? Without having to get to this remote close icon to close the detail panel?

Thanks to Bertrand who kindly ask for it, it is now easy :)

• Clicking on the grid opens the details
• Clicking again closes them !

Demo:

Talk to your doctor, it is better for your health to save those pixels of mouse movements ;)

For consistency, the same applies for sequence diagrams! Of course :)

Thanks to new Material UI features, I introduced the dark mode into Spider UI. One click in the settings, and you're ready for your long night work ;-)

Groovy :) !!

Another Bertrand request: you may now export the grid to an Excel file. All settings made to the grid are transferred to the Excel output:

• Selected columns
• Size of columns
• Duration unit
• And nice formatters

It is ready with predefined filter area and some metadata from the current search filters on Spider.

Again, Bertrand raised the fact that the map of the network is not needed for certain use cases. He asked if there was a way to hide it and go full screen for the grid.

Here it is ;) ! You may now enlarge the bottom drawer to the full height of you screen (minus search and timeline).

Requested by Michal K, Bertrand and who else, I added the possibility to choose the unit for the HTTP durations, between seconds and milliseconds.

I may look simple, but it impacts many places where i had to inject a single component (hopefully) to update all at once:

• Grid
• Sequence diagram
• Details tab
• Diff tab
• Filter histogram
• Map tooltips and arrow legend

React feels like magic: all are updated with a single switch :-D

I did not do it in stats. Too much work in current stats framework for low value.

As considered in Streetsmart, showing the resource name in breadcrumb is not always the  best, especially for generated resources.

Thus I improved the breadcrumb of Spider to display name of resource only when intelligible, and by always keeping first and previous item accessible

Try and tell me what you think! :)