On request from Alessandro (thanks! :) ), two new features have been added:

Time reset​

As you can play an go in the future, in the past or lost in the middle of nowhen, I added a new button left of the timeline to reset time to the min/max of the current selected Whisperer(s).

I'm not satisfied with the color, i may change soon ;-)

Zoom reset​

Using Spider with a touch trackpad can be difficult for zoom and pan on the map. A quick help has been to change the behavior of the 'Center map' button that now reset zoom level as well.

Time limit in the future​

You may now go in the future (many asked for it). - This is useful not to have to change the selected time to get real time communications.

However, for sake of usage I limited the future to current end of day (local time).

As Whisperers and Users lists tends to grow with authorization, I did a quick but useful improvement: I added filters :) !

And as this is based on ES searches, you have full access to ES querystring DSL, and, what's even nicer... access to cross field search. Just type in 'SIT1' to get all SIT1 Whisperers. Or type S*T1 to get both SAT1 and SIT1.

Isn't it great? ;)

It is now possible to drag the timeline while holding control key!

No more need to zoom out and in to progress in time :-) Isn't that neat? You can even get in the future from now for instance if you want to get regular updates for the current day.

You can also click on the small arrow head left and right to shift the time line in either direction.

Happy sliding!

Thibaut

Spider has evolve much this past week.

Profile management​

• Admin can now create users directly from the UI

• Users can now edit their profile and change their password.

• And administrator can also reinitialize a password.
• Password are saved encrypted in database using one way encryption.

Rights management​

• Users are associated with rights
• Rights give special features access to users such as Passwords regeneration, and Whisperer creation

Whisperer sharing​

• Last important part (for now): Whisperers can be shared across accounts
• The Whisperer of an account can be shared in read only mode to any user that has a registered account
  • The user won't be able to upload data, change configuration or remove data
  • But it will see all the data captured by the Whisperer
  • This allow easier management of Whisperers on one side, and free access on the other side

Sequence diagrams now includes intermediate times to show the time spend thinking between two calls.

Enjoy!

New processing options have been implemented to offer more secured ways to run spider:

Packets​

• Save Packets: Save packets on server side or remove them once processing is done.
  • Allow preventing the reconstruction of sensitive communications when off.
  • On by default.

HTTP communications parsing​

• Save Content: Save the reconstructed payload of communications in higher level objects (HTTP).
  • Useful when save packets is off to still be able to see content.
• Headers to filter: List of regexp that are matched against the HTTP headers of both request and response. When one matched, the headers is removed from HTTP communication.
  • Useful for Basic auth headers that you wouldn't want to have readable by everybody.
• URIs to filter: List of regexp that are matched against the complete URI. When one matched, the communication is not saved.
  • Useful to avoid saving and searching on request to login endpoints for instance.

Hostnames generation​

• Name patterns: List of regexp that are matched against the FQDN of captured hostnames. First one to match is used to generate custom short name for Spider.
  • Regexp should include at least one capture group
  • When several capture groups are present, they are concatenated by dots '.' to generate the custom name
  • It allows keeping the store position on the host on the map when the host changes IP and name, but not short name. Like in Docker Swarm.

Following a need by Remi, now Spider UI has an automatic refresh every 60s.

The automatic refresh:

• Is only performed if new data may have come (current time is in your visualized time frame, and one of your selected Whisperers is recording)
• Refreshes
  • the global time span
  • the network map
  • the timeline
  • the grid / sequence diagram
  • and many others stuff, by side effects

Manual refresh is of course still possible with the icon!

The delay will be configurable on the UI... later. Ask for it if you need it! :)

This feature was long postponed because it seemed complicated to me. But with good GUI architecture and good IDE, it was much faster to implement than I estimated!! Estimated for 1 week work... done in 5 hours !

Spider can now display/search on several Whisperers at once!

By clicking on Shift/Ctrl, you can select several Whisperers at once, and combined their data in Network Map, Grid, Sequence diagram, and all the searches/filtering.

There may still be some bugs hanging around, but it seems to work fine :-) It is a bit slower to build the map because of the pre processing done in the browser, but you shouldn't see it much.

Now, I can tell Pawel that we can put several Gateways on SIT1 and co :) !! And I'm getting ready for complex systems!

Reference data:​

As captured on the Gateway​

As captured on ElasticSearch​

Combined!!​

Notice the small squares on the map that tells what nodes has been seen by what Whisperer?

Aren't they nice?

To answer a need defined by Bertrand B., and confirmed by others, you can now adapt the effect of clicking on the 'add to filter' icon:

• Simple click: add to the filters
• Ctrl: add to the filters, but in negative mode (NOT)
• Shift: replace all filters by this one (useful when clicking on a correlation token)
• You can combine Shift and Ctrl.

I don't know the correspondence with OS X for the Mac addicts, but I bet you'll find out ;-)

This works on:

• All links on the details panel
• Host links inside the hosts tooltips
  • You can filter out hosts
• Arrows on the network map
  • You can filter out unwanted communication channels

Isn't that neat ? ;-)

To avoid having to click on 'interpreted' every time you open an HTTP communications, I changed the default display mode:

• If Spider can parse and do color syntaxing on the content, the display defaults to 'interpreted'. This works with various content as:
  • XML
  • HTML
  • XHTML
  • JSON
  • Javascript
  • CSS
• For images, they are displayed in 'browser' mode:
  • Jpeg
  • Gif
  • Png
• For the rest, they are displayed as 'raw' by default

All this is based on the content-type of the request/response body to display.

JSON​

Image​