As Whisperers and Users lists tends to grow with authorization, I did a quick but useful improvement: I added filters :) !

And as this is based on ES searches, you have full access to ES querystring DSL, and, what's even nicer... access to cross field search. Just type in 'SIT1' to get all SIT1 Whisperers. Or type S*T1 to get both SAT1 and SIT1.

Isn't it great? ;)

It is now possible to drag the timeline while holding control key!

No more need to zoom out and in to progress in time :-) Isn't that neat? You can even get in the future from now for instance if you want to get regular updates for the current day.

You can also click on the small arrow head left and right to shift the time line in either direction.

Happy sliding!

Thibaut

Spider has evolve much this past week.

Profile management​

• Admin can now create users directly from the UI

• Users can now edit their profile and change their password.

• And administrator can also reinitialize a password.
• Password are saved encrypted in database using one way encryption.

Rights management​

• Users are associated with rights
• Rights give special features access to users such as Passwords regeneration, and Whisperer creation

Whisperer sharing​

• Last important part (for now): Whisperers can be shared across accounts
• The Whisperer of an account can be shared in read only mode to any user that has a registered account
  • The user won't be able to upload data, change configuration or remove data
  • But it will see all the data captured by the Whisperer
  • This allow easier management of Whisperers on one side, and free access on the other side

Sequence diagrams now includes intermediate times to show the time spend thinking between two calls.

Enjoy!

New processing options have been implemented to offer more secured ways to run spider:

Packets​

• Save Packets: Save packets on server side or remove them once processing is done.
  • Allow preventing the reconstruction of sensitive communications when off.
  • On by default.

HTTP communications parsing​

• Save Content: Save the reconstructed payload of communications in higher level objects (HTTP).
  • Useful when save packets is off to still be able to see content.
• Headers to filter: List of regexp that are matched against the HTTP headers of both request and response. When one matched, the headers is removed from HTTP communication.
  • Useful for Basic auth headers that you wouldn't want to have readable by everybody.
• URIs to filter: List of regexp that are matched against the complete URI. When one matched, the communication is not saved.
  • Useful to avoid saving and searching on request to login endpoints for instance.

Hostnames generation​

• Name patterns: List of regexp that are matched against the FQDN of captured hostnames. First one to match is used to generate custom short name for Spider.
  • Regexp should include at least one capture group
  • When several capture groups are present, they are concatenated by dots '.' to generate the custom name
  • It allows keeping the store position on the host on the map when the host changes IP and name, but not short name. Like in Docker Swarm.

Following a need by Remi, now Spider UI has an automatic refresh every 60s.

The automatic refresh:

• Is only performed if new data may have come (current time is in your visualized time frame, and one of your selected Whisperers is recording)
• Refreshes
  • the global time span
  • the network map
  • the timeline
  • the grid / sequence diagram
  • and many others stuff, by side effects

Manual refresh is of course still possible with the icon!

The delay will be configurable on the UI... later. Ask for it if you need it! :)

This feature was long postponed because it seemed complicated to me. But with good GUI architecture and good IDE, it was much faster to implement than I estimated!! Estimated for 1 week work... done in 5 hours !

Spider can now display/search on several Whisperers at once!

By clicking on Shift/Ctrl, you can select several Whisperers at once, and combined their data in Network Map, Grid, Sequence diagram, and all the searches/filtering.

There may still be some bugs hanging around, but it seems to work fine :-) It is a bit slower to build the map because of the pre processing done in the browser, but you shouldn't see it much.

Now, I can tell Pawel that we can put several Gateways on SIT1 and co :) !! And I'm getting ready for complex systems!

Reference data:​

As captured on the Gateway​

As captured on ElasticSearch​

Combined!!​

Notice the small squares on the map that tells what nodes has been seen by what Whisperer?

Aren't they nice?

To answer a need defined by Bertrand B., and confirmed by others, you can now adapt the effect of clicking on the 'add to filter' icon:

• Simple click: add to the filters
• Ctrl: add to the filters, but in negative mode (NOT)
• Shift: replace all filters by this one (useful when clicking on a correlation token)
• You can combine Shift and Ctrl.

I don't know the correspondence with OS X for the Mac addicts, but I bet you'll find out ;-)

This works on:

• All links on the details panel
• Host links inside the hosts tooltips
  • You can filter out hosts
• Arrows on the network map
  • You can filter out unwanted communication channels

Isn't that neat ? ;-)

To avoid having to click on 'interpreted' every time you open an HTTP communications, I changed the default display mode:

• If Spider can parse and do color syntaxing on the content, the display defaults to 'interpreted'. This works with various content as:
  • XML
  • HTML
  • XHTML
  • JSON
  • Javascript
  • CSS
• For images, they are displayed in 'browser' mode:
  • Jpeg
  • Gif
  • Png
• For the rest, they are displayed as 'raw' by default

All this is based on the content-type of the request/response body to display.

JSON​

Image​

Long time promised, here they come. Official release of automatic sequence diagrams representation in Spider.

• Available for the 3 current views: HTTP, TCP and Packet.
• Model the different actors from the servers IP
• Display requests, responses and timeline of all communications
• Display time on the left axis (non linear)
• Automatic layout with clients on the left, and servers on the right. In order of communication
• Allow opening a communication by clicking on it

Here is a sample for a Controls request on Streetsmart:

As the calls way and back to the reverse proxies in HTTP sequence diagram can be annoying, I implemented an automatic filtering of the reverse proxies:

Reverse proxies hiding!

• Spider will build the diagram be using the Origin client of the request (as communicated in x-forwarded-for header)
  • This show direct communications between original client and target server.
  • BUT, the communications are not merged. They are just hidden: You only see the HTTP request behind the last proxy.
• This applies to both the network map and the sequence diagram.

To use it, click on the icon option next to the HTTP view icon.