2024.04.22-2 Release Note

April release comes with 2 major features:

• Public Links
• Data Store Policies

Upgrade risk

warning

• No API breaking changes 👍
• ⚠ Compatibility issue
  • Requires helm chart version 3.0
  • Indices lifecycle definitions needs to be adjusted with new values
• ⚠ Data loss
  • Existing index for Links service is migrated from a time based index to a static one.
  • Existing data (links) are lost.

Key changes

• Public links
• Data storage policies
• U/X improvements

How to upgrade

1. Pull new Helm chart 3.0 from repository
2. Adjust global.version field value to 2024.04.22-2 in your values.yaml
3. Transform the global.elasticsearch.stores.operations key in your value file to the streamedData and uploadedData structures to define various Data Storage Policies. See Elasticsearch indices setup
4. Deploy

• Migration of Links index will be performed automatically, as well as removal of existing template and indices.
• Creation of new Data Storage Policies ILMs templates and indices will be done automatically.
  • Previous data will still be accessible, and their current indices will be removed at their end of TTL.

tip

See Reference documentation for details.

Versions

Spider

New versions of Spider components:

Component	Version	Docker tag
Helm chart	3.0	-
Analysis UI	10.6.1	2024.04.22-2
Controllers	1.2	2024.04.22-2
Whisperers	6.1	2024.04.22-2
Back office	-	2024.04.22-2
Login UI	-	2024.04.22-2
Monitoring UI	-	2024.04.22-2

Dependencies

These components are set up in the correct versions by the Helm chart:

Dependency	Version	Docker tag
Elastic stack	7.17.4	7.17.4
Redis	7	7-alpine
Traefik	2.11	2.11

Compatibility

Spider has been successfully tested under these versions of dependencies:

3rd party software	Version
Helm	3.14
Kube	1.24 - 1.28

List of changes

Helm chart

✨ New features	Public links Configuration in Links service APIs in Login & NetworkView Purge configuration Monitoring dependencies update Data Storage policies New Elasticsearch objects creation, removal of previous ones Configuration of Pollers and UI of available DSP
⚙️ Improvements	Links index migration
🐞 Bug fixes

Analysis UI

✨ New features	Public link feature Ability to create a public link to share a link to someone without Spider account Open the link and limit access Browse, search and delete links Data Storage Policy configuration in Whisperer parsing tab
⚙️ Improvements	Request and Response content-length headers value have been added to HTTP grid and HTTP filters in menu. Add loader for grid and sequence diagram when loading next page of infinite scroll Stats UX improvements: Fix loaders on diagrams and pivot table Limit height of pivot table to panel height Add sticky header and scroll ot pivot tables Add intermediate time resolution for evolution diagrams Use asynchronous queries to avoid timeouts when querying over huge datasets Heatmap improvements Paginate heatmap loading to avoid timeouts In dashboard and stats
🐞 Bug fixes	Use etag to avoid parallel update of Teams users and rights Manage HTTP communications with URI not including '/' (HTTP timeline diagram was failing) Allow uploading HTTP communications with UNKNOWN parsing status (for HTTP2 coms)

Controllers

✨ New features
⚙️ Improvements
🐞 Bug fixes

Whisperers

✨ New features
⚙️ Improvements
🐞 Bug fixes

Back office

✨ New features	Public Links Public Links APIs in Links service Time base data access limitation on data services Access limitations Data Storage Policies New poller version to take DSP into account
⚙️ Improvements	Email notifications on Team changes now includes the editor email address instead of its @id. A user with a Controller shared directly to him will have access to it even if a team is selected.
🐞 Bug fixes	Allow uploading HTTP communications with UNKNOWN parsing status (for HTTP2 coms) When having accessfilter on a Whisperer, accessing data of another did not work all the time Fix accessing Controllers permissions (they were not fetched correctly for non admins)

Login UI

✨ New features	Public link flow
⚙️ Improvements
🐞 Bug fixes

Monitoring UI

✨ New features	New graphics for Data Storage Policies stored size New sections to group charts and limit API calls when sections are closed
⚙️ Improvements	Added dependencies to Links added with Public Links
🐞 Bug fixes

Online documentation

• Public Links manual
• Data storage policies configuration

API impacts

note

This section informs about any impact on Spider API, so you may adjust your scripts.

• New Public Links APIs
• New Public Link schema
• Updated API descriptions for Search and Get APIs of
  • Hosts
  • Web-Read
  • Tcp-Read
  • Pack-Read
  • Customers
  • Whisps-Status

Data impacts

note

The table below tells if there are data mapping changes in Elasticsearch indices, associated or not with migrations (Yes ✅ / No ❌).
Migration are automated at upgrade time, but they may leave unattended indices that you have to remove manually.

Index	Description	Migration
spider-link-2024.0*.*-0*	Time based indices for links deleted	❌
spider-link-2023-03-31	New static index to store links
spider-{dataset}-{upload|streaming}-{dsp}	New indices created by DSP settings