156 posts tagged with "features"

With Streetsmart team help and Ace one, we improved Lucene syntax highlighting in Ace, and then in Spider.

The custom parser I put in place better parses:

• Field identifiers
• Numbers
• Dates
• Wilcards
• Regular expressions
• IP adresses
• Ranges

And has better parenthesis matching.

All in all, editing queries by hand has never been so easy :-)

Before going public, I needed extended features in user account management:

• User needed to be able to reinitialize his password when he forgot it
  • Without contacting the support
  • With security and identity confirmation: getting a link by mail to enter a new password
• User needed to be able to change email in a secured way
  • Since email is used for password recovery, changing mail must be secured by something stronger than current token
  • User then needed to enter its password when changing email
• User needed to be notified by mail when changing email or password
• Since email was getting very important, emails needed to be confirmed
  • When setting an new email adress, user need to confirm the email by cliking on a link sent to him by email
  • At user account creation, or email change
• Last but not least, a new user needs to be able to create its account with or without admin confirmation

First, I needed to create an email service.

Mail sender service​

This is a simple mail over REST service, integrated with a defined account.

• It can use SMTP, secured SMTP or Gmail account.
• It uses nodemailer library
• It integrates with gmail by OAuth tokens following these tips:
  • https://nodemailer.com/smtp/oauth2/
  • https://medium.com/@nickroach_50526/sending-emails-with-node-js-using-smtp-gmail-and-oauth2-316fe9c790a1
• API

Then I needed to define and develop the flows

Password recovery​

Process is the following:

• On login screen, click on the password forgotten link, and enter your email adress

• A mail is sent to your adress (if the account exists) with a link to change your password

• The link opens a new window were you may set a new password The link has a limit in validity and won't work after a short period of time The link can only be used once

Changing an email or a password​

When changing email or password in profile edition tab, Spider is now requesting that you provide current password:

Then,

• If you change email
  • An informative email is sent to previous mail to tell about mail change

• • Another email is sent to new email address with a link to confirm new mail
• If you change password
  • An informative email is sent to confirm change of password

Checking an email​

At account creation, or at email change, an activation email is sent to user's email address to confirm his email:

This email contains a link to confirm the new address. It is valid only once and for a limited time.

• Clicking on the link validates the email change.

• Not doing so on time... locks down the account, and only an administrator can unlock it.

Creating an account​

A new user can now create an account from the login screen:

A creation form opens to input user details:

At the bottom of it, a comment input allows to give information for the administrator validating the account (if need be).

The new user will receive an email to confirm his email address.

Depending on platform settings, the account may need validation/activation by an administrator.

• The latter will then receive an email automatically, once the email is confirmed.
• He can also associates whisperers to the new account, if requested.

Thanks to Error boundaries and refactoring of errors on the saga side, I implemented something similar to what sentry.io is offering:

• Client side errors tracking

Indeed, Sentry is right: clients do not send you the errors they found, it is best to get them yourselves! If you want to troubleshoot easily your GUI errors, you have to know them.

So this is what I did:

Sending the client side errors​

• Saga error and React error are caught globally
• Once caught they are enriched with as many metadata as I can
  • Last request performed in case of Saga errors
  • What React component failed in case of React errors
  • A dump of current Redux state
    • I reused the 'share' feature :)
  • Last user action performed
    • I had it from my session tracking statistics feature
• Then the log is sent to Spider server in background.
  • If this sending fails, nothing is done, except console logging.

Reporting on the client side errors​

As I'm already using Elasticsearch to store most resources, storing the UI logs inside was peanuts. And then, I had included them in Spider monitoring UI:

• I know how many errors were done and when

• I can browse them

• I can open them to find details and troubleshoot

Here it is a safety belt when the interval of timeLine is too small comparing to time range. I need to do something on timeline to avoid having these errors. See, it is useful! :)

• I can search for similarities
• And... I can open the user state in Spider UI and reproduce the error, just by copy pasting the link :-)

Amazing !

React 16 allowed me to integrate a much better toast library than the one offered initially by Material UI: react-notifications-component

This library offers:

• Notifications types: info, warn, error...
• Customisation with images, titles and so on
• Notifications stacking
• Custom placements...

The integration was purely limited to changing the encapsulation of Material UI notifications and offering more types to saga errors. Quite quick :)

Here is an example of notifications when you try to zoom too much (with the scroll):

Long time no news! Not that Spider did not progress during 3 months, more that I was lasy to write about it.

There are been a series of improvements, and I'm about to share the most important ones.

All UIs have been migrated to lastest React: 16.8​

This was much needed and postponed, as I was still using React 14. Migration was mostly seemless witht the nice tooling offered by the community, and the scripts to converts the code (extract PropTypes, use PureComponents...)

Libraries upgrade​

In the same time, I upgraded nearly all libraries. Few libraries couldn't get upgrade as the changes were too big:

• Tabs: the library I used was not maintained (react-tab-panel), and the repo has been removed. I switched to react-tabs. Which offers less features, is simpler, but is well maintained.
• Redux
  • Redux was not upgraded to latest version as this one was using React context and was showing performance issues. I have to wait for next one.
• React-select
  • React-select did a major rework and change, and upgrade will need a seperate task. Moreover, it did not fill needed.
• React-dropzone
  • Major interface break as well, less prioritary.
• Material-UI
  • Material UI encountered many major version change, and this would impact much of the UI to upgrade it.
  • I created a separate task for this.

New feature: Error boundaries​

Coming with React 16 are improved error boundaries. The features help catching rendering errors in React component tree and handle them gracefully by rendering a fallback when the error is catched.

I encapsulated all branches of Spider UI into these new error boundaries, so that the error is nicely catch and displayed. And so that avoiding the source of the error (changing record, or timespan) would avoid the error without requiring a full refresh of the page.

• The bottom grid
• The detail view
• A detail view tab
• The map
• The timeline
• The menu

When an error happens, instead of the expected content, you will see a poor fallen spider ;)

Here is a sample of an error on an HTTP request detail:

I corrected some Javascript and CSS, and now Spider UI has been successfully tested on:

• Chrome
• Chromium
• Firefox
• Opera

I always used latest version of course. There might be some unseen issues, please inform me if you find any.

Edge will not be supported as too limited, and for Safari... I don't have an Apple laptop to test ;-), sorry!

I did some reverse engineering over ACE autocompletion and syntax highlighting library and managed to improve Lucene autocompletion without changing the library :-) Man, I wish there were better documentation!

Improvements

• Regular expressions are recognized in syntax highlighting
• Numbers and ElasticSearch operators are recognized as such

• Autocompletion do not take anymore the '.' as an identifier delimiters
  • So, no more strange behavior when selecting a subsubproperty in the completions options

• Autocompletion avoid proposing to insert identifiers in the values anymore
  • I did not manage to handle all cases, but most are

2 new charts have been added to monitoring dashboards:

Redis load/s​

Tracks the load required on Redis instances in the cluster.

Redis is fast, we can notice up to 12k requests/s on one instance. Compared to CPU usage... it is almost 1k req/s = 1% CPU. Impressive !!

Services load/s​

Tracks the services load in the cluster.

Here there is no direct relation to CPU usage. As the process of each service can be quite different.

After much use and comments, the way Whisperers were selected on the UI needed improvements.

• We are always working with more than one Whisperers in our distributed systems
• Then, having to click ctrl or shift to select them were painful
• Moreover, having to get to one Whisperer only to edit its config was also painful. Too many clicks.

So I decided to change:

• Whisperers selection is now acting as a real checkbox
• You may use ctrl or shift while clicking to select only one
  • It then works better on my tablet ;-)
• And from the Whisperer selection drop down, icons are now giving direct access to its status tab and to its config tab.
• Consequently, the Whisperer config button has been removed from the main menu.

Isn't it neat? Small changes that make life easier :-)

Goto Now​

On special request from Michal D., you may now click on the icon below to move the selection and timeline to current time. It is even more easier to troubleshoot your current work :-)

Improvements on timeline​

I took the benefit, while I was working on it to improve timeline U/X:

• When shifting / dragging the timeline while you were zoomed in, the higher zoom levels are updated (shifted) as well so that you don't feel lost when zooming out.
• When moving selection outside current domain of timeline, the timeline shifts
  • Thus, the timeline will stay up-to-date with Step backward and Step forward actions.

Bugfixes​

I solved a bug on race conditions when updating timeline and grid. This should fix the few cases when the grid / timeline were not updated.