Introducing the plugin store

Spider Plugin framework has been introduced in the beginning of year 2021.
This great feature enables to extend Spider by integrating business information into the technical data captured by Spider.

However great the feature is, it is not finished, and to make plugins available to Spider, you needed to have your own webserver and expose the files yourselves.

Spider now introduces its own Plugins Store, that allows users to upload plugins and make them available for everybody!

Context

A plugin is composed of:

• A manifest in json, with name, description, parameters list and a link to the code.
• A code file in javascript.

See: https://spider-analyzer.io/docs/extending/principles

Prior to the Plugins Store, the only way to load a plugin was with the input form below, where you input the URL of the manifest file:

Then, the plugin was downloaded and added in the users settings.
The issues are that:

• You need a web server
• You need to configure it to allow cross-origin resource sharing (CORS) with Spider website domain
• You also need it to be highly available

I wanted to change this by providing a small service allowing to upload plugins and to expose them.
As simple as a web server exposing .json and .js files.

First round

Steps	Work time
* Create a CRUD service * Deploy it to Kube * Expose in the monitoring * Manage specific upload service	3 hours
* Create a UI extension in the settings to show plugins * Tabs by plugins type * Upload tab * Diff features * Manage auth	5 hours
* Tests, deploy to SSS	2 hours

10 hours later, I add the stuff working and deployed! Then I figured that wasn't good enough 😅

Indeed, it became very easy to upload and share a plugin on a specific Spider instance, but how do you share them globally?
Ex: A plugin developed by Flowbird needs to be available to all Flowbird setups of Spider!

I needed to go global and push the plugins to a central service...

Going global

Design

Floocus already has central services to manage licenses, usage reporting and billing.

They are implemented using AWS SaaS solution to perform their operations:

• API gateway
• Lambda
• Dynamo DB
• S3
• SES
• ...

It works great at very low cost (0€/month in fact), but this comes at a price: the wake-up time of a Lambda is often around 2s.

How to make it so that plugin code would still be fast to download, and that it does not slow down Spider UX?

I could provision a small Lambda h24 to offer the service. It costs 15 / 20€ per month.
Cheap, but I'm sure I can do better ;-)

I got the idea to reuse the crazy idea I used for Floocus UIs:

1. Host the static files on S3
2. Trick the API gateway to expose them using HTTPS

note

You won't find the doc on AWS for this... they advise to use AWS Cloudfront or Amplify for this

Implementation

So I did it :)
And, as I already had all the infrastructure as code for Floocus, 4h later, I had:

• my Lambda,
• with a copy of my local service inside,
• storing to S3,
• with the same S3 being exposed by an API gateway directly!

Cheap computing on one side, H/A and full speed on the other side!
A small CQRS architecture 😁!

Steps	Work time
* Copy the service in a Lambda * Change Data Layer from ES to Dynamo and S3	3 hours
* Infrastructure to expose S3	1 hour

Next step, authorization...

There it gets complicated:

• How to give to a user of an independent system the right to manipulate a resource of another system?

Solution: build a proxy with a service account:

1. Transform the original local Plugins service to a proxy
2. Get a service account token linked to the licence in Floocus when checking the license
3. Use it for auth while managing plugins
4. And bonus: add a global feature flip to the service to use either local or global store

Steps	Work time
Service account token feature in license Lambda	0.5 hour
Adapt config service to get token and transmit it	0.5 hour
Add proxy features to local service	2 hours
Transform API and UI for a common search	1 hour
UX improvements	2 hours

Deployment

That's the easy part, as everything is already automated ;)

Steps	Work time
Package & deploy	0.5 hour

Last but not least: Documentation !

Documentation is crucial to remember everything, especially when it is done so fast!

Steps	Work time
Architecture	1 hour
User training update	0.2 hour
Website documentation	1 hour
OpenAPI update	0.5 hour
News (this one)	2 hours

To conclude

All in all, in 2 weeks, at 2h per day, 25h of work later the feature is out, tested and documented 😄 !!

Pfiou, though work!

Final architecture

Here is the final architecture of the solution, between local services and central services.

UX

Access

The Plugins Store is accessed through the Plugins tab in Spider main settings.

About

A first About tab explains more about plugins and provide links to the documentation.

Upload

The Upload tab allows to select the manifest and the code to upload.
Spider reads the files directly to provide information for the uploader to check its files.

In case of mismatch between the files, or with an existing plugins in the store, the user is visually warned.

Plugins tabs

Finally, plugins tab lists the plugins from the store available for the connected user.

Yes, I think I need to improve it a bit, I'm not satisfied yet. But it does the job!

Next steps?

Turn it to a marketplace? 😉

Any comment? Contact me !
Cheers!
Thibaut