To be able to capture more environments, Spider has moved to Cluster mode:

• Docker Swarm using Docker stacks
• ElasticSearch in cluster

Next step will be Redis architecture: ES is not fast enough in indexing.

So I'll put all input load to Redis only, and decouple performance needs by adding synchronisations pollers. Like we do on Streetsmart. I thought I could live without for longer using bulks... But, if I don't want to increase the hardware, I need to improve the architecture!

Swagger UI is available.

You can list, document and play with the client API of Spider: https://streetsmart.parkeon.com/spider/api

Cheers :)

First 'official' disclosure of Spider. I'm eager to get feedback, good or bad!

I've been working on Spider since December 2015. That's a reboot of a project I did in... guess when... 2003!

Back then, it was called RAPID Spy, because it was meant to spy and analyse RAPID architecture that was a trademark of Reuters Financial Software. It was used for analysis of functional test results and of technical and performance tests. But in fact, it revealed more powerful than that. RAPID was essentially Web based architecture. And if Web Oriented Architecture was still uncommon in 2003, I've nevertheless used RAPID Spy in all my projects since that time. It even help me decode a prioritary protocol of a COTS software we setup for one of our customers in 2008. I even built performance tests out of it.

In 2003, I bought back RAPID Spy rights from Reuters. I was sure I would reuse it, and they wouldn't do anything with it.

From RAPID Spy to Spider, the base concept is the same, but concerning architecture, technical expertise, evolutivity, scalability, there is a world! 12 years of experience building systems are in between.

Hope you'll like it! :-)