Hi,

To answer client identification problematics, I've performed a few small updates in Spider.

Now the Http Communication resource includes 2 new properties:

• stats.src.origin: the original IP address from the client. Stored as an IP in ES, so queryable like this: stats.src.origin:"10.1.22.0/16"
• stats.src.identification: the identified client. Extracted by:
  • The login from Basic Auth identification
  • The sub field from JWT

Both fields can be queried and aggregated upon. Both are accessible in grid (you need to disconnect/reconnect to refresh grids available columns), and in detail view.

What's more, 2 other fields have been added in grid: the response date and the x-forwarded-for request header.

Enjoy!

Elasticsearch, Redis, Polling Queues, Circuit breakers, REST microservices...

Got an error, a time out in logs? Who generated it, what was the load, how was the system behaving at that time? How did it recover?

If you don't have integrated monitoring, you're driving blind!

I was starting to feel this way. I dedicated this week work to develop a new app in the system that monitors all others. Other applications are existing circuit breakers stats, queue stats, and soon REST stats, and I access ElasticSearch stats and Redis stats through their APIs.

All is collected continuously,  sent to another ES index, and Kibana helped me build easy and fast reports on spider health and speed through time. Really easy when you got all your pieces already in place and easily extendable!

Yesterday,  Spider encountered a major architecture upgrade:

• I moved to one index by resource type
  • To get ready for ES 6
  • To improve shards balance in the cluster
  • To reduce IO access because smaller volume resources are separated from big ones
  • To improve ES aggregations speed
• I introduced a poller between Redis and ElasticSearch for the four main resources
  • The load constraint is now only focused on two microservices and only on Redis
  • The load on ES is smoothed
• I added some in memory cache on Whisperer configuration access from microservices to reduce drastically unnecessary calls
• I have now 20 microservices in my cluster, with many being multi instantiated 😎

Streesmart instance has been upgraded with a complete data purge (sorry).

Let's look how it behaves. It should be much more stable!

To be able to capture more environments, Spider has moved to Cluster mode:

• Docker Swarm using Docker stacks
• ElasticSearch in cluster

Next step will be Redis architecture: ES is not fast enough in indexing.

So I'll put all input load to Redis only, and decouple performance needs by adding synchronisations pollers. Like we do on Streetsmart. I thought I could live without for longer using bulks... But, if I don't want to increase the hardware, I need to improve the architecture!

Swagger UI is available.

You can list, document and play with the client API of Spider: https://streetsmart.parkeon.com/spider/api

Cheers :)

First 'official' disclosure of Spider. I'm eager to get feedback, good or bad!

I've been working on Spider since December 2015. That's a reboot of a project I did in... guess when... 2003!

Back then, it was called RAPID Spy, because it was meant to spy and analyse RAPID architecture that was a trademark of Reuters Financial Software. It was used for analysis of functional test results and of technical and performance tests. But in fact, it revealed more powerful than that. RAPID was essentially Web based architecture. And if Web Oriented Architecture was still uncommon in 2003, I've nevertheless used RAPID Spy in all my projects since that time. It even help me decode a prioritary protocol of a COTS software we setup for one of our customers in 2008. I even built performance tests out of it.

In 2003, I bought back RAPID Spy rights from Reuters. I was sure I would reuse it, and they wouldn't do anything with it.

From RAPID Spy to Spider, the base concept is the same, but concerning architecture, technical expertise, evolutivity, scalability, there is a world! 12 years of experience building systems are in between.

Hope you'll like it! :-)