Customer profile
The customer operates on the market for IP traffic classification and network intelligence technology used in physical, SDN and NFV architectures. The company provides software for vendors who embed real-time application visibility in their products for traffic optimization, service chaining, quality of service, analytics, cyber security, etc. The company is based in Paris, France, and has offices in Santa Clara, CA, Singapore and Tokyo.
Business challenge
The customer turned to R Systems for a proof of concept in developing an analytics solution for large datacenter network traffic.
The challenge encountered by the customer’s development team was the particularly large volume of information which needed to be stored. The objective was to store up to 90% of 6 million information per minute for a “live period” of 10 days or the equivalent of 5TB of information (the system’s limit), whichever occurred first.
R Systems contribution and solution
R Systems undertook the consulting and development role in finding, proposing and building a viable solution to the customer’s challenge.
The solution consisted of deep packet inspection (DPI) probes collecting traffic information and sending all data to a Kafka cluster and aggregated it for extracting relevant data using specific widgets with specific filters and on different periods of time.
R Systems experts tested for different technologies, finally suggesting an innovative solution based on a combination of Apache Kafka and Elasticsearch, with stream processing custom Java implementation based on Kafka Consumer Groups, and distributed stateless Java applications (running at least 5 times faster than a Spark streaming based implementation).
Thus the solution was designed and proven to achieve high performance parameters, such as:
- ingestion speed at 150,000 records per second;
- 90,000 records per second inserted in Elasticsearch;
- 5 concurrent users being able to view auto-refreshing dashboards with complex aggregations.
Combining their expertise in the latest technologies with a thorough understanding of the customer’s business case, R Systems specialists succeeded to develop and implement an innovative solution to the challenges faced by the client’s own development team. In addition to the solid technical skills, the customer highly appreciated R Systems’ innovative approach and valuable role as a consulting partner.
For the complete case study in .pdf format, please click here.
CONTACT US