Scalæ By the Bay has ended
Back To Schedule
Saturday, November 12 • 2:10pm - 2:50pm
Implement a scalable statistical aggregation system using akka

Sign up or log in to save this to your schedule, view media, leave feedback and see who's attending!

At Symantec email security group, a common problem we face is to aggregate multiple metrics with different time granularity in real-time from hundreds of millions emails per day. Various existing solutions try to address the problem by using batch and/or streaming algorithms. Often such approach requires the use of many different technologies and are expensive to run. Another approach is to use statistical data structures such as Count Min Sketch that can greatly reduce the overheads of storage and processing at the cost of accuracy. However, implement such algorithms at large scale poses several problems. In this talk, we introduce Algegate (algebra + aggregate) a pure statistical, distributed platform implemented using Akka. It is designed to be fault-tolerant, back-pressure compliant and easily to scale out to multiple nodes.

avatar for Stanley Nguyen, Vu Ho

Stanley Nguyen, Vu Ho

Software Engineer, Symantec
Stanley Nguyen is a software developer in Email Security group at Symantec where he helps to build a high availability big data platform, write high performance backend services and develop interactive visualisation interfaces. He writes a majority of code in Scala, Go and NodeJS. Vu... Read More →

Saturday November 12, 2016 2:10pm - 2:50pm PST