Scalæ By the Bay

At Symantec email security group, a common problem we face is to aggregate multiple metrics with different time granularity in real-time from hundreds of millions emails per day. Various existing solutions try to address the problem by using batch and/or streaming algorithms. Often such approach requires the use of many different technologies and are expensive to run. Another approach is to use statistical data structures such as Count Min Sketch that can greatly reduce the overheads of storage and processing at the cost of accuracy. However, implement such algorithms at large scale poses several problems. In this talk, we introduce Algegate (algebra + aggregate) a pure statistical, distributed platform implemented using Akka. It is designed to be fault-tolerant, back-pressure compliant and easily to scale out to multiple nodes.