One of the challenges of modern data science is handling complex data structures like structs, arrays, maps, queues, images, etc. in datawarehouse environments (as oppose to flat table representations). In this talk I will cover the existing ways to work with complex data structures with open-source projects like Pig, Hive and Impala as well as how E8 Security used Scala to simplify data pipelines with complex data structures. I will use an example of endpoint security computations. An endpoint in Enterprise Security is any computing device exposed to the clients or customers that request access to the corporate network, which yet cannot be entirely controlled or administered by the network administrators for one or another reason. The challenge is usually solved by additional monitoring of the devices themselves and the network traffic emanating from/to the device. In this particular instance, E8 built a machine learning based solution that tracks the footprint of the system and builds threat models based on the changes in the device footprints, which requires extensive use of complex and nested data structures. Currently E8 Security has customers tracking more than 0.5 million of endpoints running 10,000s different processes.
