Scalæ By the Bay

This talk will showcase an NLP pipeline we have built with Scala and Spark at SalesforceIQ to analyze and derive insights from large amounts (several hundreds of millions of examples) of text data. Our stack utilizes EMR, Spark, S3, Avro, Azkaban, OpenNLP, and many elements from the functional programming paradigm (read: semigroups, monoids, and foldMaps, oh my!) to build a scalable and powerful pipeline for extracting rich information from email content. This pipeline currently powers our suggested follow-up feature, which informs customers when they need to follow up with an important email or conversation, in addition to foundational features for other use cases. As data processing pipelines become ubiquitous, and more people turn to Spark and Scala to build such pipelines, this talk will answer some questions of how to effectively go about the task. Elements from functional programming and libraries such as Twitter's algebird, scalaz, or cats allow for natural and efficient implementations of many core aspects of distributed data processing. We couple this with the OpenNLP library to create a data pipeline for the linguistic analysis of text, primarily in the form of email content. This has given us a solid foundation for engineering text based features as well as training text-based models for a variety of supervised learning tasks that outperforms a similar pipeline in traditional Map/Reduce with Java, in a more maintainable and scalable way.