Flexible Infrastructure Supporting Machine Learning for Anomaly Detection in Big Data

Huot, Alexander J.

Student Work

Flexible Infrastructure Supporting Machine Learning for Anomaly Detection in Big Data

Public

The ever-changing landscape of both machine learning and the fields to which they apply make traditional model building methods insufficient for creating models that reflect the current state of the world. Automated model generation and deployment is quickly becoming the standard in financial technology. We designed and implemented a pipeline that supports the continuous creation, training, and deployment of models to reduce a six month process to a one hour task. We utilized Spark, Hadoop, and Hive to create a fault tolerant and scalable pipeline as a backend supported by a web application as the interface. The final architecture of our pipeline, the process of its implementation, and the evaluation of the Chronos Pipeline are described.

This report represents the work of one or more WPI undergraduate students submitted to the faculty as evidence of completion of a degree requirement. WPI routinely publishes these reports on its website without editorial or peer review.

Creator