Server-less Architecture for Enterprise-level Data Processing


Requirement:

Our client, one of USA’s largest commercial offerings, wanted to implement an organization-wide data strategy with advanced analytics capabilities (people, processes and systems), so they could make data-driven business decisions.

The Knowledge Lens Solution:
  • Loaded external data sources from various locations such as MoveIT, S3, Oracle/SQL Server, Redshift, SalesForce.
  • Used Spark's in-memory computation to reduce traditional IO problem and speed up efficiency.
  • Provided visual layout to all the orchestration processes using Airflow.
  • Converted existing applications into server-less architecture using AWS Lambda and API Gateway.
  • Developed multiple traditional engines from ground up, using Spark dataframes.
Tech Specs:
Benefits and ROI:
  • Our client could plug the server-less apps as a rest service throughout their domains.
  • Databricks was used as a Big Data Platform to utilize the integration between compute and storage.
  • Databricks's custom built run-time Spark environment was utilized to achieve job optimization.