Data Pipelines Case Study
Data Platform Architect

Data Pipelines

Project Overview

These CDC pipelines copy changes from database transaction logs across systems and stream them to analytics warehouses in real time.

Mohammed Tajul Islam designed automated schema evolution pipelines on AWS Redshift and Greenplum clusters, guaranteeing zero data loss and fault-tolerant streaming.

Data Pipelines Overview Diagram

Technical Actions & Advisory

Architectural highlights of high-throughput data streaming:

  • Stream Extraction: Guided the setup of adapters that read transaction logs and stream changes without putting any load on the source database.
  • Schema Auto-Mapping: Set up ETL schemas that adjust tables automatically when database fields change.
  • Distributed Computations: Led the plan for connecting Apache Spark and AWS Redshift warehousing.
Data Pipelines Real-time Flow Diagram