Data Engineer

Key Responsibilities

• Engineered end-to-end ETL pipelines ingesting data from diverse on-premise sources (MySQL, Oracle, SQL Server, and SAP DB) into a multi-layer BigQuery data warehouse and an S3-based lakehouse; implemented Medallion architecture and Slowly Changing Dimensions (SCD) with Open Table Formats to optimize analytical performance.
• Orchestrated and automated large-scale, cross-cloud data workflows by designing and managing 20+ Airflow DAGs with SLA monitoring and automating complex ETL using Python/Java with GCP tools (Dataproc, Dataflow).
• Led cloud migration initiatives by successfully moving 200+ tables from GCP to AWS, utilizing AWS Glue, Lake Formation, and MWAA, significantly optimizing metadata governance and performance.
• Implemented a robust CI/CD pipeline using Azure DevOps to automate the deployment and management of all data assets, including AWS Glue jobs, Athena queries, Redshift, and Airflow DAGs.

Requirement

• Minimum 3 year of experience as Data Engineer
• Knowledge of programming languages: Java, Python, Bash Script
• Hands on experience with SQL
• Familiar with some of big data stacks like Hadoop, Google Cloud Bigquery, Google Cloud, Dataflow, Apache Airflow, Kafka, NoSQL database