Data Engineer

Key Responsibilities

• Analyze, understand, maintain, debug, and enhance existing Python scripts used for data pipelines, data processing, and automation tasks.
• Design and implement processes to transform unstructured and semi-structured data, including JSON, XML, and log files, into clean, structured, and analysis-ready tabular formats.
• Develop, configure, maintain, and optimize data pipelines using Azure Data Factory (ADF) to support data integration, data movement, transformation, and workflow orchestration across multiple data sources.
• Build and manage ETL/ELT workflows to ensure efficient data extraction, transformation, loading, and delivery across systems.
• Configure pipeline activities, dependencies, triggers, and scheduling to support reliable and automated data processing.
• Integrate data from multiple sources and ensure it is processed and delivered accurately to the appropriate destinations.
• Write, maintain, and optimize complex SQL queries involving multiple joins, data granularity management, and query performance tuning.
• Implement data validation, consistency checks, and error-handling mechanisms to maintain data accuracy, completeness, and reliability.
• Monitor pipeline execution, investigate failures, identify root causes, and resolve data processing issues efficiently.
• Work closely with technical and business stakeholders to understand data requirements and deliver scalable, maintainable data solutions.
• Identify opportunities to improve pipeline performance, automation, processing efficiency, and overall data workflow reliability.

Required Skills & Qualifications

• Minimum 8+ years of professional experience in Data Engineering, with hands-on experience in data pipeline development, data integration, and data transformation.
• Strong proficiency in Python, with the ability to read, understand, debug, maintain, and enhance existing codebases.
• Strong SQL skills, including complex joins, data granularity handling, query optimization, and performance tuning.
• Hands-on experience developing, configuring, and maintaining data pipelines using Azure Data Factory. Understanding of pipeline orchestration, data movement, transformation activities, scheduling, monitoring, and troubleshooting is essential.
• Familiarity with ADF pipelines, activities, linked services, datasets, integration runtimes, and triggers.
• Solid understanding of ETL/ELT concepts, data integration, data transformation, and data pipeline architecture.
• Familiarity with tools used for data integration, workflow orchestration, and data transformation.
• Proven experience converting unstructured or semi-structured data, such as JSON, XML, and log files, into structured, tabular datasets.
• Familiarity with cloud server environments and cloud-based infrastructure for data processing and pipeline operations.
• Understanding of data validation, data consistency, error handling, and pipeline reliability practices.
• Strong analytical thinking and problem-solving skills, with the ability to investigate and resolve data pipeline issues.
• Ability to work effectively with cross-functional teams and translate business and technical requirements into practical data solutions.

Preferred Qualifications

• Practical experience implementing and managing end-to-end data pipelines using Azure Data Factory (ADF).
• Experience configuring ADF triggers, pipeline parameters, dependencies, and error-handling mechanisms.
• Familiarity with ADF monitoring, pipeline execution troubleshooting, and performance optimization.
• Experience integrating data from multiple sources into structured data destinations.
• Familiarity with dbt (data build tool) for data transformation, data modeling, and testing.
• Exposure to cloud-based data platforms and modern data engineering practices.