Job Objectives
Develop and deliver scalable ETL/ELT solutions across Teradata and Hadoop platforms by building robust data pipelines, ingestion frameworks, and automation capabilities. Ensure efficient data processing, performance optimization, and reliable integration of enterprise data sources while supporting Enterprise Data Warehouse (EDW), Change Data Capture (CDC), and Master Data Management (MDM) initiatives.
Key Responsibilities
ETL/ELT Development
- Design, develop, and maintain ETL/ELT solutions using Teradata SQL, Informatica PowerCenter, Apache Spark, QueryGrid, and Trino.
- Build data transformation and loading processes for Teradata-based Enterprise Data Warehouses (EDW) and Data Marts.
- Develop scalable and reusable ETL frameworks to support enterprise data integration requirements.
Data Engineering & Platform Development
- Develop and manage data pipelines across Teradata and Hadoop ecosystems.
- Work with Hadoop technologies including Hive, Impala, Spark, Kafka, Iceberg, Ranger, Atlas, NiFi, Flink, and related components.
- Support enterprise data ingestion, processing, and integration initiatives.
Performance Optimization
- Analyze, troubleshoot, and optimize highly complex ETL and data processing applications.
- Perform performance tuning to reduce resource utilization and improve processing efficiency.
- Ensure optimal performance of Teradata and Hadoop-based solutions.
Framework & Automation Development
- Create and enhance automation frameworks using GCFR, Shell Scripts, and BTEQ.
- Automate end-to-end ETL processes and integrate with enterprise control and scheduling systems.
- Develop configurable and reusable components to improve operational efficiency.
Data Integration & CDC
- Build and maintain in-house CDC frameworks using Java to process Oracle redo/OLR logs from SAP (Oracle) systems.
- Design configurable CDC ingestion frameworks to load data into Teradata environments.
- Ensure reliable and scalable enterprise data integration capabilities.
Master Data Management (MDM)
- Install, configure, and support Master Data Management (MDM) applications.
- Develop data validation, approval workflow, and user-upload functionalities.
- Support reference data management and governance requirements.
Required Skills & Experience
ETL/ELT Development
- Strong hands-on experience in Informatica PowerCenter and Teradata-based ETL development.
- Expertise in Teradata SQL and complex data transformation logic.
- Experience with Apache Spark, QueryGrid, and Trino.
Data Platform Expertise
- Strong knowledge of Teradata and Hadoop ecosystems.
- Hands-on experience with technologies such as:
- Hive
- Impala
- Spark
- Kafka
- Iceberg
- Ranger
- Atlas
- NiFi
- Flink
Data Pipeline & Orchestration
- Experience building enterprise-scale data pipelines.
- Knowledge of enterprise scheduling and orchestration tools.
- Experience with Shell Scripting and automation frameworks.
Performance Optimization
- Proven experience in performance tuning and optimization of large-scale ETL and data processing applications.
- Ability to identify bottlenecks and improve system efficiency.
Data Integration & CDC
- Experience implementing Change Data Capture (CDC) solutions.
- Knowledge of Java-based framework development for data ingestion and integration.
- Experience integrating SAP (Oracle) and Teradata platforms.
Master Data Management (MDM)
- Experience implementing and supporting MDM solutions.
- Understanding of reference data management, validation frameworks, and approval workflows.
Key Skills
- ETL/ELT Development using Informatica PowerCenter and Teradata
- Teradata SQL
- Apache Spark
- QueryGrid
- Trino
- Hadoop Ecosystem (Hive, Impala, Kafka, Iceberg, Ranger, Atlas, NiFi, Flink)
- Data Pipeline Development & Orchestration
- Performance Optimization & Tuning
- Data Integration & CDC
- Java Development
- Shell Scripting & BTEQ
- ETL Framework & Automation Development
- Master Data Management (MDM)
- Reference Data Management
Preferred Profile
- Strong experience in Enterprise Data Warehouse (EDW) environments.
- Proven ability to develop scalable and high-performance data engineering solutions.
- Experience working in large enterprise banking, financial services, or data-intensive environments.
- Strong analytical, troubleshooting, and problem-solving skills.
- Ability to work collaboratively with business and technical stakeholders.
