Data Engineer
Data Engineer
Terralogiq- Posted 23 hours ago
- Be among the first 10 applicants
Job Description
Data Migration & Modernization
- Serve as the data engineering lead within a larger project team focused on a full-scale on-premises infrastructure migration.
- Design, plan, and execute the end-to-end migration of on-premises relational databases (e.g., MySQL, PostgreSQL, SQL Server) to managed services like Google Cloud SQL.
- Lead the technical strategy for migrating large-scale data warehouses and data lakes from on-premises Cloudera/Hadoop ecosystems to Google BigQuery.
- Develop, test, and deploy robust and efficient ETL/ELT data pipelines using GCP services such as Dataflow, Dataproc, and Cloud Composer to support post-migration workloads.
- Perform comprehensive assessments of source data systems, including schema analysis, data profiling, and dependency mapping, to create a detailed and reliable migration plan.
- Ensure data integrity, consistency, and security throughout the migration lifecycle by implementing rigorous validation, testing, and reconciliation processes.
- Collaborate closely with client stakeholders to understand their data requirements, business objectives, and technical constraints.
- Work in close collaboration with Cloud Infrastructure Engineers to ensure data services are seamlessly integrated with networking, security, and compute resources.
- Write and optimize complex SQL queries and scripts for data transformation, validation, and performance tuning in BigQuery and Cloud SQL.
- Troubleshoot and resolve technical challenges related to data connectivity, pipeline failures, performance bottlenecks, and data quality issues.
- Implement monitoring and alerting solutions to ensure the ongoing health, performance, and cost-efficiency of the migrated data platforms on GCP.
- 3+ years of proven experience in a Data Engineer, Analytics Engineer, or similar data-focused role.
- Demonstrable hands-on experience within a large-scale on-premises to cloud migration project, with a specific focus on data platform modernization.
- Strong proficiency in SQL and experience with relational databases (e.g., MySQL, PostgreSQL, SQL Server, Oracle).
- Hands-on experience with Google Cloud data services, particularly BigQuery, Cloud Storage, and Cloud SQL.
- Experience in designing and building ETL/ELT data pipelines using programming languages like Python or Java.
- Solid understanding of core cloud infrastructure concepts, including networking (VPC), compute (VMs, Containers), and security (IAM).
- Solid understanding of data warehousing concepts, data modeling, and data lake architectures.
- Excellent analytical and problem-solving skills, with the ability to work methodically through complex technical issues.
- Google Cloud Professional Data Engineer certification.
- Google Cloud Professional Cloud Architect certification is a strong plus.
- Direct experience migrating workloads from Cloudera, Hadoop, or Hive to a cloud-native platform.
- Familiarity with GCP's Database Migration Service (DMS) or other data replication tools.
- Experience with data pipeline orchestration tools like Cloud Composer or Apache Airflow.
- Knowledge of Infrastructure as Code (IaC) tools such as Terraform for automating data infrastructure deployment.
- Experience working with geospatial data or location-based services is a significant advantage.
More Info
Key Skills
Cloud Composer
Data Lake Architectures
Google Cloud SQL
Database Migration Service
Containers
About Company
Skills:
Data Governance, Data Modeling, Sql, Python, Feature pipelines, Data quality checks, Documentation, Data pipelines, Data contracts
Skills:
Informatica, Data Integration, Sql, Bash Scripting, Hive, Confluence, Python, Etl, Big Data, Data Warehouse, HDFS
Skills:
Dml, Pl Sql, Sql, ELT, Oracle Rdbms, Sql Loader, Python, Kubernetes, Etl, Debezium, AWS cloud data services, Performance tuning of high-throughput data pipelines, Solace messaging
Skills:
Data Mart, Data Modeling, Microsoft Power Bi, Data Warehouse, Dax, Power Query, Etl, ELT, Power BI Service
Skills:
PostgreSQL, SQL Server, Apache Nifi, Etl, Airflow, Pentaho