Senior Data Engineer

Fresher

Save

Early Applicant

Job Description

Requirements

Programming language
Phyton
Advance SQL (subquery, ROW_NUMBER, RANK, PARTITION BY, Indexing, Explain Plan, dan Query Tuning, Stored Procedures, Triggers, View, Materialize View, Constraints, indexing, query optimization)
Database Systems
MySQL
PostgreSQL
Oracle
Snowflake
Unstructured Data
Data modeling dan arsitektur
Distributed Data Storage Systems (partitioned data storage, sorting key, SerDes, data replication, caching and persistence)
Distributed Data Processing Systems (partitioning, predicate pushdown, sort by partition, maintaining size of shuffle blocks, window function, leveraging all cores and memory available in the cluster to improve concurrency)
Stream Data Processing (Real-time, Stream and Batch Processing)
Tools
Data Pipelines and Automation (Airbyte)
Data ingestion in Message Queues
Data Wrangling operations Pandas,numpy,re
Data Scraping requests/BeautifulSoup/lxml/Scrapy
Interacting with External APIs and other Data Sources, Logging
Parallel processing Libraries Dask, Multiprocessing
Data engineering tools (Apache Kafka, Apache Airflow)
Cloud Native Technologies (Serverless Computing, Virtual Instances, Containering Docker, Orchestration -Kubernetes)
Data Visualization (Qlikview, Tableau)

Jobdesc

Membuat perencanaan aktivitas pekerjaan.
Membuat stream pipeline for realtime data processing.
Membuat datamart untuk report dan dashboard.
Mengumpulkan, mengurutkan, membersihkan, dan mengelola data.
Melakukan analisis dan memberikan hasil analisis kepada tim.
Mengelola data transfer (tally data) dari database transaksi ke data warehouse.
Mengelola server job scheduler.
Berkolaborasi dengan tim untuk mengidentifikasikan dan memperbaiki masalah teknis pada backend dan frontend.
Membuat laporan pekerjaan bulanan.