Data Engineer

Job Description

• Design, create, maintain and improve ETL and data pipeline to support entire cycle of data usage for data analytics team
• Create and maintain data catalog, data linkage and data life cycle for data analyst team
• Create and improve automation entire data process from data injection, ETL orchestration to model deployment
• Analyze problem and find resolution, as well as work closely with administration to monitor performance and advise any necessary infrastructure changes
• Work with business domain experts, data scientist and developers to identify relevant data for analysis

Qualification

• Strong coding skill i.e. Python, Spark, SQL
• Experience working with varies forms of data infrastructure inclusive of relational database such as SQL, Delta lake and Spark
• Experience in cloud platform, especially Azure Databrick and Power BI
• Experience with orchestration tools e.g. batch and real-time data processing
• Experience with CICD pipeline data and machine learning model deployment
• Knowledge in ETL and data pipeline framework, data patterns such as structured, semi structured, understand data in various type of database and data store such as Hive, HBase, Impala, MongoDB, Delta Lake, graph database using batch and streaming mechanism leveraging on-premise and on cloud Big data architecture
• Knowledge in data science fundamental such as database, operation system, algorithms, data structures, etc
• Interest leaning and working with new technology stack