Data Engineer

Data Engineer

Agensi Pekerjaan EPS Consultants Sdn Bhd
Kuala Lumpur
3-6 years
Not Specified

Job Description



Client Background: Financial Services Company
Location: Mid Valley City
Headcount: 1
Tenure: Permanent
Remuneration: to be discussed
Responsibilities:
·Work on highly complex projects that require in-depth knowledge within or across technical domains: technical, solutions, business or information.
·Design, build, test, deploy and maintain automated, optimized and highly available data pipelines/data workflows from multiple sources and systems into big data / cloud platforms to facilitate deeper analysis and reporting.
·Develop data models and data structures with business analyst.
·Design, build, and maintain data virtualization layer that delivers the data across disparate systems to the business users.
·Developing technical documentation.
·Perform program reviews to ensure that automation and data design elements are reusable and repeatable across projects.
·Support development of guidelines, standards, and processes to ensure the highest data quality and integrity in the data stores residing on the data lake/data warehouse.
·Work with IT and data owners to understand the types of data collected in various databases and data warehouses.
·Innovate through exploration, benchmarking, making recommendations, and implementing big data technologies for platforms.
·Assist in creation of accurate, effective, and efficient analysis and/or modification of existing analysis.
Requirements:
·2 years or more of data engineering experience - Managing data lakes/data warehouses, and working with batch and streaming data pipelines in the cloud.
·Experience delivering data solutions in AWS data stack, Databricks, and/or Cloudera.
·Experience in Big Data querying tools, such as Presto, Spark, and/or Impala.
·Experience with data virtualization tool, such as Denodo.
·Experience identifying and protecting PII - Using hashing, encryption, and/or pseudonymization to handle PII.
·Demonstrated experience in mentoring engineers and developing engineering best practices.
·Experience in API development and interaction, web scraping, and any other open source tools will be added advantage.
·Technical expertise in EDW methodologies and/or big data architectures.
·Proficient in SQL, Python, Spark, Databricks, AWS (Textract, Lambda, Glue, Athena, and etc.) or equivalent technology.
·Knowledge of Terraform, CircleCI or equivalent technology.
·Knowledge of BI tools, such as Power BI, MicroStrategy, and/or Tableau will be added advantage.

Job Details

Similar Jobs

People Also Considered

Career Advice to Find Better

Simple body text this will replace with orginal content