Job opening - Lead Data Engineer(Azure Databricks) - Insurance in Hong Kong | Randstad Hong Kong

summary

hong kong
HK$45,000 - HK$65,000 per month
permanent
full-time

specialism
information technology
sub specialism
data science & machine learning
reference number
91M0243745_0031038117595353901

job details

about the company. ...

Leading life insurance and financial services company

about the job.

Create and implement a data strategy using Azure Data tools for advanced data processing and analytics.
Lead the development of data pipelines with a focus on optimizing Databricks components and cluster performance.
Set up data integration configurations for Extract, Transform, Load (ETL) processes aligned with target data architecture.
Understand and configure enterprise data models, including CDM and departmental data marts, using hive Metastore and Unity Catalog.
Design and maintain robust ETL/ELT processes using Azure Data Factory and Azure Databricks for seamless data integration.
Enhance data platform resilience through access control frameworks and best practice education for users.
Implement data governance practices using Databricks Unity Catalog and Azure Purview to ensure compliance and data quality.
Continuously monitor and optimize the cost and performance of Databricks environments, including data partitioning and indexing.
Collaborate with cross-functional teams to understand data needs and communicate technical specifications effectively.
Implement security best practices for data management and ensure compliance with data privacy regulations and internal policies.

skills & experiences required.

University/College graduate in Computing, IT, or a related discipline.
Minimum 5 years of relevant experience, with at least 3 years of hands-on experience using Databricks.
In-depth knowledge of Azure Databricks for data engineering, including proficiency in Apache Spark, PySpark, and Delta Lake.
Familiarity with Databricks components such as Workspace, Run-time, Clusters, workflows, DLT, functions, hive Metastore, SQL Warehouse, Delta sharing, and Unity Catalog.
Knowledge of the insurance industry is preferred.
Experience in ETL/ELT and data integration with an understanding of enterprise data models like CDM and departmental data marts.
Proficiency in using Azure Data Factory (ADF) to build complex data pipelines and integrate data from various sources.
Strong experience in SQL, Python, and PySpark for building and optimizing data workflows.
Knowledge of Jenkins DevOps for implementing CI/CD pipelines and managing code repositories.
Familiarity with Azure Purview, Azure Key Vault, Azure Active Directory, and RBAC for managing security and compliance in data platforms.

Please feel free to drop me a message or share your updated resume to 📩Cheryl.lau@randstad.com.hk

show more

about the company.
Leading life insurance and financial services company

about the job.

Create and implement a data strategy using Azure Data tools for advanced data processing and analytics.
Lead the development of data pipelines with a focus on optimizing Databricks components and cluster performance.
Set up data integration configurations for Extract, Transform, Load (ETL) processes aligned with target data architecture.
Understand and configure enterprise data models, including CDM and departmental data marts, using hive Metastore and Unity Catalog.
Design and maintain robust ETL/ELT processes using Azure Data Factory and Azure Databricks for seamless data integration.
Enhance data platform resilience through access control frameworks and best practice education for users.
Implement data governance practices using Databricks Unity Catalog and Azure Purview to ensure compliance and data quality.
Continuously monitor and optimize the cost and performance of Databricks environments, including data partitioning and indexing.
Collaborate with cross-functional teams to understand data needs and communicate technical specifications effectively.
Implement security best practices for data management and ensure compliance with data privacy regulations and internal policies.

...

skills & experiences required.

University/College graduate in Computing, IT, or a related discipline.
Minimum 5 years of relevant experience, with at least 3 years of hands-on experience using Databricks.
In-depth knowledge of Azure Databricks for data engineering, including proficiency in Apache Spark, PySpark, and Delta Lake.
Familiarity with Databricks components such as Workspace, Run-time, Clusters, workflows, DLT, functions, hive Metastore, SQL Warehouse, Delta sharing, and Unity Catalog.
Knowledge of the insurance industry is preferred.
Experience in ETL/ELT and data integration with an understanding of enterprise data models like CDM and departmental data marts.
Proficiency in using Azure Data Factory (ADF) to build complex data pipelines and integrate data from various sources.
Strong experience in SQL, Python, and PySpark for building and optimizing data workflows.
Knowledge of Jenkins DevOps for implementing CI/CD pipelines and managing code repositories.
Familiarity with Azure Purview, Azure Key Vault, Azure Active Directory, and RBAC for managing security and compliance in data platforms.

Please feel free to drop me a message or share your updated resume to 📩Cheryl.lau@randstad.com.hk

show more

experience
5 years
skills
Azure Databricks, ETL/ELT, CI/CD pipelines, Azure Data Factory (ADF)
qualifications
Azure Databricks, ETL/ELT, CI/CD pipelines, Azure Data Factory (ADF)
education
Bachelor Degree

apply with randstad.
Applying with us is easy. We will review your application and see if you are a good fit for the job and the company.
we’ll give you a call.
Our consultant will call you to discuss your application and further career aspirations if you're suitable for a role.
getting you registered.
If you’ve never worked with us before, we’ll need some basic additional pieces of information to confirm your eligibility to work in Hong Kong SAR.
compliance check.
Next, we just need to verify a few things - we’ll make the relevant compliance checks and keep you posted.
reference and background check.
As part of the process in ensuring you’re perfect for the role, we’ll make contact with any relevant references you’ve provided.
the perfect job for you.
Our expert team will either arrange an interview for the role you’ve applied for, or if they believe there’s a better opportunity, they’ll suggest alternative options too.
the interview.
If a job which you have applied for requires an interview, we will ensure you are fully prepared and know exactly what to expect - good luck!
start your new job.
Congratulations, you’re ready to begin your new job. The team will ensure that you’re fully prepared for your first day.

CL

cheryl lau

data scientist - conglomerate company
- contract
about the job.Machine Learning Model DevelopmentDesign, develop, and deploy machine learning models to tackle business challenges and enhance decision-making processesContinuously monitor and optimize model performance, scalability, and accuracy in production settingsUtilize advanced machine learning techniques (such as supervised/unsupervised learning and deep learning) to create innovative solutionsAI GovernancePerform AI audits to ensure compliance with
posted 21 november 2024
about the job.Machine Learning Model DevelopmentDesign, develop, and deploy machine learning models to tackle business challenges and enhance decision-making processesContinuously monitor and optimize model performance, scalability, and accuracy in production settingsUtilize advanced machine learning techniques (such as supervised/unsupervised learning and deep learning) to create innovative solutionsAI GovernancePerform AI audits to ensure compliance with
show job details
bi engineer (up to 50k/monthly)
- permanent
- HK$40,000 - HK$50,000 per month, 13th month pay
about the company.Well known Health Insurance Companyabout the team.You will be joining the CX and Transformation Team where you will play a key role in implementing transformative changes through data analysis and operational improvements. Your main responsibilities will include analyzing transactional data, creating impactful dashboards using Power BI, and collaborating with cross-functional teams to enhance our data and operational frameworks.about the
posted 21 november 2024
about the company.Well known Health Insurance Companyabout the team.You will be joining the CX and Transformation Team where you will play a key role in implementing transformative changes through data analysis and operational improvements. Your main responsibilities will include analyzing transactional data, creating impactful dashboards using Power BI, and collaborating with cross-functional teams to enhance our data and operational frameworks.about the
show job details
senior ux/ui designer (insurance)
- permanent
- HK$35,000 - HK$48,000 per month
about the company.fast-growing life insurance company often using digital platformabout the team.You will work under the digital platform team and work closely with varies stakeholders that involve in the digital product developmentabout the job.Work closely with cross-functional teams to transform user requirements into actionable design solutions that achieve targeted outcomes.Implement design processes that integrate with agile development practices, en
posted 21 november 2024
about the company.fast-growing life insurance company often using digital platformabout the team.You will work under the digital platform team and work closely with varies stakeholders that involve in the digital product developmentabout the job.Work closely with cross-functional teams to transform user requirements into actionable design solutions that achieve targeted outcomes.Implement design processes that integrate with agile development practices, en
show job details

Thank you for subscribing to your personalised job alerts.

You need to enable JavaScript to run this app.