senior data engineer (must know python & spark).

about the company.
Our client is a MNC IT service provider, and they are seeking a senior data engineer to oversee the projects with data platform solutions.

• Design, develop, and optimize robust data pipelines using Apache Spark to process and transform large datasets efficiently.
• Collaborate with stakeholders to architect scalable and flexible data solutions that meet business requirements.
• Integrate data from various sources, ensuring seamless data flow and accessibility for analytics and reporting.
• Monitor and enhance the performance of Spark applications, optimizing query performance and resource utilization.
• Implement data quality checks and validation processes to ensure the integrity and accuracy of data.
• Work closely with data scientists, analysts, and other engineering teams to support data-driven projects and initiatives.
• Maintain comprehensive documentation of data pipelines, processes, and architecture for future reference and onboarding.
• Provide guidance and mentorship to junior data engineers, fostering a culture of collaboration and continuous learning.

• Bachelor’s degree in Computer Science, Engineering, or a related field; Master’s degree preferred.
• Minimum 8 years of experience in data engineering, with at least 3 years of hands-on experience using Spark.
• Proficiency in programming languages such as Python, Java, or Scala; experience with big data technologies (Hadoop, Kafka, Hive).
• Strong understanding of relational and NoSQL databases, data warehousing concepts, and ETL processes.
• Excellent problem-solving skills and the ability to analyze complex data sets.
• With experience of vendor/off-shore development team management
• Strong verbal and written communication skills, with the ability to convey technical concepts to non-technical stakeholders.
• Fluency in Cantonese, Mandarin and English