Big Data Lead

On-site
Posted Oct 28, 2026
1 position

Opens an external site

Employment type: Full-time
Experience level: Senior · 5+ years
Posting language: English
Working hours: 40 hours per week

Job summary

Develop and maintain scalable data pipelines using PySpark and Spark SQL to process large datasets efficiently. Collaborate with stakeholders to design ETL workflows and optimize Spark jobs for performance and reliability.

Job details

JD: Develop and maintain scalable data pipelines using PySpark and Spark SQL for processing large datasets efficiently. Write clean, reusable, and optimized code in Python for data manipulation, analysis, and automation tasks. Design and implement ETL workflows to extract, transform, and load data from various structured and unstructured sources. Collaborate with data engineers, analysts, and stakeholders to understand data requirements and deliver solutions. Optimize Spark jobs for performance tuning, resource utilization, and minimizing execution time. Work with distributed computing frameworks to process and analyze big data in a cloud or on-premises environment. Develop and maintain unit tests to ensure the accuracy and reliability of data pipelines and transformations. Utilize Spark SQL for querying and managing large datasets stored in distributed systems like Hadoop or cloud storage. Monitor and troubleshoot data pipeline issues, ensuring reliability and timely delivery of data. Stay updated with the latest advancements in PySpark, Spark SQL, and big data technologies to improve existing systems.

What you’ll do

Requirements

Requires proficiency in Python and distributed computing frameworks for big data processing. Candidates must be able to write clean, optimized code and implement unit tests to ensure data pipeline accuracy.