Big Data Lead

Job summary

Develop and maintain scalable data pipelines and ETL workflows using PySpark, Spark SQL, and Python to process large datasets from structured and unstructured sources. Optimize and monitor distributed data processing, ensure pipeline reliability through testing and troubleshooting, and collaborate with engineers, analysts, and stakeholders to deliver data solutions.

Job details

Role: Big Data Lead Location: Toronto, ON Fulltime Develop and maintain scalable data pipelines using PySpark and Spark SQL for processing large datasets efficiently. Write clean, reusable, and optimized code in Python for data manipulation, analysis, and automation tasks. Design and implement ETL workflows to extract, transform, and load data from various structured and unstructured sources. Collaborate with data engineers, analysts, and stakeholders to understand data requirements and deliver solutions. Optimize Spark jobs for performance tuning, resource utilization, and minimizing execution time. Work with distributed computing frameworks to process and analyze big data in a cloud or on-premises environment. Develop and maintain unit tests to ensure the accuracy and reliability of data pipelines and transformations. Utilize Spark SQL for querying and managing large datasets stored in distributed systems like Hadoop or cloud storage. Monitor and troubleshoot data pipeline issues, ensuring reliability and timely delivery of data. Stay updated with the latest advancements in PySpark, Spark SQL, and big data technologies to improve existing systems.

What you’ll do

Develop and maintain scalable data pipelines and ETL workflows using PySpark, Spark SQL, and Python to process large datasets from structured and unstructured sources. Optimize and monitor distributed data processing, ensure pipeline reliability through testing and troubleshooting, and collaborate with engineers, analysts, and stakeholders to deliver data solutions.

Requirements

The role calls for experience developing data pipelines and ETL workflows with PySpark, Spark SQL, and Python, as well as working with distributed computing frameworks and large datasets. It also requires the ability to optimize Spark jobs, write unit tests, troubleshoot pipelines, and collaborate with technical teams and stakeholders; no education requirement is stated.

Listed skills

• Troubleshooting · Preferred
• Python · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• PySpark
• Spark SQL
• Python
• Data Pipeline Development
• ETL
• Data Transformation
• Distributed Computing
• Big Data Processing
• Spark Performance Tuning
• Resource Optimization
• Unit Testing
• Data Quality
• Pipeline Monitoring
• Troubleshooting
• Hadoop
• Cloud Storage

Job areas

• Data & Analytics
• Technology
• Software
• Management & Leadership

More jobs from Tekgence Inc