Big Data Developer

Job summary

Design, develop, and optimize scalable data pipelines and ETL/ELT workflows using PySpark and Python to process large structured and unstructured datasets. Perform data transformation and validation, troubleshoot Spark performance, implement data quality and monitoring controls, and collaborate with technical and business stakeholders.

Job details

Role : Big Data Engineer (PySpark & Python) Location: Mississauga, ON, Canada Job Type: FTE Salary : CAD 120K + Benefits Job Summary We are looking for an experienced Big Data Engineer with strong expertise in PySpark and Python to design, develop, and optimize scalable data processing solutions. The ideal candidate will have experience working with large datasets, building robust ETL/ELT pipelines, and supporting data integration and transformation initiatives in a cloud or big data environment. Key Responsibilities Design, develop, and maintain scalable data pipelines using PySpark and Python. Build and optimize ETL/ELT workflows to process large volumes of structured and unstructured data. Perform data cleansing, transformation, aggregation, and validation. Develop efficient Spark jobs and troubleshoot performance and scalability issues. Work with SQL, relational databases, and data warehouses to support analytics and reporting. Implement data quality checks, monitoring, and error-handling mechanisms. Collaborate with data engineers, architects, analysts, and business stakeholders to deliver data solutions. Follow best practices for coding standards, documentation, testing, and data security. Support cloud-based big data platforms and CI/CD processes where applicable. Required Skills Strong hands-on experience with Python and PySpark. Solid understanding of Apache Spark, distributed computing, and big data processing. Proficiency in SQL and experience with complex queries. Experience developing ETL/ELT pipelines and data integration solutions. Knowledge of data modeling, data warehousing, and data lake concepts. Experience with performance tuning, debugging, and data quality management. Familiarity with Git, unit testing, and Agile development methodologies. Preferred Qualifications Experience with cloud platforms such as AWS, Azure, or GCP. Exposure to Databricks, Delta Lake, Apache Kafka, or Hadoop. Experience with orchestration tools such as Airflow or Azure Data Factory. Knowledge of CI/CD pipelines and automated deployment practices. Education & Experience Bachelor's degree in Computer Science, Information Technology, Engineering, or a related field. 5+ years of experience in data engineering or big data development, with strong hands-on expertise in PySpark and Python.

What you’ll do

Design, develop, and optimize scalable data pipelines and ETL/ELT workflows using PySpark and Python to process large structured and unstructured datasets. Perform data transformation and validation, troubleshoot Spark performance, implement data quality and monitoring controls, and collaborate with technical and business stakeholders.

Requirements

Requires a bachelor's degree in Computer Science, Information Technology, Engineering, or a related field, and 5+ years of data engineering or big data development experience. Candidates should have strong hands-on Python and PySpark expertise, knowledge of Spark and distributed computing, proficiency in SQL, and experience with data pipelines, data warehousing, performance tuning, and data quality.

Listed skills

• SQL · Preferred
• Git · Preferred
• Python · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• Python
• PySpark
• Apache Spark
• Distributed Computing
• SQL
• ETL/ELT Pipelines
• Data Integration
• Data Modeling
• Data Warehousing
• Data Lakes
• Performance Tuning
• Data Quality Management
• Git
• Unit Testing
• Agile Development
• Cloud Platforms

Job areas

• Data & Analytics
• Technology
• Software

More jobs from TechNeptune Consulting Inc.