Q1 Technologies, Inc.Verified Job Source

Big Data/ Python Data Engineer (Hadoop Ecosystem)

Toronto, ON
Hybrid
Posted Sep 8, 2026
1 position

$60–$70 / hour

Opens an external site

Employment type: Contract
Experience level: Lead · 10+ years
Apply by: Oct 8, 2026
Posting language: English
Working hours: 40 hours per week
Office presence: 3 days per week
Seniority: Mid-Senior level
Application method: Direct apply is available

Job summary

Design, develop, and optimize scalable data pipelines and real-time processing solutions using PySpark, Kafka, and Hadoop. Collaborate with cross-functional teams to implement ETL/ELT workflows and ensure data quality and governance.

Job details

Data Engineer Toronto, ON - Hybrid: 3 days a week in office 6 Months Contract with possibility of extension 10-12+ Experience MUST Job Summary An experienced Data Engineer with strong expertise in Big Data technologies to design, develop, and support enterprise-scale data platforms. The ideal candidate should possess hands-on experience in PySpark, Apache Spark, Kafka, Hadoop ecosystem components, and Apache NiFi, with a strong understanding of data ingestion, transformation, and real-time processing frameworks. Key Responsibilities Design, develop, and optimize scalable data pipelines using PySpark, Spark, Hadoop, and Apache NiFi. Build and maintain batch and real-time data processing solutions. Develop and support Kafka-based streaming applications and event-driven architectures. Create and optimize ETL/ELT workflows for large-scale structured and unstructured datasets. Develop complex SQL queries for data extraction, transformation, validation, and troubleshooting. Implement data ingestion solutions from databases, APIs, files, and streaming sources. Monitor, troubleshoot, and enhance the performance of Spark jobs and data pipelines. Collaborate with architects, business analysts, and development teams to deliver high-quality data solutions. Support platform upgrades, deployments, testing, certification, and production releases. Ensure data quality, governance, security, and operational excellence across data platforms. Mandatory Skills PySpark Apache Spark (Spark SQL, DataFrames) Apache Kafka Hadoop Ecosystem (HDFS, Hive, YARN) Apache NiFi SQL Python Preferred Skills Spark Streaming Airflow / Oozie Hive Scala Jenkins, Bitbucket, Git JIRA, Confluence Cloud Platforms (GCP/AWS/Azure) Data Warehousing concepts and Dimensional Modeling

What you’ll do

Requirements

Requires over 10 years of experience with strong expertise in the Hadoop ecosystem, PySpark, and Apache NiFi. Proficiency in SQL and Python is mandatory for developing complex data extraction and transformation processes.