Job Description:

On behalf of our client, Affinity is seeking a Databricks Developer to join on a permanent basis. Reporting to the Team Lead, Data Development, you will be responsible for implementing robust data pipelines using Apache Spark on Databricks, supporting advanced data transformation, and enabling scalable data products that power enterprise analytics and reporting.

You’ll collaborate closely with data engineers and analysts to deliver high-performance, reliable pipelines and ensure quality outputs. This is a hands-on development role focused on engineering scalable, maintainable, and optimized data flows in a modern cloud-based environment.

Responsibilities:

• Design, build, and maintain scalable data pipelines and workflows using Databricks (SQL, PySpark, Delta Lake).
• Develop efficient ETL/ELT pipelines for structured and semi-structured data using Azure Data Factory (ADF) and Databricks notebooks/jobs.
• Integrate and transform large-scale datasets from multiple sources into unified, analytics-ready outputs.
• Optimize Spark jobs and manage Delta Lake performance using techniques such as partitioning, Z-ordering, broadcast joins, and caching.
• Design and implement data ingestion pipelines for RESTful APIs, transforming JSON responses into Spark tables.
• Apply best practices in data modeling and data warehousing concepts.
• Perform data validation and quality checks.
• Work with various data formats, including JSON, Parquet, and Avro.
• Build and manage data orchestration pipelines, including linked services and datasets for ADLS, Databricks, and SQL Server.
• Create parameterized and dynamic ADF pipelines, and trigger Databricks notebooks from ADF.
• Collaborate closely with Data Scientists, Data Analysts, Business Analysts, and Data Architects to deliver trusted, high-quality datasets.
• Contribute to data governance, metadata documentation, and ensure adherence to data quality standards.
• Use version control tools (e.g., Git) and CI/CD pipelines to manage code deployment and workflow changes.
• Develop real-time and batch processing pipelines for streaming data sources such as MQTT, Kafka, and Event Hub.

Qualifications:

• 5+ years of experience in data engineering or big data development.
• Bachelor's degree in computer science or a relevant field, or equivalent training and work experience.
• Strong hands-on experience with Databricks and Apache Spark (PySpark/SQL).
• Proven experience with Azure Data Factory, Azure Data Lake, and related Azure services.
• Experience integrating with APIs using libraries such as requests and http.
• Deep understanding of Delta Lake architecture, including performance tuning and advanced features.
• Proficiency in SQL and Python for data processing, transformation, and validation.
• Familiarity with data lakehouse architecture and both real-time and batch processing design patterns.
• Comfortable working with Git, DevOps pipelines, and Agile delivery methodologies.

Affinity Earn:

Know someone who’s great for this, or any of our open roles? Earn up to $4,000/year for each successful referral through Affinity Earn. You can also earn up to $50,000 for helping us find new clients. Learn about our referral program at https://affinity-group.ca/earn/ or browse our jobs & follow us at https://www.linkedin.com/company/affinity-staffing/jobs/

About Affinity:

Affinity Group is a technology and business consulting and services company. We believe in creating long term relationships between clients and consultants that foster a mutually beneficial partnership. Affinity is an equal opportunity employer. We celebrate diversity and are committed to creating an inclusive environment for all employees. All employment is decided on the basis of qualifications, merit and business need.

For more information on Affinity, please visit www.affinity-group.ca

Job Number: 12619