Senior Data Engineer

Job summary

Develop and maintain scalable data pipelines using Databricks, PySpark/Spark, and SQL, integrating data from Azure storage and other systems for analytics and BI. Optimize and monitor workloads, implement governance and security controls with Unity Catalog, and deploy Databricks artifacts across environments using Databricks Asset Bundles.

Job details

Role : Azure Data Engineer with Databricks Expertise Location : Ottawa/Toronto (Hybrid) Contract: 06+ Months Job Description: Key Responsibilities: 1. Data Pipeline Development: o Build and maintain scalable ETL/ELT pipelines using Databricks. o Leverage PySpark/Spark and SQL to transform and process large datasets. o Integrate data from multiple sources including Azure Blob Storage, ADLS and other relational/non-relational systems. 2. Collaboration & Analysis: o Work Closely with multiple teams to prepare data for dashboard and BI Tools. o Collaborate with cross-functional teams to understand business requirements and deliver tailored data solutions. 3. Performance & Optimization: o Optimize Databricks workloads for cost efficiency and performance. o Monitor and troubleshoot data pipelines to ensure reliability and accuracy. 4. Governance & Security: o Implement and manage data security, access controls and governance standards using Unity Catalog. o Ensure compliance with organizational and regulatory data policies. 5. Deployment: o Leverage Databricks Asset Bundles for seamless deployment of Databricks jobs, notebooks and configurations across environments. o Manage version control for Databricks artifacts and collaborate with team to maintain development best practices.

What you’ll do

Develop and maintain scalable data pipelines using Databricks, PySpark/Spark, and SQL, integrating data from Azure storage and other systems for analytics and BI. Optimize and monitor workloads, implement governance and security controls with Unity Catalog, and deploy Databricks artifacts across environments using Databricks Asset Bundles.

Requirements

The role calls for expertise in Azure data engineering and Databricks, including PySpark/Spark, SQL, pipeline development, and data integration. It also requires collaboration with cross-functional teams and experience with workload optimization, data governance, security, deployment, and version control.

Listed skills

• Microsoft Azure · Preferred
• SQL · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• Databricks
• Azure
• PySpark
• Apache Spark
• SQL
• ETL/ELT Pipelines
• Data Integration
• Data Transformation
• Azure Blob Storage
• Azure Data Lake Storage
• Unity Catalog
• Data Governance
• Data Security
• Performance Optimization
• Databricks Asset Bundles
• Version Control

Job areas

• Data & Analytics
• Technology
• Software

More jobs from Arkhya Tech Inc.