Senior / Lead Data Engineer - Databricks, AI Engineering & Retail & Omnichannel Analytics

Job summary

You will design and implement scalable data ingestion, transformation, and data product frameworks while driving Databricks best practices. The role involves leading technical initiatives, mentoring engineers, and delivering end-to-end analytics solutions for retail and omnichannel use cases.

Job details

Looking for: Senior / Lead Data Engineer - Databricks, AI Engineering & Retail & Omnichannel Analytics Job Type: Contract Location: Remote (Canada) Description: About the Role We are seeking a Senior / Lead Data Engineer to help build the next generation of our Retail & Omnichannel Analytics platform. This role combines hands-on engineering, technical leadership, and architecture. You will help define scalable data ingestion patterns, establish Databricks best practices, develop reusable frameworks, enable self-service analytics, and drive AI-enabled engineering practices across the organization. You will work across the full analytics lifecycle, from data ingestion and transformation through semantic models, reporting, governance, and AI-powered data products. The ideal candidate has successfully implemented enterprise-scale data and analytics platforms using Databricks and can bring proven patterns and best practices from previous initiatives. Key Responsibilities • Design and implement scalable data ingestion, transformation, and data product frameworks. • Define and drive adoption of Databricks best practices, including Medallion Architecture (Bronze/Silver/Gold), performance optimization, governance, and operational excellence. • Build batch, near real-time, and streaming data pipelines using Databricks and Azure technologies. • Design and deliver end-to-end data products supporting Retail & Omnichannel Analytics use cases. • Develop trusted analytical datasets, semantic models, and governed consumption layers for reporting and self-service analytics. • Enable data democratization through capabilities such as Databricks Genie and reusable business-ready data products. • Lead proof-of-concepts and evaluate emerging platform capabilities across the Databricks ecosystem. • Define AI SDLC and AI-assisted development patterns, including the use of AI agents and engineering accelerators. • Implement data quality, lineage, monitoring, and observability practices. • Partner with engineering, analytics, product, and business teams to deliver scalable solutions and mentor engineers on best practices. Required Qualifications • 7+ years of Data Engineering experience. • Strong hands-on experience with Databricks in enterprise production environments. • Advanced expertise in: Python SQL Scala Spark / PySpark Delta Lake • Experience building Lakehouse architectures and implementing Medallion Architecture (Bronze/Silver/Gold). • Strong experience with ETL/ELT design, data integration, and scalable data pipelines. • Experience creating analytical and dimensional data models. • Experience delivering end-to-end analytics solutions from ingestion through semantic modeling and reporting. • Experience implementing CI/CD and Data Engineering SDLC best practices. • Experience with Azure DevOps, Git, and release management processes. • Strong understanding of data governance, security, data quality, and performance optimization. • Strong communication skills and ability to lead technical discussions. Preferred Qualifications • Experience with Unity Catalog, LakeFlow, Delta Live Tables, Databricks SQL, and Databricks Genie. • Experience establishing engineering standards, reusable frameworks, or platform best practices used by multiple teams. • Experience implementing AI-assisted development workflows and AI engineering practices. • Experience designing AI agents or automation solutions that improve engineering productivity. • Experience leading platform modernization initiatives and technical proof-of-concepts. • Experience in Retail & Omnichannel Analytics, Merchandising, Inventory, Supply Chain, Store Operations, Customer Analytics, or Digital Commerce. Tech Stack: Languages • Python • SQL • Scala Data Engineering • Spark • PySpark • Delta Lake Data Platform • Databricks • Unity Catalog • LakeFlow • Delta Live Tables (DLT) • Databricks SQL • Databricks Genie • Databricks Workflows Data Integration & Orchestration • Azure Data Factory (ADF) • Databricks Workflows • REST APIs • Batch & Streaming Pipelines DevOps • Azure DevOps • Git • CI/CD Pipelines Architecture & Governance • Lakehouse Architecture • Medallion Architecture • Data Products • Semantic Models • Data Quality • Data Lineage • Observability & Monitoring AI Engineering • AI-Assisted Development Tools • AI Agents • GenAI-enabled Engineering Workflows

What you’ll do

You will design and implement scalable data ingestion, transformation, and data product frameworks while driving Databricks best practices. The role involves leading technical initiatives, mentoring engineers, and delivering end-to-end analytics solutions for retail and omnichannel use cases.

Requirements

Candidates must have 7+ years of data engineering experience with advanced expertise in Python, SQL, Spark, and Databricks. Strong proficiency in building Lakehouse architectures, implementing CI/CD pipelines, and managing data governance is essential.

Listed skills

• Microsoft Azure · Preferred
• SQL · Preferred
• CI/CD · Preferred
• Python · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• Databricks
• Python
• SQL
• Scala
• Spark
• PySpark
• Delta Lake
• Azure
• Data Engineering
• Lakehouse Architecture
• Medallion Architecture
• ETL/ELT
• Data Modeling
• CI/CD
• Data Governance
• AI Engineering

Job areas

• Data & Analytics
• Technology
• Software
• Engineering
• Retail