Insight Global is seeking a Lead Data Engineer to join a large retail client based in Vancouver, BC. (1-2 Days a month on site). The data engineering team builds and maintains the data infrastructure that powers analytics, reporting, and data-driven decision making across the company’s digital platforms, retail operations, and business units. We work with cross-functional agile squads to develop reliable data pipelines, improve data quality, build self-serve analytics tools for our business stakeholders, and ensure timely access to accurate data for our analysts and data scientists. This Lead Engineer will be responsible for supporting several upcoming initiatives, including modernizing the organization’s data platform through the migration of data pipelines from Azure Data Factory to Databricks and the transition of legacy Hive tables to Unity Catalog. The ideal candidate will bring strong Databricks experience, including a solid understanding of Medallion Architecture and best practices for defining and managing Bronze, Silver, and Gold data layers to support scalable, governed data solutions.

Day to Day Activities

Architecting and implementing scalable data platforms using Azure, Databricks, and NoSQL databases Overseeing Unity Catalog adoption for centralized data governance and metadata management Designing integrations with ML infrastructure and collaborating with data science teams on model deployment Leading and mentoring a team of data engineers, balancing hands-on coding with strategic oversight Establishing coding standards, DevOps/SDLC workflows, and performance optimization practices Driving backlog prioritization, technical planning, and cross-functional alignment Implementing enterprise-grade data governance frameworks, including security (SecOps) and cost management (FinOps) Championing data quality initiatives and advanced monitoring solutions Evaluating emerging technologies (e.g., real-time processing, GenAI pipelines) for business impact Partnering with business leaders to translate requirements into technical roadmaps Leading hiring processes, performance evaluations, and team development Exemplifying the company’s leadership code (Lead Self, Lead Others, Lead Business) through innovation, accountability, and mentorship

Here are some of the things you could be working on in the future

Leading the technical implementation of data mesh architecture and domain-oriented data products Designing and implementing real-time data streaming solutions for critical business processes Establishing advanced MLOps practices to operationalize machine learning workflows at scale Developing comprehensive data observability solutions across the entire data ecosystem Creating specialized data acceleration frameworks to improve platform performance

Must Haves

6-10 years of experience in data engineering, with 3+ years in overseeing projects Strong Azure experience with Databricks, and pipeline orchestration tools (Spark) Strong experience with Unity Catalog, DLT and Asset bundles Advanced skills in SQL, Python, Pyspark and infrastructure-as-code (Terraform, ARM templates) Proven track record in designing large-scale data architectures (batch/streaming, lakehouse, NoSQL) Strong leadership skills with experience mentoring teams, supporting Jr Engineers and driving technical excellence Can communicate complex technical strategies to executives and stakeholders

Plusses

AWS Experience ML Ops experience (Databricks, ML Flow) Retail experience in the past