Senior Data Engineer

Insight Global is hiring a Senior Data Engineer for a large retail client in Vancouver, BC. This role sits within the Data & Analytics organization and will support the Guest Data Lake platform, which enables trusted, scalable access to customer and guest data across digital, analytics, reporting, and personalization initiatives. The ideal candidate brings deep experience building cloud-native data platforms, designing scalable data solutions, and supporting customer data domains using Azure, Databricks, PySpark, and SQL. You will work closely with Data Engineering, Analytics, Data Science, and business teams to deliver reliable, governed, and high-performing data products. What You'll Do Design and implement scalable data architectures, including batch, streaming, and lakehouse solutions Enhance and support the Guest Data Lake platform to ensure reliable, high-quality customer data access Build and optimize ETL/ELT pipelines for structured and semi-structured data sources Enable analytics, personalization, reporting, and digital experience initiatives through customer data solutions Lead migrations from legacy platforms to modern cloud-native architectures Improve data quality, observability, monitoring, and governance frameworks Partner with Data Science and Analytics teams to deliver ML-ready and analytics-ready datasets Translate business requirements into scalable technical solutions Drive CI/CD, Infrastructure-as-Code, and software engineering best practices Participate in architecture reviews, technical design discussions, and platform roadmap planning Mentor junior engineers through code reviews and technical guidance Optimize platform performance, security, scalability, and cloud costs Required Qualifications 5–8+ years of Data Engineering experience Strong hands-on experience with Azure, Databricks, PySpark, and Unity Catalog Advanced proficiency in SQL and Python Experience supporting customer/guest data domains and customer-centric data platforms Experience working with structured and semi-structured data, including NoSQL databases (Cosmos DB preferred) Strong understanding of Data Governance, Data Quality, and Data Observability best practices Proven experience building and maintaining large-scale ETL/ELT pipelines and cloud-based data lakes Experience with CI/CD, Terraform, and Infrastructure-as-Code practices Experience contributing to architecture decisions and mentoring junior engineers Preferred Qualifications Retail, e-commerce, loyalty, or customer data platform experience Experience with Spark Declarative Pipelines and Metric Views in Databricks Experience supporting Machine Learning or AI-enabled data products Understanding of Data Mesh architecture and domain-oriented data products AWS experience Experience with real-time streaming and event-driven architectures Future Opportunities Expand and evolve the Guest Data Lake to support new digital and customer experience initiatives Design advanced data quality and observability frameworks Support enterprise-wide data governance standards and implementation patterns Implement Data Mesh principles and domain-driven ownership models Build automation and self-service capabilities for data infrastructure provisioning Support real-time and event-driven data processing solutions Deliver scalable personalization, customer analytics, and guest engagement capabilities Optimize cloud infrastructure for performance, scalability, reliability, and cost efficiency

Design and implement scalable cloud-native data architectures and ETL/ELT pipelines to support the Guest Data Lake platform. Collaborate with cross-functional teams to deliver high-quality, governed data products for analytics, personalization, and machine learning.

Requires 5-8+ years of data engineering experience with deep expertise in Azure, Databricks, PySpark, and SQL. Candidates must have a proven track record of building large-scale data lakes and implementing CI/CD and Infrastructure-as-Code practices.

• Microsoft AzurePreferred
• SQLPreferred
• CI/CDPreferred
• TerraformPreferred
• PythonPreferred

Identified from the job description. Confirm important requirements above.

• Azure
• Databricks
• PySpark
• SQL
• Python
• Unity Catalog
• ETL/ELT
• Cosmos DB
• Terraform
• CI/CD
• Infrastructure-as-Code
• Data Governance
• Data Observability
• Data Lakehouse
• NoSQL
• Data Mesh

• Data & Analytics
• Technology
• Engineering
• Software
• Retail

Minimum experience

5+ years

Posting language

English

Working hours

40 hours per week

Seniority

Mid-Senior level

Application method

Direct apply is available