Lead Data Architect - Databricks Migration

Canada
Remote
Posted Oct 9, 2026
1 position

$114,000–$171,000 / year

Opens LinkedIn

Employment type: Full-time
Experience level: Senior · 7+ years
Posting language: English
Working hours: 40 hours per week
Seniority: Mid-Senior level

Job summary

Own the end-to-end design of an Azure Databricks data platform, establish architecture and engineering standards, and lead migration of workloads from Cloudera/Hadoop and Dataiku. Stay hands-on across both workstreams, unblock engineers, review designs and code, and ensure data quality, performance, reliability, and cost efficiency.

Job details

About the Role We partner with global enterprises to design and build large-scale data platforms that power the products and operations at the center of their business. This engagement is a modernization program that moves data workloads off legacy Cloudera/Hadoop and Dataiku environments and onto Databricks on Azure. The program runs two engineering workstreams in parallel, delivered by a joint team of client and Lumenalta engineers. Each engineer focuses on a specific workload. What the team needs is one person who owns the full picture. That is this role. You will own the end-to-end solution design, set the Databricks patterns that the rest of the team builds on, and set the pace of delivery. You will stay hands-on across both workstreams and step in wherever a blocker comes up, so everyone else can stay focused on their own workload. What You'll Do Own the end-to-end solution design for the platform, from ingestion through curated, consumable data. Define the Databricks patterns and standards the joint team follows, including medallion architecture (bronze, silver, gold), Unity Catalog structure, and pipeline conventions. Build streaming and event-driven ingestion using Azure Event Hubs, Event Grid, and Azure Data Explorer. Lead the migration of workloads off Cloudera/Hadoop and Dataiku, including assessing existing pipelines and deciding how each one is rebuilt on Databricks. Work hands-on across both engineering workstreams, and step in to unblock engineers whenever an issue comes up. Set the pace for delivery, review designs and code, and keep quality and consistency high across the team. Write clean, efficient, and maintainable Python, PySpark, and advanced SQL. Ensure data quality, performance, reliability, and cost efficiency across the platform. What We're Looking For 7+ years of experience as a Data Engineer, including time as a technical lead on a data platform team. Production experience with Databricks on Azure. Strong, hands-on experience with Python, PySpark, and advanced SQL. Proven experience designing and implementing medallion architectures and Unity Catalog in production. Experience with streaming or event-driven ingestion on Azure, such as Event Hubs, Event Grid, or Azure Data Explorer. Experience migrating workloads from Hadoop-based platforms such as Cloudera to a modern lakehouse. Comfortable leading a mixed team of client and partner engineers, setting direction while staying hands-on. Strong communication skills, with the ability to explain design decisions to engineers and stakeholders. Fluent English, both written and spoken. Nice to Have Experience migrating workflows from Dataiku to Databricks. Experience with Delta Live Tables or Databricks declarative pipelines, and Databricks Workflows. Infrastructure as code for Azure and Databricks, such as Terraform or Databricks Asset Bundles. Databricks Data Engineer Professional certification. Why Lumenalta is an amazing place to work at At Lumenalta, you can expect that you will: Be 100% dedicated to one project at a time so that you can innovate and grow. Be a part of a team of talented and friendly senior-level developers. Work on projects that allow you to use leading tech. For Databricks practitioners specifically, we are a Databricks partner with Champions and MVPs on our team, and that carries three concrete things: Delivery Partner Program qualification. Databricks co-delivers its Professional Services engagements through a short list of approved partner organizations, and practitioners have to be individually qualified before they can work on them. The qualification is granted once at the individual level and does not expire, so it stays with you. Partner organizations are the route in, so the qualified pool stays small. That scarcity is most of the value, and we sponsor it. Champion path. Databricks Champion is an individual recognition with its own badge, a dedicated Databricks solutions architect as your mentor, access to product teams, confidential roadmap briefings, and an invitation to the internal Tech Summit. Nomination has to come from a partner organization, which puts it out of reach for independents and most contractors. We nominate actively and regularly, and the Champions already on our team mentor candidates through it. Certifications covered. We pay the exam fee for any Databricks certification you want to take. Our Process A screening call, a technical interview, and a HackerRank assessment. The technical interview is a conversation rather than a whiteboard exam. We will ask you to walk through a governance design you have shipped, a federate-versus-ingest call you made, what you would change, and what you would expect security reviewers to push hardest on. We will also ask for a writing sample. Relevant project references matter more to us than certifications. Salary Salary range: CA$114,000 - CA$171,000 annually, with final compensation determined by your qualifications, expertise, experience, and the role's scope. Location: This is a fully remote position. Must have availability to work U.S. Eastern time zone business hours. Benefits In addition to competitive pay, we offer a variety of benefits to support your professional and personal growth, including: Flexible working hours in a remote environment. Health insurance (medical and dental) for T4 Employees. A professional development fund to enhance your skills and knowledge. 15 days of paid time off annually. Access to soft-skill development courses to further your career. Position Details This is a full-time position requiring a minimum of 40 hours per week, Monday through Friday. Application Deadline Applications will be accepted until October 25th, 2026. Candidates can expect feedback by November 2nd, 2026.

What you’ll do

Requirements

Requires 7+ years of data engineering experience, including technical leadership, production experience with Azure Databricks, and strong Python, PySpark, and advanced SQL skills. Candidates should have production experience with medallion architecture and Unity Catalog, Azure streaming or event-driven ingestion, Hadoop-based platform migration, and the ability to lead a mixed engineering team; fluent English is required.