Job Purpose

The Senior Data Engineer contributes to the design, build, and operation of Orca’s data ingestion, transformation, and analytics platforms. The role spans an environment that includes Microsoft Fabric, Azure Data Factory, Azure SQL, SQL Server, and Power BI, alongside Python and T-SQL for pipeline development. Working under limited guidance from the VP Technology and the Software Technical Lead, the Data Engineer owns end-to-end delivery of ingestion frameworks, data quality controls, and metadata management for freight and shipment data. As Orca’s data function scales, this role also acts as a technical reference and mentor for junior data analysts and future data engineers on ingestion, transformation, and pipeline reliability practices.

Duties and Responsibilities

• Design and build metadata-driven ingestion pipelines to onboard new carrier feeds (SFTP, EDI, email, portal extracts) using standardized, reusable patterns.
• Develop and maintain data transformation logic in Microsoft Fabric using T-SQL and Python, ensuring idempotent loads and structured error handling.
• Implement automated data quality controls (freshness, completeness, schema-drift detection) with alerting integrated to Azure Monitor and Log Analytics.
• Own retention and tiering automation across Azure SQL and SQL Server, including archival and compaction routines for high-volume tables.
• Build and maintain metadata catalog and lineage tracking for modernized datasets using Microsoft Purview or equivalent tooling.
• Contribute to large-tenant database strategy work, including performance benchmarking between SQL Server on Azure VMs and Azure SQL Managed Instance, and pilot migration execution.
• Develop reusable transformation and validation libraries that reduce time-to-onboard for new carrier feeds.
• Collaborate with the Data Analysis team to expose modernized data through Fabric semantic models and Power BI datasets.
• Contribute to CI/CD pipelines and Infrastructure-as-Code for data services using Azure DevOps Pipelines, Terraform, and Bicep.
• Act as a mentor to data analysts and future data engineers on pipeline design, data modelling, and quality practices.
• Participate in code reviews and architecture reviews, providing technical feedback across the data and software teams.
• Document ingestion patterns, transformation logic, data quality rules, and operational runbooks for the modernized platform.
• Support incident response and root-cause analysis for data-service issues, including remediation and post-incident learning.
• Other projects, duties and initiatives as assigned.

Qualifications

Technical:

• 5+ years of professional experience as a Data Engineer or similar role, building and operating production data pipelines.
• Post-secondary degree in Computer Science, Software Engineering, Data Engineering, or a related discipline.
• Strong hands-on experience with Microsoft Azure data services: Azure Data Factory, Azure SQL, SQL Server, and Azure Storage.
• Experience with Microsoft Fabric or a comparable modern data platform (Databricks, Snowflake) is required.
• Proficiency in T-SQL and Python for data transformation, orchestration, and scripting.
• Demonstrated experience designing and maintaining metadata-driven ingestion frameworks and reusable transformation libraries.
• Experience implementing data quality controls, schema-drift detection, and observability for data pipelines.
• Exposure to Infrastructure-as-Code (Terraform, Bicep, or ARM) and CI/CD using Azure DevOps Pipelines is required.
• Experience with Power BI semantic models and dataset development is considered an asset.
• Prior experience with EDI, SFTP-based data exchange, or freight/logistics data is considered an asset.
• Prior experience with Microsoft Purview or equivalent catalog and lineage tooling is a plus.

Non-Technical

• Strong communication skills; comfortable engaging both technical and business audiences with clarity.
• Comfortable with ambiguity and greenfield problem-solving; able to define patterns where none exist yet.
• Ownership mindset; sees a system through from design to production operation, not just individual pull requests.
• Mentoring orientation; willing to invest time in growing junior team members’ capability.
• Resilient in a high-autonomy environment with limited established process today.