Senior Fabric Data Engineer (ID#5544)

Job summary

Build and configure Microsoft Fabric environments, infrastructure, CI/CD pipelines, and a metadata-driven framework for onboarding data sources. Onboard and test a pilot source, document operational processes, and coach client engineers through knowledge-transfer workshops.

Job details

New Value Solutions is a national IT consulting company. We are searching a Senior Microsoft Fabric Data Engineer who is a hands on builder of Microsoft Fabric environments, the CI/CD pipeline, the metadata driven onboarding framework and the pilot source onboarding. This is a contract role, estimated to be 300-400 hours over a 3-4 month period. Definite possibilities for more hours as needed. We also have another project for a different client which may require additional assistance. Responsibilities •Build Dev, Test and Prod exactly as approved in the foundation design, using Bicep for Azure resources (capacity, gateway subnet, private endpoints, private DNS, Key Vault) and the Microsoft Fabric REST APIs for capacity assignment, domains, workspaces, private links, managed private endpoints, gateway connections and Entra ID group role assignments. •Run the connectivity proof on non sensitive data to confirm private endpoint access, gateway connectivity and mirroring behaviour under the proposed network model. •Implement source control and CI/CD: Git integration for workspaces, Azure DevOps Repos and Pipelines, fabric-CI/CD deployment from a build agent inside the client's VNet, parameter files and variable libraries, approval gates and release notes. Demonstrate a change moving Dev to Test to Prod. •Build the metadata driven onboarding framework: source registry, pipeline and notebook templates, validation and reconciliation templates. •Onboard the approved pilot source into Bronze, Silver and Gold, schedule and monitor loads, and run reconciliation, freshness, performance, access and recovery tests. •Write the operations runbook, the onboarding playbook and the environment verification checklist. •Pair with client engineers throughout and coach them while they onboard part of the pilot themselves. •Deliver knowledge transfer workshops on CI/CD and release process, onboarding a new source and operations and troubleshooting. Required experience and skills •At least two years building on Microsoft Fabric in production: Lakehouse, notebooks (PySpark and Spark SQL), Data Factory pipelines, Copy jobs, Dataflows Gen2, Mirroring, Delta tables and table maintenance. •Azure DevOps YAML pipelines and Git workflows, including branch policies and pull request review. Experience deploying Fabric items with fabric-CICD or the Fabric REST APIs. •Infrastructure as code with Bicep (or Terraform) for Azure networking and platform resources. •Practical experience with VNet data gateways or on premises data gateways and with sources behind private endpoints. •Strong SQL and Python. Able to design metadata driven, configuration first frameworks rather than one off pipelines. •Data validation and reconciliation practices: row counts, checksums, freshness checks, failure handling and alerting. •Able to explain what they build to client engineers and to document it clearly. Preferred Skills, an asset •DP-700 (Fabric Data Engineer) or DP-600 certification. •Oracle and Azure SQL Database source experience, including LogMiner or change data capture prerequisites. •Power BI semantic model basics, Microsoft Purview, Fabric capacity monitoring and cost management. •Experience in public sector environments with formal security review.

What you’ll do

Build and configure Microsoft Fabric environments, infrastructure, CI/CD pipelines, and a metadata-driven framework for onboarding data sources. Onboard and test a pilot source, document operational processes, and coach client engineers through knowledge-transfer workshops.

Requirements

Requires at least two years of production experience with Microsoft Fabric, including Lakehouse, notebooks, pipelines, mirroring, and Delta tables, plus Azure DevOps CI/CD and infrastructure-as-code experience. Candidates should also have strong SQL and Python skills, experience with private network connectivity and data reconciliation, and the ability to document and explain technical work.

Listed skills

• SQL · Preferred
• CI/CD · Preferred
• Data Validation · Preferred
• Terraform · Preferred
• Git · Preferred
• Python · Preferred

Other relevant skills

Identified from the job description. Confirm important requirements above.

• Microsoft Fabric
• PySpark
• Spark SQL
• Azure Data Factory
• Azure DevOps
• Bicep
• Terraform
• Python
• SQL
• CI/CD
• Git
• Fabric REST APIs
• Data Modeling
• Data Validation
• VNet Data Gateways
• Delta Tables

Job areas

• Data & Analytics
• Technology
• Software
• Consulting

More jobs from New Value Solutions