Sr Data Quality Engineer - Azure Focus

Sr Data Quality Engineer - Azure Focus

Chicago, IL (loop/downtown) - 2 days work from home (WFH), 3 days onsite

Summary:

This position focuses on building automated tesing frameworks to ensure data quality and reliability across data pipelines in our Azure, Databricks, SQL Server/Azure SQL and Medallion Architecture (Bronze, Silver, Gold layers) environment.

We use Python and SQL extensively on our data engineering team, and are looking for candidates who can come to the table with these skills, and a candidate coming from an Azure/Databricks environment.

We are also looking for candidates that have strong experience building automated validation frameworks using tools - if you have experience with DQX (Databricks Data Quality Framework), that would be awesome, but anything similar works.

Last, we are leaning into AI on our engineering teams, and you'll have the opportunity to leverage tools like Claude Code, so any experience in this area is also a nice to have.

What you'll be doing Design and implement scalable end-to-end testing frameworks for data pipelines

Validate ingestion from APIs, SQL Server, and flat files

Ensure data quality across Medallion architecture layers (Bronze, Silver, Gold)

Build automated checks for schema validation, data integrity, and transformations

Develop reusable validation patterns using Great Expectations or similar frameworks

Leverage Claude Code and prompt engineering to accelerate test generation and standardization

Create reusable AI-driven testing assets (.md skills, templates) and workflows for AI-assisted coding and testing

Integrate testing into CI/CD pipelines (Azure DevOps, GitHub Actions)

Collaborate with data engineers, Product, and DevOps to define data quality standards and acceptance criteria

Monitor and improve data reliability, observability, and test coverage

Investigate data quality issues and drive root-cause resolution

Skills we're seeking 5+ years of experience with Data Engineering and/or Data Quality Engineering

Must have strong experience with Data Quality Engineering Must have experience with data validation tools such as DQX, Great Expectations, etc.

Must have strong experience with Data (Quality) Engineering in Azure environments

Must have experience with Medallion Architecture

Must have strong experience with SQL Server

Python experience

Nice to haves Bachelor's or Master's Degree in an IT related field

Spark and/or PySpark experience

Experience with Claud Code

Experience with Azure DevOps, CI/CD pipeliens, etc. Bicep experience would be awesome, or any IaC for that matter