Senior Data Platform Engineer

Senior Data Engineer / Data Architect

Our client, a growing mid-market company, is investing in a modern enterprise data platform built on Azure Databricks to support analytics, reporting, and emerging AI initiatives. This is a hybrid engineering and architecture role covering everything from pipeline builds to data modeling, governance, and platform optimization within a Lakehouse framework.

Responsibilities

• Develop batch and streaming pipelines in Databricks and Spark, moving data from source to consumption

• Build out the medallion (Bronze/Silver/Gold) layer structure within the Lakehouse

• Bring data in from ERP, SaaS tools, APIs, and older legacy platforms

• Improve pipeline performance while keeping an eye on cost and scale

• Set up CI/CD workflows through GitHub

• Take ownership of data architecture direction and set the standards other teams build against

• Create logical and physical data models spanning finance, operations, and field service

• Lock in consistent definitions for shared entities like customer, job, and revenue

• Shape data structures so they work equally well for BI reporting and AI/ML training

• Run governance through Unity Catalog, covering access roles, lineage, and audit needs

• Put standards in place for naming, permissions, and data quality checks

• Keep HR and financial data protected in accordance with security policy

• Build checks that catch data quality issues before they reach reporting or downstream systems

• Design repeatable patterns for how data flows in, gets transformed, and gets consumed by BI tools, APIs, or AI models

• Link Databricks into the broader Azure ecosystem (ADLS, internal APIs, enterprise apps)

• Keep the platform observable, stable, and running smoothly

• Get data ready for AI/ML consumption, including automation-focused use cases

• Collaborate with analytics and AI groups on datasets built for predictive work

• Sit with business and application teams to turn their needs into working data solutions

• Set direction on engineering approach and architectural tradeoffs for the broader team

• Coach junior team members and contractors as the group expands

Requirements

• 6+ years building data pipelines, data architecture, or enterprise data platforms

• Direct hands-on background standing up cloud-based data pipelines

• Prior work inside enterprise systems (ERP, CRM, or comparable platforms)

• Strong Azure Databricks and Apache Spark chops

• Solid Python/PySpark skills

• Advanced SQL

• Comfortable with Azure Data Lake Storage (ADLS)

• CI/CD background using GitHub

• Familiarity with API-driven and data integration patterns

• Comfortable with both dimensional and normalized modeling approaches

• Working knowledge of data governance and security, including access roles and lineage tracking

• Delta Lake or broader Lakehouse experience is a plus

• Any exposure to ML/AI pipelines or data science work is a plus

• Manufacturing, oil & gas, or ERP-heavy backgrounds are a plus

• Experience with C#, .NET, SQL, or JavaScript is a plus

• DevOps and source control experience tied to ERP development is a plus

• Understanding of cloud ERP platforms and migration paths is a plus

• Power BI, SSRS, or Tableau exposure for integration/reporting is a plus

Do not apply unless you are authorized to work in the United States for any employer as client company cannot sponsor or transfer visas at this time.

Walker Elliott is an Equal Opportunity Employer.

For additional information, please email your resume to [email protected] or apply online.