Senior Databricks Architect / Lead Data Engineer

Senior Databricks Architect / Lead Data Engineer Location: Westchester, IL

Duration: 6-12 Months

Here are some bullet points to help narrow the search:

• 10+ years overall experience

• 5+ years on Databricks

• 2+ completed Databricks implementations

• Strong PySpark coding skills (must be able to code during interviews)

• Experience with Unity Catalog, DLT, Delta Lake, and Workflows

• Azure Databricks architecture experience

• Client-facing consulting background

Position Summary

We are seeking an experienced Senior Databricks Architect / Lead Data Engineer to lead the design, development, and implementation of modern data platforms leveraging Azure Databricks. The ideal candidate will possess deep expertise in building scalable data engineering solutions, strong hands-on coding skills, and proven experience delivering enterprise-grade Databricks implementations for clients.

This role requires a blend of technical leadership, architecture, solution delivery, and client-facing consulting capabilities.

Key Responsibilities

Solution Architecture & Design

Design and architect scalable, secure, and high-performance data platforms using Azure Databricks.

Lead the design and implementation of modern data lakehouse architectures utilizing Delta Lake, Unity Catalog, and Databricks best practices.

Define data ingestion, transformation, orchestration, governance, and consumption patterns across enterprise data ecosystems.

Establish architecture standards, development frameworks, and reusable components.

Data Engineering & Development

Develop and optimize complex data pipelines using PySpark and Databricks.

Build and maintain ETL/ELT frameworks leveraging Delta Live Tables (DLT) and Databricks Workflows.

Ensure code quality, performance optimization, scalability, and reliability of data processing solutions.

Implement data quality, monitoring, lineage, and governance controls.

Project Delivery

Lead end-to-end Databricks implementations from strategy and design through deployment and stabilization.

Collaborate with business stakeholders, data consumers, architects, and platform teams to translate business requirements into technical solutions.

Guide development teams and perform design/code reviews.

Troubleshoot complex technical challenges and provide solution recommendations.

Client Engagement & Consulting

Serve as a trusted advisor to client stakeholders on data modernization and analytics transformation initiatives.

Conduct architecture workshops, solution demonstrations, and technical assessments.

Present technical solutions and recommendations to business and executive audiences.

Support estimation, solutioning, proposal development, and pre-sales activities as needed.

Required Qualifications

10+ years of overall experience in Data Engineering, Data Architecture, or Analytics platform delivery.

5+ years of hands-on experience with Databricks.

Demonstrated experience delivering at least two (2) end-to-end Databricks implementations in enterprise environments.

Strong hands-on coding expertise in PySpark with the ability to develop solutions during technical interviews and coding assessments.

Extensive experience with:

Unity Catalog

Delta Lake

Delta Live Tables (DLT)

Databricks Workflows

Strong experience designing and implementing Azure Databricks architectures.

Experience integrating Databricks with Azure services such as Azure Data Lake Storage (ADLS), Azure Data Factory (ADF), Azure Synapse, Azure Key Vault, and Microsoft Fabric (preferred).

Strong understanding of data modeling, ETL/ELT design patterns, data governance, and performance tuning.

Proven client-facing consulting experience with excellent communication and stakeholder management skills.

Preferred Qualifications

Databricks Certified Professional or Associate certifications.

Azure Data Engineering and/or Azure Solutions Architect certifications.

Experience implementing CI/CD, Infrastructure as Code, and DevOps practices for Databricks environments.

Experience with data governance, metadata management, security, and access controls in enterprise data platforms.

Familiarity with streaming architectures, real-time analytics, and advanced analytics workloads.

Key Competencies

Enterprise Data Architecture

Data Lakehouse Design

PySpark Development

Azure Databricks

Solution Leadership

Client Relationship Management

Technical Consulting

Data Governance & Security

Problem Solving & Analytical Thinking

Team Leadership & Mentoring

This JD should attract candidates in the 12-18 year experience range who can function as both an architect and hands-on Databricks technical lead.