Azure Data Engineer

This will require an onsite interview or require the candidate to pick up their laptop in person

Requirements/Top Skills Needed:

ADF

Databricks

Python

SQL

Retail experience would be helpful, but not required

Job Title: Azure Data Engineer

Job Summary: We are seeking a highly skilled Azure Data Engineer to design, build, and optimize our next-generation cloud data platform. You will build scalable data pipelines, manage enterprise data warehouses, and transform massive datasets using Azure Cloud, ADF, Databricks, Spark, and PySpark.

Key Responsibilities

• Pipeline Orchestration: Design and implement scalable ETL/ELT pipelines using Azure Data Factory (ADF).

• Big Data Processing: Build high-performance, distributed data transformation workloads via Azure Databricks and Apache Spark/PySpark.

• Data Warehousing: Architect, model, and maintain cloud-native Data Warehouses and Data Lakes (ADLS Gen2).

• Architecture Implementation: Apply modern architectural patterns like the Medallion Architecture (Bronze, Silver, Gold layers) and Delta Lake formats.

• Performance Tuning: Optimize Spark jobs, cluster configurations, and SQL queries to minimize latency and cloud compute costs.

• Governance & Security: Implement security frameworks, data lineage, and user access policies using Unity Catalog.

• CI/CD Automation: Deploy, monitor, and automate workflows using Azure DevOps and automated alert configurations.

Required Technical Skills:

• Cloud Platform: Microsoft Azure Ecosystem (ADLS Gen2, Azure SQL, Synapse Analytics, Key Vault, Storages, Azure Functions, Logic Apps).

• Data Orchestration: Advanced proficiency in Azure Data Factory (ADF).

• Compute & Analytics: Deep hands-on experience with Azure Databricks.

• Big Data Frameworks: Strong mastery of Apache Spark and PySpark (including Spark SQL).

• Languages: Expert level in Python and SQL.

• Data Warehousing: Solid understanding of Data Modeling (Star/Snowflake schema) and warehousing concepts