Databricks Data Engineer

Required Skills & Experience

§ 6+ years of hands-on expertise with Azure Databricks, Python, PySpark, and Delta Lake.\n\t\t\t§ 6+ years of proven experience with Azure Data Factory for orchestrating and monitoring pipelines.\n\t\t\t§ STRONG T-SQL Coding! There will be a coding excersize on interview\n\t\t\t§ Strong SQL Server / T-SQL experience with a focus on query optimization, indexing strategies, and coding best practices.\n\t\t\t§ Demonstrated experience in SSIS package design, deployment, and performance tuning.\n\t\t\t§ Hands-on knowledge of Unity Catalog for governance.\n\t\t\t§ Experience with Git (Azure DevOps Repos) and CI/CD practices in data engineering projects.

Nice to Have Skills & Experience

• Microsoft Certified: Azure Data Engineer Associate (DP-203)\n\t• Microsoft Certified: Azure Solutions Architect Expert or equivalent advanced Azure certification\n\t• Databricks Certified Data Engineer Associate or Professional\nAdditional Microsoft SQL Server or Azure certifications demonstrating advanced database and cloud expertise

Job Description

• Data Pipeline Development: Design, build, and optimize ETL/ELT pipelines using Azure Databricks (PySpark, Delta Lake) and Azure Data Factory (ADF).\n\t\t• Data Flows & Transformations: Develop pipelines, data flows, and complex transformations with ADF, PySpark, and T-SQL for seamless data extraction, transformation, and loading.\n\t\t• Data Processing: Develop Databricks Python notebooks for tasks such as joining, filtering, and pre-aggregation.\n\t\t• Database & Query Optimization: Optimize database performance through SQL query tuning, index optimization, and code improvements to ensure efficient data retrieval and manipulation.\n\t\t• SSIS & Migration Support: Maintain and enhance SSIS package design and deployment for legacy workloads; contribute to migration and modernization into cloud-native pipelines.\n\t\t• Collaboration & DevOps: Work with cross-functional teams using Git (Azure Repos) for version control and Azure DevOps pipelines (CI/CD) for deployment.\n\t\t• Data Governance & Security: Partner with governance teams to integrate Microsoft Purview and Unity Catalog for cataloging, lineage tracking, and role-based security.\n\t\t• API & External Integration: Implement REST APIs to retrieve analytics data from diverse external data feeds, enhancing accessibility and interoperability.\n\t\t• Automation: Automate ETL processes and database maintenance tasks using SQL Agent Jobs, ensuring data integrity and operational reliability.\n\t\t• Advanced SQL Expertise: Craft and optimize complex T-SQL queries to support efficient data processing and analytical workloads.