Data Engineer

Job TitleJob Summary: Skills5+ years' experience in SQL (Expert)4+ years of experience in Azure Databricks with PySpark4+ years of experience in Azure Cloud platform3+ years of experience in ADF (Azure Data Factory), ADLS Gen 2 and Azure SQL2+ years of experience in Databricks workflow & Unity catalog2+ years of experience in Python programming & package buildsExperience in data ingestion, cleansing, and transformation processes from various structured and unstructured data sources like Cassandra & Mark Logic and on-prem Mainframe sources using Databricks/PySpark supporting batch and near-real-time ingestion, transformation, and processing.Manage job scheduling, orchestration, and monitoring (e.g., using Azure Data Factory, Airflow, or Databricks Workflows).Ability to design modular, reusable workflows using tasks, triggers, and dependencies.Skilled in using dynamic expressions, parameterized pipelines, custom activities, and triggers.Familiarity with integration runtime configurations, pipeline performance tuning, and error handling strategies.Strong understanding of ETL/ELT design patterns, data warehousing, and data lakehouse architectures.Good to have Azure Entra/AD skills and GitHub ActionsGood to have experience working on event-driven architectures using Kafka, Azure Event HubResponsibilities:Design develop and optimize scalable data pipelines leveraging Databricks (Spark, PySpark, SQL, Delta Lake) to support enterprise level data processing and analyticsWrite clean maintainable and efficient PySpark and Python code to support data ingestion transformationIntegrate Azure Databricks with various Azure data services to build robust and scalable data platformsImplement and maintain ETL workflows for scalability cost effective and operational efficiencyCollaborate with data analysts and stakeholders to gather requirements and deliver scalable data solutionsParticipate actively in design discussions code reviews and agile ceremonies to foster a collaborative and high performing team environment.