Technical Lead
New Brunswick, NJ, United States
Job Title
DatabricksTechnical Lead
Relevant Experience
(inYrs)
10+
Must Have Technical/Functional Skills
Certification:Databricks certified Data Engineer Professional, SQL, Azure data Engineer,Python
PrimarySkillset:
Apache Spark: Strongunderstanding and hands-on experience with Apache Spark, including SparkSQL, Spark Streaming.
Databricks: Proficiency inusing Azure Databricks, including setting up and managing clusters,notebooks, and jobs.
Python and SQL: Strongprogramming skills in either Python and SQL as these are the primarylanguages used for writing Spark applications.
Data Processing and Analysis:Experience in designing and implementing data processing and analysispipelines using Spark and Databricks.
Distributed Computing:Knowledge of distributed computing concepts and experience withdistributed computing frameworks like Spark.
Cloud Platform: Familiaritywith Azure cloud platform and its services, including Azure DataFactory, Azure access, Azure Storage, Azure Data Lake, and Azure SQLDatabase.
Data Engineering: Understandingof data engineering concepts and experience with ETL processes, datamodeling, and data warehousing.
DevOps: Experience with CI/CDpipelines, version control systems (e.g., Git), and automated deploymentof Spark applications.
Performance Optimization:Ability to optimize Spark applications for performance and scalability,including tuning Spark configurations and leveraging Sparkoptimizations.
Data Security and Governance:Understanding of data security and governance practices, including dataencryption, access controls, and compliance regulations.
Monitoring and Troubleshooting:Proficiency in monitoring and troubleshooting Spark applications,identifying and resolving performance bottlenecks or issues.
Collaboration andCommunication: Excellent communication skills and the ability tocollaborate effectively with cross-functional teams, including datascientists, data engineers, and business stakeholders.
SecondarySkillset: Machine Learning: Familiaritywith machine learning concepts and experience with implementing machinelearning algorithms using Spark MLlib or other machine learningframeworks.
Data Visualization: Kn owledgeof data visualization tools and libraries like Matplotlib, Plotly, PowerBI or Tableau for presenting insights derived from data analysis.
#J-18808-Ljbffr