Data Engineer
Plano, TX, United States
As a Big Data Application Developer, you will be responsible for developing, optimizing, and maintaining big data solutions. You will work with a team of data engineers and analysts to build scalable and efficient data processing pipelines and applications using a variety of technologies.
Key Responsibilities :
Develop and maintain big data processing applications using Spark, Python, and Scala.
Design and implement ETL processes using AWS Glue.
Build and manage workflows with AWS Step Functions and Lambda.
Develop, test, and deploy data pipelines on Databricks.
Collaborate with data engineers, analysts, and stakeholders to understand data requirements and deliver solutions.
Optimize and tune data processing jobs for performance and scalability.
Ensure data quality and integrity through thorough testing and validation.
Document processes, solutions, and provide support as needed.
Required Skills and Qualifications : Programming Languages : Proficiency in Python and Scala.
Big Data Technologies : Experience with Apache Spark for large-scale data processing.
Cloud Services : Hands-on experience with AWS Glue, AWS Step Functions, and AWS Lambda.
Data Platform : Familiarity with Databricks for data engineering and analytics.
Database Knowledge : Understanding of SQL and NoSQL databases.
ETL Processes : Strong experience in designing and implementing ETL pipelines.
Data Integration : Knowledge of integrating various data sources and systems.
Performance Tuning : Skills in optimizing big data jobs for performance.
Collaboration : Excellent communication skills and the ability to work collaboratively in a team environment.
Problem-Solving : Strong analytical and problem-solving abilities.
Preferred Qualifications : Experience with additional big data tools and frameworks.
Certification in AWS or relevant big data technologies.
Familiarity with DevOps practices and CI/CD pipelines.
Experience with machine learning frameworks and libraries.
Education :
Bachelor's degree in Computer Science, Engineering, or a related field. Advanced degrees are a plus.