Big Data Engineer

Role: Big Data Engineer

Location: Plano, TX (Onsite)

Client: Artech / JPMorgan Chase

Mode: W2 Contract / Full Time Opening

Job Description:

Job Family Description: Expert in in AWS and Client Azure technology and design techniques as well as experience working across large environments with multiple operating systems/infrastructure for large-scale programs (e.g., Expert Engineers) starting to be firm-wide resources working on projects across Client

Is multi-skilled with expertise across the software development lifecycle and toolset

May be recognized as a leader in Agile and cultivating teams working in Agile frameworks

Sought out as coach for at least one technical skill

Strong understanding of techniques such as Continuous Integration, Continuous Delivery, Test Driven Development, Cloud Development, resiliency, security

Stays abreast of cutting-edge technologies/trends and uses experience to influence the application of those technologies/trends to support the business; may give speeches outside the firm, writes articles

Technologies:

Data Bricks

Apache Spark

Python(and/or Scala)

AWS

SQL

Java

CI/CD Pipeline tools

Description: Minimum Industry experience: 10 years

Bachelor's degree or equivalent work experience

Must have experience implementing atleast 2 end to end projects with Databricks which includes components like.

Delta lake.

Unity Catalog.

Databricks Connect.

DataBricks API.

Databricks workflow Orchestration.

Must be well-versed with Databricks Lakehouse concepts and its implementation in large-scale enterprise environments.

Must have a good understanding of creating complex data pipelines and use of data ingestion tools like nifi.

Must be well versed in using Apache Spark framework and strong in SQL and spark-sql

Must have excellent coding skills in Python(and/or Scala), preferably Python.

Must have strong performance optimization skills to improve efficiency and reduce cost using techniques like parallel processing.

Must have worked on AWS and should have working experience with AWS services like S3, Lambda, DynamoDB, SQS, and CloudWatch.

Must be able to work with architects and lead engineers to design solutions to meet functional and non-functional requirements.

Must have strong written and verbal communication skills.

Must have experience in an Agile environment.

Data engineering background is a plus.