Technical Lead

City of Daşşak, NY, United States

Location Country United States

Location Region

Location City

Job Details

Spark Data Engineer

Location: Remote USA

Type: Full- time

Experience: Mid-level – Senior

Salary: $100,000 - $140,000

HIRING FOR TWO SPARK DATA ENGINEERS!

About ITI

ITI is an industry-leading information and data management solutions firm with business-driven technology designed for the world’s largest banks, financial services, and healthcare companies. Headquartered in One World Trade Center at the heart of New York City’s Financial District, ITI has over 300 employees across 9 branch and satellite offices in key global, financial services markets including London, India, and Australia. We are seeking two dedicated professional to join our team as a Technical Engineer and contribute to the overall success of the organization and team.

Job Description

We are looking for two Spark Solutions Developer and Data Engineers capable of building a data standardized solution on the Hadoop cluster for one of our Fortune 500 client programs. This is a high visibility, fast-paced key initiative that will integrate data across internal and external sources, provide analytical insights, and integrate with the customer’s critical systems.

Key Responsibilities

Ability to design, build and unit test applications on Spark framework.

Build Spark based applications for both batch and streaming requirements, which will require in-depth knowledge on majority of Hadoop and Kafka.

Ability to design & build real-time applications using Apache Kafka & Spark Streaming.

Develop and execute data pipeline testing processes and validate business rules and policies.

Optimize performance of the built Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame, and Pair RDD's.

Optimize performance for data access requirements by choosing the appropriate native Hadoop file formats (Avro, Parquet, ORC etc) and compression codec, respectively.

Build integrated solutions leveraging Unix shell scripting, RDBMS, Hive, HDFS File System, HDFS File Types, HDFS compression codec.

Experience in processing large amounts of structured and unstructured data, including integrating data from multiple sources.

Create and maintain integration and regression testing framework on Jenkins integrated with BitBucket and/or GIT repositories.

Participate in the agile development process, and document and communicate issues and bugs relative to data standards in scrum meetings.

Work collaboratively with both the onsite and offshore teams.

Develop & review technical documentation for artifacts delivered.

Ability to solve complex data-driven scenarios and triage defects and production issues.

Ability to learn-unlearn-relearn concepts with an open and analytical mindset.

Participate in code release and production deployment.

Challenge and inspire team members to achieve business results in a fast paced and rapidly changing environment.

Qualifications

BE/B.Tech/ B.Sc. in Computer Science/ Statistics, Econometrics from an accredited college or university.

Minimum 5 years of extensive experience in designing, building, and deployment of Spark-based applications.

Expertise in handling complex large-scale Big Data environments preferably (5Tb+).

Minimum 5 years of experience in building data pipelines using both Spark structured streaming and spark batch processes.

Experience in consuming and producing messages to Kafka topic.

Clear understating of Hadoop Ecosystem and Spark Architecture

Strong understanding of Functional programing and RESTful API’s

Produce unit tests for Spark transformations and helper methods.

Experience with design and implementation of ETL framework for complex warehouses.

Knowledge of large data sets and experience with performance tuning and troubleshooting

Strong knowledge and hands-on experience in SQL, Unix shell scripting

Ability to build abstracted, modularized reusable code components.

Hands-on experience in generating/parsing XML, JSON documents, and REST API request/responses.

Excellent communication skills and Good Customer Centricity.

We would love to be able to contact everyone personally; however, this is not possible. If you have not heard from us within 6 weeks of submitting your application, we thank you for your interest in joining ITI Data, but on this occasion, your application was not shortlisted.

Email your resume to: [email protected]

#itidata #jobs #Engineer

Key Responsibilities Ability to design, build and unit test applications on Spark framework.

Build Spark based applications for both batch and streaming requirements, which will require in-depth knowledge on majority of Hadoop and Kafka.

Ability to design & build real-time applications using Apache Kafka & Spark Streaming.

Develop and execute data pipeline testing processes and validate business rules and policies.

Optimize performance of the built Spark applications in Hadoop using configurations around Spark Context, Spark-SQL, Data Frame, and Pair RDD's.

Optimize performance for data access requirements by choosing the appropriate native Hadoop file formats (Avro, Parquet, ORC etc) and compression codec, respectively.

Build integrated solutions leveraging Unix shell scripting, RDBMS, Hive, HDFS File System, HDFS File Types, HDFS compression codec.

Experience in processing large amounts of structured and unstructured data, including integrating data from multiple sources.

Create and maintain integration and regression testing framework on Jenkins integrated with BitBucket and/or GIT repositories.

Participate in the agile development process, and document and communicate issues and bugs relative to data standards in scrum meetings.

Work collaboratively with both the onsite and offshore teams.

Develop & review technical documentation for artifacts delivered.

Ability to solve complex data-driven scenarios and triage defects and production issues.

Ability to learn-unlearn-relearn concepts with an open and analytical mindset.

Participate in code release and production deployment.

Challenge and inspire team members to achieve business results in a fast paced and rapidly changing environment.

Education Level Bachelor's Degree

Experience Level 5 - 7 years

#J-18808-Ljbffr