Big Data Engineer

Job Description

Candidate will be responsible for designing and developing complex requirements to accomplish business outcomes. The selected candidate should have hands-on Hadoop experience and an excitement for solving complex problems. This developer will be part of a collaborative team and will provide direction to team members. Experience in an Agile based programming environment is a plus

Position Summary

• Key responsibilities of the role include ensuring that software is developed to meet functional, non-functional, and compliance requirements. This role ensures solutions are well designed with maintainability/ease of integration and testing built-in from the outset. They possess strong proficiency in development and testing practices common to the industry, and have extensive experience of using design and architectural patterns. A Software Engineer is an individual contributor to a team.

• Contributes to story refinement/defining requirements.

• Participates in estimating work necessary to realize a story/requirement through the delivery lifecycle.

• Performs spike/proof of concept as necessary to mitigate risk or implement new ideas.

• Codes solutions and unit tests to deliver a requirement/story per the defined acceptance criteria and compliance requirements.

• Designs/develops/maintains automated test suites (integration, regression, performance).

Required Qualifications

• Strong database and SQL skills

• Good understanding on Hadoop Big Data Eco system

• Experience writing Shell scripts in LINUX or Unix

• Experience with Autosys

• Hadoop/Spark - Running Pig and Hive Queries, analyzing data with Hive, Hands on experience with Oozie, Importing and exporting the data using Sqoop from HDFS to Relational Database, Developing MapReduce Program to format the data

• Good communication skills

• Familiarity with CI/CD tools such as Ansible, Jenkins, and Artifactory