Python\PySpark Engineer

Role: Python\PySpark Engineer

Location: Jersey City, NJ

FTE only

Job Description

Must Have Technical/Functional Skills

Technical Skills

Strong proficiency in Python, packaging, dependency management, and virtual environments.

Hands-on experience with PySpark, including Spark performance tuning (partitioning, caching, broadcast joins, memory optimization).

Expertise in data ingestion (batch/stream), schema management, and robust error-handling/retry logic.

Solid unit and integration testing practices, including data quality validations.

Experience with CI/CD pipelines (Azure DevOps/Jenkins), Git branching strategies, and artifact versioning.

Working experience with Cloudera/Hadoop (HDFS, Spark, Hive/Impala) and Databricks (Delta Lake, clusters, jobs, notebooks).

Knowledge of observability techniques: structured logging, metrics, tracing, and debugging in distributed systems.

Secure coding practices including secrets management, PII protection, and compliance-aware development.

Functional Skills Strong documentation discipline for frameworks, reusable components, and best-practice patterns.

Effective collaboration with Cloud Architects and Data Ops to ensure stable and supportable pipelines.

Clear communication of technical ideas and solution approaches.

Comfort working in Agile environments with iterative development and frequent releases.

Roles & Responsibilities Design and develop scalable Python/PySpark ingestion and transformation pipelines.

Implement schema evolution logic, validation frameworks, and resilient error-handling mechanisms.

Optimize Spark jobs for performance, cost efficiency, and production readiness.

Integrate all jobs into automated CI/CD pipelines, ensuring versioning and release governance.

Work closely with Ops teams to ensure proper monitoring, logging, and operational supportability.

Participate in Agile ceremonies, sprint planning, code reviews, and demo sessions