Principal Data Engineering Lead

Houston, TX, United States

48forty Solutions is the largest pallet management services company in North America. We provide end-to-end pallet solutions, from supply to retrieval, on-site services, reverse logistics, and packaging materials. 48forty Solutions is truly Pallet Management Made Simple. Our operations workforce is the heart and soul of our business. We are currently looking for a Principal Data Engineering Lead.

Summary

You will be a hands-on technical Lead Data Engineer to our Modern Data Platform Stack to support our growth and enable new data development in a dynamic, fast-paced environment. As a Data Engineering Technical Lead, you will be responsible for leading a team of data engineers in designing, building, and maintaining data pipelines and data processing systems using various technologies such as Azure, Spark SQL, SQL Server, Azure Data Factory, Azure Synapse, Azure Databricks, Dimensional Modeling, Agile, Azure DevOps, and DataOps. You will provide technical guidance, drive recommended practices, and ensure the successful delivery of data engineering projects. You will play a critical role in shaping the data infrastructure and enabling data-driven initiatives within the organization.

Essential Duties And Responsibilities

Technical leadership: Lead a team of data engineers, providing technical guidance, mentoring, and coaching. Drive the adoption and establishment of recommended practices, standards, and methodologies for data engineering projects.

Design and architecture: Collaborate with stakeholders to understand data requirements and translate them into scalable and performant data engineering and data product solutions. Design data pipelines, data models, and system architectures using Azure services, including Azure Data Factory, Azure Synapse, and Azure Databricks.

Data pipeline development: Lead the development of data pipelines, ELT processes, and data integration workflows using Azure Databricks, Azure Synapse Pipelines, Spark SQL, PySpark and other relevant technologies. Ensure the availability, reliability, and scalability of data processing systems.

Dimensional modeling: Apply dimensional modeling techniques to design and implement data models in SQL Server, adhering to best practices for analytical reporting and business intelligence. Design star schemas, fact tables, and dimension tables to support data analysis and reporting requirements.

Performance optimization: Optimize data processing workflows, SQL queries, and transformations to improve system performance, reduce latency, and enhance scalability.

Data quality and governance: Establish data quality standards, implement data validation rules, and develop data cleansing processes. Ensure adherence to data governance policies and practices to maintain the accuracy, integrity, and security of data.

Agile project management: Collaborate in an Agile team, participate in sprint planning, backlog refinement, and other Agile ceremonies.

Azure DevOps and DataOps: Implement and oversee Azure DevOps practices for version control, CI/CD, and automation of data engineering artifacts. Drive the adoption of DataOps principles, including testing, deployment automation, and monitoring of data pipelines.

Collaboration and communication: Collaborate effectively with stakeholders, including data analysts, and business users, to understand their data requirements and align data engineering solutions accordingly. Communicate technical concepts and project progress to non-technical stakeholders.

Continuous learning and innovation: Stay up to date with the latest advancements in data engineering technologies and tools. Maintain and achieve relevant certifications. Evaluate, establish, and recommend methodologies, and frameworks that can enhance the data engineering capabilities of the team.

Qualifications And Skills

5 years of experience in data engineering, data integration, and data warehousing using Azure services such as SQL Server, Azure Data Factory, Azure Synapse, and Azure Databricks.

5 years of experience in SQL programming and expertise in working with database management systems like SQL Server.

Bachelor's Degree in Computer Science, Information Technology, or a related field required.

In-depth understanding of dimensional modeling principles and best practices for designing scalable and high-performance data structures.

Hands-on experience with Spark SQL and data processing using Apache Spark.

Familiarity with cloud-based architecture and services, especially Microsoft Azure.

Knowledge of data governance, data quality, and data security principles.

Experience working in an Agile development environment and familiarity with Agile methodologies.

Strong understanding of Azure DevOps practices and tools for version control, CI/CD, and automation.

Knowledge and experience with DataOps principles, including data pipeline testing, deployment automation, and monitoring.

Strong problem-solving and Analytical Acumen

Strong knowledge of cloud-based architectures and services, particularly Microsoft Azure.

Proficiency in data governance, data quality, and data security principles.

Experience leading and mentoring a team of data engineers, providing technical guidance, and driving best practices.

Expertise in Agile methodologies and experience working in Agile development environments.

Ability to multi-task, prioritize assignments and work well under deadlines in a changing environment with a cross functional agile team.

Experience managing and leading a small team is preferable.

Work Environment

The work environment characteristics described here are representative of those an employee encounters while performing the essential functions of this job. Reasonable accommodations may be made to enable individuals with disabilities to perform the essential job functions.

The employee will be working at the corporate office and will be required to sit for long periods of time at a desk working on a laptop.

The noise level in the work environment is usually moderate.

Benefits

Competitive Pay

Holiday Pay

Referral Bonuses

Long-Term Career Advancement

Great Team Environment

PTO

Full-time employees eligible for Medical, Dental, Vision, Basic Life, AD&D and Short-Term & Long-Term Disability insurance on the 1st of the month following 60 days of employment

48forty Solutions is an equal opportunity employer.

Privacy Policy

CA Applicant

CA Workforce

#J-18808-Ljbffr