Data Engineering Consultant

Data Engineering Consultant designs, implements, and optimizes scalable data pipelines and architectures. This role bridges raw data and actionable insights, ensuring robustness, performance, and data governance. Collaboration with analysts and scientists is central to delivering high-quality solutions aligned with business objectives. Key responsibilities include developing and maintaining real-time and batch data pipelines, integrating and transforming data, automating processes, collaborating with cross-functional teams, ensuring data governance and compliance, and optimizing performance. The role also involves staying updated with emerging technologies and contributing to innovation.

Must Have

• Design, implement, and optimize scalable data pipelines and architectures.
• Architect and maintain real-time and batch data pipelines using Apache Spark, Databricks, Snowflake, or Airflow.
• Build ETL/ELT workflows, validation checks, and cleaning steps for data reliability.
• Automate data reconciliation, metadata management, and error-handling procedures.
• Coordinate with Data Scientists, Data Architects, and Analysts for business alignment.
• Apply robust security measures (RBAC, encryption) and ensure regulatory compliance (GDPR).
• Conduct regular performance audits and optimize data pipelines and storage systems.
• 5-10 years of experience in data engineering with a strong focus on big data technologies.
• Expert-level proficiency in Python/PySpark.
• Extensive experience with cloud platforms like AWS, Azure, or GCP.
• Advanced knowledge of data warehousing solutions such as Databricks, Snowflake, BigQuery, or Redshift.
• Strong understanding of data modeling, ETL/ELT processes, and data governance principles.
• Proficiency in SQL and NoSQL databases.

Good to Have

• Relevant certifications (e.g., AWS Certified Data Analytics - Specialty, Azure Data Engineer Associate, Google Cloud Professional Data Engineer).
• Experience with real-time data streaming technologies (e.g., Kafka, Apache Flink).
• Knowledge of machine learning concepts and their application in data engineering.
• Familiarity with data lake architectures and associated technologies.
• Experience with CI/CD practices for data pipelines.
• Understanding of data security and compliance requirements (e.g., GDPR).
• Strong problem-solving skills and ability to work independently.
• Excellent communication skills, both written and verbal.
• Experience mentoring junior engineers and collaborating with cross-functional teams.

Company Description

Technology is our how. And people are our why. For over two decades, we have been harnessing technology to drive meaningful change.

By combining world-class engineering, industry expertise and a people-centric mindset, we consult and partner with leading brands from various industries to create dynamic platforms and intelligent digital experiences that drive innovation and transform businesses.

From prototype to real-world impact - be part of a global shift by doing work that matters.

Job Description

Data Engineering Consultant designs, implements, and optimizes scalable data pipelines and architectures. This role bridges raw data and actionable insights, ensuring robustness, performance, and data governance. Collaboration with analysts and scientists is central to delivering high-quality solutions aligned with business objectives.

Key Responsibilities:

1. Data Pipeline Development

• Architect, implement and maintain real-time and batch data pipelines to handle large datasets efficiently.
• Employ frameworks such as Apache Spark, Databricks, Snowflake or Airflow to automate ingestion, transformation, and delivery.
• Design and implement fault-tolerant, self-healing data pipelines to ensure continuous data flow.
• Optimize data pipeline performance through parallel processing and distributed computing techniques.

2. Data Integration & Transformation

• Work with Data Analysts to understand source-to-target mappings and quality requirements.
• Build ETL/ELT workflows, validation checks, and cleaning steps for data reliability.
• Develop and maintain data dictionaries and metadata repositories for improved data understanding.
• Implement data quality monitoring tools and processes to ensure data integrity throughout the pipeline.

3. Automation & Process Optimization

• Automate data reconciliation, metadata management, and error-handling procedures.
• Continuously refine pipeline performance, scalability, and cost-efficiency.
• Implement monitoring and alerting systems for early detection of pipeline issues.
• Develop and maintain documentation for all automated processes and workflows.

4. Collaboration & Leadership

• Coordinate with Data Scientists, Data Architects, and Analysts to ensure alignment with business goals.
• Mentor junior engineers and enforce best practices (version control, CI/CD for data pipelines).
• Participate in technical presales activities and client engagement initiatives.
• Lead cross-functional team meetings to discuss data engineering challenges and solutions.
• Contribute to the development of data engineering standards and best practices within the organization.

5. Governance & Compliance

• Apply robust security measures (RBAC, encryption) and ensure regulatory compliance (GDPR).
• Document data lineage and recommend improvements for data ownership and stewardship.
• Implement data masking and anonymization techniques for sensitive information.
• Collaborate with legal and compliance teams to ensure adherence to data protection regulations.

6. Performance Tuning & Optimization

• Conduct regular performance audits of data pipelines and storage systems.
• Implement caching strategies and query optimization techniques to improve data access speeds.
• Utilize partitioning and indexing strategies to enhance query performance on large datasets.
• Optimize resource allocation and utilization in cloud environments to manage costs effectively.

7. Emerging Technologies & Innovation

• Stay abreast of emerging data engineering technologies and methodologies.
• Evaluate and recommend new tools and frameworks that can enhance data engineering capabilities.
• Conduct proof-of-concept projects to test innovative data engineering solutions.
• Contribute to the company's thought leadership through blog posts, whitepapers, or conference presentations on data engineering topics.

Qualifications

Required:

• Bachelor's degree in Computer Science, Information Technology, or a related field; Master's degree preferred
• 5-10 years of experience in data engineering, with a strong focus on big data technologies
• Expert-level proficiency in Python/PySpark
• Extensive experience with cloud platforms, particularly AWS-or- Azure-or- GCP
• Advanced knowledge of data warehousing solutions such as Databricks -or- Snowflake-or- BigQuery -or- Redshift
• Proven track record in designing, implementing, and optimizing large-scale data pipelines
• Strong understanding of data modeling, ETL/ELT processes, and data governance principles
• Proficiency in SQL and NoSQL databases
• Familiarity with data visualization tools and techniques

Preferred:

• Relevant certifications such as AWS Certified Data Analytics - Specialty, Azure Data Engineer Associate, or Google Cloud Professional Data Engineer
• Experience with real-time data streaming technologies (e.g., Kafka, Apache Flink)
• Knowledge of machine learning concepts and their application in data engineering
• Familiarity with data lake architectures and associated technologies
• Experience with CI/CD practices for data pipelines
• Understanding of data security and compliance requirements (e.g., GDPR)
• Strong problem-solving skills and ability to work independently
• Excellent communication skills, both written and verbal
• Experience mentoring junior engineers and collaborating with cross-functional teams.

Additional Information

At Endava, we’re committed to creating an open, inclusive, and respectful environment where everyone feels safe, valued, and empowered to be their best. We welcome applications from people of all backgrounds, experiences, and perspectives—because we know that inclusive teams help us deliver smarter, more innovative solutions for our customers. Hiring decisions are based on merit, skills, qualifications, and potential. If you need adjustments or support during the recruitment process, please let us know.