Performance Engineer

Diagnose issues that an in-house performance testing team has been unable to. There are five aspects to Performance Engineering: software development lifecycle and architecture, performance testing and validation, capacity planning, application performance management and problem detection and resolution. The ideal candidate will have experience building reusable Python/PySpark frameworks for standardizing data engineering workflows, test frameworks for pipeline reliability, and data quality frameworks for monitoring and validation. Hands-on experience with Datadog or similar observability tools is required to monitor pipeline performance, optimize resource usage, and ensure system reliability. You will work within a cross-functional team, building scalable, production-grade data pipelines on cloud platforms such as AWS, Azure, or GCP.

Must Have

• Databricks Unified Data Analytics Platform expertise.
• Develop and maintain ETL/ELT pipelines using PySpark and Python.
• Build reusable, modular frameworks for data engineering workflows.
• Implement test frameworks for automated unit, integration, and regression testing.
• Design and maintain data quality frameworks for validation and monitoring.
• Optimize Spark jobs for performance, scalability, and cost-efficiency.
• Collaborate with data architects on robust data models and design patterns.
• Integrate Databricks pipelines with cloud storage (S3, ADLS, Snowflake).
• Implement CI/CD pipelines for Databricks using Git, Jenkins, or Azure DevOps.
• Monitor pipeline performance, resource utilization, and system health with Datadog.
• Build dashboards and alerts for proactive monitoring and troubleshooting.
• Partner with data scientists and stakeholders to translate requirements.
• Conduct code reviews, enforce best practices, and mentor junior engineers.
• 5-8 years in data engineering or software development.
• 3+ years hands-on experience with Databricks and PySpark.
• Strong Python programming skills, including reusable libraries.
• Proficiency in SQL and cloud data warehouses like Snowflake.
• Working knowledge of CI/CD, Git, Docker/Kubernetes, automated testing.
• Strong understanding of medallion/lakehouse data architecture patterns.
• Minimum 5 years experience in Large Language Models.

Good to Have

• Experience with Airflow, Prefect, or Azure Data Factory for orchestration.
• Exposure to infrastructure-as-code tools like Terraform, CloudFormation.
• Familiarity with MLflow, Delta Live Tables, or Unity Catalog.
• Experience designing frameworks for logging, error handling, or observability.
• Knowledge of data security, access control, and compliance standards.

Job Description

Project Role : Performance Engineer

Project Role Description : Diagnose issues that an in-house performance testing team has been unable to. There are five aspects to Performance Engineering: software development lifecycle and architecture, performance testing and validation, capacity planning, application performance management and problem detection and resolution.

Must have skills : Databricks Unified Data Analytics Platform

Good to have skills : NA

Minimum 5 year(s) of experience is required

Educational Qualification : 15 years full time education

summary: The ideal candidate will have experience building: Reusable Python/PySpark frameworks for standardizing data engineering workflows Test frameworks to ensure pipeline reliability and correctness Data quality frameworks for monitoring and validation Additionally, hands-on experience with Datadog or similar observability tools is required to monitor pipeline performance, optimize resource usage, and ensure system reliability. You will work within a cross-functional team, building scalable, production-grade data pipelines on cloud platforms such as AWS, Azure, or GCP. Roles & Responsibilities:- Data Engineering & Framework Development Develop and maintain ETL/ELT pipelines in Databricks using PySpark and Python. Build reusable, modular frameworks to accelerate development and enforce standards across pipelines. Implement test frameworks for automated unit, integration, and regression testing of pipelines. Design and maintain data quality frameworks to validate ingestion, transformation, and output. Optimize Spark jobs for performance, scalability, and cost-efficiency. Collaborate with data architects to define robust data models and design patterns. Cloud & Platform Integration Integrate Databricks pipelines with cloud-native storage services (e.g., S3, ADLS, Snowflake). Implement CI/CD pipelines for Databricks notebooks and jobs using Git, Jenkins, or Azure DevOps. Ensure pipelines follow best practices for modularity, reusability, and maintainability. Monitoring, Observability & Optimization Use Datadog to monitor pipeline performance, resource utilization, and system health. Build dashboards and alerts for proactive monitoring and troubleshooting. Analyze metrics and logs to identify bottlenecks and improve reliability. Collaboration & Delivery Partner with data scientists, analysts, and business stakeholders to translate requirements into scalable solutions. Conduct code reviews, enforce best practices, and mentor junior engineers. Promote knowledge-sharing of reusable frameworks, testing practices, and data quality approaches. Professional & Technical Skills:- Bachelor’s or Master’s degree in Computer Science, Engineering, or related field. 5–8 years of experience in data engineering or software development. 3+ years hands-on experience with Databricks and PySpark. Strong Python programming skills, including writing reusable libraries and frameworks. Experience designing and implementing test frameworks for ETL/ELT pipelines. Experience building data quality frameworks for validation, monitoring, and anomaly detection. Proficiency in SQL and experience with cloud data warehouses (Snowflake, Redshift, BigQuery). Familiarity with Datadog or similar monitoring tools for metrics, dashboards, and alerts. Experience integrating Databricks with AWS, Azure, or GCP services. Working knowledge of CI/CD, Git, Docker/Kubernetes, and automated testing. Strong understanding of data architecture patterns — medallion/lakehouse architectures preferred. Nice to Have Experience with Airflow, Prefect, or Azure Data Factory for orchestration. Exposure to infrastructure-as-code tools (Terraform, CloudFormation). Familiarity with MLflow, Delta Live Tables, or Unity Catalog. Experience designing frameworks for logging, error handling, or observability. Knowledge of data security, access control, and compliance standards. Soft Skills Strong problem-solving and analytical skills. Excellent verbal and written communication. Ability to work in agile, cross-functional teams. Ownership mindset, proactive, and self-driven. Additional Information:- The candidate should have a minimum of 5 years of experience in Large Language Models. - This position is based at our Bengaluru office. - A 15 years full-time education is required.

Qualification

15 years full time education

Additional Information

Equal Employment Opportunity Statement

All employment decisions shall be made without regard to age, race, creed, color, religion, sex, national origin, ancestry, disability status, veteran status, sexual orientation, gender identity or expression, genetic information, marital status, citizenship status or any other basis as protected by federal, state, or local law.

Please read Accenture’s Recruiting and Hiring Statement for more information on how we process your data during the Recruiting and Hiring process.

Important Notice

We have been alerted to the existence of fraudulent messages asking job seekers to set up payment to cover various costs associated with establishing employment at Accenture. No one is ever required to pay for employment at Accenture. If you are contacted by someone asking for payment, please do not respond, and contact us at india.fc.check@accenture.com immediately.