Job Description
Altoros is seeking a part-time QA Automation Engineer to define and implement comprehensive test strategies across front-end, back-end, and LLM/agent workflows. The role involves building and maintaining automation for UI/E2E, API, data, and integration tests, as well as focusing on LLM quality, evaluations, and non-functional testing. Key responsibilities include CI/CD integration, test data management, quality operations, and documentation. This remote position is based in Poland.
Good To Have:
- Experience with agent frameworks, tool calling, retrieval/RAG, or tracing.
- Familiarity with evaluation/observability tools for LLM apps (e.g., prompt testing frameworks, tracing dashboards).
- Performance testing (e.g., k6/JMeter) and basic security/safety testing practices.
Must Have:
- Own the test strategy across FE, BE, and LLM/agent workflows; define the test pyramid and coverage goals.
- Build and maintain automation for UI/E2E flows, API/contract testing, and data & integration tests.
- Design golden datasets and rubrics for LLM output evaluation.
- Set up regression/eval harnesses for prompts, tools, and agents; track drift and hallucination rates.
- Implement guardrails and adversarial/red‑team tests for safety, privacy, bias, and toxicity.
- Perform non‑functional testing to baseline and monitor performance, reliability, and resilience.
- Take CI/CD ownership: integrate tests into pipelines, gate releases, and surface quality KPIs.
- Manage test data & environments, including fixtures, synthetic data, and stable seeds.
- Conduct quality operations: triage defects, drive root‑cause analysis, and communicate proactively.
- Maintain clear test plans, playbooks, and release criteria documentation.
Description
Altoros - an established provider of software development services, is looking for a QA Automation Engineer.
Requirements
Responsibilities & Responsibilities:
It's a part-time position.
- Own the test strategy across FE, BE, and LLM/agent workflows; define the test pyramid and coverage goals.
- Build and maintain automation for:
- UI/E2E flows (critical paths, multi‑step agentic scenarios).
- API/contract testing (REST/GraphQL), schema/compatibility checks.
- Data & integration tests across services and external partners.
- LLM Quality & Evals
- Design golden datasets and rubrics for LLM output evaluation (pass/fail thresholds, human‑in‑the‑loop review).
- Set up regression/eval harnesses for prompts, tools, and agents; track drift and hallucination rates.
- Implement guardrails and adversarial/red‑team tests for safety, privacy, bias, and toxicity.
- Non‑functional testing: baseline and monitor performance, reliability, and resilience for key user journeys.
- CI/CD ownership: integrate tests into pipelines, gate releases, reduce flakiness, and surface quality KPIs (e.g., failure rate, MTTR, coverage).
- Test data & environments: manage fixtures, synthetic data, and stable seeds for deterministic runs.
- Quality operations: triage defects, drive root‑cause analysis, and keep crisp, proactive communication with PM, Design, and Engineering.
- Documentation: maintain clear test plans, playbooks, and release criteria.
Nice to have:
- Experience with agent frameworks, tool calling, retrieval/RAG, or tracing.
- Familiarity with evaluation/observability tools for LLM apps (e.g., prompt testing frameworks, tracing dashboards).
- Performance testing (e.g., k6/JMeter) and basic security/safety testing practices.
Set alerts for more jobs like QA Automation Engineer
Set alerts for new jobs by altoros
Set alerts for new Quality Assurance jobs in Poland
Set alerts for new jobs in Poland
Set alerts for Quality Assurance (Remote) jobs
