Senior QA Tester - AI Observability & Monitoring

Job Title: QA Tester – AI Observability & Monitoring

Duration: 6 Months Contract with a possibility for an extension

Location: Offshore - India / 100% Remote

Rate: INR 1200 per hour

Experience Required: 8+ years in QA and at least 2 years in AI ML related projects. Should have Observability and Monitoring experience.

Role Overview

We are seeking a QA Tester specializing in AI Observability and Monitoring to support the validation and continuous monitoring of AI/ML solutions in a regulated enterprise environment.

This role will focus on ensuring that AI systems are traceable, explainable, and continuously performing as expected by validating observability frameworks, monitoring pipelines, and model performance metrics. The candidate will work closely with AI engineers, data scientists, and validation teams to ensure AI solutions meet quality, compliance, and audit readiness standards.

Key Responsibilities

• AI Observability Validation
• Validate observability instrumentation across AI systems, including:
• Input/output tracing
• Telemetry data (latency, token usage, cost, etc.)
• Ensure all observability signals are captured, linked, and auditable
• Verify traceability and explainability of model behaviour across workflows
• AI Model Monitoring & Drift Testing
• Validate monitoring frameworks for:
• Model performance (accuracy, confidence, consistency)
• Drift detection and threshold-based alerts
• Test alerting mechanisms and escalation workflows for:
• Performance degradation
• Anomalous outputs
• Support continuous monitoring validation in production environments
• AI Behavior & Functional Testing
• Design and execute test scenarios covering:
• Edge cases and ambiguous inputs
• Prompt variations and response consistency
• Bias and fairness validation
• Validate model outputs against expected results and SME benchmarks
• Perform comparative validation (AI vs. baseline/manual outputs)

Interested in remote work opportunities in QA & Testing? Discover QA & Testing Remote Jobs featuring exclusive positions from top companies that offer flexible work arrangements.

• Observability Tools & Integration Testing
• Test integration between AI applications and observability tools (e.g., Langfuse or similar platforms)
• Validate data pipelines feeding observability dashboards and KPI metrics
• Ensure end-to-end visibility across AI lifecycle (development → QA → production)
• Non-Functional & System Quality Testing
• Validate non-functional requirements including:
• Performance and latency
• Reliability and resilience
• Logging and auditability
• Ensure monitoring coverage aligns with enterprise quality and governance standards
• Audit, Compliance & Documentation
• Maintain audit-ready documentation for:
• Test cases, execution results, and validation evidence
• Ensure alignment with:
• SDLC validation processes
• AI governance and compliance requirements
• Support inspection readiness and audit responses as needed
  

Required Qualifications

• Bachelor’s degree in Computer Science, Data Science, Engineering, or related field
• 3–7 years of experience in QA / Testing / Validation
• Experience working with AI/ML systems or data-driven applications
• Exposure to monitoring systems, logging frameworks, or observability platforms
  
  

Technical Skills

• Strong understanding of:
• AI/ML concepts (LLMs, model behavior, drift, evaluation metrics)
• Experience with:
• API testing and backend validation
• SQL / data validation techniques

Browse our curated collection of remote jobs across all categories and industries, featuring positions from top companies worldwide.

• Familiarity with:
• Observability tools (e.g., Langfuse, logging/monitoring platforms)
• Test management tools (e.g., QTest, ALM tools)
  

QA & Validation Skills

• Experience designing:
• Functional and non-functional test scenarios
• Edge case and negative testing scenarios
• Understanding of:
• Test automation concepts (Python preferred)
• End-to-end validation lifecycle
  

Preferred Qualifications

• Experience in GenAI / LLM testing
• Knowledge of:
• Prompt engineering and evaluation methods
• Familiarity with:
• GxP / regulated industry environments
• AI governance, explainability, and Responsible AI frameworks
  

Tips: Provide a summary of the role, what success in the position looks like, and how this role fits into the organization overall.

Responsibilities

[Be specific when describing each of the responsibilities. Use gender-neutral, inclusive language.]

Example: Determine and develop user requirements for systems in production, to ensure maximum usability

Qualifications

[Some qualifications you may want to include are Skills, Education, Experience, or Certifications.]

Example: Excellent verbal and written communication skills

Skills: validation,logging,skills,data,ml,testing,metrics,platforms,drift,compliance