AI Quality Assurance (QA) Analyst

AI Quality Assurance (QA) Analyst

100% Remote Working - USA

Eastern Time Zone Working Hours

Contract-To-Hire

**Suitable candidates will need to be legally authorized to work in the USA - H1B Visa, Sponsorship etc are not being accepted for this role***

Role Summary

The AI QA Analyst will ensure the quality, reliability, and regulatory soundness of AI-driven capabilities used to support business process automation, data validation, and intelligent decision support. This role combines standard QA responsibilities with focused testing of AI/ML models, OCR pipelines, and workflow integrations in enterprise applications.

Key Responsibilities

Core QA & Testing

• Design, document, and execute test plans, test cases, and test scripts for web applications, APIs, and batch processes supporting data intake and business rule validation.
• Perform functional, regression, integration, system, and UAT testing across multiple environments, ensuring defects are logged, prioritized, and retested to closure.
• Validate end-to-end workflows from intake (documents, portals, email) through decision-ready application views, including queue indicators and evidence/drill-down screens.
• Collaborate with product owners, business analysts, developers, data scientists, and business users to clarify requirements, reproduce issues, and refine acceptance criteria.
• Contribute to test automation (UI, API, and data-level checks), maintaining stable regression suites and monitoring results.

Interested in remote work opportunities in QA & Testing? Discover QA & Testing Remote Jobs featuring exclusive positions from top companies that offer flexible work arrangements.

AI / Data / Model-Focused QA

• Develop and execute test strategies for LLM- and ML-based features that support business rule validation, recommendation engines, and automated triage.
• Test model behavior on both structured and unstructured inputs (PDFs, scanned images, OCR output, free-text applications) to verify extraction accuracy and business guideline alignment.
• Assess AI outputs for correctness, consistency, explainability, and stability over time, with a focus on recommendations, reason codes, and evidence presented to business users.
• Partner with data and digitization teams to validate data pipelines, OCR quality, and field mappings from document repositories into downstream stores consumed by AI services.
• Track and analyze model and pipeline defects, helping define quality metrics (accuracy, precision/recall on business rule checks, OCR error rates) and acceptance thresholds before production scaling.

Workflow, Controls, and Compliance

• Verify that business rule validation triggers at the correct stage of the application process and that configuration (turning treatments on/off by product/segment) behaves as designed.
• Ensure auditability of AI decisions by checking logging of inputs, outputs, decision reasons, and user overrides for downstream reporting and governance.
• Participate in defect triage and root-cause discussions, recommending process or control enhancements to reduce recurring issues and production incidents.
• Support performance and scalability testing for AI services and OCR jobs, monitoring cost/compute-sensitive behaviors (e.g., high-volume processing).

Required Qualifications

• 3+ years of software QA or QA analyst experience testing web applications, APIs, and back-end services in an Agile environment.
• Hands-on experience writing test cases, test scripts, and defect reports in modern test and defect management tools (e.g., Jira, Azure DevOps, qTest).
• Experience validating data flows, APIs, and integrations, including basic SQL skills for test data setup and verification.

Preferred Qualifications

• Experience testing AI/ML systems (LLMs, classification models, or NLP pipelines), including model validation and data quality checks.
• Exposure to OCR or document-intelligence solutions and testing unstructured document flows (PDFs, images, scanned forms).
• Domain experience in business process automation, decision support systems, or data analytics.
• Experience designing tests for recommendation systems (e.g., scores, triage queues, eligibility filters).
• Strong analytical skills and attention to detail, with the ability to interpret guideline documents, business rules, and regulatory constraints.

Key Skills and Competencies

• Quality mindset with a strong sense of ownership for end-to-end application experiences.
• Excellent written and verbal communication, able to translate complex behaviors and AI outputs into clear defect descriptions and test evidence.
• Collaborative and proactive, comfortable working closely with business users, product managers, engineers, and data scientists.
• Comfortable in fast-paced, iterative delivery environments with evolving requirements and solution designs.