AI Summary
We are seeking a Data Quality Tester to ensure data integrity in a modern Databricks environment. The ideal candidate will design and execute automated checks on incoming data streams. Strong proficiency in Python, PySpark, and SQL is required.
Key Highlights
Automated data and integration testing
End-to-end pipeline verification
Data validation and reporting
Key Responsibilities
Develop and run automated data and integration tests within Databricks using Python, PySpark, and SQL.
Conduct end-to-end testing of the data journey—from SCADA through AMQP and into Databricks.
Verify complex transformation logic and ensure data compliance with predefined business rules.
Technical Skills Required
Benefits & Perks
100% remote work
6-month contract with extension possibility
Nice to Have
Previous experience with SCADA systems or the Energy/Renewables sector
Job Description
co.brick talents — powered by AI, powered by people.
Our client, a major player in the Energy industry, is integrating a second-layer SCADA system for renewable assets into a modern Databricks environment. We need a precision-focused Data Quality Tester to act as the gatekeeper of data integrity. You will design and execute automated checks on incoming data streams, ensuring that the transformation logic is sound and the business rules are strictly followed before the data hits the lake.
Details
- Start Date: ASAP
- Duration: 6 months (with a high probability of extension).
- Location: 100% Remote.
- Language: English.
Interested in remote work opportunities in QA & Testing? Discover QA & Testing Remote Jobs featuring exclusive positions from top companies that offer flexible work arrangements.
- Automated Testing: Develop and run automated data and integration tests within Databricks using Python, PySpark, and SQL.
- E2E Pipeline Verification: Conduct end-to-end testing of the data journey—from SCADA through AMQP and into Databricks.
- Data Validation: Verify complex transformation logic and ensure data compliance with predefined business rules.
- Reporting & Architecture: Use Azure services (Data Lake, Blob Storage, SQL) to capture, store, and prepare test results for analysis.
- CI/CD Integration: Work with DevOps teams to bake automated tests into the Azure DevOps pipelines for continuous validation.
- Collaboration: Act as the bridge between Data Engineers, SCADA specialists, and Cloud Architects to ensure full traceability and documentation.
Browse our curated collection of remote jobs across all categories and industries, featuring positions from top companies worldwide.
- Technical Stack: Strong proficiency in Python, PySpark, and SQL.
- Platform: Hands-on experience with Databricks.
- Azure Ecosystem: Familiarity with Azure services for data storage and result preparation.
- DevOps: Proven experience with Azure DevOps for test management and CI/CD integration.
- Domain Knowledge: Previous experience with SCADA systems or the Energy/Renewables sector.
Similar Jobs
Explore other opportunities that match your interests
Senior QA Engineer
••••••
••••••
••••••
Job Type
••••••
Experience Level
••••••
RTB House
Poland
Visa Sponsorship
Relocation
Remote
Job Type
Full-time
Experience Level
Mid-Senior level
itrex group
Poland
Visa Sponsorship
Relocation
Remote
Job Type
Full-time
Experience Level
Mid-Senior level
act digital
Poland