Validate large-scale ETL and data pipelines using Databricks and PySpark, perform data quality checks and reconciliation, and develop automated QA coverage. The role also uses GitHub Copilot, GitHub Actions, AI agents, prompt engineering, and AI-driven knowledge bases to improve testing, documentation, and development efficiency. Responsibilities include test planning, defect management, regression testing, anomaly detection, and collaboration across business, operations, and application development teams.
Role Overview
We are looking for an ETL QA Engineer with strong experience in
Databricks and PySpark, combined with hands-on exposure to AI-assisted
development and prompt engineering. The ideal candidate will be responsible for
validating large-scale data pipelines, ensuring data quality, and leveraging
tools like GitHub Copilot, GitHub Actions/Runners, and AI agents to improve
testing efficiency and coverage. You will also contribute to building and
validating AI-powered knowledge bases.
Required Skills & Experience
- Core
Technical Skills
- Strong hands-on experience in Databricks (notebooks, jobs, clusters, Delta Lake, SQL).
- Expert-level PySpark skills, including transformations, actions, window functions, and
optimization techniques.
- Solid understanding of ETL/ELT
concepts, data warehousing, and data modeling (star/snowflake
schemas, dimension/fact tables).
- Proven experience in data
validation, data quality checks, and reconciliation techniques.
- Practical experience with GitHub, GitHub Copilot, and GitHub Actions/Runners for CI/CD
pipelines.
- AI &
Prompt Engineering
- Experience crafting effective
prompts for large language models (LLMs) for coding, testing, and
documentation.
- Exposure to agent-based
architectures or tools (e.g., workflow/agent frameworks that
orchestrate multi-step AI tasks).
- Familiarity with AI-based
tools for code analysis, test generation, or knowledge management.
- Knowledge
Base & Documentation
- Experience building AI-driven
knowledge bases or documentation systems (e.g., using vector search,
embeddings, or LLM-based retrieval).
- Strong documentation skills,
with the ability to translate complex data/QA concepts into clear
knowledge articles.
- Testing & QA Practices
- Strong understanding of QA
methodologies: test planning, test design, defect management, and
regression testing.
- Experience with automated
testing frameworks (Python-based testing libraries such as pytest, unittest, or similar).
- Familiarity with data quality
tools/practices (e.g., validation rules, thresholds, anomaly detection).
Key Attributes
- Strong analytical and
problem-solving skills with a detail-oriented mindset.
- Passion for data quality,
automation, and continuous improvement.
- Ability to work in an agile,
fast-paced environment and collaborate across multiple teams like Business
and operations team and AD team.
- Curiosity and openness to adopting new AI tools and
practices for QA.
Kumaran Systems Toronto, Ontario, CAN Office
703 Evans Avenue, Suite 605 , Toronto, ON , Canada, M9C 5E9
Similar Jobs
Blockchain • eCommerce • Fintech • Payments • Software • Financial Services • Cryptocurrency
Own end-to-end analytics engineering for AI and developer experience data. Instrument telemetry, build governed production pipelines, define trusted metrics, and create executive-facing dashboards and analyses. Partner with data engineering, AI, and data science teams to improve data quality and connect raw signals to decision-making. Present clear recommendations on AI investment, developer productivity, and ROI while using AI coding agents to build and maintain pipelines, dashboards, and analyses.
Top Skills:
Ai Coding AgentsCi/CdGitLlm-Based Classification ToolingMetrics StorePythonReactSnowflakeSQLTypescript
45 Minutes Ago
Blockchain • eCommerce • Fintech • Payments • Software • Financial Services • Cryptocurrency
Graduate research interns will own end-to-end projects to build customer world models and proactive intelligence: developing representation learning, foundation model, RL/agentic methods, evaluation frameworks, and shipping research into production to improve customer decision-making.
Top Skills:
Deep LearningFoundation ModelsJaxOptimizationPythonPyTorchRepresentation Learning
Blockchain • eCommerce • Fintech • Payments • Software • Financial Services • Cryptocurrency
Build and deploy agentic and generative AI systems that automate risk and compliance investigations. Own full-stack ML systems, including data integrations, prompt tuning, orchestration, evaluation, productionization, monitoring, and transformer-based modeling across transaction, graph, text, and profile data. Collaborate with product, engineering, platform, and operations teams while leading strategic roadmaps and delivering scalable production systems.
Top Skills:
Agentic AiAmazon Web Services (Aws)Apache AirflowGenerative AiGoogle Cloud Platform (Gcp)Large Language Models (Llms)Natural Language Processing (Nlp)PythonPyTorchSnowflakeTransformer Models
What you need to know about the Toronto Tech Scene
Although home to some of the biggest names in tech, including Google, Microsoft and Amazon, Toronto has established itself as one of the largest startup ecosystems in the world. And with over 2,000 startups — more than 30 percent of the country's total startups — Toronto continues to attract new businesses. Be it helping entrepreneurs manage their finances, simplifying business operations by automating payroll or assisting pharmaceutical companies in launching new drugs, the city's tech scene is just getting started.
