Design and maintain automation frameworks to test AI/LLM-driven features. Develop test strategies for non-deterministic outputs, run functional/regression/performance/API tests, track AI defects, integrate CI/CD pipelines, and collaborate with engineering, product, and UX teams to ensure AI quality and responsible AI practices.
At OnePlan, we specialize in creating AI-enabled solutions that make strategic portfolio, financial, resource, and work management seamless. We help businesses bridge the gap between strategy and execution by offering solutions that boost business agility, streamline project management, and optimize resources.
What Makes us Unique?
What truly makes OnePlan stand out is our commitment to delivering powerful solutions and fostering a culture of collaboration. We combine robust analytics with a platform that integrates seamlessly into the tools businesses already know and trust. Our high-trust, team-focused environment allows us to innovate quickly and deliver solutions that drive meaningful results for our clients. We're passionate about exceeding expectations, working together to empower organizations to succeed in a rapidly changing business landscape.
About the Role
As a QA Automation Engineer - AI at OnePlan, you will play a critical role in ensuring the quality, reliability, and performance of our AI-powered features and capabilities. This role goes beyond writing test scripts. It’s about pioneering quality assurance practices for intelligent systems, defining how we validate non-deterministic AI outputs, and shaping the future of our product through rigorous automation and deep technical collaboration. You will work closely with AI engineers, product managers, and UX designers in a dynamic, agile environment to design, build, and scale automation frameworks that keep pace with our rapidly evolving AI capabilities. This is a deeply AI-focused role: the majority of your work will center on testing, evaluating, and improving AI systems, not general software, and requires both a quality engineering mindset and a genuine curiosity about how large language models and intelligent features behave in production.
About the Role
As a QA Automation Engineer - AI at OnePlan, you will play a critical role in ensuring the quality, reliability, and performance of our AI-powered features and capabilities. This role goes beyond writing test scripts. It’s about pioneering quality assurance practices for intelligent systems, defining how we validate non-deterministic AI outputs, and shaping the future of our product through rigorous automation and deep technical collaboration. You will work closely with AI engineers, product managers, and UX designers in a dynamic, agile environment to design, build, and scale automation frameworks that keep pace with our rapidly evolving AI capabilities. This is a deeply AI-focused role: the majority of your work will center on testing, evaluating, and improving AI systems, not general software, and requires both a quality engineering mindset and a genuine curiosity about how large language models and intelligent features behave in production.
What You’ll Do at OnePlan
- Design, build, and maintain robust automation frameworks specifically tailored for testing AI features, including LLM-powered outputs, recommendation engines, and intelligent workflows.
- Develop and execute test strategies for validating AI model outputs, including accuracy, relevance, consistency, and bias evaluation across diverse input scenarios.
- Perform functional, regression, performance, and scalability testing on AI-driven features, including API-level testing of AI service integrations and end-to-end automated test suites.
- Collaborate with AI engineers, data scientists, product managers, and UX designers to define acceptance criteria and quality standards for AI features throughout the development lifecycle.
- Identify, document, and track defects in AI behavior, including edge cases, hallucinations, and unexpected model responses through resolution using appropriate tools and workflows.
- Champion quality and responsible AI practices throughout the development lifecycle, contributing to prompt engineering reviews, model evaluation rubrics, and AI safety checklists.
- Build and maintain CI/CD-integrated automation pipelines that continuously validate AI feature quality, using tools such as Playwright, Claude, or equivalent frameworks.
- Participate in team stand-ups, sprint planning, retrospectives, and continuous improvement activities, bringing an AI-quality-first mindset to every discussion and helping establish best practices for testing intelligent systems.
Our Ideal Fit
- 2-4 years of experience in QA automation engineering, with demonstrated hands-on work building and maintaining automated test frameworks.
- Proficiency with automation tools and frameworks (e.g., Playwright, Claude, or similar) and API testing tools (e.g., Postman, REST-assured).
- Demonstrated experience testing AI/ML-integrated features or services, with a solid understanding of the unique challenges of validating non-deterministic outputs, including LLM responses, probabilistic recommendations, and generative content.
- Bachelor’s degree in Computer Science, Engineering, or a related technical field, or equivalent practical experience.
- Strong analytical and critical-thinking skills, with the ability to design test cases for complex, probabilistic AI behaviors.
- Excellent verbal and written communication skills, with the ability to clearly articulate AI quality issues and their business impact to both technical and non-technical stakeholders.
Preferred Qualifications
- Hands-on experience with LLM evaluation techniques, including prompt testing, output scoring, and benchmarking AI model responses against defined quality criteria.
- Familiarity with Microsoft Azure AI services, Azure OpenAI, and cloud-based AI infrastructure (e.g., Azure Machine Learning, Cognitive Services).
- Experience with data-driven testing approaches, including generating and managing synthetic test datasets for training and validating AI model behavior at scale.
- Understanding of responsible AI principles, including fairness, transparency, and explainability, and experience incorporating these into QA testing practices.
- Proficiency JavaScript for scripting test utilities, data processing pipelines, and custom AI evaluation harnesses.
- Background in integrating automated QA into CI/CD pipelines (e.g., GitHub Actions, Azure DevOps) with experience in shift-left testing and continuous quality monitoring for AI-driven products.
More Reasons Why You Should Apply!
- We’re a remote-first company with team members across the USA, Canada, UK, and India!
- OnePlan has been recognized as the Global Microsoft Partner of the Year in Project Portfolio Management in 2019, 2020, 2021, 2022 and 2023.
- We’ve been named a "Strong Performer" in the latest Forrester Strategic Portfolio Management WAVE report.
- We offer comprehensive health, dental, and vision benefits, with additional insurance options.
- Employer RRSP and 401K matching programs.
- A fun, collaborative, and diverse environment with regular health and team challenges to keep things light and enjoyable!
At OnePlan, we are committed to equal employment opportunities regardless of race, color, ancestry, religion, sex, national origin, sexual orientation, age, citizenship, marital status, disability, gender identity or expression, or veteran status. We are proud to be an equal-opportunity workplace.
Upon receipt of an offer letter, candidates will be subject to a standard background check process.
Disclaimer: We’ll only contact candidates who have applied directly through our official channels. Any communication about job offers will always come from an email address linked to OnePlan Solutions, and we’ll follow our standard hiring process every time. You’ll never be asked for money or personal information during the interview process. If something feels off, don’t hesitate to reach out to us to confirm.
Ready to Apply?
Check out what it’s like to work at OnePlan and learn more about us at https://oneplan.ai/
Disclaimer: We’ll only contact candidates who have applied directly through our official channels. Any communication about job offers will always come from an email address linked to OnePlan Solutions, and we’ll follow our standard hiring process every time. You’ll never be asked for money or personal information during the interview process. If something feels off, don’t hesitate to reach out to us to confirm.
Ready to Apply?
Check out what it’s like to work at OnePlan and learn more about us at https://oneplan.ai/
About
OnePlan offers a strategic portfolio, financial and resource management solution that fits the needs of every PMO. Optimize your portfolio, financials and resources enterprise wide. Connect OnePlan with Microsoft Project, Office 365 Planner, Azure DevOps, Jira, and more for a complete view into all work across the enterprise.
Similar Jobs
.png)
Big Data • Cloud • Digital Media • Machine Learning • Mobile • Software • Industrial
Design, build, and maintain automated test suites, evaluation scenarios, and CI/CD quality gates for AI-powered applications. Test APIs, validate ML model outputs, create measurable evaluation rubrics, mentor teams, and contribute to shared QA frameworks and codebases.
Top Skills:
Ai-Powered Testing ToolsAirflowAWSAzureCadCometGithub ActionsGoogle Cloud PlatformGreat ExpectationsJenkinsMetaflowMlflowPythonPyTorchTensorFlowWeights & Biases
Fintech • Financial Services
The Staff Software Engineer - Applied AI will develop and integrate AI-powered systems and tools, driving innovation through experimentation and collaboration with cross-functional teams to deliver impactful solutions.
Top Skills:
AWSAws BedrockClaudeGoHugging FaceLangchainOpenaiPythonRubySnowflake
Fintech • Financial Services
The Group Product Manager will lead the digital lending strategy, optimize customer experiences, integrate AI, govern data analytics, and collaborate cross-functionally.
Top Skills:
AIAmplitudeLookerMixpanelSalesforceSQLTableau
What you need to know about the Montreal Tech Scene
With roots dating back to 1642, Montreal is often recognized for its French-inspired architecture and cobblestone streets lined with traditional shops and cafés. But what truly sets the city apart is how it blends its rich tradition with a modern edge, reflected in its evolving skyline and fast-growing tech industry. According to economic promotion agency Montréal International, the city ranks among the top in North America to invest in artificial intelligence, making it le spot idéal for job seekers who want the best of both worlds.
Key Facts About Montreal Tech
- Number of Tech Workers: 255,000+ (2024, Tourisme Montréal)
- Major Tech Employers: SAP, Google, Microsoft, Cisco
- Key Industries: Artificial intelligence, machine learning, cybersecurity, cloud computing, web development
- Funding Landscape: $1.47 billion in venture capital funding in 2024 (BetaKit)
- Notable Investors: CIBC Innovation Banking, BDC Capital, Investissement Québec, Fonds de solidarité FTQ
- Research Centers and Universities: McGill University, Université de Montréal, Concordia University, Mila Quebec, ÉTS Montréal
