Evaluate Gemini’s ability to retrieve and use personalized context from connected Google applications. Assess AI-generated responses for accuracy, relevance, helpfulness, personalization, consistency, and unsupported assumptions. Compare responses, identify subtle strengths and weaknesses, and provide clear, structured feedback according to project guidelines. The role is remote in Canada, requires Google app connectivity, strong written English, analytical thinking, privacy compliance, and availability for at least four hours daily with PST overlap.
About Gramian
Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.
About the Role
We are seeking AI Personalization Raters based in Canada to support Gemini Personalization Evals. In this role, you will assess how effectively AI models retrieve and use relevant context from connected Google applications, including Gmail, Google Calendar, Google Drive, and Google Photos. You will evaluate accuracy, relevance, helpfulness, and personalization quality while providing structured feedback to support model improvements.
This opportunity is suitable for detail-oriented individuals with strong analytical thinking, excellent written communication, and active use of Google applications.
Engagement Details
CONTRACT: Contractor
LOCATION: Remote — Canada only
DURATION: Up to 16 weeks
Key Responsibilities
- Evaluate AI-generated responses using personalized prompts and information retrieved from connected Google applications.
- Assess whether responses are relevant, accurate, helpful, and appropriately personalized.
- Identify incorrect personalization, unsupported assumptions, irrelevant recommendations, and inconsistencies.
- Compare multiple AI-generated responses and assess their overall user experience.
- Provide clear, detailed, and well-reasoned evaluation feedback.
- Follow project guidelines to ensure consistent and high-quality ratings.
- Identify subtle strengths and weaknesses in AI-generated responses.
- Maintain data privacy and comply with project confidentiality requirements.
Requirements
- Currently based in Canada.
- Willing and able to connect relevant Google applications to Gemini, subject to required permissions and consent.
- Active use of Google Workspace or consumer Google apps, with sufficient personal data or history to support personalized retrieval evaluations.
- Strong analytical thinking and attention to detail.
- Excellent written English and ability to provide structured feedback.
- Ability to work independently in a remote environment.
- Access to a desktop or laptop with a reliable internet connection.
- Bachelor’s degree or equivalent practical experience in any field.
Similar Jobs
Cloud • Information Technology • Security • Software • Cybersecurity
Drive new business for Cloudflare’s Enterprise services across Eastern Canada. Develop territory plans, identify target accounts, build sales pipelines, present technical value propositions, negotiate contracts, achieve revenue targets, maintain strategic customer and partner relationships, and engage stakeholders from technical teams through senior executives.
Top Skills:
Cloud SolutionsCloudflareComputer NetworkingCybersecurityIaasInternet InfrastructurePaasSaaS
Artificial Intelligence • Healthtech • Machine Learning • Natural Language Processing • Biotech • Pharmaceutical
Leads health technology assessment, value, and evidence strategy for Pfizer’s genitourinary oncology portfolio. Manages HEOR, real-world evidence, economic models, global value dossiers, registries, and evidence dissemination to support reimbursement and patient access. Partners with global, regional, country, and cross-functional oncology teams, oversees vendors and project teams, and communicates findings through publications and conferences.
Aerospace • Information Technology • Software • Cybersecurity • Design • Defense • Manufacturing
Leads business systems analysis and data engineering support for military sustainment digital services. The role gathers and documents requirements, maps workflows, defines reporting and data needs, supports ETL/ELT, SQL analysis, integrations, dashboards, testing, governance, and implementation readiness. It partners with government, program, supplier, operational, product, and technical stakeholders to improve readiness and operational performance, while mentoring junior analysts and maintaining requirements, process, data, and delivery documentation.
Top Skills:
AgileAmazon RedshiftAPIsAWSAzure SynapseConfluenceData ModelingDatabricksEtl/EltGitJIRALookerMiddlewarePower BIQlikSnowflakeSplunkSQLTableau
What you need to know about the Toronto Tech Scene
Although home to some of the biggest names in tech, including Google, Microsoft and Amazon, Toronto has established itself as one of the largest startup ecosystems in the world. And with over 2,000 startups — more than 30 percent of the country's total startups — Toronto continues to attract new businesses. Be it helping entrepreneurs manage their finances, simplifying business operations by automating payroll or assisting pharmaceutical companies in launching new drugs, the city's tech scene is just getting started.
