Position Description

The Psychometrics Lead is responsible for the objective measurement of skills, knowledge, and competencies through the design, construction and validation of assessment instruments such as questionnaires, and performance exams. The Psychometrics Lead will support the design and oversight of quality assurance activities at Touchstone Institute that will be informed by evidence and research.

The Psychometrics Lead provides leadership and guidance to all program areas on design, analysis, reporting and interpretation of exams and assessments at Touchstone Institute.

Key Accountabilities

Assessment Development & Validation

• Works closely with the Senior Director, Assessment and Senior Director of Program Design & Implementation and relevant SMEs to develop and operationalize clinical and written examinations and assessments that are deemed appropriate and psychometrically defensible

• Create examination and assessment blueprints based on feedback and content provided by SMEs

• Develops and leads workshops collaborating with SMEs in exam and assessment activities

• Validates the process for developing exam questions and cases in consultation with SMEs and Lead Examiners, ensuring that all development is in line with industry standards and best practices

• Develops a standardized and consistent method for assessing and scoring candidates

• Ensures exams and assessments comply with industry standards (e.g. AERA, APA, NCME guidelines)

• Serves as a key resource to the Assessment Oversight Committee, providing technical expertise and evidence-based recommendations.

• Prepares materials, presents findings, and contributes to governance and decision-making related to assessment strategy and integrity.

Quality Assurance and Quality Improvement

• Continually oversees the review of examinations, scoring systems and methodologies to ensure ongoing psychometric integrity

• Leads the development of quality assurance standards and procedures that align with industry best practices

• Collaborates with cross-functional teams inside the organization to turn data into ideas/insights for organizational growth

• Maintains documentation/records of QA activities, audit findings, and corrective actions to support transparency and accountability

Data Analysis & Interpretation

• Analyzes assessment data to generate insights that inform program design, candidate selection, and performance evaluation.

• Analyzes assessment data to generate insights that inform program design, candidate selection, and performance evaluation.

• Prepares technical reports and presentations for internal and external stakeholders.

• Develops opportunities to integrate research and quality assurance reports to inform and support strategies and demonstrate effectiveness in measurement

• Support Building networks and accessing expert advisors

Position Requirements

Knowledge

• Superior understanding of exam theory, learning theory, and assessment methodologies

• Strong understanding of measurement theory and methods

• Knowledge of relevant health profession terminology regarding competency-based assessment

• Demonstrated expertise in examination development and test analysis

• Good judgment, critical thinking skills, and close attention to detail applied in this field

Skills

• Strong time management skills and ability to prioritize multiple assignments and work demands

• Strong analytical and report writing skills and the ability to translate statistical information for clients and regulatory bodies

• Excellent verbal and written communication skills

• Strong conceptual, analytical and problem-solving skills

• Well-developed critical thinking skills to identify key issues and propose solutions to program setbacks

• Strong interpersonal and collaboration skills to work with others to coordinate information

• Strong data visualization skills, and comfort applying them in new and innovative ways

• Strong leadership skills: experience managing a team of analyst staff is an asset

• Proficiency in the use of word processing, and presentation software (e.g. Word, Excel, MS Project and PowerPoint)

Abilities

• Ability to deal with emergent issues in a professional and timely manner

• Ability to work independently or as part of a team, with a range of clients at all organizational levels

• Demonstrated ability to process multiple types of information, perform multiple tasks simultaneously, and prioritize

Qualifications

• Master’s degree in Statistics, Biostatistics, Quantitative Psychology, Developmental Psychology and Education, or other equivalent degrees

• At least 5 years of experience in an educational or testing environment

• Related educational assessment and measurement experience as well as experience in test development and analysis

Closing date October 3, 2025

Touchstone Institute is an equal-opportunity employer that is committed to fostering a diverse work environment and exercises accessible employment practices. In accordance with the Ontario Human Rights Code, Accessibility for Ontarians with Disabilities Act (AODA) and Touchstone Institute’s policies, a request for accommodation will be accepted in all aspects of our hiring process. If you anticipate needing any type of accommodation during the recruitment process, please email hr@tsin.ca in advance of your appointment.