The Challenge
Undergraduate Medical Education (UME) and Graduate Medical Education (GME) programs share a common challenge in North America: Finding the best evaluation tools for interviewing and assessing applicants.
In Canada, applicants rank residency programs in their order of preference then they get interviewed, assessed, and ranked by the GME programs. Finally, they get matched with a residency program based on fit by the Canadian Resident Matching Service (CaRMS).
For UMEs, applicants need an MCAT exam score. From there, the process differs depending on where you apply but most include Multiple Mini Interviews (MMIs). MMIs put candidates through several stations, typically in a timed circuit, to obtain an aggregate score. It is designed to assess verbal and nonverbal communication skills that are difficult to measure using standardized exams or transcripts.
In terms of similarities in the GME and UME admissions process, both employ a structured interview methodology, but the processes differ greatly from there. The timeline is similar, but each school and program uses different interview types (panel, MMIs, one-on-one, round robin, etc.) and evaluates based on their own specific criteria and competencies.
In addition to the ongoing need for better evaluation tools, all GME and UME programs were faced with the additional challenge of finding innovative ways to interview when social distancing protocols were in place due to COVID-19.
Dell Medical School’s UME Program in the US and Dalhousie’s Internal Medicine GME Program in Canada turned to structured digital interviewing. Even though remote interviewing is no longer required, both programs continued to use pre-recorded video interviewing because it offers a number of specific advantages associated with interviewing and ranking.
The Solution
Pre-recorded video interviewing answers many of the interview challenges GME and UME programs face, better enabling candidate assessment.
Here are four reasons why pre-recorded interviewing is a powerful evaluation tool.
1. Pre-recorded interviewing increases the number of applicants getting an interview
Dalhousie’s Internal Medicine GME Program and Dell Medical School’s UME Program have far more applicants than they can invite to a synchronous interview, so selecting the right candidates to move forward with is critical.
Both programs had a pre-qualifying step in their process, but it didn’t effectively ensure the live interviews were being offered to the right candidates. Dalhousie’s Internal Medicine GME Program and Dell Medical School’s UME Program incorporated pre-recorded interviewing through VidCruiter to address this challenge.
In addition to increasing the number of applicants that can be interviewed, pre-recorded interviewing also removes the need to schedule interviews.
Scheduling was an added layer of complication that was part of why both programs were unable to interview more applicants. With a pre-recorded format, raters rate when it’s convenient for them. This process increases flexibility for staff while removing the stress of having to manage complex schedules.
Dalhousie’s Internal Medicine Residency Program receives 400 to 500 applications and has limited interview slots. To interview in person, they had to cut the pool of applicants by half, and because of this, candidates felt the process was unfair.
Pre-recorded interviewing streamlines the process, giving every candidate who applies an equal opportunity to interview. Dalhousie’s Internal Medicine Residency Program now interviews double the applicants it did prior to VidCruiter, and no applicant is filtered out based on what may or may not be relevant criteria.
The main goal of the admissions interview for GME and UME programs is to find out if each candidate has the competencies and meets the criteria the school is looking for. This is easier to achieve given the standardization that pre-recorded interviews offer. Each question is asked exactly the same way. This removes the risk of interviewers influencing the answer to the questions by placing undue emphasis on a particular part of the question or omitting nuanced details.
“It’s actually a much more inclusive interviewing strategy. To interview hundreds of people, you need a big army of people that show up, and you don’t really know their potential biases and can’t really control what they’re doing. By removing the interviewers, and providing training to raters, you can ensure there are no unfair questions.”
Dr. Ian Epstein
Associate Professor, Past Internal Medicine Program Director, and Assistant Dean of Professional Affairs,
Dalhousie Medical School
Dell Medical School’s UME Program found that less than 10% of all applicants earned an interview, and a significant number of those people would fail the 60-point interview evaluation. This meant these opportunities could’ve been given to other candidates who were a better fit.
Because of the volume of applicants and the department’s limited interview capacity, they were using a written application as a form of pre-screening before the interview stage. To identify these candidates earlier, Dell Medical School’s UME Program used pre-recorded interviewing in addition to the written application. The program found that there was a strong correlation between how the applicants performed on the pre-recorded interview and how they performed on the in-person MMI (prior to COVID-19).
With VidCruiter, both programs now have confidence that the candidates they are advancing underwent a consistent evaluation process.
2. Pre-recorded interviewing enables a more equitable process
An equitable process gives every candidate an equal opportunity to succeed, eliminating any barriers they may face. Pre-recorded video interviewing allowed Dalhousie’s Internal Medicine Residency Program and Dell Medical School’s UME Program to increase standardization and remove any obstacles to interviewing based on resources, geography, or competing interview offers, making their processes more equitable.
“If you ask a candidate to record their personal statement and they are of means, maybe they have a very nice studio they can do that in and have somebody edit it. If you don’t have a lot of means, then you don’t really have those resources. In some ways, pre-recorded interviewing levels the playing field.”
Dr. Joel Daboub
Assistant Dean of Admissions and Assistant Professor,
Dell Medical School The University of Texas at Austin
Dalhousie’s Internal Medicine Residency Program and Dell Medical School’s UME Program use structured interviewing, but even within a structured process, there’s still room for interviewer variation, in addition to rating deviation.
In a structured pre-recorded video interview, everyone has the same experience and opportunity, which is the fairest way to evaluate applicants. Digitizing the workflow also allows more people to rate a single candidate which can help incorporate diverse opinions and minimize bias.
A pre-recorded interview process can deploy a variety of accessibility accommodations to ensure it’s as fair and inclusive as any other interview format. Dell Medical School’s UME Program found that pre-recorded interviewing delivers the same consistent experience to each candidate and even “reduces a little bit of stress and also ensures the student’s answer is the only thing being evaluated,” says Daboub.
Pre-recorded interviewing is also more equitable because it allows applicants to complete the interview on their own time. By removing the need for scheduling in the interview process and the need for candidates to travel to meet in person, pre-recorded interviewing increases flexibility significantly and reduces participation costs.
“Candidates have overwhelmingly said that they prefer not to have to spend the expense of travel. They like having the ability to interview at more schools and not having to schedule weeks in advance to do their interviews.”
Dr. Joel Daboub
Assistant Dean of Admissions and Assistant Professor,
Dell Medical School The University of Texas at Austin
3. Pre-recorded interviewing increases the quality of the evaluations
Dell Medical School used the greater volume and depth of candidate data to increase the quality of UME admissions evaluations. The results showed, with significant validity, that every aspect of Dell’s UME admissions interview process, including the pre-recorded interviewing steps, measured distinct characteristics. This variety of data collected was used to enrich its committee’s evaluation and decision-making process.
In addition to the quantity and quality of data points improving with pre-recorded interviewing, the quality of the evaluation also improved. Dell Medical School’s UME Program found that pre-recorded interviewing was also a fairer process for screening with equal fidelity to live video interviews.
“The other question that we had when we switched to virtual was: are we getting the same fidelity in our evaluations as we did when we ran this process in person? The scoring of the asynchronous MMIs showed similar distribution to the in-person MMI interviews. This suggests that the asynchronous MMIs are getting us the same evaluation data with the added benefit of the flexibility that pre-recorded interviews provide.”
Dr. Joel Daboub
Assistant Dean of Admissions and Assistant Professor,
Dell Medical School The University of Texas at Austin
Pre-recorded video interviewing also produces a record that can be reviewed by as many evaluators as necessary, which isn’t possible when candidates are evaluated in person. This increases transparency in the interview process to help ensure consistency and compliance.
Dalhousie’s Internal Medicine Residency Program had a single interviewer conducting 200 interviews a year out of necessity until it brought in an assortment of raters to share the interviewing task.
Instead of training people on how to conduct a structured interview, now Dalhousie’s Internal Medicine Residency Program can train evaluators on how to follow the structured rating process. This program now finds that using multiple raters is smoother and more efficient.
“The fact that it’s asynchronous, from a practical point of view, does allow you to do some rater training, and makes it easier to have re-reviews if there are questions or concerns. I like the standardization of our question set in our rubric, that’s much better…you’re always at the whims of the individual running it if it’s synchronous. They can go off script, even if you give them a script.”
Dr. Ian Epstein
Associate Professor, Past Internal Medicine Program
Director, and Assistant Dean of Professional Affairs,
Dalhousie Medical School
Part of evaluating competencies is being able to distinguish authenticity. If an answer feels authentic, there’s a higher degree of certainty in the candidate’s response.
With written assessments, there is more of an opportunity for someone to have external assistance, from humans or AI, and the benefit of unrestricted time to create a response. Compared to a candidate statement, which is a written narrative of the person’s accomplishments, a pre-recorded video interview introduces spontaneity.
The challenge that is often raised with pre-recorded interviewing is that candidates can’t engage or interact with an interviewer. Dalhousie’s Internal Medicine Residency Program found that by not giving the questions in advance and providing a 45-second opportunity to prepare before answering, the experience is consistent with the previous live interview process. Raters also focus their entire attention on the rating and not the rapport or banter between questions, increasing the effectiveness of the evaluation.
“Having done thousands of live interviews, and having watched hundreds of pre-recorded interviews, I can tell you that even in a standardized synchronous interview, most of the answers are going to seem rehearsed. These candidates can kind of guess what they’ll be asked, especially if they’re interviewing multiple places. The candidates who have the most authentic, least rehearsed-sounding answers will do better in either format.”
Dr. Ian Epstein
Associate Professor, Past Internal Medicine Program
Director, and Assistant Dean of Professional Affairs,
Dalhousie Medical School
Dell Medical’s UME Program also found that by time-restricting the responses, the answers themselves were much more authentic and less edited or rehearsed, much like they would be in an in-person interview, contributing to a quality increase in the candidate responses.
4. Pre-recorded interviewing helps fill spots in residency programs
Addressing the crucial problem of the doctor shortage starts with filling residency programs. Instead of interviewing 100 candidates, pre-recorded interviewing allows programs to interview hundreds of applicants. This creates more opportunities to fill in the first round and increases the likelihood of filling in the second round by removing barriers like time, access cost, and cross-cultural communication.
However, when spots are unfilled in Dalhousie’s Internal Medicine Residency Program, they get offered to a second iteration of global CaRMS applicants.
The pre-recorded interview format eliminates scheduling and geographical barriers and also makes it easy to have interview questions reviewed by an industrial and organizational (I/O) psychologist. They can address the cultural sensitivity of the questions and make the process more inclusive, especially for the secondary iteration of applicants.
Dalhousie’s Internal Medicine Residency Program had success increasing inclusivity in their pre-recorded interviewing process by bringing in raters that are either specialized or have experience rating from a position of cultural competency. The pre-recorded format allows for more opportunities to find a match given the challenges when interviewing international candidates.
“The risk of an unfilled spot, vs. someone who doesn’t complete the program, is worse in my opinion. You’re losing the opportunity to create a doctor who will serve the Canadian public, on the off chance they might not be successful. Most candidates can be successful if they’re supported by the proper training.”
Dr. Ian Epstein
Associate Professor, Past Internal Medicine Program
Director, and Assistant Dean of Professional Affairs,
Dalhousie Medical School
The Results
Dell Medical School’s UME Program found that adding pre-recorded interviewing in two steps of its interview process helped it assess candidates for core competencies and select those that were the right fit more often.
Dell Medical’s UME Program designed its MMI on VidCruiter to identify certain attributes that are relevant to its mission and built a rubric that maps back to each one of those relevant attributes. It was very important that the technology that delivered the evaluation didn’t affect the ability to evaluate the candidates.
“The asynchronous video interview platform is just delivering that in the same way that if we had it here in-person, we would deliver it as a scenario on a door that they would read and then walk in and then begin to answer the question,” says Daboub. “What we saw there is that it did not impact our ability to evaluate the candidate in the MMI.”
In implementing pre-recorded interviewing, Dalhousie’s Internal Medicine Residency Program found that its rating rubric became much more standardized and the scores they received were more reliable. After it started using VidCruiter for admissions, the quality of the interview was as good, if not better than the in-person due to an increase in standardization. Additionally, the increased candidate pool provides a greater opportunity to evaluate candidates from diverse backgrounds.
Dalhousie’s Internal Medicine Residency Program found that pre-recorded interviewing supported the staff by offering greater flexibility. Also, the candidates they’ve matched have been “excellent.”
“We’ve had really good results…the cohorts of residents that we’ve had in the VidCruiter years are the same or better than the ones we had in the old system,” says Dr. Epstein.
Using pre-recorded interviewing allows UME and GME Programs to collect a wider variety of quality data points throughout the admissions process that come together to support decision making and outcomes. If a decision was ever contested or called into question, having the interviews and ratings recorded would be key for compliance, defensibility, and the preservation of candidate rights.
Delivering a structured interview process via a pre-recorded interview is more equitable and inclusive of all candidates, and it enables medical schools to select the candidates that best fit their programs. While some candidates and interviewers miss the sense of personality that came with an in-person process, pre-recorded interviewing allows evaluators to focus on the assessment criteria. This is what matters most.
Grab the Free eBook.