Does the experience of the writer affect the evaluative components on the standardized letter of recommendation in emergency medicine?

Background The Standardized Letter of Recommendation (SLOR) was developed in an attempt to standardize the evaluation of applicants to an emergency medicine (EM) residency. Objective Our aim was to determine whether the Global Assessment Score (GAS) and Likelihood of Matching Assessment (LOMA) of the SLOR for applicants applying to an EM residency are affected by the experience of the letter writer. We describe the distribution of GAS and LOMA grades and compare the GAS and LOMA scores to length of time an applicant knew the letter writer and number of EM rotations. Methods We conducted a retrospective review of all SLORs written for all applicants applying to three EM residency programs for the 2012 match. Median number of letters written the previous year were compared across the four GAS and LOMA scores using an equality of medians test and test for trend to see if higher scores on the GAS and LOMA were associated with less experienced letter writers. Distributions of the scores were determined and length of time a letter writer knew an applicant and number of EM rotations were compared with GAS and LOMA scores. Results There were 917 applicants representing 27.6% of the total applicant pool for the 2012 United States EM residency match and 1253 SLORs for GAS and 1246 for LOMA were analyzed. The highest scores on the GAS and LOMA were associated with the lowest median number of letters written the previous year (equality of medians test across groups, p < 0.001; test for trend, p < 0.001). Less than 3% received the lowest score for GAS and LOMA. Among letter writers that knew an applicant for more than 1 year, 45.3% gave a GAS score of "Outstanding" and 53.4% gave a LOMA of "Very Competitive" compared with 31.7% and 39.6%, respectively, if the letter writer knew them 1 year or less (p = 0.002; p = 0.005). Number of EM rotations was not associated with GAS and LOMA scores. Conclusions SLORs written by less experienced letter writers were more likely to have a GAS of "Outstanding" (p < 0.001) and a LOMA of "Very Competitive" (p < 0.001) than more experienced letter writers. The overall distribution of GAS and LOMA was heavily weighted to the highest scores. The length of time a letter writer knew an applicant was significantly associated with GAS and LOMA scores.