philosophy meta-forum
From Leiter:

The Differing "Codes" in Which Letters of Reference are Written

MOVING TO FRONT, SINCE IT'S LETTER-WRITING SEASON (originally published January 10 2010, and moved to front periodically since) A young philosopher writes: We all know how important letters of reference are and how much weight they have in the decision who to interview at the APA or at least...

Comments

Comments in green on the left are from Leiter, comments on the right are from the philosophymetaforum.

Manuel Vargas

2831 day(s) ago

This is, at best, a partial reply to some of the above questions, but folks thinking about the nature of letters of recommendation might want to look at: FRANCES TRIX and CAROLYN PSENKA "Exploring the Color of Glass: Letters of Recommendation for Female and Male Medical Faculty" Discourse Society 2003; 14; 191 DOI: 10.1177/0957926503014002277 It is focused on Medical faculty appointments, and may not have perfect relationship to philosophy letters, but the article does a nice job of presenting some of the tropes that more frequently recur in the context of letters that are judged to be stronger. Moreover, it takes a look at differences that crop up in letters written for men vs. women. If memory serves, however, it does not speak to differences across perceived color/ethnic lines.

Anon

2831 day(s) ago

Given these problems, plus the fact that relying on them essentially reinstitutes the old boy networks and the various biases inherent in them that most nominally oppose, why DO letters have so much weight? Why not abandon them altogether? What information do they provide that is both important and that cannot be found out by reading candidates' dossiers and interviewing them? (A related issue is that we applicants have no idea whether those writing for us are actually saying things that will help us. Why shouldn't we be able to put our work, teaching evals., etc. forward without having to worry about back-stabbing or lukewarm or simply not enthusiastic enough letters cutting our legs out from under us.)

Aldo Antonelli

2831 day(s) ago

I tend to find these comparisons uninformative, and never use them is the letters I write for students. Even when told that X is the top n-th student the letter writer has ever had, what am I to make of that information, with no knowledge of the contrast class? The "codes" you mention are partially useful: although it's difficult to tell whether "recommend with enthusiasm" is better than "highest recommendation" it is usually possible to tell a guarded recommendation from one that is not. But it seems to me we are missing the point: what is by far most useful in a letter is an explanation of the candidate's work, why it is important, and how it relates to other people's work in the same field. Explain the student's contributions to the field and let the readers make up their own mind.

mark lance

2831 day(s) ago

Regardless of how one puts the comparative judgments in a letter, the fact is that any rational reader will have approach these with enormous skepticism. Everyone does their best to make their students look good in letters and this runs a range from reasonable gilding to outright outrageous lies. So most everyone I know has learned that this comparative comments are just of very very limited use. What is most valuable in letters is a detailed description of what the candidate has actually done. What is new in their work, how they argue for it. If a letter doesn't say anything substantive that is exciting, I don't really care that the teacher says they are smarter than a combination of Kant and Gauss. I just figure they couldn't think of anything in the work worth describing. But if one paints a picture of a fascinating project, well there it is.

Brian

2831 day(s) ago

Certainly comparisons are no substitute for substantive discussion of the candidate's work and why it is significant, so that is not the issue. And of course if one doesn't have a handle on the comparison class, then the comparisons are useless. But if X is one of the three best students in philosophy of mind from Rutgers in the last ten years, that is informative (assuming one credit the assessment and assuming it fits with the description of the work). And if one can't evaluate the import of a comparison class consisting of other philosophers in that sub-field, how in the world is one going to evaluate the substantive description of the work and its originality? If one knows the sub-field well enough, then one can do the latter. But then one can also assess the import of a comparison with other specialists in that sub-field.

Christopher Gauker

2831 day(s) ago

I'm with Aldo and Mark on this. If the letter writer shows that the candidate's work is interesting and important, through a detailed account of it, then that renders the comparisons to others irrelevant. If the letter writer can't do that, then his or her opinions will not count for much with me. Through your question, Brian, you seem to say that one cannot understand what the letter writer says about why the work is interesting without knowing who's who in the biz. Could you please clarify? [BL reply: If you don't know how to assess a comparison class of philosophers--i.e., you don't know what it means to say that a particular philosophy of biology candidate is as good as Godfrey-Smith and Dupre--then what are the odds you can figure out whether the dissertation of the philosopher of biology candidate is an original contribution to the field?]

Chandran Kukathas

2831 day(s) ago

On a couple of occasions in the UK I have been asked by the hiring university for a letter of reference for the successful candidate, who had already accepted the job. Presumably this means that the candidate was offered the post subject to suitable endorsements from his/her referees; and that the letters served not to help the appointments committee to make its decision but to allow the university to show that it had done what was necessary to establish the bona fides of the person they were hiring. This seems like a reasonable practice. Is it?

Mark

2831 day(s) ago

I'll row against the current here a bit. I think comparisons and superlatives are often, even usually, very important. This is particularly true in cases where the hiring department doesn't have a lot of expertise in the hiring field; they look to the letter-writers for assessment abilities, especially comparative, that they don't really have. Also, it is often pretty easy for a letter writer to rattle off a description of someone's project in a way that makes it sound as though it could well be really interesting and ingenious--something that can be hard to judge in a couple of pages, especially if it's not a field close to your own. I don't need to see four descriptions of the same project; give me the comparisons and explanations as to why they compare as you say they do--the description of the work should be in service of that. The inscrutability of the "code" is inevitable, given that writers recommending non-stellar students will always look for positive-sounding stuff that isn't an outright lie. (By the way, I especially enjoy the "Now, unlike others, I am totally straightforward in my recommendations" trope.)

varol akman

2831 day(s) ago

Here's an obvious point but I think it needs to be reiterated: Letters that paint a realistic picture, that is, talk about the strengths as well as the weaknesses of the candidate, are the most useful and trusthworthy in my view. I'm sorry to say that 90 percent of the letters I receive nowadays basically say that the candidate is identical to Superman or Supergirl.

David Wallace

2831 day(s) ago

I think it misses the point slightly to ask "what do references show that interviews, and careful study of dossiers, do not"? Suppose that serious study of a candidate's writing sample (say) would put the interview panel in as good a position as the referee to assess the candidate. "Serious study" probably means at least several hours. With 100+ applicants for the job, it's impossible to give them all that level of scrutiny. (The most striking thing I learned when I moved from being an applicant to being on hiring panels was how little time any dossier will be studied for at the initial stage.) My impression is that referee reports don't play a huge role in the late stages of the process, where there are only a few candidates and they're all getting very detailed attention. It's earlier on that they get used a lot. On a separate point: I agree that giving explicit comparisons is good. But it's tricky for people like me who have only just started writing references and don't have access to a large comparison class. Possibly that's a(nother) reason that applicants should try not to ask too-junior people, I guess.

Eric Schliesser

2831 day(s) ago

Comparison classes contain entertaining and sometimes useful information, although they often reveal more about the letter writer's sense of self than the canidate. I am with Aldo, Mark, and Christopher--I find detailed information about the content of the work more useful.

mark lance

2830 day(s) ago

Brian: The issue is not with the ability to understand the reference class. I am perfectly capable of understanding what it means when someone says that a new graduate student is better than the following 3 famous people in the field. Many of my undergrads are capable of this. The problem, as many of us have said, is that the state of the letter writing art is such that I should not believe these comparisons. But if they explain the new contribution that x made to the literature, one that has not been made by those 3 people, I do believe them - or at least come a lot closer to believing, close enough to make it clear that I should read the paper carefully. (David is clearly right. The point of all this is to cut the pile down to a manageable number that your resources will allow serious study of.) [BL reply: Obviously the comparative judgments at issue are rarely of the form, "X is the next Kripke," which of course anyone can evaluate. Typically they involve comparisons with junior scholars who may be well-known within the specialty but not otherwise. Institutional affiliations may then supply some guidance as to what the comparison is supposed to mean. But the challenge for a non-specialist in evaluating a detailed explanation of what the candidate's work contributes in a specialized field can be substantial. So the question becomes whether the comparative judgments aid the interpretation of the recommendation]

mark lance

2830 day(s) ago

Just noticed the poll. It is worth noting that the 3rd and 4th answers are compatible, and I think both true. Of course comparisons CAN be helpful. If you have antecedent reason to think someone is more honest in this than most, they are. (And "helpful" is a very very weak standard. Maybe even most of them add a bit of useful information - just way way less than substantive discussion.) It seems to me that they can be seriously useful, but rarely are. Just to mention a particularly amusing case using Brian's hypothetical example - in one year I was in possession of letters for 4 candidates each claiming that the candidate was one of the 3 best phil mind students at Rutgers. Of course each letter writer may have honestly believed what they said - actually I have had contradictory letters from the same writer, but I'll eschew mentioning online anything that would suggest who that is - but it certainly makes it hard for a search committee to lend much weight to such statements.

Steven Hales

2830 day(s) ago

There are plenty of complaints about "recommendation inflation," excessive praise, etc. What no one has commented on is that we are in a Prisoner's Dilemma. The competition for jobs and grad school admissions is so fierce that everyone is compelled to give their maximum praise. Yes, it would be better if we all cooperated and wrote fair, critical judgments of our students. However: the temptation payout for defection is really high. Our students get primo jobs and grad school admittances. The sucker payout is terrible-- we can't place any of our students. Therefore we all defect and get the punishment payout, the worst collective option. Therefore it's a bit silly to just gripe about recommendation inflation, as if it a character defect, when we really face a collective action problem.

Margaret Atherton

2830 day(s) ago

However problematic the use of comparisons may be, at least they are written in straightforward English prose, whereas if you forget or never knew the damn codes, you can end up inadvertently condemning a candidate you meant to praise. So my question is, if not comparisons, then what will do the job? I am assuming the presence of discussions of the work, which are of course essential but do not in the way the codes do signal "This should be taken seriously" or "This one is not so warm as it might seem."

David Sobel

2830 day(s) ago

One serious problem with letters is that they are inflated and not uniformly so. But another problem is that we may not understand each other's code for explaining how strongly we are recommending someone. I tend to assume that "warmly" and "without reservation" are certainly less strong than "highly" or "strongly" recommended. Eager to know if others agree. Also, while it is certainly good for letters to explain the excitement of the philosophical project, the problem is that letter writers tend to see themselves as advocates for the person they are writing for and so provide mainly the most positive things they can say about a candidate. Given that, it is helpful to have some sort of summing up of the strength of one's recommendation rather than just an explanation of the positive features of the candidate.

J.C. Berendzen

2830 day(s) ago

Last year I served on a search committee for the first time, in an area fairly far from my AOS (though one in which I have some side interests, so I wasn't totally at a loss). I was really surprised to find that evaluation letters were really helpful, and that I relied on them a lot. The main parts of the letters that helped me, though, were more descriptive than evaluative. In this regard, I found letters from dissertation advisors particularly useful, because they were more apt to provide really good descriptions of the candidate's current research projects, and of their potential for further work (obviously this last point is also evaluative). These descriptions were often more helpful than those provided by the candidate because 1) I found that the letter writers were often more articulate in their descriptions than the candidates and 2) although the letter writers have an interest in the candidates being hired, they are still, I would think, a little better placed to provide a dispassionate assessment. In the end, I found this kind of thing to be more helpful than the comparative claims being discussed here (though not entirely separable from them). Of course, I am just one example, so I don't mean to claim that those comparative claims aren't useful. Rather, I just want to point out, in support of some of the claims above, that letters of reference can be really helpful even if questions of the "code" and comparative judgements are set largely to the side.

Christopher Hitchcock

2830 day(s) ago

I have a number of comments. 1. Letters of recommendation play a role primarily at two different stages in the process. (i) Making the first cut. I would say that in our searches, roughly 25% of candidates make it to the point where their written work gets read. Even weeding out 3/4 of the candidates, reading the work of job applicants is a full-time job for 5 - 6 weeks. Even at this stage, letters are only likely to make a difference in borderline cases. (E.g. if someone is a graduate of a top program, or top in their specialty area, is working squarely in the AOS in which we're searching, and has one or two publications in good places, she will make the cut. If someone is 5 years out from the Ph.D., and does not have any kind of publication record, she will not make the cut.) (ii) Once the hiring committee has made a decision, the letters may play a role in convincing others in the department, and others in the institution, to go ahead with the appointment. 2. To cut against the grain a little bit, I never find detailed descriptions of the research all that useful. Every candidate will include a statement describing her research (such as a dissertation précis). I would much rather see how the candidate herself describes her project, whether she can convey enthusiasm and a sense of the importance of the project. Sometimes, however, descriptions of the project from letter writers can convey a kind of negative information: the letter writer might reveal that she really does not understand the candidate's work, or the area in which the candidate is working. 3. I find explicit comparisons helpful. That kind of talk is less cheap. The comparisons needn't be overall comparisons -- candidate X is just as good as Y. One could say, e.g., that candidate X's paper on topic A is as good as Y's influential paper on the same topic. 4. We have done 4 searches in the past 5 years -- I don't know if that is quite typical, but certainly it is something that most of us do frequently. We start to see letters from the same people. People do earn reputations as reliable letter writers or as gross exaggerators. So the situation is not quite the Prisoner's Dilemma that Steven Hales describes. It is, at least, an *iterated* Prisoner's Dilemma. If Professor Z writes a glowing letter for candidate X, we read the work, and find it trite, ill-informed, or otherwise mediocre, we will harbor a resentment at Professor Z for wasting our time, and Professor Z's students are liable to suffer in the future. (Remember the story about crying 'wolf'?) Also, while we're at the evil smokers, we gossip with our colleagues at other schools who are doing searches. "Boy, professor Z's letter for candidate X was really over the top. But what did you really think of X's work?" So reputations do spread.

Christopher Hitchcock

2830 day(s) ago

Phrases to avoid in letters of recommendation: "I cannot recommend this candidate too highly" "I would waste no time in making him an offer" "You would be very fortunate to get this candidate to work for you" "I recommend this candidate without qualifications" "No one would be better for your department than this candidate"

Bence Nanay

2830 day(s) ago

These comments neatly demonstrate that there is no consensus on what one should be looking for in a letter. But this is a HUGE problem. If the norm is to give a comparative statement, then a letter without a comparative statement is not a strong letter. If the norm is not to give a comparative statement, then it can be a very strong letter indeed. So in order to know whether a letter is strong or not, we need to know what norms the letter-writer follows. But this is impossible unless the whole letter-writing community follows by and large the same norms.

David Chalmers

2830 day(s) ago

Comparisons are often phrased in terms of codes that are as ambiguous as noncomparative evaluations. I find the old favorite "compares favorably to" particularly unhelpful in this regard, and "comparable to" is also problematic. It's also worth noting that given the often-embraced norm that one should write as strong a letter as one can while remaining honest, writers can easily game the comparison system by making comparisons with individuals that they evaluate less highly than the rest of the profession. So the most helpful comparisons are explicit rankings within canonical comparison groups of a decent size. These rankings are also slightly less distasteful than one-on-one comparisons with named individuals. But for obvious reasons these rankings tend to be made only for the very best candidates. Still, from a broader perspective, ambiguity and variation in norms aren't entirely bad things. If everyone had the same norms and values across the board, then even given variation in departmental situations, a much smaller group of candidates would tend to receive job offers, with obvious ill effects. The variation in letter-reading norms is at least one factor in facilitating a broad distribution that gives chances to more people. One might argue that even the inflationary conventions (accentuate the positive, play down the negative, don't lie) serve this distributive goal. The conventions reduce the effects of negative information that would tend to eliminate too many candidates from too many positions too soon, and they provide the opportunity for as many candidates as possible to put their best foot forward. It's also worth noting that at least in my own experience over the last 15 years or so, inflation has stayed fairly constant and conventional. The prisoner's dilemma arms race is held in check by the reputational penalties of defecting: in precisely the cases where defecting helps, by advancing a candidate to further consideration, the letter-writer tends to be penalized by that very further consideration. Of course the system is far from perfect. There are still occasional egregious defectors, and the variation in norms tends to hurt those whose letter-writers conform to norms nearer the low end of the inflationary realm. I suppose that overall this speaks in favor of a system with inflationary norms that vary qualitatively but that don't vary too much "quantitatively".

Bence Nanay

2830 day(s) ago

Dave, are you really suggesting that the present system is good because some job seekers get better jobs than they deserve and some others get worse jobs than they deserve? This may in fact be beneficial for some departments (maybe even my own). But for those job-seekers for whom the question is not about whether they get a top 15 or a top 30 job, but whether they get a job at all, this is hardly a consolation...

Roberta Millstein

2830 day(s) ago

I'll add my voice to those who find the comparisons unhelpful and add some reasons to those already mentioned. People have said that comparisons of X to Y can be helpful for a letter reader who isn't in X's and Y's field. But if the letter reader isn't in X's and Y's field, then they may not be terribly familiar with Y's work, either -- unless Y can only be a person at the very top of his/her field, so that any philosopher would have heard about him/her. But then, very few candidates will genuinely earn such comparisons, leaving the rest uncompared (and thus seen as unworthy?) or with an inflated comparison. As a letter writer, there are problems, too. Maybe the most well-known people in your field are not the people who you think are actually the best people in your field. So then, do you make the accurate comparison or the comparison that is more likely to be successful (even though, ironically, you are in your mind downplaying the accomplishments of the subject of your letter!) As for the "codes," we could seek to standardize things. It wouldn't solve the inflation problem, but at least it would prevent letter writers from inadvertently harming candidates they meant to praise.

David Chalmers

2830 day(s) ago

Bence: "not entirely bad" is not the same as "good". I suggested that the system is better in at least one respect than a system with no variations in norms and no inflation, in that a broader distribution of candidates make it further in the process than would happen otherwise, thereby avoiding logjams of offers to a small group of candidates. This broader distribution may or may not have the additional consequence that some candidates will get better or worse jobs than they deserve, but if it does (and if the alternative system doesn't), then this will simply be one of the various respects in which the system is worse.

Anon

2828 day(s) ago

I believe that, in the UK at least, centralised funding competitions (e.g. for British Academy postdocs) demand very structured references, responding to a very specific set of questions. I'm at the wrong end of the process (as an applicant, I don't actually see the form for the report) to know whether these provide adequate scope for letter writers to express their views, but I do wonder whether a standardised form might be helpful. It could, for example, force referees to rank all candidates relative to their year group. If all referees filled in the same report form issued by the APA, it would surely make comparisons easier. Does anyone with experience of this kind of report have a view?

John Alexander

2828 day(s) ago

This may have been mentioned earlier but why not have a standardized form letter utilizing a liker scale that defines each element? There could be room for additional comments, but that would augment rather then dictate the recommendation.

Michael Kremer

2828 day(s) ago

Anon and John Alexander: NO standardized forms, please! These impose ridiculous comparisons and quantification of what is largely unquantifiable. They also provide the illusion that difficult comparisons become easier. We already chafe under these in writing letters of reference for applicants to graduate school. It is my sincere hope that admissions committees largely ignore them and read my discursive comments.

John Alexander

2827 day(s) ago

Micheal Given the wide range on how to phrase comments and various interpretations to these comments in the previous responses on what certain phrases mean it seems clear that 'discursive' comments are no easier to evaluate then what you fear would happen with a standardized form. If the APA formed a committee to reach agreement on how to structure the form and interpret that phrases, and/or attach values to the numbers, then a reasoned uniformity would be possible. Of course that assumes the we could reach such an agreement, but if we cannot then this discussion, at any level, seems moot and we will muddle on as we are now appear to be doing. Of course, coming from a business background we do tend to look for pragmatic solutions so I would also suggest that interviews are a good practice for evaluating potential grad students. After all, admission committees are deciding on future professional colleagues and 'co-workers,' universities will spend large sums on them, and expect them to produce to some standard of excellence so it might be helpful to actually talk to potential students. Can you imagine a company hiring a person without interviewing them? One question that I have is how many of you have experience with getting a grad student who did not live up to expectations? What is the percentage of those students to those who meet or exceed expectations? This seems a relevant question in so far as if the percentage is low then then is there a real problem?

Matias Vernengo

1858 day(s) ago

I now give letters with explicit comparisons (in economics not philosophy), and also do not say anything about potential for growing if for example a candidate is still young and is not yet a great teacher (but in my opinion has great potential; by the way my view is that rarely someone is a good teacher at the beginning of their careers, and I certainly was not). The reason is that people read any notion that somebody is not perfect as a critique. Being on the other side, reading a letter, I find that the only way to know for sure what a person means sometimes is to call and talk directly to them.

IU Undergrad

1858 day(s) ago

I have a question about this "notorious UK/US difference" mentioned by the author and in a comment or two. I am an American student who studied abroad in England and secured a letter-writer while I was there. What should I know, or let my writer know, about the differences between UK and US letters-writers so that the differences will not harm me in my applications to primarily American schools? Your help is greatly appreciated.


Allowed tags: 'p', 'b', 'em', 'blockquote'. URLs are automatically linkified.