"...of course the student’s “reaction paper” was appropriate. Anything would have been appropriate. No brain work was required, nobody seems to have noticed." Love this take! I haven't seen anyone else make this argument about the controversy. It was bad pedagogy and a bad essay. Bad all around.
_X_ It offers students a "wide variety of options" for how to complete the assignment
_X_ The course is online, which means it's cheaper for admin and more convenient for students
_X_ Standards for content are lower and therefore more equitable
_X_ Word counts for such an assignment meet some arbitrary Gen Ed standard for "rigor"
_X_ The reading covers some kind of gender/race/colonial/disability aspect (often still required)
_X_ The students' own reactions and opinion must be treated as more important than any "objective" critique
This is exactly the kind of assignment that's applauded by Centers for Teaching and Learning, hall monitor faculty, and Education School pedagogy "experts."
Your post resonated with me because it provoked a long time annoyance I had with social sciences. It seems that undergraduate social science education is built upon students learning the theories, with the expectation that they will apply the theory to real life (work, politics, media) or build new theories upon it, while there is never enough evidence behind the theories themselves in the first place. Statistics based research suffers from many limitations such as unrepresentative sample, unrealistic model, statistical strength, confounding factors, and small effect size even when statistically significant. It seems to me that any conclusion must be drawn from a large body of literature, and even in that case, the conclusion will always be nuanced and noisy. In assignments like this, the student is encouraged to take the theory as given and freely extrapolate from it, ignorant of the uncertain nature of the theories themselves.
This is exactly what goes on in much of the humanities, at least those disciplines that have been influenced by sociology and/or critical theory. The most intellectually demanding task a student is likely to be assigned is to explicate and apply a theory to 'critique' or comment on something (e.g. a 'cultural object' or text), without ever critiquing the theory itself.
Philosophy has largely avoided the full-scale adoption of this kind of capital-t "Theory", although the problem here is that even if it's not in what they teach - and even though some of what is taught directly goes against or undermines it - professors themselves are often sympathetic to "woke" positions, wanting to be good (i.e., left-wing) people, and so don't stand up to policies like discrimination in hiring (e.g. "diversity" statements), etc. Because of this, most philosophy departments after 2020 will have one or two radicals embedded in them, with the rest of the faculty not being inclined to stand up to their radical ideas, so this might not last for much longer.
Classics and some history departments are likely also to be stand-outs, or at least won't have been quite as affected (infected?) as literature, sociology, or any of the "X studies" departments.
Since retiring, I have lost hope that Psychology could be taken as a credible degree with the lack of scientific rigor and absence of bare minimum education in statistics and research methodology. Maybe, if professionals consistently call out the poor quality of current education received, Psychology may move a bit closer to the science to which it once aspired. All goes back to the education/training received.
possibly. but consider the gender ideology that is pervasive in Psychology that has encouraged a lack of attention to research. Yes, sociology is a serious problem, but Dissonance theory really is replicable, yet never taken seriously in the present.
As Hollis Robbins implies, a paper enters the literature by being read, gets read by being assigned, gets assigned by being cited, and the loop never returns to the foundational question: was the original finding actually robust?
When the same readings, interpretations and citation clusters also support the intellectual authority of the course convener, pedagogy risks becoming institutional self-confirmation. The canon is taught, cited, examined and renewed by the very structure that benefits from its survival.
That is not peer review. It is citation replication.
Or, to paraphrase William Shakespeare through the lens of Jean Baudrillard:
“It is a tale told by an idiot, full of sound and fury, signifying simulacra.”
This problem isn't new. When I was teaching a humanities elective for non-arts majors around 18 years ago, I happened to see an assignment that one of the other lecturers had given her class, which was an elective on the figure of the vampire in literature and film (already not a promising sign of real rigor, though the topic could be handled properly). The assignment was literally just asking the student to find a website on vampires, say what the website's take was on whether vampires were real or not, and they say what they thought (about the reality of vampires, I assume, though the wording left this ambiguous). While this was a minor assignment, it was still worth something like 15% of the course grade.
This might not have been as bad as the final exam for a remedial course on the mechanics of writing, a copy of which I found left in the departmental photocopier. Every question - on the final exam for the writing course - was multiple choice. (This was also around 2008-ish.)
But, after reading this, I am pleased that during my daughter's college work, she once called me to discuss one of her psych course assignments. The question posed was to assess the methodology of the study - which was as poor as the one cited in this post.
As a researcher trained in the prior generation, I suggest that the skill to assess the credibility and validity of a study was more critical than the study itself. I had a good talk with her and her team; they had capability found the many egregious study design and data interpretation errors.
Anyway... glad my daughter had a professor that still promoted critical thinking...
“Everyone assumes that a psychology BA from a public flagship is evidence that the holder has been trained in the discipline.” Exactly! Degrees do not always equal full competency in the area.
The biggest problem with this assignment in the current era and configuration (large online class) is that the majority will have ChatGPT do it for them. Even if the assignment isn't perfect, asking students to read an empirical article and think about it (make connections and applications) is a reasonable learning activity for this level. Presumably, this is in addition to the course texts and lectures that provide the overview of scientific knowledge -- this is aimed at developing some analytical skills. At the level the course is pitched at, that's probably all one can do, but at least, in the old days, it would get students to actually read a research article. And, they're not going to assume this is state of the art -- they're all going to say the sample size is too small and have other critiques. The assignment could be improved -- more guidance about how to incorporate and react to content from the article could make it more rigorous -- but the aims are not out of line.
Having spent an enormous amount of time teaching undergraduate research methods in psychology AND being a developmental psychologist myself, I have alot of thoughts to offer here, and some questions about the Oklahoma case in particular. I have no beef with the methodological critique of the assigned paper. But I think that there may be an error of a sort, or a missed and important nuance, in making inferences from what is assigned in undergraduate coursework to the state of a field at the level of what experts are doing. The problem here, in my view, is not with the science, but with the undergraduate education happening within that field. (Note that there are also problems with the science - I don't dispute that - but I am vehemently opposed to the notion that those problems are unique to social sciences - the replication crisis in biomedical sciences is equally large and being addressed with infinitely less transparency). In psychology, and perhaps many other fields, there is a sharp disconnect (my opinion, my colleagues might disagree) between how we teach undergraduates and the scholarly work that most of us think of as "the field." A hundred or so citations means, in essence, that this paper is not being treated as important within the science. So why assign it? Undergraduate psychology majors are revenue generators - the major is large, the faculty have been willing to teach 'efficiently' - meaning large lectures, multiple choice tests, and a kind of branching major structure that maximizes flexibility at the expense of knowledge and skill development - this willingness is connected to an emphasis on research and scholarly work, and graduate training of the next researchers. Given the size of psychology undergraduate majors, very very few of those majors are in the category of the next generation researchers - so investment in the major is low. The flexibility structure means, in turn, that students taking a lower-division, transferable lifespan development course have not had enough methodological education to read a paper using structrural modelling or other more advanced statistical techniques (most of which are not taught at all for undergraduate psychology majors); it also means that the instructor cannot rely on a foundation of knowledge about correlation/causation, statistical power and representativeness of samples, questionnaire and measurement standards, etc. The instructor also cannot rely on help for grading (though AI may be changing this rapidly or already, as I type!). This produces a situation where you assign a bad paper because it is one students can generally understand, create an assignment with the capacity to accommodate an unknowable range of preparedness and therefore no capacity to actually cultivate skills or knowledge, and deal with the messy result in whatever way possible. It's not good, but it's not a problem with the SCIENCE. Redesigning an undergraduate curriculum to do better is completely do-able, and in fact, happens at small liberal arts institutions and elsewhere (i have SO MANY IDEAS here, but there is a reason I'm in the Honors College and not in my home department), but the incentives for faculty (grant-writing, research productivity, graduate training) and administration (efficiency, scale, transferability) are not well-aligned with doing so, and therefore, it will not happen. A further issue with the Oklahoma case might be - and here, I was never able to really figure this out from the coverage I read - this was not a faculty instructor, I don't think? (My recollection is that it was a graduate instructor - but again, this could be wrong). If a graduate instructor, that is a further element of the perverse incentives here, where farming out lower-division courses (the gen-eds that need to transfer) to less skilled, less experienced instructors often happens with little training and less support - I used to oversee our graduate teaching practicum. So, none of this is divorced from the many issues you often raise about higher education in our times, but I think it highlights a particular kind of disconnect between the research "enterprise" of universities and the undergraduate educational mission that is perhaps under-explored, and which may be quite variable across disciplines.
“Right-wing defenders of a wronged student seem not to want to touch the possibility that the credential she was working toward wasn’t substantial.”
Very good piece.
I learned quite a bit, and agree with the main theses.
Then you seemed to feel the need to put in this cheap shot as a way to show “balance” in what is generally a critique of leftist professors and departments.
You have no evidence at all that these people not of the left - whom you injudiciously blanket label as “right-wing” - don’t “want to touch” said issue.
You merely have evidence that they commented on the obvious thing here that did in fact bother them.
In fact, I’d be willing to bet that the majority of the “defenders” you cite would agree with your claim about the lack of substance of the credential she was working toward.
The fact that it is not the focus of every - in fairness to you, almost certainly *any* - of the “right-wing defenders” you cite is not evidence they “don't want to touch” the issue.”
Merely that it is not *their* issue.
[P.S. I’m quite glad that it seems to be one of *yours*. Kudos to you for that.]
I appreciate your comment and even your pointing out of cheap shots. Look, the right are the only people who seem to be able to make change in higher ed. I want to point people who can fix things in the right direction. If I have to say blunt things to get their attention, I will. I hang my head a little bit but only a little bit.
I don't think I agree that "the right are the only people who seem to be able to make change in higher ed", since they hold almost no power in higher ed.
Christopher Rufo and Bill Ackman - the latter hardly a man of the right - are the only two people right of center I've seen able to "make change" in higher education.
Unless you count the Trump Administration via executive orders.
Maybe Sian Leah Beilock at Dartmouth is of the right, though it's at least as likely that she is centrist or even center-left as of 2001.
But with her as the possible exception, it seems to me that only the left-leaning leaders of higher education and their Boards of Trustees can actually change things in academia. All the right can do is try to apply pressure, or maybe change how federal funding works.
The idiocacy of one under-supervised graduate student is just an anecdote, the sort of thing that predictably gets people riled up. (Please, God, may I never write a persuasive essay that makes so much of an N of 1.) The replication crisis is more concerning and worthy of attention and, indeed, scorn. I've taugh undergraduate methods (in sociology) for twenty-five years and I imagine most of my undergrads are surprised that the crisis is so serious, as the result of things we definitely don't teach, like p-hacking, some outright (but I hope rare) fabrication, and running as many experiments as necessary to get a signficant result. What I taught, many times over: validity, reliability, confounding variables, observer effects, statistical signficance, social desirability, etc., etc.
It's easy to make a strong case for the intellectual merits and practical importance of social science. It's getting harder to defend how we've gone about doing it. That might be because we're bad at it, or because the incentives are perverse, or because we're looking for the wrong kind of order with our finicky statistical models. (Fortunately, I rarely do that kind of research and when I once did, was willing to admit that I'd failed to discover anything worthwhile.)
I agree I am making much of an n of 1. But seriously, that class and that assignment is beyond ridiculous. Even you have to agree with that! And disciplines should police their own, without having people like me outside the discipline point it out!
“This small glimpse of one assignment in one course should be enough to make employers, graduate schools, licensing bodies, and federal lending programs start asking hard questions.” - - Yes, the crux of the matter. Without any of these hard questions and rejection of the credentials, there’s no incentive to fix this dismal masquerade.
"...of course the student’s “reaction paper” was appropriate. Anything would have been appropriate. No brain work was required, nobody seems to have noticed." Love this take! I haven't seen anyone else make this argument about the controversy. It was bad pedagogy and a bad essay. Bad all around.
This assignment is, in fact, gold-standard:
_X_ It offers students a "wide variety of options" for how to complete the assignment
_X_ The course is online, which means it's cheaper for admin and more convenient for students
_X_ Standards for content are lower and therefore more equitable
_X_ Word counts for such an assignment meet some arbitrary Gen Ed standard for "rigor"
_X_ The reading covers some kind of gender/race/colonial/disability aspect (often still required)
_X_ The students' own reactions and opinion must be treated as more important than any "objective" critique
This is exactly the kind of assignment that's applauded by Centers for Teaching and Learning, hall monitor faculty, and Education School pedagogy "experts."
Your post resonated with me because it provoked a long time annoyance I had with social sciences. It seems that undergraduate social science education is built upon students learning the theories, with the expectation that they will apply the theory to real life (work, politics, media) or build new theories upon it, while there is never enough evidence behind the theories themselves in the first place. Statistics based research suffers from many limitations such as unrepresentative sample, unrealistic model, statistical strength, confounding factors, and small effect size even when statistically significant. It seems to me that any conclusion must be drawn from a large body of literature, and even in that case, the conclusion will always be nuanced and noisy. In assignments like this, the student is encouraged to take the theory as given and freely extrapolate from it, ignorant of the uncertain nature of the theories themselves.
This is exactly what goes on in much of the humanities, at least those disciplines that have been influenced by sociology and/or critical theory. The most intellectually demanding task a student is likely to be assigned is to explicate and apply a theory to 'critique' or comment on something (e.g. a 'cultural object' or text), without ever critiquing the theory itself.
“… at least those disciplines that have been influenced by sociology and/or critical theory.”
There actually exist humanities disciplines *not* so influenced?!?
Could you point them out to me?
Genuinely curious, as I find the idea unfathomable.
Though I suppose some philosophy departments might at least come close.
Philosophy has largely avoided the full-scale adoption of this kind of capital-t "Theory", although the problem here is that even if it's not in what they teach - and even though some of what is taught directly goes against or undermines it - professors themselves are often sympathetic to "woke" positions, wanting to be good (i.e., left-wing) people, and so don't stand up to policies like discrimination in hiring (e.g. "diversity" statements), etc. Because of this, most philosophy departments after 2020 will have one or two radicals embedded in them, with the rest of the faculty not being inclined to stand up to their radical ideas, so this might not last for much longer.
Classics and some history departments are likely also to be stand-outs, or at least won't have been quite as affected (infected?) as literature, sociology, or any of the "X studies" departments.
Since retiring, I have lost hope that Psychology could be taken as a credible degree with the lack of scientific rigor and absence of bare minimum education in statistics and research methodology. Maybe, if professionals consistently call out the poor quality of current education received, Psychology may move a bit closer to the science to which it once aspired. All goes back to the education/training received.
…and yet psychology is actually more based, relatively speaking, than is sociology, let alone all of the grievance studies.
possibly. but consider the gender ideology that is pervasive in Psychology that has encouraged a lack of attention to research. Yes, sociology is a serious problem, but Dissonance theory really is replicable, yet never taken seriously in the present.
As Hollis Robbins implies, a paper enters the literature by being read, gets read by being assigned, gets assigned by being cited, and the loop never returns to the foundational question: was the original finding actually robust?
When the same readings, interpretations and citation clusters also support the intellectual authority of the course convener, pedagogy risks becoming institutional self-confirmation. The canon is taught, cited, examined and renewed by the very structure that benefits from its survival.
That is not peer review. It is citation replication.
Or, to paraphrase William Shakespeare through the lens of Jean Baudrillard:
“It is a tale told by an idiot, full of sound and fury, signifying simulacra.”
This problem isn't new. When I was teaching a humanities elective for non-arts majors around 18 years ago, I happened to see an assignment that one of the other lecturers had given her class, which was an elective on the figure of the vampire in literature and film (already not a promising sign of real rigor, though the topic could be handled properly). The assignment was literally just asking the student to find a website on vampires, say what the website's take was on whether vampires were real or not, and they say what they thought (about the reality of vampires, I assume, though the wording left this ambiguous). While this was a minor assignment, it was still worth something like 15% of the course grade.
This might not have been as bad as the final exam for a remedial course on the mechanics of writing, a copy of which I found left in the departmental photocopier. Every question - on the final exam for the writing course - was multiple choice. (This was also around 2008-ish.)
This post.... no words...
But, after reading this, I am pleased that during my daughter's college work, she once called me to discuss one of her psych course assignments. The question posed was to assess the methodology of the study - which was as poor as the one cited in this post.
As a researcher trained in the prior generation, I suggest that the skill to assess the credibility and validity of a study was more critical than the study itself. I had a good talk with her and her team; they had capability found the many egregious study design and data interpretation errors.
Anyway... glad my daughter had a professor that still promoted critical thinking...
“Everyone assumes that a psychology BA from a public flagship is evidence that the holder has been trained in the discipline.” Exactly! Degrees do not always equal full competency in the area.
The biggest problem with this assignment in the current era and configuration (large online class) is that the majority will have ChatGPT do it for them. Even if the assignment isn't perfect, asking students to read an empirical article and think about it (make connections and applications) is a reasonable learning activity for this level. Presumably, this is in addition to the course texts and lectures that provide the overview of scientific knowledge -- this is aimed at developing some analytical skills. At the level the course is pitched at, that's probably all one can do, but at least, in the old days, it would get students to actually read a research article. And, they're not going to assume this is state of the art -- they're all going to say the sample size is too small and have other critiques. The assignment could be improved -- more guidance about how to incorporate and react to content from the article could make it more rigorous -- but the aims are not out of line.
Having spent an enormous amount of time teaching undergraduate research methods in psychology AND being a developmental psychologist myself, I have alot of thoughts to offer here, and some questions about the Oklahoma case in particular. I have no beef with the methodological critique of the assigned paper. But I think that there may be an error of a sort, or a missed and important nuance, in making inferences from what is assigned in undergraduate coursework to the state of a field at the level of what experts are doing. The problem here, in my view, is not with the science, but with the undergraduate education happening within that field. (Note that there are also problems with the science - I don't dispute that - but I am vehemently opposed to the notion that those problems are unique to social sciences - the replication crisis in biomedical sciences is equally large and being addressed with infinitely less transparency). In psychology, and perhaps many other fields, there is a sharp disconnect (my opinion, my colleagues might disagree) between how we teach undergraduates and the scholarly work that most of us think of as "the field." A hundred or so citations means, in essence, that this paper is not being treated as important within the science. So why assign it? Undergraduate psychology majors are revenue generators - the major is large, the faculty have been willing to teach 'efficiently' - meaning large lectures, multiple choice tests, and a kind of branching major structure that maximizes flexibility at the expense of knowledge and skill development - this willingness is connected to an emphasis on research and scholarly work, and graduate training of the next researchers. Given the size of psychology undergraduate majors, very very few of those majors are in the category of the next generation researchers - so investment in the major is low. The flexibility structure means, in turn, that students taking a lower-division, transferable lifespan development course have not had enough methodological education to read a paper using structrural modelling or other more advanced statistical techniques (most of which are not taught at all for undergraduate psychology majors); it also means that the instructor cannot rely on a foundation of knowledge about correlation/causation, statistical power and representativeness of samples, questionnaire and measurement standards, etc. The instructor also cannot rely on help for grading (though AI may be changing this rapidly or already, as I type!). This produces a situation where you assign a bad paper because it is one students can generally understand, create an assignment with the capacity to accommodate an unknowable range of preparedness and therefore no capacity to actually cultivate skills or knowledge, and deal with the messy result in whatever way possible. It's not good, but it's not a problem with the SCIENCE. Redesigning an undergraduate curriculum to do better is completely do-able, and in fact, happens at small liberal arts institutions and elsewhere (i have SO MANY IDEAS here, but there is a reason I'm in the Honors College and not in my home department), but the incentives for faculty (grant-writing, research productivity, graduate training) and administration (efficiency, scale, transferability) are not well-aligned with doing so, and therefore, it will not happen. A further issue with the Oklahoma case might be - and here, I was never able to really figure this out from the coverage I read - this was not a faculty instructor, I don't think? (My recollection is that it was a graduate instructor - but again, this could be wrong). If a graduate instructor, that is a further element of the perverse incentives here, where farming out lower-division courses (the gen-eds that need to transfer) to less skilled, less experienced instructors often happens with little training and less support - I used to oversee our graduate teaching practicum. So, none of this is divorced from the many issues you often raise about higher education in our times, but I think it highlights a particular kind of disconnect between the research "enterprise" of universities and the undergraduate educational mission that is perhaps under-explored, and which may be quite variable across disciplines.
This is exactly the conversation I wanted to provoke. There is so much here that that was never talked about in December.
Didn’t the piece note the course was ‘online’, with no associated instructor?
just graduate student graders, yes.
“… the replication crisis in biomedical sciences is equally large and being addressed with infinitely less transparency”
I claim no expertise in the replication crisis in biomedical sciences. So I am not disputing that it might be quite large.
But I don’t see how - nor could ChatGPT when I asked it - it could be as large as in the social sciences.
On what basis I do you make that claim?
“Right-wing defenders of a wronged student seem not to want to touch the possibility that the credential she was working toward wasn’t substantial.”
Very good piece.
I learned quite a bit, and agree with the main theses.
Then you seemed to feel the need to put in this cheap shot as a way to show “balance” in what is generally a critique of leftist professors and departments.
You have no evidence at all that these people not of the left - whom you injudiciously blanket label as “right-wing” - don’t “want to touch” said issue.
You merely have evidence that they commented on the obvious thing here that did in fact bother them.
In fact, I’d be willing to bet that the majority of the “defenders” you cite would agree with your claim about the lack of substance of the credential she was working toward.
The fact that it is not the focus of every - in fairness to you, almost certainly *any* - of the “right-wing defenders” you cite is not evidence they “don't want to touch” the issue.”
Merely that it is not *their* issue.
[P.S. I’m quite glad that it seems to be one of *yours*. Kudos to you for that.]
I appreciate your comment and even your pointing out of cheap shots. Look, the right are the only people who seem to be able to make change in higher ed. I want to point people who can fix things in the right direction. If I have to say blunt things to get their attention, I will. I hang my head a little bit but only a little bit.
What a great response!
I greatly appreciate your honesty.
I don't think I agree that "the right are the only people who seem to be able to make change in higher ed", since they hold almost no power in higher ed.
Christopher Rufo and Bill Ackman - the latter hardly a man of the right - are the only two people right of center I've seen able to "make change" in higher education.
Unless you count the Trump Administration via executive orders.
Maybe Sian Leah Beilock at Dartmouth is of the right, though it's at least as likely that she is centrist or even center-left as of 2001.
But with her as the possible exception, it seems to me that only the left-leaning leaders of higher education and their Boards of Trustees can actually change things in academia. All the right can do is try to apply pressure, or maybe change how federal funding works.
I loved this piece.
After reading it, I went to read the link you cited, but it’s paywalled.
So I’m finding it very, very hard to believe your claim that said piece went viral.
And if I can’t trust that word, Why should I trust anything you say? 😏
Hahah here's a tweet that was seen apparently 100K times. https://x.com/minzlicht/status/2054709095681200212?s=20
The idiocacy of one under-supervised graduate student is just an anecdote, the sort of thing that predictably gets people riled up. (Please, God, may I never write a persuasive essay that makes so much of an N of 1.) The replication crisis is more concerning and worthy of attention and, indeed, scorn. I've taugh undergraduate methods (in sociology) for twenty-five years and I imagine most of my undergrads are surprised that the crisis is so serious, as the result of things we definitely don't teach, like p-hacking, some outright (but I hope rare) fabrication, and running as many experiments as necessary to get a signficant result. What I taught, many times over: validity, reliability, confounding variables, observer effects, statistical signficance, social desirability, etc., etc.
It's easy to make a strong case for the intellectual merits and practical importance of social science. It's getting harder to defend how we've gone about doing it. That might be because we're bad at it, or because the incentives are perverse, or because we're looking for the wrong kind of order with our finicky statistical models. (Fortunately, I rarely do that kind of research and when I once did, was willing to admit that I'd failed to discover anything worthwhile.)
I agree I am making much of an n of 1. But seriously, that class and that assignment is beyond ridiculous. Even you have to agree with that! And disciplines should police their own, without having people like me outside the discipline point it out!
“This small glimpse of one assignment in one course should be enough to make employers, graduate schools, licensing bodies, and federal lending programs start asking hard questions.” - - Yes, the crux of the matter. Without any of these hard questions and rejection of the credentials, there’s no incentive to fix this dismal masquerade.
And a specific reason why colleges and universities are diminishing in intrinsic value, no? Just being eroded from the edges…
Many, many are getting hurt all along the continuum.
I had so hoped for a renaissance in the social sciences, but I think it is too late.