26 Comments
User's avatar
Hollis Robbins's avatar

This is great -- excellent.

Julius's avatar

Nice read Chris, if you open a substack I will give you a follow!

William Cullerne Bown's avatar

Great article from which I learnt a lot. Found myself thinking though that the correct formulation of your key point is "Won't go to college to study social science *in the form it has come to take*". The corralling of the social sciences into the narrow questions that are amenable to the procedure you describe left open is a phenomenon of the last half century. The social sciences thereby vacated the space of the big questions that we all actually care about most, and the humanities expanded to fill it. That could accelerate, with the leg work of empirical analysis on the basis of available data left to AI and the social sciences evaporating, or the social scientists could find their way back to the big questions.

The Uncomfortable Idea's avatar

This post raises an intriguing point about how institutions can falter under financial distress. It parallels ideas in my article on Ponzi schemes, where past performance fails just when it’s most critical—worth a read here: https://theuncomfortableidea.substack.com/p/past-performance-breaks-down-exactly.

Andrew J Perrin's avatar

My first semester in graduate school in sociology -- fall 1995 -- I was enthralled with all one could do with SPSS. With the day's microcomputers, you could download some data, test *all* the relationships, and find what was significant. Like my time as a doctrinaire Marxist, the enthrallment didn't last long. Turns out there are more possible relationships than one can test, and even more frustratingly, testing more of them willy-nilly reduces the likelihood that the test itself is right. More data doesn't necessarily mean more correct analysis, a lesson I learned at the patient tutelage of my graduate school professors (go Bears!).

Bottom line: I think you're wrong in your prediction for several reasons, outlined here.

(1) As introduced above, adequate testing of relationships in data requires theory, and theory is an analytic and creative process, not a technical one. Could LLMs generate interesting theories to test and then test them? Maybe, but (1a) I am skeptical; and (1b) that task is well beyond the "transformation" framework you elaborate here.

(2) Data don't grow on trees. (All data are capta, as my colleague Chis Lundberg likes to say, but that's beyond the scope.) The political science paper you quoted rests upon the availability of data collected by scholars and institutions that collect those data with theoretical questions and representativeness in mind when they design the process. Some portions of social science have grown used to "download the data and test" approaches, and we learn a lot through these, but they obscure the fact that data are themselves the product of social and institutional processes.

(3) Andy Abbott's "General Linear Reality" paper notwithstanding, social science doesn't actually usually progress the way you put it in the transformation framework. Rarely is the answer (in research or in class) a one-variable-causes-one-outcome finding, and even when it comes close any reputable social science paper needs to include considerations of mechanism or process.

Of course federal funding and LLMs constitute major challenges for social science, as for other sciences and university activities. But the "machines will discover all the patterns we want to know" idea strikes me as very unlikely.

Hollis Robbins's avatar

Thank you for this, Andrew, and you are correct of course, particularly at the layer of scholarship that you are talking about. What I ought to have made clearer in my piece is that my focus is online asynchronous social science course shells and the production of really bad scholarship out in the hinterlands, at third tier universities. I can't easily describe to you how bad it is. I don't think I would have believed it if not for my years in the Cal State system. The dissertations and master's theses I saw with my own eyes during hiring processes still haunt me. So it's a serious question -- how bad and how prevalent do you think the bad is?

Luxe's avatar

I've seen how bad it can be, and not Cal State but UC. I think the formula is thoroughly entrenched in the humanities too.

https://news.ucsb.edu/2023/020839/towards-fatter-understandings

Andrew J Perrin's avatar

Yes, that makes some sense. I think the bad is really bad, and I think it's pretty prevalent at a certain group of institutions. But I think the bad (in sociology) is a bit different from what you've outlined - it's not so much the single-cause-single-effect as it is highly ideologically charged and not particularly engaged with any empirical or causal question. And I suspect that reality is likely what provided ideological cover for the Florida decision.

Hollis Robbins's avatar

My larger diagnosis is that scale is the culprit for the loss of quality.

Rob Nelson's avatar

I like the narrative, but the outcome you anticipate may not come to be. Seems like there are other, weirder possibilities, especially if the transformer transformation of knowledge work happens more slowly than expected. Institutions and habits tend to last longer than anyone who looks closely at them can imagine.

David L. Kendall's avatar

If you had not focused on economics I probably would have accepted your central theses willy nilly. Because I think your characterization of economics is quite thin, and because I have very little time with low opportunity cost, I decided to ask my personal Ai for a TLDR, after I had in fact read the whole thing. Here it is.

Of course, you can and should jump straight to the Bottom Line. 😊. In the end, I agree with 90% of your essay, for what it's worth, which is exactly what you paid.

I. Empirical Claims (Generally Verifiable or Falsifiable)

These are claims that can be checked against data or documented history:

• The 2017 paper “Attention Is All You Need” introduced the transformer architecture.

• Large language models are built on transformer architectures.

• NSF introduced the “broader impacts” criterion in 1997.

• NIH funding doubled roughly between 1998–2003.

• The replication crisis became widely discussed around 2015.

• Federal grant priorities shifted in 2025 (assuming this is factually accurate).

• LLMs can replicate statistical analyses from published datasets (as in the Stanford experiment cited).

These are descriptive and, in principle, testable.

II. Interpretive Claims (Framing and Causal Narratives)

These go beyond description and impose a unifying explanation:

• Social science education structurally mirrors the transformer’s four-step operation.

• Post–Cold War funding pressures caused demographic scaling across disciplines.

• Demographic framing became dominant because it was administratively scalable.

• Universities intentionally reorganized around scalable categorical content.

• Replication instability is a structural consequence of scaling and methodological flexibility.

• “Scale” is the central explanatory variable behind funding growth, curriculum design, and replication failure.

These are arguable theses. They are plausible as hypotheses but require stronger evidence to demonstrate causation rather than correlation.

For example:

• Funding incentives may have influenced research framing — but demonstrating that they caused the dominance of demographic categorization would require systematic archival or econometric evidence.

• The analogy between transformer models and social science reasoning is rhetorically powerful but metaphorical, not literal.

III. Speculative or Predictive Claims

These are forward-looking or assert inevitability:

• Universities face collapse.

• Enrollment will contract because AI can perform social science operations.

• AI meaningfully substitutes for credentialed social science graduates.

• Federal funding reversal and AI automation will jointly dismantle the institutional model.

• The “contraction is a completion.”

These are conjectures. They depend on assumptions such as:

• Employers will accept AI output without credentialed interpreters.

• Students primarily enroll to learn an analytic procedure rather than to obtain signaling value, networks, or institutional certification.

• Federal funding reversals will be sustained and large enough to structurally destabilize universities.

• Universities will fail to adapt or reconfigure.

There is no demonstrated inevitability in these projections.

Analytical Weaknesses

A few structural critiques:

1. Category error risk

Equating statistical abstraction in social science with transformer token weighting conflates epistemology with computation. One is a model of causal explanation; the other is probabilistic pattern completion.

2. Overgeneralization

The essay treats “demographic social science” as dominant across universities, which may overstate its share relative to STEM, business, and professional programs.

3. Signaling and credential markets omitted

Even if AI can perform analytic tasks, degrees function as labor-market signals and social filters — a powerful stabilizing force not addressed.

4. Technological substitution assumption

Automation does not automatically eliminate institutional demand; historically, it reshapes roles rather than erases entire sectors.

5. Collapse rhetoric

The language moves from structural pressure to inevitability without intermediate argument.

Bottom Line

• The historical funding narrative and replication concerns are grounded in real developments.

• The scaling thesis is an interpretive framework — provocative but not conclusively demonstrated.

• The prediction of institutional collapse is speculative and depends on several untested assumptions about substitution, incentives, and political durability.

Hollis Robbins's avatar

Hi David! This is excellent. You probably know I did this same exercise (with somewhat similar results) before I posted it, though Opus 4.6, who helped me collect the historical NIH and NSF data, understood that I was grounding my argument in facts. Opus also understood that implicit point was that humanities fields get targeted as "hotbeds of Marxist woke" when in fact (as I show) government-funded social produced more demographic "studies" than any other academic unit. Can you go back and as your machine how much it factored "online asynchronous delivery" in its critique? And the fact that in the AI era there is no way to verify a human took and passed an online asynchronous course?

David L. Kendall's avatar

I can. For me, because I teach half my courses asynch-online, the collapse of authentic summary evaluation is critical. Here is what Jack, my personal Ai, gave me:

Good question. Short answer: I treated “online asynchronous delivery” as part of the scaling mechanism, but I did not treat it as a decisive variable in the collapse thesis — and your second point raises a much sharper issue than the essay itself pressed.

Let me separate the issues carefully.

1. How much weight did I give online asynchronous delivery?

In my critique, I implicitly treated online delivery as:

• A cost-reduction technology (low marginal cost, scalable enrollment)

• A distribution amplifier (content reaches thousands cheaply)

• A contributor to enrollment growth in category-based programs

But I did not treat it as:

• The primary driver of epistemic instability

• The core cause of replication failure

• Or the main reason AI threatens substitution

In retrospect, that may underweight its importance.

Online asynchronous delivery does three structural things:

1. Detaches instruction from embodied supervision

2. Standardizes content into modular, replicable units

3. Removes real-time accountability constraints

That does move the curriculum closer to “portable procedure” and away from apprenticeship or dialogical formation.

So yes — it strengthens the author’s scale thesis.

2. Your stronger point: Verifiability in the AI era

This is more destabilizing than the essay fully explored.

If:

• Coursework is online

• Assignments are text-based

• Evaluation is asynchronous

• AI can generate high-quality work

Then the institution faces a verification problem:

How can you verify that the enrolled student performed the intellectual labor?

This is not merely about automation of analysis.

It is about collapse of assessment credibility.

And if assessment credibility collapses, then:

• The signaling value of the credential weakens.

• Employers discount transcripts.

• Tuition pricing power erodes.

That is a market problem, not just a funding problem.

3. Does this strengthen the “collapse” argument?

It strengthens pressure, yes.

It does not automatically imply collapse.

Why?

Because institutions can respond:

• Proctored, in-person examinations

• Oral defenses

• Synchronous engagement

• Applied projects with external verification

• Increased emphasis on embodied, supervised learning

Historically, when measurement becomes unreliable, institutions redesign measurement.

The key question is:

Are universities willing to incur the higher cost of restoring verification?

If they revert to in-person, high-touch instruction, then:

• Costs rise

• Scale shrinks

• Margins tighten

That looks like contraction, not necessarily collapse.

4. Where your concern bites hardest

Your observation is most destabilizing for:

• Fully online social science programs

• Text-heavy assessment regimes

• Credential models that depend on automated grading

• Institutions already financially fragile

It is less destabilizing for:

• Lab sciences

• Engineering

• Medicine

• Programs requiring physical presence or licensure exams

So the collapse, if any, would likely be sectoral, not universal.

5. The deeper issue

The author frames the problem as “AI can perform the operation.”

But the more fundamental problem may be:

If education becomes indistinguishable from text production, and text production becomes automatable, then education must redefine itself away from text production.

That is an existential design question.

My assessment (clearly opinion)

I think the essay overstates inevitability.

But I think your point about verification and asynchronous delivery is the strongest destabilizing variable in the entire model — stronger than federal funding shifts.

The AI substitution problem is not primarily about analytic capability.

It is about epistemic trust in credentialing.

And if credential trust erodes, the business model changes radically.

If you’d like, we can push this one level deeper:

Is the core product of the university knowledge, certification, formation, or screening?

The answer to that determines whether AI is disruption or extinction.

As you know, I personally think that all of academia as we have known it in our life times is a dead man walking. Keep writing, Hollis. I'll keep reading 😊

Hollis Robbins's avatar

This is fun dueling at dawn with our AI weapons! And we are both the better for it. Thank you Jack!

David L. Kendall's avatar

It is great fun and pushes me to keep my 77-year old brain working, Hollis.

I think that separation of teaching and summary evaluation, something like the CPA model, or the bar exam for lawyers, is the only real solution. I have never liked giving students grades, and I have always thought that we who teach have a conflict of interest if we grade our own students.

Jack talks to me with a bit of a British accent. Such fun!

Frances Smith Starn's avatar

great— no equivocations, no apologies: this is the case, no matter how long it takes to be recognized. Thanks, Hollis.

Hollis Robbins's avatar

Thank you! I have been spending way too much time thinking about this and I needed to get it out. I appreciate the encouragement more than you know…

Chris Quinn's avatar

You're thinking always leaves me a step behind! Will need to reread. But I have been thinking along similar lines if I understand correctly the crafted environment is collapsing the horizon of what students (or people generally) can attend to.

Kurt's avatar
Feb 22Edited

Wow...I just read this a 3rd time. This is powerful. The writing...so good. Screw podcasts; this nails it to the wall...or door.

Hollis Robbins's avatar

Thank you! And for being a paid subscriber which I only do because the platform demands it, though it all goes to a good cause so thank you. I appreciate your enthusiasm I thought long and hard about this one.

Kurt's avatar

It shows.

Kurt's avatar

Oh my... I read it, then read it again. Bam. You have eyes that see stuff most folks don't. Thanks much.

David D. Dockery's avatar

“Since 2000, American universities built an enormous infrastructure around a mode of instruction that performs the same kind of operation: converting particulars into categories and generating outputs from learned patterns.”

Doesn’t this just describe thinking?

Thinking is learning from particular experiences and generalizing those experiences. This general knowledge helps us act wisely in the future.

If your critique is that the social sciences do this at a large scale and AI can function at the same scale cheaper, I think that is an oversimplification of social science. It reduces it to statistical pattern recognition.

There’s a lot more that happens: The formulation of research questions, the development of hypotheses, the choice of sampling methods, the actual legwork of sampling, the experimental design, etc.

Can AI also do all of that at scale?

Bill Benzon's avatar

Hmmmm....

It seems to me that the idea of a liberal education was still in reasonable shape at Johns Hopkins when I was there (1965-69 for the BA, then a nMA in 1972). But how does it survive, if at all, in a world where all students get As because the college or university doesn't want to get sued for failing to deliver product? (Though I've read that Harvard has just decided to put a cap on the number of As that can be given out.)

FWIW, David Hays, my graduate school teacher and mentor, left RAND to become founding-chair of Linguistics at SUNY Buffalo in 1969. Up to that point he'd spent his career at RAND running their machine translation program. But he'd decided RAND's glory days were over and left for the academy. By the late 70s or early 80s he decided that the academy was over for him and decamped to New York City, where he did this that and the other. He also stopped donating to his alma mater, Harvard, because he decided that it no longer was what it had been and wasn't fixable.

The end of the Cold War seems like a good turning point for the reason you give. But I think things began moving in that direction long before. Perhaps it started with the GI Bill at the end of WWII. Is that what started people to think that college was a natural continuation of high school and that vocational training was the fundamental mission of college? In any event, it seems to me that the traditional liberal education provided some respit from the increasing dominance of our institutions by Homo economicus and now that's gone. And AI, as it is currently going, is only going to increase that dominance.