It's a really interesting concept, and I can see how I could do it in a class of 20 students.
What I'm having trouble with figuring out is how it possible at scale.
For example, in the following context: 1) 100+ students in class together for three hours/week (no additional contact beyond those three hours), 2) students don't have even basic knowledge about the subject, and 95% avoid doing the assigned work necessary to develop the basic knowledge that would be needed for RE, 3) there are objectively right and wrong answers, and it is important that students learn the difference (the feedback isn't as straightforward as a circuit not working), 4) TAs who don't know the material either (and therefore can't provide the feedback), 5) administration-required midterm and final exams (not papers or projects).
I really like the idea, and would appreciate suggestions for how to do it in that context.
I would like that too. I know how much more my students would leave class understanding if there were 20-30 of them in class rather than 100.
I also know that there aren't enough classrooms to house the students who are at the university, and that the university isn't willing to reduce class sizes because it would cost more to offer a class than they want a class to cost.
So the (current- and near-future) reality is that I have 100 students/class, and everything. My question is now to do RE when the students don't have enough content knowledge to self-correct, the subject matter doesn't have direct feedback (e.g., there isn't the kind of feedback you get from mis-wiring something), and there's little time for the professor input to the RE process given the number of students in the class.
You inspired me to move ahead with an assessment technique for my law students that I had been mulling but had not consolidated into a specific assignment. In essence: turn the classic law school "analyze a set of hypothetical facts in light of the legal principles we have discussed" assessment into a "figure out what the [AI-assisted] client did, didn't do, and might do differently in order to align with present and emerging legal principles." Diagnose, in other words, rather than prescribe, or judge. We'll see what my students think, pun intended.
You mention biology, but more to the point medical research is reverse engineering the mechanism behind the presentation of disease in hopes of disrupting it.
I agree on the importance of the concept of reverse engineering. I know Steven Pinker used the concept in How the Mind Works (1997):
"Reverse-engineering is what the boffins at Sony do when a new product is announced by Panasonic, or vice versa. They buy one, bring it back to the lab, take a screwdriver to it, and try to figure out what all the parts are for and how they combine to make the device work. We all engage in reverse-engineering when we face an interesting new gadget. In rummaging through an antique store, we may find a contraption that is inscrutable until we figure out what it was designed to do. When we realize that it is an olive-pitter, we suddenly understand that the metal ring is designed to hold the olive, and the lever lowers an X-shaped blade through one end, pushing the pit out through the other end. The shapes and arrangements of the springs, hinges, blades, levers, and rings all make sense in a satisfying rush of insight." (pp. 21-22)
But I'm pretty sure the term was introduced into cognitive psychology before that. I have the vague sense – but, alas, no citation, that Donald Broadbent used it in the late 1950s.
"I examine a set of stories that are organized on three levels: 1) the entire story trajectory, 2) segments within the trajectory, and 3) sentences within individual segments. I conjecture that the probability distribution from which ChatGPT draws next tokens seems to follow a hierarchy nested according to those three levels and that is encoded in the weights of ChatGPT’s parameters. I arrived at this conjecture to account for the results of experiments in which I give ChatGPT a prompt with two components: 1) a story and, 2) instructions to create a new story based on that story but changing a key character: the protagonist or the antagonist. That one change ripples through the rest of the story. The pattern of differences between the old and the new story indicates how ChatGPT maintains story coherence. The nature and extent of the differences between the original story and the new one depends roughly on the degree of difference between the original key character and the one substituted for it. I end with a methodological coda: ChatGPT’s behavior must be described and analyzed on three strata: 1) The experiments exhibit behavior at the phenomenal level. 2) The conjecture is about a middle stratum, the matrix, that generates the nested hierarchy of probability distributions. 3) The transformer virtual machine is the bottom, the code stratum."
Note that my story paradigm was inspired by the work Claude Lévi-Strauss:
"The procedure I have been using is derived from the analytical method Claude Lévi-Strauss employed in his magnum opus, Mythologiques. He started with one myth, analyzed it, and then introduced another one, very much like the first. But not quite. They are systematically different. He characterized the difference by a transformation – a term he took from algebraic group theory. He worked his way through hundreds of myths in this manner, each one derived from another by a transformation."
Thus, I would start with a short five-paragraph fairy tale generated by ChatGPT. The protagonist was Princess Aurora. When I asked that the new protagonist be Prince Henry, the resulting changes were limited and predictable. But when I asked that the new protagonist be XP-708-DQ, the entire story was shifted from a fairy tale universe to a science fiction universe. Something particularly interesting happened when I asked that the protagonist be a colorless green idea. ChatGPT refused to create a story because that was not a proper protagonist. It also informed me that the phrase was from Noam Chomsky (which, of course, I new).
It's a really interesting concept, and I can see how I could do it in a class of 20 students.
What I'm having trouble with figuring out is how it possible at scale.
For example, in the following context: 1) 100+ students in class together for three hours/week (no additional contact beyond those three hours), 2) students don't have even basic knowledge about the subject, and 95% avoid doing the assigned work necessary to develop the basic knowledge that would be needed for RE, 3) there are objectively right and wrong answers, and it is important that students learn the difference (the feedback isn't as straightforward as a circuit not working), 4) TAs who don't know the material either (and therefore can't provide the feedback), 5) administration-required midterm and final exams (not papers or projects).
I really like the idea, and would appreciate suggestions for how to do it in that context.
Thanks!
I am hoping that the notion of scale in education will please go away soon. It doesn't work, it has never worked, I can't imagine it ever working.
I would like that too. I know how much more my students would leave class understanding if there were 20-30 of them in class rather than 100.
I also know that there aren't enough classrooms to house the students who are at the university, and that the university isn't willing to reduce class sizes because it would cost more to offer a class than they want a class to cost.
So the (current- and near-future) reality is that I have 100 students/class, and everything. My question is now to do RE when the students don't have enough content knowledge to self-correct, the subject matter doesn't have direct feedback (e.g., there isn't the kind of feedback you get from mis-wiring something), and there's little time for the professor input to the RE process given the number of students in the class.
Any suggestions would be greatly appreciated!
You inspired me to move ahead with an assessment technique for my law students that I had been mulling but had not consolidated into a specific assignment. In essence: turn the classic law school "analyze a set of hypothetical facts in light of the legal principles we have discussed" assessment into a "figure out what the [AI-assisted] client did, didn't do, and might do differently in order to align with present and emerging legal principles." Diagnose, in other words, rather than prescribe, or judge. We'll see what my students think, pun intended.
You mention biology, but more to the point medical research is reverse engineering the mechanism behind the presentation of disease in hopes of disrupting it.
100% yes
I agree on the importance of the concept of reverse engineering. I know Steven Pinker used the concept in How the Mind Works (1997):
"Reverse-engineering is what the boffins at Sony do when a new product is announced by Panasonic, or vice versa. They buy one, bring it back to the lab, take a screwdriver to it, and try to figure out what all the parts are for and how they combine to make the device work. We all engage in reverse-engineering when we face an interesting new gadget. In rummaging through an antique store, we may find a contraption that is inscrutable until we figure out what it was designed to do. When we realize that it is an olive-pitter, we suddenly understand that the metal ring is designed to hold the olive, and the lever lowers an X-shaped blade through one end, pushing the pit out through the other end. The shapes and arrangements of the springs, hinges, blades, levers, and rings all make sense in a satisfying rush of insight." (pp. 21-22)
But I'm pretty sure the term was introduced into cognitive psychology before that. I have the vague sense – but, alas, no citation, that Donald Broadbent used it in the late 1950s.
I've used it in my investigations of ChatGPT. I've developed a whole research program around it. The key document is: ChatGPT tells stories, and a note about reverse engineering: A Working Paper, Version 3, https://www.academia.edu/97862447/ChatGPT_tells_stories_and_a_note_about_reverse_engineering_A_Working_Paper_Version_3
Here's the abstract:
"I examine a set of stories that are organized on three levels: 1) the entire story trajectory, 2) segments within the trajectory, and 3) sentences within individual segments. I conjecture that the probability distribution from which ChatGPT draws next tokens seems to follow a hierarchy nested according to those three levels and that is encoded in the weights of ChatGPT’s parameters. I arrived at this conjecture to account for the results of experiments in which I give ChatGPT a prompt with two components: 1) a story and, 2) instructions to create a new story based on that story but changing a key character: the protagonist or the antagonist. That one change ripples through the rest of the story. The pattern of differences between the old and the new story indicates how ChatGPT maintains story coherence. The nature and extent of the differences between the original story and the new one depends roughly on the degree of difference between the original key character and the one substituted for it. I end with a methodological coda: ChatGPT’s behavior must be described and analyzed on three strata: 1) The experiments exhibit behavior at the phenomenal level. 2) The conjecture is about a middle stratum, the matrix, that generates the nested hierarchy of probability distributions. 3) The transformer virtual machine is the bottom, the code stratum."
Note that my story paradigm was inspired by the work Claude Lévi-Strauss:
"The procedure I have been using is derived from the analytical method Claude Lévi-Strauss employed in his magnum opus, Mythologiques. He started with one myth, analyzed it, and then introduced another one, very much like the first. But not quite. They are systematically different. He characterized the difference by a transformation – a term he took from algebraic group theory. He worked his way through hundreds of myths in this manner, each one derived from another by a transformation."
Thus, I would start with a short five-paragraph fairy tale generated by ChatGPT. The protagonist was Princess Aurora. When I asked that the new protagonist be Prince Henry, the resulting changes were limited and predictable. But when I asked that the new protagonist be XP-708-DQ, the entire story was shifted from a fairy tale universe to a science fiction universe. Something particularly interesting happened when I asked that the protagonist be a colorless green idea. ChatGPT refused to create a story because that was not a proper protagonist. It also informed me that the phrase was from Noam Chomsky (which, of course, I new).
I have just issued a report on that work that covers material in 11 different papers: ChatGPT: Exploring the Digital Wilderness, Findings and Prospects, https://www.academia.edu/127386640/ChatGPT_Exploring_the_Digital_Wilderness_Findings_and_Prospects
As an engineer, I am absolutely blown away with her approach to using AI in Higher Ed.
This makes a lot of sense and is a terrific idea. I will admit that I’ve never thought about this approach in any systematic way.