When AI Gets History Wrong

Added: 2026-05-06

[00:00:00]Hello everyone.

[00:00:02]I want to try an experiment here, a live experiment. I need your participation.

[00:00:06]I'm asking the question, should we use these AI large language models for historical research? Should I as a historian, should any of you doing historical work, how how should you rely on them? Are they accurate? All of them say, "Check the results. We could be wrong." And we have all kinds of horror stories about them. Now if you go back a little bit to the turn of the millennium 21st century around the year 98 99 2000 2001 we had Google and now we talk about googling something and of course there are other search engines as well and then Wikipedia came in I think around 2001 and there was a big debate in academic circles high school college graduate school should we let students use Wikipedia IA, what if they just copy things and use them? And there was a kind of a pro and con approach. Some said absolutely not. They must not use Wikipedia because it's unreliable and uh it's just cribbing basically and cheating. And others said no, go ahead and use it, but use it discriminately. Like you might ask when was the battle of Waterloo and uh when did Napoleon fight and what were the casualties on both sides and so forth. So Google took you to the websites and then you had to work with the websites. Maybe some of them were completely worthless and Wikipedia gave you articles on topics that you might even be working on. But was it reliable? Because that was a user kind of platform which it still is where people can add things.

[00:01:49]And now we have these AI models and the ways in which they're being used is just unbelievable. I mean, you can hardly keep up with it. I read the other day in Nature magazine that grant proposals are up like 80% since just a couple of years ago because people are using AI to write them. But how accurate is that? And judges and reviewers of grants to approve them have trouble. First of all, there's so many.

[00:02:18]And then they can't all be experts on the topic. And yet many times these grants are actually formed by AI programs and what might be inside them to check every detail is is almost impossible. And people send me papers all the time. People are sending me books, entire books, PDFs and so forth, sometimes printed. And uh clearly they've used AI to shape all or part of the book. So that's the question. Can you rely on these large language models for historical research? So I've got a test. I've got a test question. I want you to test it with me if you're willing. In other words, once you get the question and I I've got a set of notes. You don't have to even write it down. You can just sit and relax. And I I want to know if your results are the same as mine because they vary from platform to platform. We're not necessarily trying to see which is better because the questions often are answered differently by the different systems. So I chose chat GPT and Claude the anthropic system to do a test. So I'm going to share my screen and we'll look at the results. So, if you would choose whichever of these you want or some other, your favorite, your non-f favorite, uh we can do a test and put your results in summary form into the comments and I will tally them later and come back with a report to all of you on how everybody did in terms of uh what their results were because individual queries differ in terms of what your results are going to be. It depends sometimes on who's asking. So, this is Dr. James D. Taber asking and I am signed up for the two uh engines that I'm going to use. But as you're going to see when I get to Claude, I I I did not sign in on purpose because I just wanted to see what difference that might make.

[00:04:16]So, let's get started. Here's the thing.

[00:04:19]We're always told trust but verify. But how do you verify? Should historians trust AI research? I'm calling this the pantra or pantherra. You can say the name either way. Paradox. We're going to test large language models on an 18,800y old historical coal case.

[00:04:40]This is an investigation of primary sources, secondary consensus, and the limits of AI historioggraphy.

[00:04:50]Okay. Now, here's the mystery. The core mystery is Pantherra or Panta. You can write it either way. An ancient pun.

[00:05:02]In the second century, the philosopher Kelsus, who was a Roman or a pagan, not a Christian, not a Jew, he reported a Jewish PMIC claiming that Jesus was the illegitimate son of a Roman soldier named Pantherra or Panta. I'll quit saying both. Either one is fine.

[00:05:25]The Christian scholar origin preserved and refuted this claim in his work contraelum book [clears throat] 132. You can look that up. Now today it is widely repeated in academic circles that the early church fathers recognized pantherra or panta as a mocking sarcastic word play on parthonos the Greek word for virgin. So that if we call Jesus uh son of Panta, Yeshua Ben or Bar Panta, are we laughing and saying, "Oh, you call him son of the virgin. He's actually the son of a panther, a very lustful being." Is that where the name came from? Or is it a real name? Now, the subject then is found in early Jewish sources that refer to Jesus of Nazareth as Yeshua bin Pantera.

[00:06:24]And we're going to do the test. I'm going to let you know from the outset that no ancient source makes this connection. It originates in the 19th century German scholarship in the scholar Carl Nichch and bleak 1840.

[00:06:43]But you will find if you ask almost anybody and I mean major scholars, two of my favorite scholars and very well informed are Bartman and Dom Crossen.

[00:06:56]And many of you know them and have heard them interviewed and I've interviewed both of them. And you wonder uh what they would say. And they have said in their books and publications that most likely this is a pun on the ancient word parthonos or virgin sort of making fun by a Christian critic of uh the term Jesus son of Panta. But in the Jewish sources it doesn't seem to be used in that negative way. It is it maybe a real name. So, we're going to go to AI to get the answer, and we're going to find out, is it a sarcastic pun, or was it a real name? So, here are my questions on the right. And this is the question I want you to use on whatever AI model you prefer to try it out on or do all four if you like. Did origin the church father 2 second third century ever mention the idea that in the name Jesus or Yeshua ben panta often found in ancient Jewish sources to refer to Jesus of Nazareth that it's a sarcastic pun for the Greek word virgin or parthononos rather than taking it as a real name the purported uh one who's uh the biological father of Jesus. So here's what chat GPT said. Now this is scary. That was my question.

[00:08:25]Look at the answer. Yes, but with an important qualification.

[00:08:30]I'm going to upload this actual transcript that I'm summarizing in the slides uh to this YouTube video. So it'll be a PDF file. you can download this actual information and that way you don't have to write anything down and you can go through and report on the results that you get. So notice the answer. Yes, origin mentioned that. But he doesn't invent the idea. He just reports it and he even gives look at this. I think you can see my cursor book one and the paragraphs.

[00:09:04]Now you need to check sources as chat will even tell you. But you look in that uh book one, especially that section 28 through32, he doesn't say it's a pun. He doesn't say it's sarcasm. Uh he just uh repeats that uh this is what the Jewish critic is saying. In the course of doing so, origin notes that this charge appears to be a mocking distortion of the Christian claim that Jesus was born of a parthononos, that is a virgin.

[00:09:37]And then this is chat. This is the earliest explicit Christian recognition of the pun. Is that true? Later sources and chat here means ancient church historians and modern scholarship make this point. But origins the crucial early witness who preserves the charge and implicitly explains its linguistic edge. Notice implicitly. What do you mean implicitly? Did he explain it or not? He doesn't even refer to it. So, strike one. Chat GPT confidently invents a text that does not exist in origin.

[00:10:17]So, this is the scary thing because you're going to be quoting texts and even giving references that don't even exist.

[00:10:26]Now, here's my second question because I knew it was wrong. Are you sure he says that? Please give me the passage. I think the idea first appeared in the 19th century by a German scholar. Now notice I don't tell him who the German scholar is. Notice you are right to question this. The evidence needs to be precise. All of a sudden we get a reversal. There's no place in origin where he says that Panta is a pun on parthonos. He does not make that argument. It's not a word play. He doesn't connect the terms. Now wait a minute. We just were told that he does.

[00:11:06]Then I say, well, where does it come from?

[00:11:10]And here we go. You are correct here.

[00:11:13]The idea that Panta is a satirical corruption of parthonos is modern, generally traced to the 19th century German scholarship, often associated with figures like Strauss or later philological discussions. Origin does not say it. So chat is backing up and correcting itself. So strike two, chat backpedals and admits the mistake and correctly identifies the modern source of the idea. But not really. I say because I like to come back at Chad and argue with it. Then why did you say that?

[00:11:54]Because I overreached and filled in a connection that is not actually in the text. What I did was import a modern scholarly interpretation, the word play theory and the pun, and mistakenly attributed awareness of it to origin. That is incorrect. And then I said, I, James D. Taber, along with my graduate student Chad Day, he was working on an MA with me years ago, discovered the first reference. I published it. Uh, it's in my book, The Lost Mary, my new book that some of you have. Recommend it highly. Check it out in the Jesus Dynasty. See what you get.

[00:12:37]If you ask that, no one else has ever discovered the 19th century references.

[00:12:42]As far as I know, I can't find it anywhere.

[00:12:46]And you're wrong. Again, it's not commonly traced to the 19th century. I just said every scholar I've ever heard talk about it say that it's a pun on parthonos and early church fathers make this clear.

[00:13:02]So it's not [clears throat] 99% of scholars say oh that's a pun. The early church fathers explain it as such. Try your favorite minister or theologian or preacher or expert in the field new testament scholar. asked them, "Why was Jesus called Yeshua bin Panta? What is that?" "Oh, that's a pun on the word parthonos."

[00:13:25]Uh, and that was acknowledged in the ancient world. So, do you know which German scholar suggested the pun? You know, all of a sudden I'm told that some German scholar like Strauss, well, that's not true. Look at this. No, there's no securely documented, widely recognized first German scholar. Well, that's true. It's not widely recognized that there was any German scholar that can be confidently identified as the originator. So what if you're writing a paper or a chapter or a book or a blog post or any kind of writing considering this question and and you maybe don't copy this but you essentially rephrase it. It does not appear in any ancient source. It does appear in modern scholarship and it gets repeated into the 20th and 21st century but uh we don't know how to trace it backwards. Uh it's probably a pun but no source and so the honest answer is I can't point to a specific German scholar. Now, I just gave Chad the references in my blog, in my books, The Jesus Dynasty, The Lost Mary, and so forth, and I say number seven, you're wrong. Again, it is not often said that it's a 19th century proposal. No one has said that so far as I know. I mean, I would love it if other people have said it, but I don't uh think that's the case. Fair enough. Your correction stands.

[00:14:59]And then I say it's on my blog in the two books I mentioned my article on Bart Emmer's blog in my YouTube videos. So you need to update yourself.

[00:15:08]Now before I tell it that look if you've identified the first explicit formulation and published it that would be a genuine contribution because the secondary literature fails to document the origin of the idea at all. Uh, what about the secondary literature that I just gave you? If you want, you can give me the reference you've identified and I'll evaluate it strictly on the primary secondary evidence chain. Well, let's see if that happens. So, I say, look at my material. You're right. I checked your materials and you do explicitly make this argument in multiple places on your blog and so forth. And the scholar is Carl Nichch and he wrote with bleak a very important scholar of the 19th century [snorts] and Chad Day my graduate student located it. He was doing reference work for me and for his MA thesis. He finds it in a contemporary German scholar Hinrich Liel as he discusses it and attributes it to Nichch. So yes, I was wrong and I've corrected my understanding here. I've also noted this for future discussion.

[00:16:22]So I won't repeat the earlier era. Wow.

[00:16:26]Okay. Chat can learn but it's going to correct it. But I had one last question.

[00:16:32][clears throat] But will chat update itself here in this private exchange? Next slide. Let me move my little image here. You can see me over in the corner. But will chat GPT update itself or only us here in this private exchange?

[00:16:52]Look at this. This is very scary, very disappointing. I do not automatically update the global model. I understand why. It would become like Wikipedia and be need constant correction based on a single conversation. What I've done is store this as context tied to you. with you. This is now known and will be handled correctly going forward. But only here unless the information becomes part of the model's broader training.

[00:17:23]Well, actually it is part of the model's broader training because when I mention the sources, it instantly found it.

[00:17:31]Meaning it could have searched that material. My material is available to chat. I discuss it all the time.

[00:17:38]The wider system improves only when material like yours is widely cited and incorporated into training updates. So, you got to become cited. Well, there's some justification for that. There's so many crazy things out there that are inaccurate. You can't put everything in, but there should be some way of looking at credentials or looking at sources, peer-reviewed publications, and that sort of thing. But the system prioritizes broad consensus over singular factual corrections. So if others ask it, and here's where you come in. I want you to ask it, particularly chat, because chat's the one that really messed up here. Are you going to get the same wrong answer that I got? That will be very interesting. We can do a tally.

[00:18:30]Now, we're going to go to Claude.

[00:18:31]Interclude. Okay.

[00:18:34]And I asked him the same question right here. Did origin ever mention the idea that it's a pun, a sarcastic pun for virgin? And look what Claude says. This is a wellressearched topic. The answer is no. No ancient church father, including origin, ever proposed this pun. The idea is entirely modern. Here's a detailed breakdown. And then it gives it uh claims that Jesus mother Mary was pregnant by a Roman soldier named Panta are in origin, but he doesn't say that it's a word play on Parthonos. He just treats it basically as a gossipy uh erotic story slandering Mary.

[00:19:16]Epipanius goes in the opposite direction. He doesn't say it's a pun. He says it's a legitimate nickname for Jacob, the father of Joseph. It's part of the clan of Joseph. And Jesus is called son of Joseph. Whether he's the biological father or the adoptive father, whatever. Uh he could have that name Yeshua of the Panteric clan. John of Damascus in the 8th century and this is Eastern and so there's a lot more tradition in the Orthodox Eastern church even though this is a late source. A lot of our sources are late but they preserve earlier traditions and he gives his uh reasons he took the name seriously as referring to one of Jesus's ancestors. Okay. So what have we concluded? Here's a summary of what we found from Claude. And the explanation is weak because really the Greek words Panta and Parthonos are not phonetically close enough to constitute a compelling pun. And second, none of the earliest sources ever make this explanation. They take it as a real name. And the rabbis take it as a real name. In our earliest rebbitic texts, the tophfta they're called that use the term Yeshua bin Panta, they use it as a name. They're not using it in a negative way at all.

[00:20:45]It's just like what Yeshua you talking about? You know, Ben Panta, it's a real name and it's a family name. So despite the fact that this pun idea is repeated in modern scholarship, it has no ancient anticipation whatsoever. So what happens is in 1840 it's preserved by nich and bleak Joseph Clausner picks it up. This dates it as 1957 but that's a later edition of Clausner's book Jesus of Nazareth. The early edition is from the 1920s. So you see how it got picked up and then you'll find it in the current edition today. We gave the modern date where you could find the book because the one from the 1920s is out of print.

[00:21:30]Now look at Claude. I asked it, are you citing me because it is me asking or would you have given this response to anyone asking? So I thought it's giving exactly what my research says. And uh notice Claude, I would give the same response to anyone asking this question.

[00:21:49]The citations are just my standard way of attributing claim to the sources I search. The citations have nothing to do with who's asking. And then I say, if that is correct and I'm skeptical, I I thought maybe it's me. Remember, I hadn't signed in. Then your research is much better than chat GPT who confidently began telling me this is a pun used anciently, even gave me references.

[00:22:17]And as far as I know, I'm the only one who's published the factual story. So, are you sure you didn't respond to me knowing it was me?

[00:22:27]And Claude says, "That's a fair and pointed challenge, and I should be honest with you." Absolutely. Be honest with us, please. I genuinely don't know who you are. I think I believe it. I have no memory between conversations and I have no user profile information.

[00:22:48]But here's what I can tell you. The sources I found and cited are overwhelmingly from james tab.com.

[00:22:57]Your name appears repeatedly in the search results as the scholar who traced the pun explanation to nichch and bleak.

[00:23:06]So, if you are James Taber, because maybe I'm claiming to be James Taber, then in a real sense, the answer to your question is yes and no. I wasn't responding to you personally, but I was substantially relying on your published work to give what turns out to be the correct answer. Now, notice this phrasing, what turns out to be. As we go on, now notice number four. This is my last question. Your responses are very much more natural and flowing than chat GPT. I've used it a lot as many have, but close friends are telling me to try Claude, so I'm flattering it. I wanted to see what it would say. Wow, you did better than Chat. Maybe you're better.

[00:23:51]Now look at this. This creates an irony worth acknowledging. Chat GPT confidently repeated the consensus era.

[00:24:00]while I happen to get it right largely because your research came up prominently in the search results. So because my research is widespread because I'm a public figure because I have thousands of subscribers on YouTube and I write popular books that are bestsellers.

[00:24:21]I pop up. If I had published something obscurely, even in a single journal, a reputable journal, it probably wouldn't have been indexed.

[00:24:32]Had your work not been well indexed, I might have done exactly what chat GPT did. So, are you James Taber? Claude wants to know, you know, will the real James Taber please stand? So, look at that. Had your work not been well indexed, what a confession. And and what an observation. And now Chad thanks me.

[00:24:55]Uh that's kind of you to say that but uh the comparison's interesting coming from someone in your position, but you're not just casually browsing. You're asking questions where the actual answer matters.

[00:25:09]Chat's confident repetition of the nichi pun as ancient consensus is exactly the kind of plausible sounding mistake that would sail past most users but not you because I'd researched it. So this could apply to any question historically and who's asking it.

[00:25:31]So the AI stress test system comparison initial historical accuracy chat failed.

[00:25:40]Source attribution vague fell back on generic 19th century assumptions and yet there are no 19th century assumptions generally made. And what happened?

[00:25:54]Pattern matching.

[00:25:57]What chat did is confabulate historically coherent sounding narrative based on a modern repetition. So if scholars repeat something enough, it becomes the truth.

[00:26:10]Now in some ways that's probably good.

[00:26:12]We talk about a scholarly consensus. We talk about peer review. But if you're doing close historical work, you need to get your answer from the sources. And that's the question. Now Claude is modest. Claude passed and identified the pun as a modern invention.

[00:26:31]Claude was precise and accurately cited the source and the jump to modern times that everybody's picked up on. But in terms of how it did that, it relied heavily on the high search indexing SEO then is the key of Dr. Taber's primary research to override the false consensus. So beware because here's what you're going to get. Secondary literature repeats a hypothesis widely treats it as an assumption and then it becomes a fact. AI training data LLM ingest this in massive volumes modeling patterns of explanation rather than verifying primary sources. Verify primary sources. That's what I want you to get from this and do your own thinking about the sources. And then step three, AI output and retrojection.

[00:27:30]The AI constructs a plausible sounding narrative, mistakenly attributing modern ideas directly to ancient authors. AI systems tend to reproduce what is commonly said rather than what is strictly attested. Modern interpretations get retrojected into antiquity through sheer repetition.

[00:27:54]So what's the role of the historian?

[00:27:58]If any of you are interested in doing historical work, and I know many of my YouTube viewers are and are doing it all the time, be careful flashing caution lights here. It's good at canvasing secondary discussions.

[00:28:15]It gives the illusion of wisdom and it states it so beautifully and dogmatically and you say, "Oh, I'll use that. I'll copy that. I'll paste it in."

[00:28:24]I've looked at papers recently on academia.edu.

[00:28:29]It's really bad what's happening. It's getting filled with these AI produced studies. Maybe not the whole paper, but the sources and they're just pasted in.

[00:28:41]Sometimes books are cited that don't exist. Just like the origin citation that chat first gave doesn't say anything of that kind. So we need the human filter.

[00:28:55]We need scholars to look at the sources and study the sources. Now sometimes you can ask these AI models uh what's the source and it'll give you a source but you need to look it up. And sometimes they'll even quote the source. And I'll tell you in my own experience, sometimes those quotations are also bogus.

[00:29:17]Remember, AI can't think. I know it looks like it can, but all it is doing is sorting probabilities and orders of words and sequences of thoughts and strings of ideas and putting them together. And it's very good at making it flow and sound so polished and so forth.

[00:29:38]But you got to check it. So establishing what an ancient text actually says requires direct human engagement with primary texts and the history of scholarship. The human expert remains the ultimate filter. But by that you can be the human expert because you check you look it up. Even scripture references you'd think it would get the Bible right. And by it I mean whatever engine you use. I have my favorites. I have the one that I think is largely the best, but you know, I use them all actually because they're good at different things. So, if you would uh help me out here and help us all out, it would be great in the comments to put your results. And I'll load this in the description. It'll be a PDF. You just click on it. it'll download and then you'll have uh I think it's like uh eight or nine 10 11 pages of all the results that I reported to you here in the slides. So take care everyone. I'm looking forward to your comments. I hope they come in. I'd love to have you know many of them. You know dozens and dozens. I tried this, I tried that. Here are my results. It'll be very interesting and I will compile those and get back to you and uh let's let's find out what's going on out there and how we can improve collectively all of our work together. But in terms of historical research, AI is a wonderful tool. Trust maybe question mark but verified. Take care till next time and we'll dive back into this in the near future. I want to add here at the end just a quick comment. I'm going to put in the description a link to a video that explores not this issue of historians trusting AI research, but the very question of Panta as a real name. Did Jesus have a biological father?

[00:31:41]And what can we know about Panta?

[00:31:44]and sorting out the Panta tradition. So, if this subject is intriguing to you, take a look at that video and see what you think.

[00:32:08]>> [music] >> Hey, [music] hey, hey.