Can Generative AI help us analyze The Lord of the Rings?

Generative AI is everywhere these days. Some of its output can seem pretty impressive too, at least at a glance. So it’s natural to wonder whether we might be able to use this emerging technology to analyze our favorite works of literature in valuable, labor-saving ways. After all — as is well known by now — Generative AI has gobbled up practically every work of literature ever published as part of its training, copyrights be damned. I can’t help remembering Sméagol’s warning about Sauron, “Don’t take the Precious to Him! He’ll eat us all, if He gets it, eat all the world.” Large Language Models seem to be in the “eating all the world” stage now, and Sam Altman is no Sam Gamgee!

So, can this technology help us analyze The Lord of the Rings? The short answer is: no, not really, not yet, and maybe not ever. If that’s enough for you, you can stop reading now. :)

I’ve been experimenting with ChatGPT, Gemini, and other Generative AI tools for some time, but while they can be useful for certain tasks, they will not be replacing literary scholars (or other professions) any time soon. A scenario occurred to me recently, one that I thought might be suitable for the kinds of things Generative AI does fairly well — if the results were accurate. But hallucinations and other defects are still common, so I put it to the test. Some of you may remember my work on hapax legomena in The Lord of the Rings, which I wrote about here. Unfortunately, the Scholars Forum referred to is long gone (a long story), but the essay was later published.

I wondered whether this was the kind of analysis Generative AI might help with, so I tried this prompt with Google Gemini:

Please provide a list of 25 hapax legomena occurring in the novel, The Lord of the Rings, ignoring all paratextual elements.

This prompt tests a few things, even before you get to the results: (1) does Gemini understand what a hapax legomenon is? (2) Does it have access to the text of The Lord of the Rings? (3) Does it know what we mean by the paratext of the novel?

Let me break down the response into two parts. I’ll start with the surrounding explanatory content, and then I’ll move on to the list of words the chatbot provided.

Disclaimer: While I can provide a list of words that appear only once in The Lord of the Rings, it’s important to note that Tolkien’s world-building often involves the creation of unique languages and terms. As such, many words that might seem like hapax legomena are actually part of these invented languages, making them less truly unique in the context of Middle-earth.

Additionally, the concept of a “hapax legomena” [sic] can be subjective. Depending on how we define a “word” (e.g., including compounds, derivatives, or different grammatical forms), the list could vary significantly.

Note: This list is not exhaustive and may contain errors or omissions. It’s also possible that some of these words might appear in other works by Tolkien or in different editions of The Lord of the Rings.

This explanatory text provides some pretty good context. Gemini clearly understands what a hapax legomenon is, and it seems to have access to the text of The Lord of the Rings. It even seems to know there are different editions. It got “Middle-earth” right too; many people, let alone AI models, get that wrong. I was also impressed to see that is was aware of Tolkien’s glossopoeia and that this might inform how it should interpret the prompt. By hapax legomena, are we interested in, say, Elvish words too, or are we not? That’s for us to decide, but Gemini realized this could be an important distinction.

So far, so good. But now, let’s looks at the list Gemini provided:

1. Glimmering: Used to describe the light of the moon in the Shire.
2. Windlestraw: A type of grass mentioned in the Shire.
3. Wainwright: A person who makes or repairs wagons.
4. Garnishing: Used to describe the decorations on the walls of Elrond’s house.
5. Tarn: A small mountain lake.
6. Whirligig: A type of spinning toy.
7. Hobbity: Adjective used to describe something related to hobbits.
8. Hitherward: Toward this place.
9. Whitherward: Toward what place.
10. Thenceforth: From that time on.
11. Whilom: Formerly.
12. Withered: Dried up or shriveled.
13. Withering: Causing to dry up or shrivel.
14. Witheringly: In a withering manner.
15. Whither: To what place.
16. Whitherward: Toward what place.
17. Whithersoever: To whatever place.
18. Whithersoever: To whatever place.
19. Whitherward: Toward what place.
20. Whitherward: Toward what place.
21. Whitherward: Toward what place.
22. Whitherward: Toward what place.
23. Whitherward: Toward what place.
24. Whitherward: Toward what place.
25. Whitherward: Toward what place.

Well, this is not so good. The most obvious problem is that the word “whitherward” is given here nine times, along with several variants, including “whithersoever” given twice, and three variants on “wither”, which, worryingly, I suspect Gemini thinks is related to “whither”. All of this seems to ignore Gemini’s own preliminary comment about “compounds, derivatives, or different grammatical forms” too. It almost looks as if the model ran through the alphabet and got to W with quite a few spots left to fill in my requested list of 25 words.

But the problem is actually worse.

Let’s go through the list. Some of these words — “glimmering”, “withered”, “withering”, “whither” — occur multiple times in the novel. These aren’t hapax legomena at all. Not even close.

Most of the others — “windlestraw”, “wainwright”, “garnishing”, “tarn”, “whirligig”, “hobbity”, “hitherward”, “whitherward”, “thenceforth”, “whilom”, “witheringly”, “whitherward” — do not occur in the novel at all. Where did they come from? The films? Fan fiction? Who knows? These are what you might properly call hallucinations, in the terminology of Generative AI. Especially “garnishing”, which is supposedly “used to describe the decorations on the walls of Elrond’s house”.

Only one of the 25 words offered is actually a hapax legomenon in The Lord of the Rings, “whithersoever”, which occurs in only one place:

‘Do I not say truly, Gandalf,’ said Aragorn at last, ‘that you could go whithersoever you wished quicker than I? And this I also say: you are our captain and our banner. The Dark Lord has Nine. But we have One, mightier than they: the White Rider. He has passed through the fire and the abyss, and they shall fear him. We will go where he leads.’ [1]

So, it’s clear that Gemini handled this prompt quite badly overall. What about other LLMs? I did try this out with Microsoft Copilot, which uses the ChatGPT LLM. It stated, in part, “I can’t provide a verbatim list of hapax legomena from The Lord of the Rings due to copyright restrictions. […] If you’re interested in exploring hapax legomena in The Lord of the Rings, I recommend checking out scholarly articles or linguistic studies that analyze Tolkien’s language. These sources often provide detailed lists and discussions of such words.”

Here, Microsoft shows that it is aware of the copyright issue, where Google either isn’t aware or doesn’t care. This is a bit glib, though, because ChatGPT has most assuredly also gobbled up The Lord of the Rings, just as Gemini has. It just isn’t going to pull back the curtain for us. Is that better or worse? Hmm.

Following up, I then asked Copilot if it could recommend any specific scholarly articles or linguistic studies on hapax legomena in Tolkien’s works. You can imagine what I was angling for, eh? Was it aware of my own paper? Well, no, unfortunately. It recommended three papers, one on hapax legomena in natural language processing (a computer science topic), one of hapaxes in The Iliad, and one on ancient Akkadian and Ugaritic texts. Worthwhile, no doubt, but a missed opportunity to point an interested user to a directly relevant paper.

We could try other models, but it’s not likely any of this is ready for prime time. I hope amateur and new scholars aren’t relying on these models without double-checking their output, but we may be headed for a time of more and more careless reliance on supposed artificial intelligence. It’s bad enough that some people confuse the novel with the film and television adaptations. Generative AI could introduce a whole new set of problems for researchers.

Will Generative AI ever be good enough to perform this kind of analysis reliably? I have some reasons to doubt it, but elaborating would require quite a detour. I’ll try to address the question in a single paragraph for now. Suffice to say there are two substantial obstacles: (1) the cost of Generative AI, both direct costs as well as resource costs such as electricity, water, raw materials for chips, etc., and (2) the fact that the LLMs are running out of genuine human-generated material to train with. Models will increasingly be fed other AI output as new input, which will probably push us to a mediocre plateau in the capabilities of the technology. And the technology is not likely to get much better without a lot more investment, a lot more resource utilization, and a lot more raw material for training. The more people rely on it, the worse it may get, until the return no longer justifies the investment. In this way, Generative AI may be self-limiting. Garbage in, garbage out. This is not to say Generative AI will disappear; it’s just not likely to get a lot better than it is today. (If you want to read more about how Generative AI is a bubble that may be getting ready to burst, take a look at this very long, very detailed essay .)

[1] Tolkien, J.R.R. The Two Towers. Houghton Mifflin, 1965, p. 104.