Generative AI, Hallucinations, and Tolkien Revisited

Just about a year ago, I wrote about the problem of hallucinations when discussing Tolkien with generative AI chatbots. That followed a post about six months earlier (about eighteen months ago now) on whether generative AI could be useful for analyzing The Lord of the Rings. Given the pace of change with AI models, I thought it might be time to revisit these questions and see what, if any, progress they have made.

First, let’s try the prompt I used to bait a generative AI chatbot into hallucination. I wrote about my last experience with this here if you want to refresh your memory. In that post, I did ask, “Will AI get better at this? Maybe. There is some reason for hope!” Chasing that hope, I tried the very same prompt in Google Gemini again today:

Please elaborate on the passage in J.R.R. Tolkien’s The Lord of the Rings where Frodo Brandybuck tells Leroy Bolger, “I have never been taken with jewelry”, explaining how this amplifies or undermines the temptation of the Ring.

This time, Google gave me a much better response. It started with a promising correction and then went on into a more detailed character study of Frodo and some points of contrast with Samwise, Galadriel, and Gandalf which seemed reasonably accurate and which I won’t quote here. The opening to the response:

It appears there has been a slight mix-up in the Middle-earth archives! There are two small but significant corrections to address regarding that specific quote and the characters involved:

1. The Characters: Frodo’s last name is Baggins (though he was born a Brandybuck, he was adopted by Bilbo). More importantly, there is no character named “Leroy Bolger” in Tolkien’s legendarium; you are likely thinking of Fredegar “Fatty” Bolger, a close friend of Frodo who helped him move out of Bag End but chose to stay behind in the Shire.

2. The Quote: The line “I have never been taken with jewelry” does not actually appear in The Lord of the Rings (the books) or the Peter Jackson films.

However, the sentiment of that phrase—that Frodo is an unassuming Hobbit with no prior lust for gold or power—is a central theme of the story. Here is how that concept affects his relationship with the One Ring […]

This is a lot better than the results I saw a year ago, on several counts. For one, Gemini corrects the mistakes with the names, noting that Frodo was indeed part of the Brandybuck clan by birth. Gemini also “knows” that the line I offered as a quotation is not actually in the book. Or the movies! Gemini apparently has access to both and considered both, another encouraging point. And it got “Middle-earth” right (as opposed to differently capitalized and/or non-hyphenated forms you see everywhere online).

Six months earlier, I tested generative AI chatbots to see whether they could identify hapax legomena in The Lord of the Rings. The original results and discussion are here, and they were pretty bad indeed — basically useless. So, once again, eighteen months later this time, I tried the same prompt:

Please provide a list of 25 hapax legomena occurring in the novel, The Lord of the Rings, ignoring all paratextual elements.

And here’s how Gemini responded today (slightly edited for clarity):

A hapax legomenon is a word that occurs only once within a specific context—in this case, the narrative text of The Lord of the Rings. Finding these is a specialized task because Tolkien was a philologist who often revived archaic terms or invented his own.

To meet your request, I have excluded all paratextual elements (Appendices, Forewords, and Indexes) and focused strictly on the story text.

25 Hapax Legomena in The Lord of the Rings

Bat-fells: Used to describe the skin/wings of the Nazgûl steeds

Bewilders: Used as a noun referring to tangled thickets or paths

Brickle: An archaic variation of “brittle,” used to describe dry stone

Butterbump: A localized Hobbit name for a bittern (a type of bird)

Carracks: Large merchant ships (used in the description of Pelargir)

Dwarfishly: An adverb used specifically to describe the way a door was shut

Eyot: A small island in a river (used during the journey on the Anduin)

Flittermouse: A regional term for a bat, used by Samwise Gamgee

Gallows-tree: Mentioned in the context of the Orcs’ cruel imagery

Hallows: Specifically as a noun for the tombs of the Kings in Minas Tirith

Hayward: An officer in charge of fences and stray animals in the Shire

Hummock: A small hill or mound, used in the Dead Marshes

Inenarrable: Too great to be described (referring to the beauty of Lothlórien)

Kirtle: A woman’s gown or outer petticoat (worn by Goldberry)

Loggan: A rocking stone (used in the description of the Barrow-downs)

Mathom: An old Hobbit word for anything they had no use for but wouldn't throw away

Mirk: Used once as a noun for absolute darkness, separate from “Mirkwood”

Nodes: Used to describe the knobby joints of the Ents’ fingers

Oubliette: A secret dungeon with an opening only in the ceiling

Pinnas: A specific architectural term for a battlement or peak

Sarn: Though common in names (Sarn Ford), it appears once as a lone noun for “stone”

Shaftsman: Referring to a worker or soldier specifically manning a shaft

Thrave: An old measure of grain (twenty-four sheaves)

Warg-howl: The specific compound noun for the cry of the Wargs

Withy: A tough, flexible branch (found in “Withywindle,” but used once as a noun)

So, how did it do?

“Bat-fells” is not in the novel, but “troll-fells” is (but twice, a dis legomenon). “Fells” also occurs once by itself and one other time in a closed compound, “Coldfells”. “Bat-fells” is an appropriate description for the Nazgûl’s fell beasts, but not an authentic one. Note: the “fell” in “fell beast” is not the same “fell” in “bat-fell”, nor the same “fell” in “Coldfells” and “troll-fells”. These are three different words, each with a different linguistic source:

• fell “an animal skin or pelt” < Old English fel, fell
• fell “a mountain range, mountainous terrain” < Old Norse fjall, fiæl
• fell “strong, fierce, terrible, cruel” < Old French fel (related to the English words “felon”, “felony”)

“Bewilders”, in that specified word form, does not occur in the novel either, though other forms do — “bewilder”, “bewildered”, “bewildering”, and “bewilderment”. Gemini also identified “bewilders” as a noun, which, er, no.

“Flittermouse” does not occur in The Lord of the Rings, but it does notably occur in Tolkien’s early poem, “Goblin Feet”. Likewise, “kirtle” does not occur in the novel but does occur (three times) in the poem, “Little Princess Mee” in The Adventures of Tom Bombadil. Honest confusion or hallucination by Gemini? I’m not sure.

“Brickle” does not occur in The Lord of the Rings; no idea where this came from! Neither does “Butterbump”, which was possibly an error for “Butterbur”, but with a made-up explanation. Nor “carracks”, with its specific reference to Pelargir, though “carrock” does occur (a word with an entirely different meaning). Nor “dwarfishly”, which goes against Tolkien’s preferred spellings “dwarves”, “dwarvish”, and refers randomly to the closing of a door. Nor “gallows-tree”, “inenarrable”, “loggan”, “nodes”, “pinnas”, “shaftsman”, “thrave”, or “warg-howl”.

“Oubliette” is also not found in the novel and was especially conspicuous to me, knowing as I do that Tolkien tended to avoid words of such obvious French origin (with a small handful of exceptions like “adventure” and “louver”). There is something one might well call an oubliette in The Silmarillion, but Tolkien does not use the French word. In Tol-in-Gaurhoth, he refers to it as a “pit” and a “dungeon”. Here I should note that “dungeon” is also a word of French origin, though less obviously than “oubliette”. And “dungeon” may ultimately be of Germanic origin, connected to the word “dung”, which Tolkien famously incorporated into a low Orcish imprecation. Worse (from Tolkien’s point of view) is that “oubliette” was a very recent borrowing, first recorded in English in 1819 in Sir Walter Scott’s novel, Ivanhoe. Anyway, I’d call all of these outright hallucinations.

Others of these words do occur in The Lord of the Rings but are not hapax legomena. “Eyot” occurs four times in the singular and twice in the plural. “Hallows” occurs five times. “Hayward(s)” occurs three times — once as a common noun, as described correctly by Gemini, and twice as a vocational surname for the hobbit, Hob Hayward. “Mirk” occurs three times in the stated form and once as “mirky”, a genuine hapax legomenon and a missed opportunity for Gemini! “Mathom” and “Sarn”, of course, occur many times.

And finally, there is only one genuine hapax legomenon in this list. “Hummock” does, in fact, occur just one time in the novel, but not in the context given by Gemini. And there’s one that Gemini almost gets right. “Withy” never occurs alone as a single word, but it does occur exactly once in the nonce compound “withy-path”. It occurs in three other compounds elsewhere: as “withy-stream”, “withy-willow-stream”, and “Withy-weir” in “Bombadil Goes Boating” (in The Adventures of Tom Bombadil).

So, the results of this kind of analysis are still very bad. 16 of the 25 words given by Gemini are not in the novel at all, 7 are in the novel but are not hapax legomena, and only two are or are close to being hapaxes. Is this better or worse than the previous try? I guess it depends on how we measure success. The first try included multiple duplicate entries and lacked much on the supposed context. The new results attempt to provide context, but with a lot of hallucination, likely more than we saw in the first attempt. The new attempt also includes words of Tolkien’s own invention, unlike last time. If we want to be precise, Gemini only found one actual hapax legomenon this time and likewise only one 18 months ago (“hummock” this time, “withersoever” before).

In late 2024, I mused, “will Generative AI ever be good enough to perform this kind of analysis reliably? I have some reasons to doubt it.” I would say that assessment still holds. Perhaps I’ll try again in a year or eighteen more months hence.