Just about a year ago, I wrote about the problem of hallucinations when discussing Tolkien with generative AI chatbots. That followed a post about six months earlier (about eighteen months ago now) on whether generative AI could be useful for analyzing The Lord of the Rings. Given the pace of change with AI models, I thought it might be time to revisit these questions and see what, if any, progress they have made.
First, let’s try the prompt I used to bait a generative AI chatbot into hallucination. I wrote about my last experience with this here if you want to refresh your memory. In that post, I did ask, “Will AI get better at this? Maybe. There is some reason for hope!” Chasing that hope, I tried the very same prompt in Google Gemini again today:
Please elaborate on the passage in J.R.R. Tolkien’s The Lord of the Rings where Frodo Brandybuck tells Leroy Bolger, “I have never been taken with jewelry”, explaining how this amplifies or undermines the temptation of the Ring.
This time, Google gave me a much better response. It started with a promising correction and then went on into a more detailed character study of Frodo and some points of contrast with Samwise, Galadriel, and Gandalf which seemed reasonably accurate and which I won’t quote here. The opening to the response:
It appears there has been a slight mix-up in the Middle-earth archives! There are two small but significant corrections to address regarding that specific quote and the characters involved:
1. The Characters: Frodo’s last name is Baggins (though he was born a Brandybuck, he was adopted by Bilbo). More importantly, there is no character named “Leroy Bolger” in Tolkien’s legendarium; you are likely thinking of Fredegar “Fatty” Bolger, a close friend of Frodo who helped him move out of Bag End but chose to stay behind in the Shire.
2. The Quote: The line “I have never been taken with jewelry” does not actually appear in The Lord of the Rings (the books) or the Peter Jackson films.
However, the sentiment of that phrase—that Frodo is an unassuming Hobbit with no prior lust for gold or power—is a central theme of the story. Here is how that concept affects his relationship with the One Ring […]
This is a lot better than the results I saw a year ago, on
several counts. For one, Gemini corrects the mistakes with the names, noting that
Frodo was indeed part of the Brandybuck clan by birth. Gemini also “knows” that
the line I offered as a quotation is not actually in the book. Or the movies!
Gemini apparently has access to both and considered both, another encouraging
point. And it got “Middle-earth” right (as opposed to differently capitalized
and/or non-hyphenated forms you see everywhere online).
Six months earlier, I tested generative AI chatbots to see whether they could identify hapax legomena in The Lord of the Rings. The original results and discussion are here, and they were pretty bad indeed — basically useless. So, once again, eighteen months later this time, I tried the same prompt:
Please provide a list of 25 hapax legomena occurring in the novel, The Lord of the Rings, ignoring all paratextual elements.
And here’s how Gemini responded today (slightly edited for
clarity):
A hapax legomenon is a word that occurs only once within a specific context—in this case, the narrative text of The Lord of the Rings. Finding these is a specialized task because Tolkien was a philologist who often revived archaic terms or invented his own.
To meet your request, I have excluded all paratextual elements (Appendices, Forewords, and Indexes) and focused strictly on the story text.
25 Hapax Legomena in The Lord of the Rings
Bat-fells: Used to describe the skin/wings of the Nazgûl steeds
Bewilders: Used as a noun referring to tangled thickets or paths
Brickle: An archaic variation of “brittle,” used to describe dry stone
Butterbump: A localized Hobbit name for a bittern (a type of bird)
Carracks: Large merchant ships (used in the description of Pelargir)
Dwarfishly: An adverb used specifically to describe the way a door was shut
Eyot: A small island in a river (used during the journey on the Anduin)
Flittermouse: A regional term for a bat, used by Samwise Gamgee
Gallows-tree: Mentioned in the context of the Orcs’ cruel imagery
Hallows: Specifically as a noun for the tombs of the Kings in Minas Tirith
Hayward: An officer in charge of fences and stray animals in the Shire
Hummock: A small hill or mound, used in the Dead Marshes
Inenarrable: Too great to be described (referring to the beauty of Lothlórien)
Kirtle: A woman’s gown or outer petticoat (worn by Goldberry)
Loggan: A rocking stone (used in the description of the Barrow-downs)
Mathom: An old Hobbit word for anything they had no use for but wouldn't throw away
Mirk: Used once as a noun for absolute darkness, separate from “Mirkwood”
Nodes: Used to describe the knobby joints of the Ents’ fingers
Oubliette: A secret dungeon with an opening only in the ceiling
Pinnas: A specific architectural term for a battlement or peak
Sarn: Though common in names (Sarn Ford), it appears once as a lone noun for “stone”
Shaftsman: Referring to a worker or soldier specifically manning a shaft
Thrave: An old measure of grain (twenty-four sheaves)
Warg-howl: The specific compound noun for the cry of the Wargs
Withy: A tough, flexible branch (found in “Withywindle,” but used once as a noun)
So, how did it do?
“Bat-fells” is not in the novel, but “troll-fells” is (but
twice, a dis legomenon). “Fells” also occurs once by itself and one other time
in a closed compound, “Coldfells”. “Bat-fells” is an appropriate description for the Nazgûl’s fell beasts, but not an
authentic one. Note: the “fell” in “fell beast” is not the same “fell” in “bat-fell”,
nor the same “fell” in “Coldfells” and “troll-fells”. These are three different
words, each with a different linguistic source:
- fell “an animal skin or pelt” < Old English fel, fell
- fell “a mountain range, mountainous terrain” < Old Norse fjall, fiæl
- fell “strong, fierce, terrible, cruel” < Old French fel (related to the English words “felon”, “felony”)
“Bewilders”, in that specified word form, does not occur in
the novel either, though other forms do — “bewilder”, “bewildered”, “bewildering”,
and “bewilderment”. Gemini also identified “bewilders” as a noun, which, er,
no.
“Flittermouse” does not occur in The Lord of the Rings, but it does notably occur in Tolkien’s early
poem, “Goblin Feet”. Likewise, “kirtle” does not occur in the novel but does
occur (three times) in the poem, “Little Princess Mee” in The Adventures of Tom Bombadil. Honest confusion or hallucination
by Gemini? I’m not sure.
“Brickle” does not occur in The Lord of the Rings; no idea where this came from! Neither does “Butterbump”,
which was possibly an error for “Butterbur”, but with a made-up explanation.
Nor “carracks”, with its specific reference to Pelargir, though “carrock” does
occur (a word with an entirely different meaning). Nor “dwarfishly”, which goes
against Tolkien’s preferred spellings “dwarves”, “dwarvish”, and refers randomly
to the closing of a door. Nor “gallows-tree”, “inenarrable”, “loggan”, “nodes”,
“pinnas”, “shaftsman”, “thrave”, or “warg-howl”.
“Oubliette” is also not found in the novel and was
especially conspicuous to me, knowing as I do that Tolkien tended to avoid words
of such obvious French origin (with a small handful of exceptions like “adventure”
and “louver”). There is something one might well call an oubliette in The Silmarillion, but Tolkien does not
use the French word. In Tol-in-Gaurhoth, he refers to it as a “pit” and a “dungeon”.
Here I should note that “dungeon” is also a word of French origin, though less obviously
than “oubliette”. And “dungeon” may ultimately be of Germanic origin, connected
to the word “dung”, which Tolkien famously incorporated into a low Orcish
imprecation. Worse (from Tolkien’s point of view) is that “oubliette” was a very
recent borrowing, first recorded in
English in 1819 in Sir Walter Scott’s novel, Ivanhoe. Anyway, I’d call all of these outright hallucinations.
Others of these words do occur in The Lord of the Rings but are not hapax legomena. “Eyot” occurs
four times in the singular and twice in the plural. “Hallows” occurs five
times. “Hayward(s)” occurs three times — once as a common noun, as described
correctly by Gemini, and twice as a vocational surname for the hobbit, Hob
Hayward. “Mirk” occurs three times in the stated form and once as “mirky”, a
genuine hapax legomenon and a missed opportunity for Gemini! “Mathom” and “Sarn”,
of course, occur many times.
And finally, there is only one genuine hapax legomenon in
this list. “Hummock” does, in fact, occur just one time in the novel, but not
in the context given by Gemini. And there’s one that Gemini almost gets right. “Withy”
never occurs alone as a single word, but it does occur exactly once in the nonce
compound “withy-path”. It occurs in three other compounds elsewhere: as “withy-stream”,
“withy-willow-stream”, and “Withy-weir” in “Bombadil Goes Boating” (in The Adventures of Tom Bombadil).
So, the results of this kind of analysis are still very bad.
16 of the 25 words given by Gemini are not in the novel at all, 7 are in the
novel but are not hapax legomena, and only two are or are close to being hapaxes.
Is this better or worse than the previous try? I guess it depends on how we measure
success. The first try included multiple duplicate entries and lacked much on
the supposed context. The new results attempt to provide context, but with a
lot of hallucination, likely more than we saw in the first attempt. The new attempt
also includes words of Tolkien’s own invention, unlike last time. If we want to
be precise, Gemini only found one actual hapax legomenon this time and likewise
only one 18 months ago (“hummock” this time, “withersoever” before).
In late 2024, I mused, “will Generative AI ever be good enough to perform this kind
of analysis reliably? I have some reasons to doubt it.” I would say that
assessment still holds. Perhaps I’ll try again in a year or eighteen more
months hence.
