Discussion about this post

User's avatar
Coagulopath's avatar

I wonder if the LLMs are intentionally underperforming because the incident is so strongly connected to bad poetry (Claude even gives McGonagall a shoutout!). It would be interesting to test with poetry about an unrelated disaster...

R1's effort was definitely the best, though some of its lines are a bit cringeworthy: "Plunged into Tay’s black throat, where darkness dwells." Yes, black throats often have a bit of darkness dwelling inside them, if you look closely.

It's fascinating how these are all fundamentally the same poem. They have all have AABB or ABAB rhymes, four lines to a verse, 6-8 verses, and they have a rigid sense of narrative about them—they describe the disaster in linear time, before concluding with a preachy, moralizing lesson about the folly of human pride and hubris.

There's nothing wrong with this, but the fact that ALL the poems are like this is telling. (McGonagall's poem exhibits more creative freedom than all of the AI poems put together—almost certainly due to the author's incompetence, it must be said!)

It's a great example of "mode collapse"—the way AI models trained on human preferences end up converging on the same kind of bland samey slop. It used to be even worse—GPT4, at launch, would refuse to write poems that didn't rhyme. Even when instructed not to rhyme, it would just ignore the user and write rhymes anyway. A clear artifact of human preference training—the thousands of Kenyans OpenAI paid $2/hr preferred rhyming poems, and thus GPT4 ended up with a pathological fixation that ALL poems must rhyme, come hell or high water.

Expand full comment
Petter's avatar

The new ChatGPT 4.5 from yesterday produced the following:

***

Upon the Tay stands sturdy might,

A bridge of iron in stormy night,

Engine's scream and whistle's breath,

Unaware it rode towards death.

***

December's sky was dark and stern,

The wind that night no mercy earned,

Gale and tempest wildly spun,

Eighty lives on board undone.

***

Iron girders twist and strain,

Buckled under wind and rain,

Structure trusted, yet betrayed—

Into river's arms they laid.

***

Shattered beams beneath the wave,

Testament to bridges brave,

Yet lessons hard, disasters taught,

Sturdier spans thereafter wrought.

***

Today, beneath calm Scottish skies,

A silent monument abides,

Whispering still in tides below,

Stories tides forever know.

***

For bridges built between the banks,

Let's pause to give our quiet thanks—

But ne'er forget all lost that day,

When storm laid claim upon the Tay.

***

Will be interesting to see what it can do when it starts reasoning like the o models and Deepseek.

EDIT: Of course the formatting was ruined. Maybe this will at least be readable

Expand full comment
7 more comments...

No posts