Poetry Generating AI using GPT-2

About the project

I love reading poetry, but I'm hopeless at writing it. Thankfully, I'm (a little) better at writing code than poetry, so I wanted to see if I could develop a natural language processing (NLP) model to write poetry at the level of my favorite poets.

What was the goal of the project?

My aim was to create an ML model that could be trained on the poems of any specific author and generate poetry that sounds like their work. I planned to train the model on two of my favorite authors, Mary Oliver and Edgar Allen Poe, as their styles are very different and I was interested in seeing if the model was versatile enough to analyze both poets' writing.

Project execution

I initially tried to create my own transformer model to perform the NLP, but quickly realized that it would be more effective to use OpenAI's GPT-2 model, which is a massive pre-trained neural network with over 1.5 billion parameters. It's available for free on HuggingFace, and I used the smallest version of GPT-2 to save on computational time/energy, which has 124 million parameters.

For Edgar Allen Poe, I based my model on one developed by Scott Duda, who also trained GPT-2 on Poe. The model is relatively simple: process the text data, train/validate the model on the text, and then output the poem. Duda had two models, one trained on an individual lines of a poem and another trained on whole stanzas. I tested both and found that the line-based model performed better (in my opinion). Unlike most of the ML work that I do with biological data, in this project, there are no right or wrong "predictions", only subjective opinions of the results.

I also learned to not train GPT-2 on my local computer CPU. Not only did it take 12 hours, but it almost fried my computer.

To train on Mary Oliver, I scraped the text from an ebook of Devotions, which I split line-by-line. The same model was used on this data, although I used GPUs from Paperspace to train it. I used the Nvidia A6000 GPU, which trained the entire model in less than 3 minutes. I'd never used a GPU with that much power before and was shocked to see the epochs finishing every second.

Project results

The results of the model were pretty good! I was pleasantly surprised that the poems captured the style and emotion of each poet, although they did not have much meaning behind it. One of my poet friends said that she liked them, although "the ideas feel constrained by the language". Not terrible for an artificial intelligence!

Here are some of the poems I liked best:

Mary Oliver:

Poem 1:

When the sun stretches forward,
turning toward the sky,
I know what it is to feel
in the pulsing indigos of darkness.
I don’t doubt what a person wants to be—
to not only flutter but to have a pulse
where the great energies of the world
change us both into something better.

Poem 2:

All time and stars everywhere in the sky.
I hope to live forever,
I hope to climb a slippery sloping hill.
How many angels must you be?
How many fields you can see?
And how many trees can you grow
In the center of darkness?

‍

Edgar Allen Poe:

Poem 1:

The soul and the breast
Are more than flowers; their flowers are all ours
On earth the moon will fall
In a strange and curious moonlit sepulchre
In thy garden: and on the wall of this garden
Have you seen the moon?
I am not the moon. There are stars
On Earth that lie, dreaming,
In the sky that lie, dreaming,
On earth the moon must fall.

Poem 2:

We love thy name, and cherish thy life——or—but alas! alas!
I have known nothing of it—but thee alone—
With thy angelic wing, and thy pride—
With the holy angels, and the devil.
But alas! we are still haunted,
With fears and ill-gotten hours—
And of the time that now lies upon the Earth—
In the wild, unbroken, and unbroken.

‍

With AI becoming increasingly powerful (GPT-3 is already much better at NLP than GPT-2!), the lines between human and computer generated art are becoming more blurred. Hope you enjoyed this project and the poetry that the model created!

‍

Code on Github here: github.com/changbenjamin/Poetry-AI

‍

Some (real) Mary Oliver poems for you: