GPT-Neo, the Generative Pre-trained Transformer for the masses
In May of 2020 OpenAI, a San Francisco-based artificial intelligence research laboratory introduced to the world its third-generation language prediction model with 175 billion machine learning parameters. That is almost 10 times more than the nearest other language models that have 17 billion parameters. Text generated by GPT-3 shows the unprecedented quality and can not be distinguished from text written by a human. A lot of prominent personalities have expressed their concerns that "misinformation, spam, phishing, abuse of legal and governmental processes, fraudulent academic essay writing and social engineering pretexting" can be easily done with this language model. Because of this access to GPT-3 is restricted. Usually, the mandate of OpenAI was to provide opensource access to artificial intelligence but GPT-3 disruption possibility was deemed too high for access to it to be without any constraint. Besides, the GPT-3 model is so huge it cant be run locally on most machines.
You can play around with GPT-3 on OpenAI playground but you need to join a waitlist on their website.
There are smaller models that we can play around with to get a sense of what such large language models work. Let’s look at EleutherAI, a grassroots collective of researchers working to open-source AI research. Their models were thought on The Pile: An 800GB Dataset of Diverse Text for Language Modeling and can be found on their GitHub. The models implement GPT3-like models using the mesh-tensorflow library.
Google Colab file for your testing and experimentation can be accessed here. For more on how to use Google Colab check here.
The tutorial uses GPT-Neo. There is a newer GPT model provided by EleutherAI called GPT-J-6B it is a 6 billion parameter, autoregressive text generation model trained on The Pile. Google collab is provided as a demo for this model. Check it out here.
But here we will use GPT-Neo which we can load in its entirety to memory.
There are two main versions of GPT-NEO, one with 1.3 billion parameters and one with 2.7 billion parameters. This is way less than other language models but it’s enough that we can load it in default Google Colab jupyter notebook and can utilize Google CPU and GPU for computation. The only model we can load in Colab is the 1.3b model, the 2.7b is too big for the free version of Google Colab. If you have Pro access to Colab you can load the 2.7b model just by instead of
generator = pipeline('text-generation', model='EleutherAI/gpt-neo-1.3B')we change the model number from 1.3 to 2.7 and a larger model will load.
generator = pipeline('text-generation', model='EleutherAI/gpt-neo-2.7B')But before that, we need to install PyTorch and transformer library for python.
pip install torch==1.8.1+cu111 torchvision==0.9.1+cu111 torchaudio===0.8.1 -f https://download.pytorch.org/whl/torch_stable.html
pip install transformersTransformers provides thousands of pretrained models to perform tasks on texts such as classification, information extraction, question answering, summarization, translation, text generation and more in over 100 languages. Its aim is to make cutting-edge NLP easier to use for everyone.
The python code is super simple as libraries are doing all the work for you.
from transformers import pipeline
generator = pipeline('text-generation', model='EleutherAI/gpt-neo-1.3B')
prompt = "When elephants began to fly people were confused as to how"
temp = 0.9
output_length = 200
res = generator(prompt, max_length=output_length, do_sample=True, temperature=temp)
print(res[0]['generated_text'])That’s it. The model will be downloaded from the list that transformers curate and loaded in memory. We set up our parameters. The main one is Prompt. Enter any line of text you can think of, adjust the confidence level which is by default set to 90% or 0.9, and set how many words to output. Once all of this is set up click the play button on the left. It will take some time to run, usually a couple of minutes.
Here are some of the prompts and their output:
When elephants began to fly people were confused as to how.
There was the legend of Taranis, the flying serpent that was killed and its body split into two halves. Another claimed it was the body of a monster or giant, not a bird.
On a smaller scale, people believed that the first creatures to fly were those that lived in the forests – that is to say, animals that weren’t people.
To account for this, people came up with the idea that the first flying creatures were birds. This idea is still used in some myths.
It was not until the late 12th century that the first written account of a flying animal was found. The story of the flying serpent was first told by a Portuguese writer called Alonso Álvarez de Toledo.
The writer came across a small island in the Bay of Panama and he took a closer look at the islands inhabitants. He wrote in a Latin manuscript:
Should have let it return longer output because I’m very interested in what was written in the manuscript but it is what it is. Rerunning it would give different output. Regardless, the returned text is more or less coherent, and just imagine, this is 100 times smaller than GPT-3 model and still, the possibilities of these technologies can be seen.
After returning the code it returned:
When elephants began to fly people were confused as to how. They didn’t believe they could fly so long as people had control over them. And when the first great gilded statues of the fabled beast were built in China, people were even more baffled until it was realized that they were flying automatons with a human master in attendance. Gilded statues were built of a mixture of bronze, ivory, and sandstone. Over time people began to believe that the fabled flying elephants could travel without being observed by the person in charge. After the Gilded City of Angkor, people began to realize that they also couldn’t ride or walk through the city. As a result they started building smaller versions of the great golden statues. But soon the great Chinese were realizing that the flying machines they were building were not even that great an improvement on the great temples of the past. There were several reasons for this. First, the Chinese had a long
To end with something silly. Here is a text about Unicorns in the forest.
There are Unicorns in the forest
There are Unicorns in the forest and people in the village. Some people believe them to be a manifestation of the God’s love, but others have them as their friends. But if you cross paths with the unicorns, they will take you to the heavens.
One time, I was on a visit to my friend’s parents and I wanted to go to the lake for a small drink and relaxation on a sunny afternoon. But there was a time when there was no lake, neither was there a river nearby. On our way, we saw a herd of unicorns. We stopped and talked to them. One of the unicorns started to speak. So, I asked him the reason for his calling. The unicorn told me about a place far away and asked me to get there. I said yes. He took me to the place where there was a lake. The lake was surrounded by blue and white clouds. There I saw a forest with a waterfall.