The emergence of transformers in 2017 raised the standard of neural networks. Until now, the core concept of transformers has been rebounded and remixed in multiple models. The results concluded that it surpassed the state-of-the-art benchmarks on several machine learning applications.
Nevertheless, currently, the transformers-based models are dominated in the field of natural language processing. Some of them are ALBERT, BERT, and GPT-N.
About GPT-NEO
Many might have heard of the most powerful AI tool, GPT-3, with potential applications such as writing emails, summarization, chatbots.
Although it’s not an open-source model and available only through beta API, one must have a complete application to join it.
But what if the developer doesn’t want to overhead with GPT-3? That’s where GPT-Neo comes in – an open-source model that is pretty much related to GPT-3 in terms of performance and design structure.
Open AI increases the computation power and amount of training data in the transformer series of models. Moreover, nearly fifty million were used for computing cost and five hundred billion tokens to train the GPT-3 model. GPT-3 is also ultimately more fitting for a company to purchase and utilize with a team rather than a single individual.
In 2019, Open AI introduced the GPT-2 model with 1.5B parameters but initially refused to release the trained model due to security concerns. Then later that year, they released it. Instead of offering it as an open-source model, they allowed developers to use the API.
In March 2021, EleutherAI introduced their first collective success as GPT-Neo, a GPT-like system. It was trained on TPU’s by using the library Mesh TensorFlow.
Comparing GPT Models
The best way to compare different versions of GPT is by measuring accessibility, performance and model size.
In terms of model size, GPT-Neo comprises 2.7 billion parameters. Whereas the GPT-3 offers 2.7 billion parameters to 175 billion parameters.
Table 1: GPT-Neo and GPT-3 parameter sizes as reported by EleutherAI
| Model Name | No. of Parameters |
|---|---|
| GPT-2 | 1.5B |
| GPT-3 Ada | 2.7B |
| GPT-Neo | 2.7B |
| GPT-3 Babbage | 6.7B |
| GPT-3 Curie | 13B |
| GPT-3 Davinci | 175B |
According to EleutherAI, GPT-Neo performed better than GPT-3 Ada model based on the NLP reasoning standards. The three benchmarks used to compare the performance of these transformer models are Piqa, Hellaswag and Winogrande.
Piqa is based on the reasoning parameter as to which sentence is more appropriate among two options, while Hellaswag is the completion of a multi-choice statement that has a context paragraph and multiple endings. Lastly, Winogrande impressively uses common sense to tackle pronouns in a sentence.
Table 2: Model performance measures as reported by EleutherAI
| Model | Winogrande | Hellaswag | Piqa |
|---|---|---|---|
| GPT-3 Ada 2.7B | 52.90% | 35.93% | 68.88% |
| GPT-Neo 2.7B | 56.50% | 42.73% | 72.14% |
Eleuther also introduced an open-source model named GPT-J with 6 billion model parameters. It’s worth noting that GPT-3 has more model parameters than GPT-J, but model parameters aren’t everything.
JPT-J performs better than the GPT-3 in case of code generation problems. Their primary goal is to facilitate safety research, particularly in low-resourced individuals.
In terms of computing power and availability, the whole trend towards GPT-N has changed since to train GTP-3, a significant amount of finance and resources related to cloud computing are required. Developers dealing with those growing demands should consider operationalizing gpt3 for AI applications with spell.
However, Eleuther planned distributed computing by incorporating multiple companies like Google, CoreWeave and TensorFlow research cloud. Which enabled them to split the total compute power for separate machines.
We already talked about the availability of resources for the GPT-3 and GPT-Neo models. The weight and configs of the trained GPT-Neo model are publicly available and can be easily downloaded.
You can view GPT-Neo progress on their GitHub repo here. With the help of GPT-Neo implementation, developers and researchers are getting results scaled up to GPT-3.
Future Development
Currently, the research community is working towards advancing GPT-Neo as GPT-NeoX with the help of computing and hardware capacities of CoreWeave. It will provide multiple features as model structuring, straightforward configuration, and 3D parallelism. GPT-NeoX is progressed by using DeepSpeed library.
The number of model parameters will also be much greater than the GPT-Neo. These are cardinal issues that we expect to resolve over time with the openness of AI tools, a price reduction of computation power resources, and fine-tuning of models with a large amount of data.
Artificial intelligence has a potential blow to deal with a significant language generation space.
Although these transformer-based models cannot cover all the nuances of human language – the type of applications and the level of accuracy still surpass present model capabilities. The rapid advancement in the GPT models shows more likely that the next breakthrough may be just around the corner.
