Abekek Notes

GPT-4 Will Be 500x Smaller Than People Think

Классный пост на тему каким будет GPT-4. Заметил много постов с инфографикой показывающей относительные размеры модели GPT-4 в сравнении с GPT-3 – полный треш по словам этого поста.

What GPT-4 Might Look Like:

To properly fit a model with 100T parameters, open OpenAl needs a dataset of roughly 700T tokens.
Given 1M GPUs and using the calculus from above, it would still take roughly 2650 years to train the model.

So, here is what GPT-4 could look like:
• Similar size to GPT-3, but trained optimally on 10x more data
• Multi-modal outputting text, images, and sound
• Output conditioned on document chunks from a memory bank that the model has access to during prediction
• Doubled context size allows longer predictions before the model starts going off the rails

Regardless of the exact design, it will be a solid step forward. However, it will not be the 100T token human-brain-like AGI that people make it out to be.

https://www.reddit.com/r/learnmachinelearning/comments/10fw2df/gpt4_will_be_500x_smaller_than_people_think_here/

Reddit

From the learnmachinelearning community on Reddit

Explore this post and more from the learnmachinelearning community

👍8

www.tgoop.com/abekek_notes/797

2.05K viewsJan 21, 2023 at 14:31

GPT-4 Will Be 500x Smaller Than People Think

Классный пост на тему каким будет GPT-4. Заметил много постов с инфографикой показывающей относительные размеры модели GPT-4 в сравнении с GPT-3 – полный треш по словам этого поста.

What GPT-4 Might Look Like:

To properly fit a model with 100T parameters, open OpenAl needs a dataset of roughly 700T tokens.
Given 1M GPUs and using the calculus from above, it would still take roughly 2650 years to train the model.

So, here is what GPT-4 could look like:
• Similar size to GPT-3, but trained optimally on 10x more data
• Multi-modal outputting text, images, and sound
• Output conditioned on document chunks from a memory bank that the model has access to during prediction
• Doubled context size allows longer predictions before the model starts going off the rails

Regardless of the exact design, it will be a solid step forward. However, it will not be the 100T token human-brain-like AGI that people make it out to be.

https://www.reddit.com/r/learnmachinelearning/comments/10fw2df/gpt4_will_be_500x_smaller_than_people_think_here/

BY Abekek Notes