Why ChatGPT logic fails for video

[00:00:00]In the most straightforward implementation, we configure our neural network to take in the RGB pixel values from a sequence of video frames and then project the pixel values in the next frame, just as the GPT models are trained to predict the next token in language.

[00:00:14]However, when we use these models to predict the next frame, the results are blurry.

[00:00:18]And this blurriness compounds dramatically in longer horizon predictions. [music] Large language models are auto regressive. When ChatGPT answers a question, it generates one token at a time. At each step, it's feeding its latest generated token back into its input to create the next output. If we try this auto regressive approach with a next frame video prediction model, the results quickly devolve into blurry nothingness.

[00:00:41]Now, the blurry frames produced by our generative video prediction approach are not some huge mystery.

[00:00:46]Language is complex and unpredictable, but it's nothing compared to video.

[00:00:51]Language models use fixed-size vocabularies.

[00:00:54]GPT-2 has 50,257 discrete outputs, one for each token that the model could say next.

[00:01:01]This complete enumeration approach is hopeless in video.

[00:01:04]For full HD video in the most general case, each pixel can take on 256 discrete values, and we have 1920 * 1080 * 3 color pixels, meaning there are something like 10 to the power of 15 million possible next video frames, dwarfing the number of atoms in the observable universe.

[00:01:23]So, there's no way our video prediction model can have a discrete output for each possible next video frame.