Paradigm Shifts in Data Processing for the Generative AI Era: Robert Nishihara of Anyscale & Ray.io

[00:00:04]all right so welcome to the generative AI in the real world podcast today we have Robert nishihara co-creator of Ray and co-founder of any scale so Robert welcome to the podcast thank you so much good to see you again so uh we'll talk obviously about Ry uh and for people who are not familiar Ray is an open-source project that came out of Berkeley that has become kind of like the default AI compute engine the default Foundation layer and I've been writing about Ray for many years and recently I kind of write a series I wrote a series of post about how iOS observing companies building custom AI platforms and a common substrate seem to be rain across many companies but so uh the reason I bring this up is uh because Robert then has a great perch because he gets to work with a lot of companies that are on The Cutting Edge of AI but are also tackling some of the hardest problems in AI so I wanted to uh pick his brain about some uh key AI Trends so the first thing is about um data okay so uh we we hear about that people are running out of data uh the the teams training these Foundation models are running about data but then also that the data tools themselves are are lacking because everyone is focused on training and not enough on on on the data tool so give us Ro Robert kind of your uh high level view of the State of Affairs as far as data in in AI yeah well the the importance of data in AI is something that of course many people have have um identified and have known for a while but this is actually a really big paradigm shift in how the field of machine learning has been thinking about um Ai and if you go back a decade to The imet Benchmark imet is a uh one of the famous computer vision benchmarks where you have tons of images and you try to build models that can classify the images into different categories like cars cats dogs trees and so on and a huge amount if you think about the progress that this Benchmark drove in AI this Benchmark really drove progress in computer vision and all of AI for a long period of time and all of the innovation in this Benchmark was really around building better better model architectures uh trying to innovate on the optimization algorithms um to you know take your imag net data set train a model and and ultimately do better on object recognition but the interesting thing was that in this Benchmark you know the data set was considered static right you take your image net data set you split it into your train and test data uh then you train your model all the Innovation is on the model architecture now um you know the we're in a different Paradigm where um you know the the model architecture is is not completely static there's still Innovation there but it's more Transformers are very common there are a few standard model architectures at this point the optimization algorithm is much more static like we're all using variations of stochastic gradient descent um and where all of the Innovation is happening is on the data side is like um acquiring different sources of data evaluating which sources of data makes sense to train the model on curating the data um generating synthetic data like extracting um you know filtering out lowquality data so I actually think that um especially given how we know how important the quality of the training data is for the to impact the quality of the resulting model um and the way so the way people are thinking about curating training data and acquiring training data has really shifted over the years right if back to imag net the kind of data preparation or processing that people did was very limited you might crop the images you might scale the images to try to um generate you know to augment your data set now it's you're using AI to really process and curate the data that's the big shift so for example imagine you're working with a bunch of video data right you want to filter out low quality data how do you do that that's an AI task right to determine which or you're working with an autonomous uh vehicle company right you have self-driving cars they have hundreds um you know millions of miles of driving uh tons of video footage and you're trying to um find the most interesting data to train on a lot of the data might just be driving down an empty road for a long stretch of time nothing interesting happening some of the data is a pedestrian doing some crazy thing um something you've never seen before that is not all data is created equal right some data is far more informative or valuable than others and so a lot of the process is um taking all of your data and extracting the most informative most interesting highest quality data to train on and that is very much an AI task there's a lot here and and uh Robert for for the most part uh a lot of the tools at least uh in the in the Resurgence of data engineering tools five years ago which was centered around the pipelines and orchestration tools for uh structure data mainly in that warehouses and lake houses right yeah so now we have unstructured data but increasingly as you point out multimodal data yes so what uh what What's the challenge as far as the tooling around working with that same data and trying to do the same sorts of data processing and data prep that uh people are accustomed to lots of companies have lots of data right and they've acquired all of this data and they're storing all of this data because they believe they can get insights out of the data right they want to get value out of it they want to make decisions and the way they get value out of the data today is by running SQL queries they'll run SQL queries they'll run some simple analytics uh to learn about their data and to explore it for structured data for structured data right and because not because they don't have unstructured data but because you can't really touch that with your SQL queries your SQL queries are not going to operate on PDFs or on videos or images or you know arbitrary text documents um but the reality is that what you can do with SQL queries is quite limited right the amount of insight you can get if I compare imagine I'm a uh a company and I'm trying to get insight about my sales pipeline right the kind of SQL queries that are easy to run are things like asking about which companies have been in certain stages of the pipeline for what certain durations um things pre-built fields and categories that I've already thought about but all of the Insight is in the notes that the sales representatives are taking in the videos of the recorded meeting calls all of these kinds of things and you're not um getting any value out of that today so I think the way that people will get insights out of data in the future is with AI right they'll have ai read the data reason about it and draw conclusions and that is a very different Paradigm so I imagine data processing shifting from really being a SQL Centric workload to an AIC Centric workload and if it's an AI Centric workload it's going to be a GPU Centric workload or perhaps mixed CPU and GPU workload and the tooling around workloads that are uh very GPU and AI intensive and very data intensive is practically non-existent right this is a or it's very immature this tooling for multimodal data processing so that's a challenge yeah Cu uh in in a in a future pipeline you can imagine I'm you know I'm I'll draw a few examples here right so I have a piece of text and then I remove personally identifying information then I correct grammatical errors or translate foreign foreign terms and then I embed right so so so there's like a this complicated pipeline that uh is not really uh going to be addressed by kind of the data engineering tools from the structured world right yes you're gonna have ai to do that task that you just described yeah yeah and then and then secondly as you point out uh for in certain parts of the pipeline maybe you can use CPU instead of GPU right so so to say that okay uh we're now in the AI world it's unstructured I'm only going to use gpus for data processing I don't think that's going to be economically feasible right no I don't think that's feasible um data processing is not just running inference with an llm or with some model right there's lots of other regular processing if you have um video data you might want to decompress the video right you might want to re-encode the video and formats that are better for Downstream consumption right you might want to find scene changes between uh different clips in the b in in the movie or the or the the scene uh you may want to do transcription right you may want to do run a vision language model to uh generate text descriptions you probably want to run a lot of classifiers to extract uh to annotate uh the video and extract structured information right there's some uh stages will be GPU bound some will be Memory bound because videos are massive right there's some will be CPU bound and you're um going to want to move toward an architecture where you can disaggregate these different resources where if you if gpus are the bottleneck for a certain stage of processing you can scale those up if CPUs are bottleneck for a certain stage of processing you can scale those up and of course you want to be able to stream all of the stages of processing together so that you can keep your uh gpus busy while um the whole time right you're not you don't want to just do one stage of processing wait for that to finish finish then do the next and so on and and all of these things are areas where the tooling um has a long way to go and and most likely in in the types of data that you're describing which you know I mean you can think about a billion images or a few hundred hours of video I mean it's it's almost uh assumed that you you will have to go distributed and scale out right yes there is uh no choice but to really scale the computation the data is not going to fit on single machine it's not going to you're not going to be able to process it in any reasonable amount of time and so everything you've discussed uh so far around the this uh data challenges around multimodal data is this a real problem today or is this a are you forecasting in the future this is something people are are running into today and a lot of by the way um in the past we were really mostly getting value out of structured data because we could we had the tools to process structured data and and query structured data um we weren't really thinking about what we could do with unstructured data with other types of multimodal data with and and if videos I keep mentioning video as an example because um there it's some a type of data where there's so much value in it and yet we are not even scratching the surface of what we could be doing with video data and as a result in when previously you're in this regime where you can't get much value out of video data or images or other types of data and so people weren't collecting it as much right because there's less incentive to collect it now that generative AI is really unlocking the ability to get value out of all these different types of data we're going to collect a lot more data right because we're getting more value out of it and when you talk about the companies we see that we talk to many of them are looking around uh at the scaling laws seeing that um wow if I train a bigger model with more compute and more data I can get better results that's almost a a formula for just getting better results right there's subtlety to it but um it kind of gives you a road map and so all of these companies we're talking to are saying hey we need to enable training on 100x more data right because that's the path to getting better results um and when you of course the challenge there is that you didn't build your internal systems to hand it handle 100x more scale right you don't build that build that far in advance um so now to really enable to build the next model and release get the next um you know ship the next AI feature all of these ml platforms ml infrastructure is really on the critical path for for building and shipping that model right and of course it'll be on the critical path for the next model after that because you need even more data and so we're finding a lot of responsibility and and stress on these ml infrastructure teams who are now um you know on the critical path for delivering these kinds of results so actually as I was ask uh asking you that previous question I I started uh remembering it at this at the last race Summit in September in San Francisco there were two talks that caught my attention that I sat in on both of these talks one was from bite dance uh where they were processing petabyte scale uh data which was mainly audio and video pipelines and then the other one was from Pinterest uh and I think that was for uh uh model training or model training for their large really large scale recommender system so again they had to do kind of this dance between CPUs and and gpus but uh I guess my question is uh both companies I just named are super tech forward companies right so digital first companies so do you imagine um what you're describing is something that regular Enterprises will also have to Grapple with maybe not today but soon I mean there are always companies that are early adopters right and I would say when it comes to AI um a lot of the market is still very early right a lot of companies are very much in the exploratory phase of figuring out how to use AI in their business right should they build a chat bot do they build inter internal co-pilots how do they really incorporate AI in a native way into their products um so that is true now companies like bite dance and Pinterest are very much at The Cutting Edge of at the frontier of running Ai and production they've been doing this for a long time um and of course if anyone um knows how to work with multimodal data it's going to be these companies because of just the nature of their products now that said every company or nearly every company has a lot of data has its own data has data about its specific business and the way its customers use its product um that they can use to improve um that they can use to improve uh their uh their their platform so this is something that we see every company is going to want to get insights out of data every company is going to want to use dat to uh make decisions and the that the value is there the potential is there the challenge is just the tooling and the infrastructure to really make that possible actually there's another uh interesting angle around data in scale which which ties into this uh uh uh new world of of really large scale uh AI models which is experiments and experiment right so so take for example you're building an application an agent or retrieval augmented generation the reality is uh you're probably going to have to play around with different knobs and and run experiments and some of those experiments may start early in the pipeline like uh how do I chunk and embed and and pre-process my data right so uh sure you can you can probably use the default settings but you're likely going to be leaving a lot of insights on the table so all these data tools that Robert is describing uh I think it seems to play into you know if you want to do AI well you have to be able to run experiments and data processing and preparation is part of experimentation absolutely um the customization is not just at the level of the model of fine-tuning the model or the uh The Prompt right those are important things that require customization or require experimentation but there are really there's so many decisions to make at every stage of the pipeline exactly like you said from what data to collect to how to segment or chunk the data to how to embed it to what in what embedding functions to use you know how to do retrieval how to what to stick into the context if it's a rag application right how to rank the context the stuff that goes into the context what model to use how to fine-tune the model what data to use to fine-tune it you know whether you factchecking or um the outputs of the model there's so many decisions to make and in order to really iterate quickly you need to be in a position where you can try out different choices and evaluate how well they work right and that means you know we often recommend the companies over invest in evals early on um do all the steps by hand like evaluate not just the end to-end thing but uh individual stages of the pipeline and because that's ultimately very important for moving quickly as you as you develop and and uh if you don't have the right foundation computational Foundation these experiments will actually be impossible to pull off right that's right that's right and I think the need for scale changes depending on where you are in the uh you know AI maturity life cycle um you can off you can prototype and experiments at a very small scale um using apis using a small amount of data and and really iterate on just the nature of the product You're Building does the product make sense does it have product Market fit but once you validate the thing you're trying to build you're trying to get to production the considerations tend to change right people start to care about cost about scale about reliability about upgrading the model all of these types of things and so we see a very different set of challenge at the production phase versus the uh ideation phase so what what's your sense of uh uh what's the next data type to get popular so in other words now obviously unstructured text super popular and we're all talking about multimodal but in order for it to be multimodal there's got to be another modality so what's your sense of the next shoe to drop well images are going to be uh image data is starting to be and and I think will be ubiquitous uh I think it's for all of the common AI use cases think about um or many of the common llm use cases like chat Bots customer support co-pilots um a co-pilot will benefit from seeing a screenshot of what you have on your screen right think about like uh the computer use type model um a customer support representative will be will benefit from being able to see an image of a damaged product or a video of a damaged product so I think um I do think text images video audio are the big ones um and there are a lot of things that you can lump into here I think people will do a lot with PDFs right just a lot of um uh data like that but in the you know that will be converted to text under the hood and images under the hood um I think the one that will one that will be the most challenging which we are furthest away from really uh capitalizing on is video video is everywhere video is kind of a very powerful modality in that it combines um well images video audio uh there can be text in the videos as well and it just is a massive there's so much information there and it is a massive uh in terms of the size of the data it's just absolutely enormous so it both poses a very hard infrastructure challenge um but I think it's also a format that people very naturally interact with and so once I think we have the tooling to really take advantage of video data uh it'll be indispensable yeah and there there's some uh I think there's also some kind of uh modeling uh challenges involved and and people are working on it which is around video understanding in the true sense because right now video understanding is really here's a video produce a transcript I'll try to understand the video that way but the reality as Robert points out there's much information in a video from gestures from slides in the video or whatever right so uh that uh are not being mined properly now Robert you also have uh been working with a company that does the reverse which is to generate video yes right so so can you give our audience an overview of what's the state of the art there and when will we start prompting our way into the next Academy awardwinning movie that's a great question so uh one of the companies we work with is Runway they have some of the leading video generation models out there if you've played with Gen 3 alpha or any of these um they're which by the way uh let me just tell our audience they are also behind what's now becoming famous the AI Film Festival yes yeah yes and their models are incredible I think if you think about the overall stage of video generation and it's a rich ecosystem there's a lot of promising startups in this space um the models model quality is still the most important thing that they need to improve right if you think about what is the gap the we're at a stage where the videos that we generate are awesome um but they're still difficult to control right it's still difficult to go from a clear idea you have in your head to really materializing that uh in the video and how long are how long are the videos that you've seen that uh have caught your attention well I think a lot of these companies generate um you know five or 10 seconds at a time maybe more um of course you can continue the generation you can stitch together uh a number of segments and generate like longer sequences so there's a lot you can do there I think I don't know how far away the Academy Award wi winning film is but I do think these tools are going to um to dramatically will will be widely used um in various parts of the uh you know video creation process very soon so uh one of the phrases you dropped in the conversation earlier was scaling loss so we're speaking this week when there's a bunch of headlines about companies you know not saying that the scaling loss are over but scaling lws have slowed down until maybe there's some new model breakthrough in which case they they assume that they will pick up again but uh so can you give our audience an an update on what's happening in the scaling laws well first of all this concept of scaling laws is really one of the major breakthroughs in AI over the past decade if you go back a decade um or more it wasn't obvious to people at least not to everybody that scaling was the path to really improving uh AI capabilities right A lot of people were looking more at the algorithmic level how do we come up with more clever algorithms and strategies um versus taking simple techniques and and just scaling them and of course deep learning falls into this sweet spot of benefiting from increased scale and uh you know being able to leverage that so this is a huge break breakthrough the idea that putting more compute and more data and bigger models into into the mix can lead to better results now um this is powered by ideas right I think um there are many different techniques and strategies that are needed to make this work well and um when any given idea or model architecture or um perhaps strategy like generating synthetic data or strategy like um you know uh inference time uh you know how you do inference time compute may have a finite window of of um scaling right you take some technique you push it as far as it goes that gets you another order of magnitude or two orders of magnitude or more uh but any given technique has it's it's uh diminishing diminishing returns right um and of course the continued progress and continued Improvement come from uh new ideas and new strategies I've heard people say that we're running out of data right well um we can generate data right that's a of course we have to have good ideas we have to figure out how to do it but um but that's that's very much doable right or you or these companies may have to license data that's behind the firewall right so for sure licensing data and taking existing data and really um improving it like exactly like you said there are mistakes in the data out there there's uh incorrect data identifying that correcting it um there's a lot of of room to improve the quality of the data with the existing data that we have um just determining you know separating high quality from low quality data um you know on the reasoning front I think there's a lot more mileage to get there by um figuring out the right techniques for to train these models to enable them to really leverage inference time compute to reason well so um there will you know I think AI capabilities are going to continue to grow dramatically and to make that happen we're going to have to come continue to come up with good ideas so uh as I mentioned uh just to close right so as I mentioned uh you have a unique Birch and that uh Ray is actually the foundation for many of the uh uh AI platforms that these Leading Edge companies are using including the compan is building some of the uh Foundation models we've come to love and use so can you share some of the things you're hearing so what what should uh what should we expect uh in 2025 better better reasoning multi-agent architecture so what are you what are you excited about well all of those things you mentioned I think reasoning capabilities would be far better I think um um just specifically one Benchmark of reasoning is math and I think uh there's a huge amount of interest in and uh promise in building models that are really good at doing math um multimodality like you've mentioned will be ubiquitous like this is something that's today most of us are using text based models right text input text output um but multimodal models at least text and image is going to be ubiquitous um I think you covered a lot of ground with the data side of things people are going to be using AI across the board to get insights out of data out of types of data that they didn't have a way to use before that was just sitting there right um and companies are going to find they have a lot of internal data documents design documents you know recorded Zoom calls that they weren't doing anything with before that they assumed had no value that actually have huge amounts of value and that's beyond that's just internal stuff um I think the amount of energy and money that people spend on curating data on on data processing and data preparation is going to uh explode you know far more than actually the amount of there will be a lot of training but it's explode far more than the amount of uh money and energy people are spending on training um there's a lot to look forward to and and actually uh uh just to wrap things up so data also includes data for post training yes which uh uh post trining can include fine-tuning distillation quantization reinforcement learning with human feedback but some people assume that okay so if I'm doing post training which is fine-tuning I don't need that much data well actually uh it depends on how much fine-tuning you want to do how how extensive the fine tuning you want to do so maybe there's still an advantage Robert to even in posttraining to have the sorts of data tools you described right absolutely I think there's people do continue continued training for a reason which sort of blurs the line between pre-training and fine-tuning uh because there's a benefit to doing it if you really want to get more knowledge and and um uh intelligence into the model weights you're going to benefit from having more highquality data to do that so I agree there yeah and for most companies actually this the their platform should really be optimized for post training because very few companies will actually train models from scratch correct a very few companies will do the pre-train the the very largest models I think many companies will train lots of smaller models and lots of companies will do Post training like you said I think um a lot of the quality or what we P what we really like about a lot of the foundation models that we use uh can be attributed to the post training stage which like you said includes is quite complex and there's growing complexity around post training so a year from now exactly 12 months from now the foundation models will be much better I hope what do you think I I believe so okay I and with that thank you Robert thank you so much e