AI Building Blocks

  |  Compiler Team  
AI/ML
Open source

Compiler • • AI Building Blocks | Compiler

AI Building Blocks | Compiler

About the episode

There’s no one AI model to rule them all. Each project has its own requirements. Where do you get started building your own model?

Compiler continues its conversations with big dreamers about their big projects—and how they’re piecing together the building blocks of their AI models.

Compiler team Red Hat original show

Iscriviti

Subscribe here:

Listen on Apple Podcasts Listen on Spotify Subscribe via RSS Feed

Trascrizione

In our last episode, we spoke to Ty McDuffie about Griot and Grits, an oral history project he's working on, one that's going to make use of AI to preserve and supplement submitted stories. But it's not a project he could do alone. So he called on his friend Sherard for some help. So we go pretty far back, and we've done a lot of things together in the startup world. You know, he always has fantastic ideas. He's an idealist. A lot of times, he comes to me with ideas in order to ground them in terms of what's reality and what we can actually deliver from a technical perspective. However, with this project, Sherard was just as caught up in exploring the big ideas because he knew what AI is capable of and what it would take to build it. Today, we're going to hear more about what it takes to make those AI projects from our previous episode a reality. This is Compiler, an original podcast from Red Hat. I'm Johan Philippine. I'm Kim Huang. And I'm Angela Andrews. On this show, we go beyond the buzzwords and jargon and simplify tech topics. We're figuring out how people are working artificial intelligence into their lives. Today we're talking about the technical side of AI projects. Let's get into it. Alright, let's do a real quick recap from our last episode. We talked about a couple of different AI projects that are in the works. First up, we have Griot and Grits, which Ty McDuffie introduced us to. He's kind of cooking that up with Sherard from more just now. We also had Kris Kersey, who told us about the Iron Man suit and the Jarvis AI assistant that he's building with it. We're going to revisit both of these in turn. First up is Griot and Grits. So we just heard from Sherard Griffin a few moments ago. He's a senior director of software engineering here at Red Hat and a friend of the pod. Sherard and Ty have worked on many projects over the years. For Griot and Grits, for reasons we're about to get into, they don't want to grab a complete package like ChatGPT. They want to build something custom. Luckily, that doesn't mean it's particularly out of the ordinary. From a technical standpoint, this ends up becoming kind of a prototypical AI project. The first thing you have to think about is the data itself, the source of the data, the quality of the data, how you store the data, and how you label and tag that information. So one of the things that we're trying to figure out is the source of the data. It all comes back to data, as it so often does. For Griot and Grits, they're combining many different sources of data. They're collecting raw stories directly from black communities. They're also supplementing those recordings with historical data from a variety of sources like museums and formats including audio, video, pictures, and text. That's an organizational challenge that, for now, they have to curate and organize manually. That's right. They have to go out and manually collect those stories and then try to basically design a system that can tag if there's relevant historical context, like World War II or the Great Depression, to tie in with the large language model. We know it's possible, but it does sound like a really daunting task. Yeah, it seems like it's a lot of work, especially if you do that for just pictures. That would be a lot. But you add in the variety of different media that they're working with, and it just explodes in complexity. It becomes a super time-intensive and manual project that is very difficult to do. Yeah, I imagine it's like an information architecture kind of challenge as well. How does a machine differentiate between different titles for historical events? For example, how does it know that the War Between the States and the Civil War are the same thing? How does it know if a person uses abbreviations or jargon? It's interesting. Well, part of the way that they're getting to that is even though they're looking to have AI start doing some of that heavy lifting, they're going to do a lot of that by hand because it really sets it up for that future integration of AI. And a lot of this, I told Ty at the early stages, if you can't do it manually, then it doesn't make sense to try to use AI to do it. That's why we have a group of people going through the manual process of cataloging and documenting what these videos and oral histories will be, and then looking at ways in which we can manually augment the experience to figure out what will resonate with the users of Griot and Grits. So, in other words, don't just rush in and build an AI to do the thing. We've been hammering home this point in the last couple of episodes that AI can't really create something completely original, but an AI is very good at replicating a process and applying logic that's presented to them. If you don't have that end goal modeled for the AI, you can't be sure of what you're going to get. Angela, how does this compare to broader automation projects in general? Does this sound kind of familiar to that kind of project management, or is this completely different? This is completely different. This is taking the term AI and all of its taxonomies and trying to make sense of it. This is not automation in the traditional sense. This is more like we're trying to build a model that understands the language and the nuances. We're trying to train it in such a way because, as Kim Huang just stated, people speak very colloquially, and things can mean certain things. This is a different kind of automation and a different kind of training. So, being able to do it manually and make it understand sounds like the right thing to do because you don't want it to go off on a tangent, and you don't want people to be misinterpreted. I see why they're using such kid gloves with this to make sure that the message does not get lost inside of the technology. Okay. So we want to do it ourselves first. Then we'll know what's good, what resonates with the audience, and make sure that it's accurate. Then you have something that you can show to your model and how to tune it and build it... Exactly. ...and make sure that it is going to serve your needs properly. Now that we kind of know the process that you have to take before actually building the model, let's get to maybe some of the building blocks that they plan to use for Griot and Grits. So you have things like vLLM, Ray, and Kubernetes, all these technologies that are going to be leveraged in order to bring this to life. The models themselves are going to be fascinating because you have a lot of different models out there, both foundational models and predictive models. But then you also... we're looking at multi-modality; things where you can describe something in one format, but then the model outputs it in a different format. Okay. So that was a lot of technologies thrown together in pretty quick succession. Let's go over them real quick. So, vLLM is a library to help serve large language models to the rest of your stack. Ray is an AI compute engine... Oh... ...which is used to manage computation resources for your AI models. Okay. And Kubernetes, I think we generally know what it is. I think we know what that is. But just in case you're new to the show, Kubernetes is the orchestration platform that manages everything else and allocates resources to make sure that the whole deployment environment is running well. Those are some of the main components that you can use in your deployment to get an AI model up and running. Okay. Angela, do you have any insights about these tools or this stack or anything along those lines? Actually, I don't. You lost me at Kubernetes. I mean, I can talk to you about that until we're blue in the face. I haven't delved into large language models and things like that yet. That is still on the horizon for me. But there are tons of folks that are really starting to dig in here. It's just not on my horizon just yet. I have some other things I'm trying to figure out first. That's completely understandable. What I think is a little interesting is it's kind of mixing some of these really common technologies that we know pretty well, along with some of these other new specialized tools that can be built on top of them. Well, that makes sense. I mean, there's really nothing new under the sun. We know Kubernetes is really good at what it's doing. It's an orchestration platform. If you're talking about building large language models, we all know that OpenShift is the perfect platform for running LLMs, so it just kind of makes sense that they're going to use these technologies in conjunction with one another to get this model up and running. Okay. So we have a high-level look at the kind of tools that you need to build an AI stack. You also need to figure out where you're going to be running these models. Sherard is going to give us a quick description of how they're tackling that particular problem. The technical aspect of this project, we're taking this in multiple stages. The first pass of this is going to be a few of us with our laptops. I have one with an NPU. We're trying to play around with these models and figure out what's the right model. You can do things like sparsity and quantization to where the model isn't quite accurate on your local machine, but it'll give you an idea. Some of these models can be very accurate. If you use quantization, you can run them locally on your machine. If you have maybe 32GB of RAM or something like that, you can fit those models into memory on your local machine, ask it some interesting questions, or do some things like speech to text or text to speech. You can get pretty far. So you can start small on your own machine and see how far you can take it. Now that's assuming that your machine has 32GB of RAM, which not all of them do at this point. But that is kind of within the consumer range of computing hardware at this point. So that's something that you can set up at home. When you've got that proof of concept working, you can then think about scaling it up to the monster computers or the cloud deployment stack with all the GPUs and all that. But if you're really just trying to dip your toes in that AI water, or if you're trying to get that project started and see if it's even going to work, you can get an idea of whether or not the AI is going to be producing the kind of work that you want by running it on a smaller machine at first. Monster computers. I'm just stuck on that word. I'm sorry. Yeah, this is dipping your toes in the AI water, and I don't know how that's going to come out, but for today, we're going to ignore that one. Very well. Okay. So you've got your AI tools. You've got your hardware. Check. Next comes choosing an actual model. There are plenty of open-source options available. A plethora. When you pull these foundational models off the shelf, whether that's through Hugging Face or some other repository, they only get you so far. A lot of times, what you want to do is add additional information to it and tune that model so that it's more accurate based on what you're trying to do. Asking it to tell you a joke or write an essay is a very generic model. Llama 3 is a very big model. What we want to do is look at ways in which we can introduce smaller, more domain-specific models. Okay. In the first place, the ability to ask one of these huge models a wide variety of questions and get an intelligible answer is nothing short of amazing to me. That's just wild. But, of course, there's a cost to building, training, tuning, and running these huge models. Just because it has that wide breadth of capabilities, it needs a whole lot of data. A whole lot of compute power. Exactly. It requires an unfathomable amount of compute power to manage all that data, read through it, and then generate responses. That's necessary so that it can serve all these features to you. However, not everyone needs all of those features and capabilities in their own AI projects. Smaller models seem to be a pretty good choice for a lot of people. I think that's where the world is going to go as a whole. You're going to see more of this inflection of smaller models that are tuned towards domain-specific tasks. If you correlate that to what's going on in the industry, if you're, let's say, a financial institution or an investment banking firm, do you really need a model that tells a joke? Do you really need a model that plans your itinerary for a trip? No, you likely don't. You probably need something that's more geared towards being fine-tuned to answer questions about fraud, anti-money laundering, or getting a mortgage and the requirements around it. So that's what he means by domain-specific—keeping the information very specific to the domain you're talking about. You don't need it to tell you jokes or provide all that other stuff. You just need to stay within the confines of what you're trying to train it for, which in this case is oral histories. That's right. It's going to be specifically about taking those stories that users submit and then bringing in historical documents, documentaries, images, and a whole other set of supplemental historical information to enhance that original story. The idea is that you're not planning a trip or looking for fraud or money laundering. You're just trying to build this historical experience that really enhances the story of your own families. So starting small seems to be the way. Starting small and then keeping it small. You start with a small model and a small amount of hardware. Eventually, that data is going to expand a lot, and you're going to need to scale up the operations. But in terms of the domain of the model itself, you're not going to want to expand the feature set too much. You're just going to expand the amount of data that it's working with. Gotcha. Alright, let's do one more quick recap here. Step one: come up with your project. Pretty simple, pretty straightforward. Step two: do it yourself a little bit to see if the output is interesting, to determine if AI is even an appropriate tool for this, and to figure out what can be automated with AI and what can't. You need to assess if it's going to be accurate enough for your needs. Then, you build your initial platform and choose your appropriate model, and then create that small-scale project that you can scale up later on. Congratulations! Now you have the parts you need to build your model. However, there are a few more steps, like training the model in some cases and then tuning it, which is probably something you'll be doing in most cases. When that's done, your work isn't over yet. There's one more crucial component that you just cannot do without. Humans are at the point in the AI journey where we still cannot be removed from the process, and Griot and Grits will be no exception to that. We will fully utilize human-in-the-loop feedback, where even if we use generative AI to do very interesting things, there will still be a person for every single piece of content that we use generative AI for to validate and ensure that what we're delivering is of the highest standard. That's very key because history is a very touchy subject. We have to make sure it's objective and accurate. Yes, absolutely. History is touchy enough when it's just human-focused and human-centered due to the different points of view. You present the information, and there are all sorts of biases that can come out. You want to ensure that if you're going to introduce AI, those discussions don't get lost, and you are still producing the best possible quality content. AI can help you be more productive and efficient, but if you care about accuracy and quality, you cannot trust the model to do everything independently. Speaking of doing things yourself, we're going to check in with Kris Kersey and his Iron Man suit when we get back. Kim, could you do me a favor and remind us a little bit about Kris's project? Absolutely. Kris Kersey is an Atlanta-based technologist who is really active in cosplay. He noticed that many Iron Man cosplays were just not practical and didn't incorporate the modern technology that resembles what has been portrayed in comics and films. So, he decided to take on the Herculean task of building a real-life Iron Man suit. That was perfect. That was great, yeah. I'm just remembering from the last episode that the kinds of things he wants to do with the suit are all very cool, but to me, the logistics seem pretty daunting. When we first saw the Iron Man suit in the movie, it didn't seem remotely possible. But now... Now it's a different story. When Kris recently thought about what it would take to make it all real, it seemed very much within reach. There was no idea about what hardware you'd need or how you would even accomplish what we were seeing with Jarvis from the Iron Man movies. Once OpenAI became available, and once you had ChatGPT, which has an API that you can access, that's when it became feasible to include this. We had all the components at that point. We already have speech-to-text and text-to-speech. So, you just put the AI in between it, and you have a full AI assistant. Once I realized that all of these components were finally available, I knew I had to incorporate it. Recreating the experience of Iron Man and that Jarvis assistant that can help you out and answer questions about what you're trying to accomplish. So in other words, we have the tools and the technology; it's just a matter of putting it together now, right? Yeah, that's always the easy part. Is it though? Well, it's not exactly like Tony Stark's in a cave with a box of scraps, but there are different components. Kris mentioned the OpenAI API, which opens up a world of possibilities for building an AI assistant embedded into a local device. That's the foundation of the work he's trying to do here. As we've talked about with Sherard earlier, it's about figuring out the different components that can be used and then determining the tech stack you need to put together. Yeah. Very intriguing. Yes. Having the plan, knowing the parts, and having them available is a huge step. But as with a lot of creative work, it's going to take time to make it into what you imagined. Kris's first attempts at this Iron Man suit and the Jarvis assistant worked, but they were a little clunky. There was a significant delay between him giving a command and the suit reacting. Yeah. But it did work, and the technology has improved rapidly. He's been working on this for about a year and a half since the AI came out, and he's made significant progress. He also built a workaround to speed things up and provide more consistency that was lacking in his initial attempt. Sitting in front of the AI assistant right now is a command and control interface that intercepts stuff before it goes to the AI. That's what's handling suit control. I wrote a command and control engine that understands when I'm asking it a question that requires an answer, not AI-based answers, but basic questions like, "What's the temperature in the suit?" or "What time is it?" These are rudimentary questions that it can get from the system without needing to go to AI, as AI typically doesn't have these sorts of answers. That's interesting. So it's the AI that you're asking questions of, and then the system itself is doing its own self-checks, asking about the temperature of the suit, which wouldn't be in the AI. How is he differentiating between the two? Yeah, I'm not sure how they differentiate between questions that can be answered by local feedback or information and those that need to go into the cloud. What Kris has done is essentially reverse prompt engineering, creating preset questions that he knows people will ask if he's demoing the suit at an event. He's feeding these questions manually to show proof of concept and demonstrate how the system can differentiate between basic inquiries like the temperature or time, which don't require an LLM. So what that looks like is that this command and control interface is doing exactly what Kim just described. He's had to come up with and hardcode a set of commands for his helmet to recognize. He also set it up to listen constantly for a wake word, just like Alexa or Siri. That interface then interacts with the assistant to produce the output he wants. That sounds like a really extensive, time-intensive process, right? Hardcoding all those questions and ensuring you're picking up the questions at the right time. Sounds a little bit familiar because he did the same thing that Sherard just advised to us, right? You do the thing manually first. But of course, that's not the end goal. The dream, the next thing I actually probably I'm working on from the AI perspective is not having to have that command interface in between anymore. It's allowing the AI to understand what I'm asking and then giving it access to functionality that it can call when I'm asking it to do things. It becomes... it's actually a lot more powerful. Then you don't have to code in every single thing you might ask it. You just have to tell the AI what it has access to, and then it can query it or activate it when it thinks that's what you're asking for. Okay. So more like agentic AI is what he's talking about. Yeah. So eventually he's not going to need that kind of control interface where he's putting in every question and the answers that he's looking for and all that. He's just going to be able to talk to the AI, and it's going to figure out what he wants and hopefully interpret it correctly and then perform the action that he wants. And that's probably not that far off at this point, especially with some of the ready-made models that are available on the internet. Here's the thing, though. In order to get this done, you have to keep the project's context in mind. It's an Iron Man suit. It's going to be used at convention centers. And you can have your suit call on an internet service like ChatGPT to power the AI portion of it. But if you're in a convention center when it's crowded during a Comic-Con or something similar, where you've got a lot of people running around, things aren't going to be able to provide that reliable, consistent internet connection that you need to power your suit's AI functionality. Mhm. And so anything you're trying to access will either cut in and out on you or it will just be unreliable. You don't know what your bandwidth will be, which really does present a problem with anything you want to be interactive. And so once these LLMs, these large language models, started to become available, you started to see companies also come out with smaller versions of these large language models that can be run locally on smaller devices, devices with less memory. We could put together some bingo cards at this point about small AI models, right? Sherard and Ty are planning to run their initial versions of the project on a laptop with many fewer parameters and features than what's available with huge LLMs, and they'll probably stay on a more focused model. And again, that's exactly the kind of thing that Kris is trying to do as well, with his Iron Man suit and the Jarvis AI system. I love that. Very, very technologist solutioning there. Like, what's the solution? Bring it local. Bring it closer to you. Yeah. And Kris is thinking even smaller than Sherard and Ty. It's a huge challenge, right? Because again, all you've got is the suit. So you're running everything on the suit locally. Locally. Yeah. So your hardware limitations are going to be a huge consideration. To get those AI models shrunk down small enough to run on something that you can wear on your body underneath those pieces of armor that you've got. Now, that's nothing that's completely new to hardware or software development, right? Hardware limitations were a huge consideration in development for a very long time. But then computing resources kind of outpaced the software requirements by a big margin. And it's maybe not as big of a consideration for a lot of people these days. However, with edge computing, it's becoming relevant again, and with how power-hungry AI models can be, it's a real balancing act between the resources you have available and the functionality that you want to provide. There were really two things I had to do, which was, number one, you have to shrink your memory footprint as much as possible on all of the processes that are running on the device. You have to get that as tight as possible. Because again, the large language models take up a pretty good chunk of RAM. And so I'm really running the device up to its memory limit. I'm using every byte and going into swap space a little bit occasionally when you're a little too close to the edge. So that's the first big thing you've got to overcome: how do I fit so much functionality onto one device? That's literally a whole other level that you have to keep track of when developing your project, right? It's not just, how do I make it work? It's how do I make it work? And then what are the constraints that I have to work on with? And how do these two things intersect? Right. It's a huge, huge undertaking for him. But wait, there's even more to keep in mind. You can tweak how you use power consumption. How do I want, like, do I give more power to the CPUs and so I have more CPU power, or do I give more power to my GPU, which is actually doing the AI processing? And so there was some of that tweaking that had to be done because the responsiveness of your AI has a lot to do with how fast your GPU is running and how fast it can process that information and give it back to you. And so there was definitely a balancing act that I had to do, not only with the memory, but with the processing. That's a lot, right? That's great. It reminds me of the kinds of things that I would read about in books, like "Soul of the Machine," or about how game developers for early consoles, especially like the Super Nintendo or the Sega Genesis or the Atari, they had to come up with super creative ways to make their games fit on an incredibly limited amount of memory space. Tank controls and JPEGs. Sky boxes, baby. But this just means that they have to go back to the beginning and become more creative in how they're utilizing the resources that they have. They just have to be more creative. Yeah. Absolutely. It's been a bit of a lost art, which hasn't left completely, obviously. I mean, Kris works on a lot of these smaller machines, and he's kind of bringing that stuff back for entertainment reasons. One question that kind of came to mind is, is it worth bringing back? I mean, we have this issue of internet connectivity, right? But are Wi-Fi and cellular network improvements going to make this resource juggling obsolete again? Are we going to need to keep running things locally, or are we going to get to the point where it's not just compute resources that are getting really good, but also networking resources that are? Good question. We'll see. Time will tell. In the meantime, Kris is still building his project on very small machines and doing some really amazing work. There are, however, some other considerations at play. You know, the other direction I was going to take it is cost. Because if you're using a commercial AI solution, every part of that conversation is costing you pennies. And so you end up, in this case, kind of like early cell phone usage, where every time you wanted to make a phone call, it was going to cost you a minute or two. And so you thought about how you wanted to use your minutes. Am I going to call my buddy, or am I going to call my mom because I only got 300 minutes this month or whatever the case may be? And so same thing with AI, right? If I'm using a personal assistant, whether it be in the suit or whether it be in your home, how are these going to cost me? Whereas if you bring, again, if you bring that home, if you run it locally on your own hardware, it's not costing you anything but the electricity to run the device. That's something I don't miss about old phone plans is trying to figure out... Me neither. Who are you going to call and how long you can call them for, and... And rushing. Everyone's calling you after 8, 9 p.m. That's when it's free. Yeah. They will never know the struggle. They will never know the struggle. Truly. So that's a big advantage of being able to run things locally. Right. Again, you could even if it's the Wi-Fi technology or cellular technology got good enough that when you're in a congested space, you could use it without any delays. You're still thinking about, oh, okay, this is a commercial solution. How much do I want to be paying for my AI assistant to be doing these things for me? Or if I run it on my own machine, you know, I get it working, and I don't need to have those considerations in mind. All right. That was a lot for the suit. It's, you know, I find it to be a fascinating project because it's, number one, just this really cool recreation of what we saw in movies, you know, however many years ago, that at first seemed impossible. And now is very much within reach. And also getting kind of this closer look into the considerations of how to get it done has been really eye-opening because even though, again, we have the technology, it still takes a lot of work to get it working properly. I think we're right. I think we're lucky. Right now we're at the precipice of all of this technology kind of converging together and making what we thought wasn't even remotely possible just a few short years ago. Think about when Iron Man came out... 2000 and... Don't tell me... The early aughts? Yeah, early aughts. It's like 2003, 2004? Something like that. Yeah. And now we're like, yeah, no problem. We have an AI assistant for that. You know... 2008! The possibilities are there. And I love the fact that we can see those technologies right in front of us. So we just have to hold on just a little while longer before we have our own Iron Man suits. Oh man. That would be fun! I would love my own Iron Man suit. And Iron Man, the original Iron Man movie, came out in 2008. 2008. Okay, so... 17 years ago-ish. Give or take a couple of months. Yep. All right, let's bring it home. What do we need to keep track of when planning an AI project? What was the number one thing we each learned from this episode? We need to keep track of... you don't always have to start big. You can start small. Smaller, more localized models might be the way. And doing things manually, like you can't underestimate the human in the loop, so to speak. It still needs a lot of manual work to set these things up and to figure out all these components and how they fit together to get the solution that you want. And for me, it's just what's possible within the limitations. Right. Like just the idea that you can run something as powerful as an AI assistant on the small little edge device that you can wear under a suit is amazing. Now, it comes with its drawbacks. Right. You're not going to have that full Llama 3 model that Sherard was talking about that's telling you jokes and planning in AI territory for you and all these other things, right? That's not going to fit on whatever resources you're going to fit on your body, but you can still get something that's incredibly useful and incredibly entertaining on a very small form factor. And that, to me, is nothing short of amazing. Absolutely. We want to know what you thought about this episode. I mean, the Iron Man suit right in our hands. I mean, who would have thunk it, right? You have to hit us up on our socials at Red Hat using the #compilerpodcast. Do you see any other applications for AI tools in your projects? We'd love to hear about it. And that does it for this episode of Compiler. This episode was written by Johan Philippine. Victoria Lawton could build a fellow AI in her sleep. Thank you to our guests Sherard Griffin and Kris Kersey. Compiler is produced by the team at Red Hat with technical support from Dialect. Our theme song was composed by Mary Ancheta. If you like today's episode, please follow the show, rate the show, leave a review, and share it with someone you know. It really helps us out. Thank you so much for listening. Until next time.

More about artificial intelligence

Usually referring to processes and algorithms able to use data to simulate human intelligence, artificial intelligence can provide insights based on acquired knowledge.

Articolo del blog

Pensare in grande iniziando in piccolo: perché l'IA focalizzata è destinata a vincere nel 2025

4 marzo 2025  |  

...

AI/ML

Articolo del blog

Suggerimenti per i modelli di prompt LLM (Large Language Model) per l'IA generativa

1 luglio 2024  |  

...

AI/ML

About the show

Compiler

Do you want to stay on top of tech, but find you’re short on time? Compiler presents perspectives, topics, and insights from the industry—free from jargon and judgment. We want to discover where technology is headed beyond the headlines, and create a place for new IT professionals to learn, grow, and thrive. If you are enjoying the show, let us know, and use #CompilerPodcast to share our episodes.