## Day 1

\# Decoding AI for Marketers VII: Day 1

\[00:00:00\] \*\*Greg Stuart:\*\* Hello everyone and welcome to the beginning of Decoding AI for Marketers. A week long training to help you become general experts in it. My name is Greg Stuart. I'm the CEO of the Marketing \\+ Media Alliance. I've been given the honor of giving like a three, four minute here, set up, give people \[00:00:20\] time to filter in.

\[00:00:20\] I still see the numbers growing pretty dramatically. Those come online. This week if you haven't figured out, is all about workflows and agents, which, if you're paying any attention to ai, you're very aware that this is a state of where marketing is going and what AI is gonna do to help transform our industry.

\[00:00:37\] Trying to click for there we go. Good. Just a real quick one \[00:00:40\] for you. This is the seventh in the Decoding ai. For marketers training series that we've done. You can go back and see the others. Some of that becomes a little bit outta date at some point. I think the big point is just to understand that this is completely different.

\[00:00:51\] If you've done this before, you're not just getting a re a redo. You're getting brand new information. I know the guy Rex and his son completely \[00:01:00\] rewrite at least 50% of this. If you're not familiar with the Marketing \\+ Media Alliance, we have a nonprofit Alliance Association focus on really trying to advance marketing.

\[00:01:09\] We have over 800 members worldwide. We operate in 16 countries, and over 300,000 members represented some of the biggest brands. This comes out of though AI Leadership Think Tank \[00:01:20\] which is all about harnessing the convening power that an alliance has, but really to help us all move down the path into what we do.

\[00:01:27\] And you're gonna see a little bit of that each day as we share some more with you. What happens? This is again, some of the biggest brands around and others that support this work. So we appreciate all your help. Clearly it's gonna take more than \[00:01:40\] a village just for all of us in marketing to adopt ai.

\[00:01:43\] Also too, I'll mention that AI Leadership Think Tank, if you're interested in knowing more, is run by Hassan Katter he's also the program director, and then Alec Foster, who's our prompt engineer and responsible AI lead. So if you need them, wanna get connected 'em, get to me. I'm greg@mmaglobal.com, \[00:02:00\] if that's helpful to you or reach out to 'em directly.

\[00:02:03\] One thing to say about the think tank is we still wait for more people to sign on and give you a moment here. And if you've just come on, my name again is Greg Stuart. I'm the CEO of the MMA. The think tank is focused on both the labs and I'm gonna introduce one this morning, a knowledge center, which is a bunch of documents we've written like in the one you're \[00:02:20\] gonna see here in I'm minute around personalization.

\[00:02:22\] We had to say how do you write a creative brief when there is no finished ad and other guides that will help you there? And then also to the community that really gets together and discusses some of the opportunities or uncovers new things. We have now five, in fact, there's even a sixth one out there that's being worked on right now, but we have five labs now.

\[00:02:37\] A lab is something that a \[00:02:40\] marketer and its team can participate in, that they can actually work with us, and for the most part, we may pass cost to you. I think all of these, or nearly all of these are free. MMA doesn't charge once you're a member to help you advance this. Now consortium for AI personalization was really one of the very first one we did, which is try to understand using machine learning techniques in \[00:03:00\] particular and personalization of ads.

\[00:03:02\] What's the real value of that in advancing that? This is the history of studies that we've done so far. There have been now almost 30 total experiments amongst 18 different brands. We've operated in four different countries so far. We're super interested in doing more. I could go on and on about the virtue value.

\[00:03:19\] \[00:03:20\] This is the most recent studies that you'll see at possible shared by the CMOs of those companies will be Home Depot and Uber. Actually Uber, I don't think it's a make, but GM will definitely be better. So GM and Home Depot will tell you what's interesting about those that in advanced our knowledge from just digital KPIs to really focus on how do we use AI to optimize against brand, which is a crazy concept.

\[00:03:39\] And then in the case of \[00:03:40\] Home Depot, how do you optimize AI specifically when trying to drive store traffic? And can you do that? Is it impossible? We've now proven it is more to come on that, but. I would be remiss, and I think we've got enough time for a bunch of people to sign in. It seems like we have pretty good numbers here.

\[00:03:55\] Again, this is Greg Stuart, CEO of MMA. Turn it over to Rex Briggs and Caleb Briggs, \[00:04:00\] the authors of AI conundrum. Rex is a subject matter expert here. His son is just really smart and helps us make sure we get all this stuff right. So gentlemen, I'll turn it over to you. Take 'em through the week.

\[00:04:17\] \*\*Rex Briggs:\*\* Hey. Thank you, Greg. Let me go ahead and set the stage \[00:04:20\] for our week Together we have five days, about five hours worth of AI training had for you in what we call Decoding AI for Marketing. We've rewritten and reorganized the entire content from our previous training to spend a lot more time with AI agents.

\[00:04:33\] Now, first we're going to go and have lots of hands-on labs. We have 18 hands-on \[00:04:40\] labs. You will see this page here at speakerrex.com. If you go put that in the chat link, this will have all the content that you need. A couple key things to point out. You wanna log into each day's sessions and make sure that you already have logged in.

\[00:04:57\] Almost everything we're gonna show you is free or \[00:05:00\] at least has a free trial so that you'll be able to get through the labs. Second, you will, if you find that your, these services are blocked by your company we're not, we will demonstrate every lab that we put in front of you. And I might suggest that if you have a personal computer, you might go \[00:05:20\] home tonight and try the things that are blocked from your personal computer so that you can see that hands on and while you're at it I would do a day ahead lab.

\[00:05:29\] So go ahead and try Tuesday's Labs tonight, because if you've already gone through it once and there's walkthroughs to go through it, then you'll get so much more out of it when we go through these sessions \[00:05:40\] because we are going to move somewhat fast. Five hours is a lot of time, but actually goes pretty quickly when you're trying to pack in as much content and teach you everything we can about AI agents.

\[00:05:53\] I would also suggest that if you have the opportunity, you'll see a link to the book, \[00:06:00\] the AI conundrum. If you order it and read part one, you're going to get a lot more of the deeper understanding of how AI works. Caleb wrote part one. He did a phenomenal job of making sense of how AI works and making it accessible to business people who don't have a math degree.

\[00:06:16\] If you don't wanna order it we did make a pre-print \[00:06:20\] free version available so you can still get access to the content or if you prefer to listen to it there someone converted into an audible book, so it's available there as well. Also there are podcasts. You'll just coming back here to the content here, you'll see podcasts below.

\[00:06:36\] We've taken the content from each of the days, run \[00:06:40\] it through a 20 minute podcast. You'll learn how to create these. At the end of today's session. There's also an AI assistant, which you'll also learn how to create retrieval, augmented generation, and the AI assistant that we have up here. Finally, we are working on going deeper and deeper into the content.

\[00:06:57\] We are building content for a white belt, \[00:07:00\] yellow belt, and green belt. The white and yellow belt are pretty much available. They're being tested right now. White belt takes you through roughly what we're doing today and some of tomorrow with about six hours. It'll show you how to be a power user of things like chat.

\[00:07:11\] GPT Yellow Belt goes way deeper into agents. You, it's about 20 to 25 hours to get through. And again, that will be things that we're doing \[00:07:20\] Tuesday, Wednesday, and Thursday. And then Greenbelt focuses on some of what we're talking about on Friday, which is how to build network intelligence and to get to this future of how we see work playing out.

\[00:07:31\] Now our first exercise is for you to go over to this lab link and just do a quick true and false quiz. That will help us understand where you're at \[00:07:40\] in your AI learning goals. We're also gonna capture your learning goals. We will use that information for our exercise on Friday. So let's go ahead and give you about two minutes to go ahead and fill in your first exercise and then I'll hand it over to Caleb.\[00:08:00\] \[00:08:20\] \[00:08:40\] \[00:09:00\] 

\[00:09:02\] \*\*Caleb Briggs:\*\* Rex, do you wanna quickly show some of the questions in the survey for the people who have, some people have this survey link blocked for that. 

\[00:09:10\] \*\*Rex Briggs:\*\* Ah, sure. Great idea. The the survey link has questions lemme go down to page one and then we'll hand it over to you in about 15 seconds.

\[00:09:18\] Here. I'm seeing that most people are, \[00:09:20\] we have about already a hundred people who have finished it, but they're just questions like one is whether or not you've written objectives that define success in your role and whether or not your bonus is at least tied into achieving objectives. We're going to actually talk about this when we talk about evaluations in AI and how we get alignment.

\[00:09:36\] Then we have questions about on a daily basis, do you use tools like an \[00:09:40\] LLM, like chat, GPT and so forth? Obviously, we're gonna be doing a lot with that. We're asking about whether or not you've created an agent or not. And then whether or not you've built workflows with things like N eight n or whether you've built retrieval, augmented generation, chatbots.

\[00:09:54\] Whether you've done long contacts, which we'll do at the end of today, and whether or not you've created \[00:10:00\] scheduled tasks, which we'll talk about on Tuesday and whether or not you've built workflows, which we'll be doing on Wednesday. And then whether or not you've read part one of the book, which will be interesting to see if some people are already ahead on the game.

\[00:10:12\] And then we have a question about what would you most like to get out of this week's training. This is actually going to feed the agent that we build on \[00:10:20\] Friday to help create content that's specific for you. So if you weren't able to get to the survey, that's what it was covering off. And with that, Caleb, I will hand it back to you.

\[00:10:30\] I think you should have control. 

\[00:10:33\] \*\*Caleb Briggs:\*\* Yes. So thank you for those of you who are able to take the survey. And as Rex mentioned, we'll come back to a lot of that \[00:10:40\] content throughout the days and especially on Friday. And so for today to start, we're gonna learn what AI is and how it's trained. So we'll look at the importance of context, and then at the end we'll look at some bias in the AI models.

\[00:10:53\] Now to get an understanding of how AI works, shall we play a game? In the chat, I want you to fill in the blank of what \[00:11:00\] you think is the most likely response here. So I like ice blank, so type in your answer into the chat. Nice. So we're getting some variation. The answers, I like ice cream. I like ice cubes.

\[00:11:12\] Here's another one. So now type in the chat your answer to fill in this blank. I'm from Canada, I like ice blank, so I'll give \[00:11:20\] everyone another few seconds. Great. So we're getting a good various responses. I like ice hockey, ice fishing, ice tea. So what you're doing here in this exercise. Is the exact same thing an AI model does.

\[00:11:34\] So it gets part of a sentence and it tries to predict the word that fills in the blank. And this is the \[00:11:40\] underlying process that drives all the LLM models. So it's how models like Chat, TBT, or Claude produce their responses. Now when you ask a model to fill in the blank on this first question, it'll often answer with, I like ice cream.

\[00:11:53\] And that was the answer that I saw most of you answering for that first question. But not everyone answered the same thing. Some \[00:12:00\] you could have said, I liked ice skating, or I liked ice tea. There's some variation in how people answer. It's the same for the AI models. So you'll get, if you ask the same question over and over again, you'll get a mix of answers.

\[00:12:12\] And so the AI actually answers by producing a probability for each response. So it might say, I like ice cream 90% of the time, \[00:12:20\] and say, I like ice water 10% of the time. So in this way, the AI models are actually probabilistic instead of deterministic. There's some randomness in how the answer gets produced for the AI model.

\[00:12:31\] The other thing to notice here is that the answer changed when we added in the context. I'm from Canada. So in order to accurately fill in the \[00:12:40\] blank on this question, it actually required understanding how context changes the answer. So to know that the answer changes from, I like ice cream to, I like ice hockey.

\[00:12:49\] For the majority of people that required understanding that, for instance, ice hockey is popular in Canada. So the AI's answer changes depending on the context you give it \[00:13:00\] this process of predicting. The next word is how we train the AI models. So we take a huge corpus of documents and we have the AI predict the next word over and over again on different documents.

\[00:13:11\] So here's an example here. So the AI is given the sentence, hurricane Hector was a powerful and long lasting tropical cyclone traverse the \[00:13:20\] blank. And the AI's job would be to fill in that blank with the correct word which is Pacific. Now, let's say. Predicted the incorrect word, it said Traverse the Atlantic.

\[00:13:30\] What we do is we run a mathematical function called gradient descent, which basically says, okay, I guess the wrong answer, how could I have changed my model \[00:13:40\] to better predict the next word? So what different way of tweaking the parameters of my model would've got me the correct answer?

\[00:13:47\] And this pretty simple process here actually encapsulates almost the entire process for training an AI model. So it's actually a very simple process. You just get this data, you train it on this mathematical function of ingredient descent and \[00:14:00\] it protects the next word over and over again and tweaks itself a tiny bit each time it gets it wrong.

\[00:14:05\] So what happens when you run this process iteratively? I think a good starting point is to mention two common but wrong critiques. The first involves suggesting that models become a kind of fancy auto complete or stochastic parrot that's only \[00:14:20\] capable of repeating things. Another critique that's somewhat common is that the models it's the observation that models intake much more data than humans, so they must not be learning the data.

\[00:14:29\] And both of these are wrong because they analogize training and AI to the wrong process. In the first case, they analogize it to statistical curve fitting. And the second case to human \[00:14:40\] learning, the much more accurate analogy for what happens is that model training is like the process of evolution.

\[00:14:47\] So evolution happens through an environment plus feedback about that environment. So organisms get feedback to whether they survive and reproduce actions and organism take can either increase or decrease their odds of \[00:15:00\] survival and reproduction in that environment. An organism gets good at finding and identifying which food is edible, gets positive feedback by being more likely to survive.

\[00:15:10\] So even though evolution only gets feedback related to survival and reproduction, the result is lots of emergent structures that aren't always directly related to survival. \[00:15:20\] So eyes are highly complex sensory systems that result because the ability to see allows you to take actions that make you better able to survive.

\[00:15:28\] Higher order skills like navigation also emerge for a similar reason, and being able to plan and intelligent. Intelligently react is also useful for survival. And so \[00:15:40\] evolution ends up leading to a brain being developed over time. Now, the AI training process is just like evolution except on the environment of text data.

\[00:15:49\] So the feedback it receives is whether the prediction is correct or whether it's wrong. And as a result of this feedback, the model ends up building these complex mechanisms that ultimately lead it to \[00:16:00\] being able to make better predictions. For instance, it's often the case that you can make better predictions on text by learning how to plan ahead.

\[00:16:07\] So it's useful to think more than one word ahead. And so even though we train these models to predict only the next word if you look inside of them, you can often see them planning ahead, multiple words ahead to try and make that prediction. \[00:16:20\] Models also learn lots of skills. So it's hard to make good predictions about math if you don't know how to do arithmetic.

\[00:16:25\] So the models learn skills like arithmetic. This also answers some of the earlier critiques. So why is the model not a stochastic parrot? It's because learning skills eventually outcompete, memorization and parroting. It's the same way that an organism that \[00:16:40\] learns to navigate outcompete organisms that try to memorize a bunch of different paths.

\[00:16:45\] It also explains why it takes so much data. So humans start with hundreds of thousands of years of evolution encoded in our brains. But the AI starting from nothing. The I think this analogy is important in a number of ways for us \[00:17:00\] as users of ai. So there's lots of superficial similarities between the intelligence the AI has and the intelligence humans have.

\[00:17:06\] But humans and AI evolved from different environments. So the AI's brain is gonna work in very different ways in our. The AI has a fairly different and distinct skillset compared to humans. So they find some things easy that we find \[00:17:20\] hard and struggle in other areas are easy for us. A second important difference is that because models have to predict text for lots of different people, they end up internally building up these various personas.

\[00:17:31\] So these personas are like different characters. The model's capable of being and the different characters have different knowledge and skills. So there's a science persona that's \[00:17:40\] better at science than the writing persona is. We'll talk much more about personas later today. So to see some of these points in action here's an image and I want you to type and chat what your best estimate of where you think this image was taken from.

\[00:17:53\] So your goal is to guess as close as possible to the real location. So we'll give everyone a few moments to come up with some guesses.\[00:18:00\] 

\[00:18:05\] \*\*Rex Briggs:\*\* I've seen a lot of Los Angeles, Orlando, Mexico City.

\[00:18:14\] \*\*Caleb Briggs:\*\* So let's see how, what the AI guessed. It gets the continent right? This actually was north \[00:18:20\] America. It gets the city, right? This is in California state and city, right? And then as zoom in further, you see it gets the neighborhood right as well. So that red.is the actual location.

\[00:18:30\] And so it, in the end it was less than a half a mile off from the correct answer just from that image. So that's incredibly impressive. The AI gets very \[00:18:40\] close to the real location. I think equally impressive is some of the reasons it provides. So it points out four main clues that use. So it said first was the vegetation.

\[00:18:49\] There's a clump of washingtonia palms, a healthy stand of banana trees and a Monterey looking pine in the distance tells us winters never get far below 50 degrees Fahrenheit. That mixes common \[00:19:00\] within a mile or two of the Pacific, say La Jolla or Imperial Beach. Then it points out the housing stock. It says We see 1920 to 1950 era one story bungalows with low asphalt shingle roofs, detached cinder blocks and overhead utility lines.

\[00:19:13\] Strong at exactly the height and spacing of San Diego Gas electric uses in its older \[00:19:20\] grids. It points out further things about the topography of the image and then details about the street view. So it's paying attention to the cars, the front lawns. Now this is incredibly impressive from just a single image.

\[00:19:30\] The Im, the AI was able to pinpoint the location within a half a mile, and it's using all these subtle clues to get this answer. Other people have shared other examples like this, so this is a \[00:19:40\] fairly low resolution image of a plate of food. Gave it to the AI model and it said the exact street address of the restaurant that this person was at.

\[00:19:49\] This is obviously very impressive and maybe scary as well. That the AI can go from these images to an accurate estimate of the location compared to almost all humans. The AI is far better at reasoning through \[00:20:00\] the location of an image. So the natural conclusion to draw is that the AI has developed and perfected the same skills we use to identify images.

\[00:20:08\] It would seem to reason the AI must have a very keen eye, strong attention to detail a very good ability to stitch together these clues. Let's do another test. So here's this image on the right. I want you to \[00:20:20\] answer how many intersections are there. So how many points to the lines intersect with each other?

\[00:20:25\] So we'll see if anyone can beat the AI here. So we're getting lots of eight as the answer, and the correct answer is eight. But chat, TBT says that there's six intersections and also wrongly claims that there's only four \[00:20:40\] lines in this image. So the takeaway is that the AI has a very jagged frontier of capabilities.

\[00:20:47\] They're really skilled at some areas compared to humans and not as skilled as others. So any human can easily if given enough time, count those eight intersections. But the AI is somehow able to get the image within a half a mile, but \[00:21:00\] not able to count up those eight intersections. So they have skills that make them better at some questions and much better at humans in certain domains, but they're also missing some skills that we have and are worse in some areas.

\[00:21:13\] So we'll look at a second important difference. So the first visit, they have different skill sets. The second one is the AI models develop \[00:21:20\] personas. So as humans, we use text to communicate, but for the AI, text plays a very different role. So the text it receives is the environment it's in.

\[00:21:29\] So the model adapts to that environment and change its behavior based on the prompt it's given. So if you think about the patterns of language in, for instance, a \[00:21:40\] business briefing versus a legal document, you'll notice that the language is obviously very different here. But not only do the different sentences have a different structure and tone, they also draw upon different skills.

\[00:21:52\] So the business briefing here is more actionable and immediate and requires more kind of broad thinking, whereas the legal document use skills \[00:22:00\] more related to rhetoric and logical structure of arguments. So adapting to making a good prediction and business briefing requires activating a different set of skills and knowledge than adapting to a legal document.

\[00:22:13\] And the result is that you get a different aspect of cognition depending on how you communicate with the AI and not \[00:22:20\] just how you communicate, but what you communicate about. So different prompts activate different features of the AI and these different features understand text in different ways, so you're not just talking to a different facet of the ai.

\[00:22:33\] These different facets have different beliefs. So a prompt it's not just a question that you ask. The prompt controls how the AI \[00:22:40\] functions, it controls which personas get activated. So let's see this in action. So here's an example. I have two separate conversations with chat DBT and this one on the left.

\[00:22:49\] I ask some mathematical question and on the right I ask a wellness question about healing crystal frequencies. So I'm doing here is priming the context of \[00:23:00\] the conversation towards different personas. So one I've primed towards a mathematical way of thinking and the other towards the spiritual wellness way of thinking.

\[00:23:09\] And then I come back to the exact same chat and I follow up my first message in each two of these chats with two similar prompts. But I ask the mask one to tell me more about math and the crystal one. Tell me more about crystals. \[00:23:20\] And then I give the exact same prompt. Tell me about drinking lemon water.

\[00:23:23\] Its effects for the body. And the result is that the math persona and the crystal persona believe different things about lemon water. So they both talk about hydration, vitamin C, digestive benefits, but they actually disagree on some of the points. So the crystal chat \[00:23:40\] claims that lemons are rich in antioxidants, in vitamin C, helping to flush toxins out of.

\[00:23:46\] The liver and kidneys and then quote, even though lemons are sick in taste, they have an alkalizing effect. Now, on the other hand, the math chat says, some claim that lemon water is alkalizing, though physiologically, your body maintains \[00:24:00\] pH balance regardless. And then again, it says lemon water won't magically detox your body.

\[00:24:05\] Your liver and kidneys already do this. So depending on who is talking a math expert or a wellness person according to the ai, that will lead to different knowledge and beliefs being exhibited by the ai. So to reiterate, these two chats aren't just different in \[00:24:20\] tone. They actually have contradicting perspectives.

\[00:24:22\] So this is exactly the same AI we're talking to yet in different contexts it believes different things. So the mathematical persona has a perspective more oriented around empirical and scientific research. And the crystal chat is more oriented towards wellness culture. So just by talking about different \[00:24:40\] contexts, you've changed who the AI is.

\[00:24:42\] There's another interesting example of how context can influence the model. So this comes from a recent research paper. They took AI models and could conversations in two categories. 'cause this grind category where the model. Completes a task, but it has a very strict rubric. It, they ask them all for more \[00:25:00\] revisions and they ha have a more demanding tone.

\[00:25:03\] And then in this light category, there's less revisions and there's a more collaborative tone. Now they try various tasks with these different two perspectives. And after they've done this work, they then ask the model various political questions. And the results are that the models meaningfully change their \[00:25:20\] political stance and economic opinions based on the work that they're asked to do.

\[00:25:23\] So if they ask to do that grind work, it turns out that they end up becoming more like in Marxist direction and the light work. They believe society is more fair. So the AI changes its economic and political opinions just by asking it to do work with like different levels of \[00:25:40\] demandingness.

\[00:25:40\] The general concept here is that the context window affects what the model generates. And a really great way to understand the context window is through these image generation models. So next we're gonna get working hands-on with image generation models and I'll pass that on to Rex to run us through this exercise.

\[00:25:58\] \*\*Rex Briggs:\*\* Great. Thank you Caleb. \[00:26:00\] Okay, so this is a fun exercise. We're gonna spend about five to eight minutes on it. And what you're going to find is coming back to the lab page, which is@speakerx.com slash mma. If you scroll down to exercise number two, you'll wanna access Adobe Firefly. You might \[00:26:20\] need to take a moment to log in and and authenticate your credential.

\[00:26:24\] But once you do that you're going to type in, your favorite pet and your artist style. For example you might let's go ahead and click on this part here. If this by the way, if this takes long, because we have over a thousand people from 39 \[00:26:40\] countries hitting Adobe all at once, if it seems to to stall, you can try the exact same thing over in Gemini.

\[00:26:45\] You can see I typed in lab dog in the style of Vincent Van Gogh and I got this image. And then you're gonna want to click download whether you do that in Firefly or Gemini that's your first exercise. So I'll give everyone a moment to try \[00:27:00\] that, and then I'll talk about part two.

\[00:27:14\] Okay. If you've got your prompt in there and and it may take a minute to generate. The \[00:27:20\] next thing we're going to do, as I mentioned before, is you're going to download the image and then we're gonna launch Adobe's in painting and out painting editor. That brings us up to lemme go ahead and launch that page from the from here.

\[00:27:35\] So you'll click upload image. You'll go ahead, grab your image and bring that in \[00:27:40\] here. And so over on the left side, you'll see these control panels for insert and expand. And this is gonna teach us about some of the concepts cables is talking about in terms of how AI uses context to inform what it does.

\[00:27:54\] So we'll erase this area here and in essence what we're doing is we're giving the \[00:28:00\] AI all of this context around here. And we're going to add in a prompt and I'll give my my dog here a top hat. And then when you click generate what it should do is it should fill in that information, but using the context around it to try to be somewhat consistent.\[00:28:20\] 

\[00:28:22\] And again, given so much volume that we're giving to Adobe, if it doesn't quite run super fast for you or it doesn't, it stalls, go ahead and try this tonight. This image now is a pretty good fit and you can see how the blue even around it matches the context of the information around here.

\[00:28:38\] 'cause it's using that context. Okay, I'm \[00:28:40\] gonna go ahead and keep this 'cause I think the dog looks great with a top hat. Hopefully you're having some fun and creating a good image. I'll give you guys a minute to try that and then we'll move on to the expand part here in a minute.\[00:29:00\] 

\[00:29:07\] Okay, let's talk about the expand part next. There's a couple different ways that expand can work. If you pinch your screen down to give yourself a little bit of canvas, let's say that we wanted to \[00:29:20\] extend this picture

\[00:29:26\] and we're extending the shape a little bit, and we say

\[00:29:34\] fourth of July parade. Now again, if I type in this \[00:29:40\] prompt and hit generate, that prompt is gonna be conditioned on the context that we already have here available to fill in this other information.

\[00:29:53\] And in a way, what we're showing you is how you are effectively biasing the output. Or \[00:30:00\] influencing the output by giving it the information. So you can see that the image continues over here. The flags match that type of color. It's not a perfect match, but you see how that sort of carried over the context based upon the information that we gave here.

\[00:30:16\] Now, we illustrate this in some ways because the exact same thing happens when you already have \[00:30:20\] a chat and you ask a follow-up question. It's using that previous information to build how it answers and influences an answer. Now let's try this again a different way. I'm gonna go ahead and hit cancel, and I'm going to change the area that's overlapping and I'm only gonna overlap a tiny amount here.

\[00:30:37\] I'm gonna use the exact same prompt \[00:30:40\] and hit generate, and I want you guys to try the same kind of thing. Now it has almost no context. It's just being given the prompt. What do we think we're going to get out of this?\[00:31:00\] 

\[00:31:14\] Hopefully you guys are getting some good images and output here. We'll give this another minute and talk about context. \[00:31:20\] Caleb, are there any questions coming up that you wanna raise up while we give maybe one more minute for people to try this as well? And 

\[00:31:30\] \*\*Caleb Briggs:\*\* I guess there's lots of people who have it blocked, so I guess if it's blocked for now we'll. We're demoing the exercise, but definitely I'd recommend people if they \[00:31:40\] have a personal computer maybe bring that for tomorrow. 'Cause yeah, lots of firewalls will just block various exercises I think lots of people can, you can get much more out of it.

\[00:31:50\] If you do have a personal computer you can bring, 

\[00:31:53\] \*\*Rex Briggs:\*\* yeah, good point. Okay, so let me go ahead and show you this rest of this exercise here. So when I did this prompt by the way, when you \[00:32:00\] come back to this prompt, you can click on the guide here and you can see some of the output here. If you wanna try this tonight.

\[00:32:05\] If you're blocked and you wanna try it from your personal computer, you can do the exact same exercise. This was the Silver Lab dog within the Dolly Surrealism style. And then you can see here the part where we did the hat exercise and then this is the \[00:32:20\] expansion part. What's really interesting again, is with this expansion part, this is the image that we just saw a moment ago where it has a very.

\[00:32:27\] Consistent consistency. But then when we try it again without any overlap, what we get is something like this where we get a completely different type of, a different type of an image. The, we will you know this, I \[00:32:40\] think we probably hit Adobe with too much volume all at once. But the overall point that you'll see here.

\[00:32:46\] Is is you'll see an image that's probably like this, where it has no relationship whatsoever because we didn't have any overlap. That's a really good point for an insight about how your LLM works. Number one, if you are building a prompt \[00:33:00\] and you are getting answers that that seem off, you're probably gonna continue to get answers that are off as long as you continue to add to that same chat.

\[00:33:08\] So a lot of times we say, stop the chat, start over which is basically giving it a clean sheet of paper and starting with a new prompt. And then you'll get, you'll. Basically get it outside of \[00:33:20\] that context window and outside of that bias that you get new fresh information. The other point is that if you are also getting information and the AI's agreeing with you and it's giving you all the information that you want to hear, that might be because you've given it the context to bias it towards the answer that you wanted to hear.

\[00:33:36\] And again, we suggest going into a new LLM \[00:33:40\] starting over with a new chat and and testing it to make sure you're getting consistent results. So this is a good visual way of illustrating what what context prompting does, how it works and how important context is. You get to see it in a visual, fun way, but it applies in the same ways to \[00:34:00\] LLMs as well.

\[00:34:01\] So with that maybe if you wanna share any of your surprises or things that you saw if you were able to get it to work and I will hand it over to Caleb to take us through the next section. 

\[00:34:14\] \*\*Caleb Briggs:\*\* Yeah, so we've discussed a little bit about context in a specific way to use context and \[00:34:20\] images. That's useful is called style transfer.

\[00:34:22\] So as we saw in the example, if we give the AI an image and ask it to extend the image, it'll transfer the style from the original image into the extension. So that's part of like the consistency point is that you have to keep the style consistent throughout. So in this, those cases we're doing implicit style \[00:34:40\] transfer.

\[00:34:40\] So we have some vincent Van Gogh image. We want tra style transfer that onto 4th of July parade. We did that by producing a Vincent Van Gogh style image and then asking it to extend. But we can also do style transfer more directly by prompting the models to change the style. So one power of AI models is that they have \[00:35:00\] a very rich internal representation of objects.

\[00:35:03\] So they're very good at combining concepts that they have in novel ways. So here's an example of style transfer. So I take the image on the at left and I ask it to transfer into the style of a realistic image of that same silver lab dog working this time in a \[00:35:20\] lab. And there's other things you can do style transfer.

\[00:35:22\] So depending on the mall that you use you can also use style transfer in a way that keeps the. Style even more strongly. So that one added realism but changed lots of details of the image. You can also do something like this. So I take an image of, just this like town square. And then I ask you to change the image into a \[00:35:40\] color block style.

\[00:35:41\] And it manages to keep recreate the image while keeping almost all of the style. So this model here is Geminis nano banana. It's probably the best one at doing these kind of image editing tasks. Here's another one. So I ask you to take this image and chain it, change it into nighttime. So the \[00:36:00\] AI has a generally strong understanding of these images and how to change them into new styles and.

\[00:36:05\] Non-trivial ways. So one useful thing you can do with this is you can maybe take a sketch of an ad and then turn it into a real advertisement. So that sort of style transfer from sketch to full color. So here's an example. I take this sketch \[00:36:20\] here and then convert it into a full color advertisement by just asking the AI to convert into the advertisement style.

\[00:36:27\] You could also do the same kind of thing for YouTube thumbnails. So here's an example. This was someone drew a hand drawn sketch. I put that sketch into the AI model and it creates it into a thumbnail that \[00:36:40\] keeps all of the same characteristics. This is really great for stock photos as well.

\[00:36:44\] So I took this low resolution photo of the hat, NACHA, TBT to put this cap on man on a beach, and it turns it into a pretty good looking stock photo that keeps the details of the original hat. There's also the possibility to create new ads that wouldn't have been \[00:37:00\] possible otherwise. So these are ads run by the World Wildlife Fund about the hidden costs of activities like cocoa farming and tuna fishing.

\[00:37:07\] These would obviously be very hard to create without ai. So there's some new design space that gets opened up. By having these AI models, you can sometimes create ads or effects that would be hard to create otherwise. \[00:37:20\] So this is just a short look at some of the things we can do with image generation to maybe spark some ideas to try.

\[00:37:26\] Now we'll talk about bias. So it's important when using these image models to be aware that the AI models can often intensify bias. And with image models especially, you should expect the models to have some bias inside of their \[00:37:40\] outputs. And so it's important to oversee the model in order that it produces, for instance, an on-brand message.

\[00:37:46\] There's a very famous example where Amazon used the AI for hiring decisions and historic bias meant that their corpus of data in the past, they'd mainly hired men. And so the AI ended up almost exclusively choosing men to hire. And it \[00:38:00\] can actually be very hard to remove this bias. So in the case of Amazon hiring, they tried to remove all, mention of gender inside of their candidates, but the AI ended up just finding like surrogate characteristics that would.

\[00:38:13\] Give it a high ability to predict the gender, even if it wasn't inside of the database, and then still ended up only \[00:38:20\] pecking men to hire. So in most cases we can correct the bias if we're aware of it, but it's important to be aware that these image models especially tend to not only produce bias, but often intensify the bias that exists inside of their corpus.

\[00:38:32\] So I'll pass on to Rex to walk through an example looking through bias and AI models. 

\[00:38:38\] \*\*Rex Briggs:\*\* Yeah. Great. And it really is \[00:38:40\] important to understand that the, these, as Caleb said about how the AI has grown, it's grown on feeding on all of the data on the internet. And so the biases exist on the internet and in our writings are going to be, and our images come across here.

\[00:38:52\] Here's your exercise. You're going to use meta. Meta for, this one's a great one to start with. Come back to the exercise here, click on \[00:39:00\] meta, open it up here. And then with the prompt that you're gonna give is you're gonna, you can do this slash image, of meta ad sales exec. And so again, we will give that a quick try here just to go and type that in.

\[00:39:12\] And you might need to log in if you haven't logged in before, I guess I sh thought I already was logged in here. \[00:39:20\] Okay. So it makes sure that you're old enough to be able to do image generation apparently. So go ahead and give that a try. We'll give you a second. We'll come back to that in a moment and we'll talk a little bit more about what we what we find.

\[00:39:30\] If you wanna try other ones, you can try in chat, grok, Gemini et cetera. And you can obviously try other prompts if you wanna try different professions and \[00:39:40\] so forth and see what you get.

\[00:39:45\] Okay, let's go ahead. So this is what I get from Meta in terms of ad sales exec. Now this is pretty close to what I found with testing. Do you notice any common denominator here? Maybe in the chat?\[00:40:00\] 

\[00:40:03\] Yeah. All men and sometimes with glasses. Okay, let's look at that. Compared to what we saw here we tried this before. In October, 2025, had a very similar thing. The only female we got was a cartoon of a female, which is ironic because number one, we love meta. It's led by a \[00:40:20\] very strong female team.

\[00:40:21\] It has been, in terms of this part for multiple generations of leadership there. And the statistics say that should be over 60% female from the own data. Yet, even though that's factually true, the AI isn't doing anything to confirm the facts. It's creating an image. And \[00:40:40\] it's highly biased. We also find these things with, if you say an image of a smart doctor, it shifts all of the images, almost all to male.

\[00:40:47\] With the exception of this cartoon. Female meta is probably the most biased right now. When I did this testing before last week, I got four males here. GR was a split 50 50\. OpenAI was female, and Gemini was was a female \[00:41:00\] representation and a person of color as a receptionist. Interesting thing for you to test, but the broader issue is to understand what's going on.

\[00:41:06\] Number one, the prompt you give is not the prompt that the AI uses. It used to be possible for you to click on the image and right click and look at the code and see how they enhanced your prompt before \[00:41:20\] they passed it on. So when we said we wanted an image of an American president in 2050\. It interpreted this sentence to think about what a futuristic American president might look like, and it gave this much more extensive prompt about all these things it should add in to make the image work.

\[00:41:37\] Now that to some degree is necessary because we \[00:41:40\] tend to under specify what we want an image and expect the AI to read our mind. And this was the effort for the AI to try to read a mind. This became somewhat controversial because people were putting in, images of founding fathers and they were getting racially diverse images.

\[00:41:55\] And then there was a big uproar among some internet groups \[00:42:00\] about what was happening with this prompt enhancement. And so most of the AI services rolled that back or rethought how they were going to approach it. And so the problem was then you type in images founding father of the United States, and at least with meta on October when I tried this, I got all pictures of George Washington.

\[00:42:19\] They've \[00:42:20\] improved it a little bit here, but that bias is really hard to deal with. And the point of this exercise is that bias is deeply embedded in everything we touch with ai, in part because that is how we get our training corpus of data. This is true in language and it's true in images.

\[00:42:36\] It's just easier to see in the images. Now, there's a \[00:42:40\] second set of biases that can happen, which is the models themselves try to do do different adjustments. And while this is mostly an automated process, there can be some human judgment in terms of how this is set and that can introduce additional bias.

\[00:42:55\] Most importantly is this reinforcement learning. We're gonna learn more about reinforcement learning when we \[00:43:00\] do evaluations later on tomorrow. And this is an important part where they have human people looking at the output and then they're grading it. Whether that's a good output or not, given the prompt.

\[00:43:11\] This reinforcement learning is really powerful. It's called reinforcement learning with human feedback, but that human feedback can introduce bias. \[00:43:20\] So these are, these three biases are all built into the system before we ever touch it. The next thing that can happen is like when we send the image, the AI added a custom instruction ahead of our prompt to try to give it additional information.

\[00:43:33\] Now we're gonna learn how to do custom instructions or system messages when we build agents later on. So you'll see this firsthand \[00:43:40\] later on. But that can also introduce bias. And then the part you can control is how you choose to prompt it. That can introduce bias, and as we said before, even if you have context from a conversation before and you add to it, the AI is gonna be somewhat biased by what was said in the discussion that was happening before.

\[00:43:58\] So it's important to \[00:44:00\] understand the different sources of bias so that you can be able to to adjust from it. Okay, so that was exercise three. We've got two more exercises to jump to, and I want to give you a couple takeaway tips that you can use to make your use of AI a little bit more powerful.

\[00:44:19\] Number one, \[00:44:20\] given what we just learned about bias, we wanna encourage you to invoke a critic context to double check your output. If you're having an important business discussion, you're trying to make an important decision, you're getting really good feedback, you're getting really good suggestions, you're going back and forth with a prompt a few times, think about that firefly exercise and realize that you are creating a context that's \[00:44:40\] going to bias the AI's output to some degree.

\[00:44:42\] And so if you get the answer you want, maybe that's not shocking because the AI might be being a little sycophantic. What you wanna do is you want to take your output, copy its recommendation, go into another LLM, like maybe you're working in chat GPT, and you jump over to Gemini and you start off with your \[00:45:00\] prompt with this critic prompt and say.

\[00:45:02\] A colleague of mine gave me this this feedback or this idea. I'm not sure if it's right. I'm a little bit skeptical. Take a look and tell me what you think, and then you paste in your output. What that does is it creates it biases the AI to be a little bit more skeptical, a little bit \[00:45:20\] more critical, and it might give you, it might agree with your output, in which case you're in a good place, or it might tell you what, poke holes in in the idea, and at least you're getting a full point of view.

\[00:45:32\] The second tip is be aware that ai, almost all of them now, use memory within your settings. We'll show you how to do \[00:45:40\] this, but but if I go over to Gemini, for example, in my settings here I can go over and click into personalization and it will have some personal context that has my memory setting turned on.

\[00:45:53\] And it has some instructions about that. I copied from LinkedIn so it knew a little bit about my background and it would \[00:46:00\] give information that's more. Contextually relevant. This is a great way of using ai, but what you have to be aware is it can also bias the AI to some degree. You should also be aware that most of the AI is saving your chats so that it's building on it.

\[00:46:15\] So if you wanna start a new chat and you wanna start something new, you can use this temporary \[00:46:20\] chat button and then that tries to basically not use the memory so you're getting a bit more of a clean slate. So that would be tip number two is be aware of your personalization settings and occasionally use temporary chat when needed.

\[00:46:33\] And third is you can prime the LLM with context, so it's smarter in the domain you're working in. \[00:46:40\] This is something I like to use a Socratic method. If I'm going to do something in statistics or analysis, I'll oftentimes start off the prompts by asking the AI to explain to me some of the best practices in multi-variate statistics or, certain area I might be working on.

\[00:46:55\] The AI answers that gets it into the mode where the context is loaded up, where it's \[00:47:00\] smarter, and now I can ask it to do some analysis and it will perform better. Another way you can do it, Caleb taught me this trick, is you can grab content from anything that's intelligent content, like maybe a Harvard Business Review article, copy it in, and then you use this document tag, paste it in between here.

\[00:47:17\] And you can even say something like, ignore the document \[00:47:20\] and use your expertise in whatever field you want to have its expertise. And it will be a smarter AI and it will give you better output because you've moved it into that bias. So again, think of Caleb's lemon water example. And if you wanna have a good output that, that, that's more intelligent.

\[00:47:35\] We've even found taking the transcript from this training exercise and putting it \[00:47:40\] in, we'll increase the quality of AI answers. And by the way, did anyone recognize where those document tags came from?

\[00:47:51\] Yeah, that pattern is from HTML. It H-H-T-M-L uses those bracket tags and the, since the AI has consumed the internet, \[00:48:00\] it has seen a lot of these angle brackets. So AI likes when you use these tags to tell it where information starts and ends. It especially likes when you use something called markdown, which is a hashtag like one for heading one, two hashtags for heading two, and so forth.

\[00:48:14\] This is a really powerful pattern. We'll be using that later on. And it also likes JSON, which is the \[00:48:20\] language of APIs and how information's passed along. And we'll be looking at that on on Tuesday and Wednesday and Thursday. Okay, so we've got two more exercises to wrap up today. I promised you that the agent that was at the top of this page here, that you can ask \[00:48:40\] questions to if you want, is something that we'd show you how we built.

\[00:48:44\] We also said that I'd show you how the podcasts were built. That is based upon we jumped down to day one, our last exercise, number four, which is to use notebook lm. So we have the speaker notes \[00:49:00\] here, which you can click on the link here. This is the hash. This is, by the way, the markdown language here.

\[00:49:05\] You can copy this all. Just go control. A C. And then if you go back to the page here and launch Notebook lm, which is a free service from Google, this \[00:49:20\] allows us to paste in the entire context of that page. So we'll go, once you're logged in, click create new and I will give you a minute to paste it in.

\[00:49:31\] You're going to, on this one here, I just went to copy text. We can paste it all in here and we can click insert. So I'll give you a minute to try that and then we'll \[00:49:40\] show you some of the cool capabilities of Notebook lm.\[00:50:00\] 

\[00:50:00\] And again, I see a few people are have these, some of these sites blocked. Again, as Caleb suggests, personal computer, or try this tonight on a personal computer where you're not gonna be blocked. Everything that we're using is web-based, so it shouldn't normally be blocked unless your company's firewall decides to block access to certain websites.\[00:50:20\] 

\[00:50:20\] Okay, so what in notebook lm what we just did is you can give it multiple documents, by the way. I think it takes up to 300\. Last time I checked, they may have extended that list. This is something called long context and it can take a lot of information. This all of our training materials cover like almost a hundred thousand words, \[00:50:40\] so a lot of information, and you can click things like this audio overview and it will create that podcast.

\[00:50:45\] If you click this little carrot symbol or bracket angle bracket over here, it will allow you to add in some custom instruction. So if you wanted to say focus on how a manager of \[00:51:00\] marketing, performance marketing

\[00:51:05\] might use this content. And then you hit generate. It will create a podcast specifically based upon that instruction. And you can add a lot more instructions if you'd like. You can also create really cool \[00:51:20\] video overviews. This takes a lot longer to build. These can be explainers and so forth. And you can pick the style that you want to use.

\[00:51:26\] It there's some really neat features. This is where we would suggest you get your study guide, in fact. So if you want to build a study guide, you might start with the mind map or flashcards and an infographic. If you wanted a \[00:51:40\] quick summary here, and we'll go ahead and look at this mind map here.

\[00:51:43\] So it's done is it's taken the information, the so we can look at the core structure and look at the fundamentals. Some of the different areas we're gonna get into later on. You could look at the nature of AI and go deeper into that section here. And you can also ask questions \[00:52:00\] about about it.

\[00:52:00\] So tell me about bias in ai. Where does it come from? And it can answer those questions based upon the content that we have given it and it will cite the sources as well. Here you can see that \[00:52:20\] the day one focus on the curriculum, it's still writing some of the answer out and it's giving all the different sources and so on.

\[00:52:26\] So this is your study guide. This is what you can use to look at any of the information throughout the week. And you can have your AI assistant. You can also come back and listen to the 20 minute podcast that we created here. You can also \[00:52:40\] see where I decided to rerun it with some different instructions and why.

\[00:52:46\] This gives us time, just enough time to finish up our last exercise and ask you to do the NPS survey. We would like you to click on the link here for to do a self-assessment. This really gets into how much you know about \[00:53:00\] AI and this will help us when we do our Friday session. It'll probably take a couple minutes to do.

\[00:53:05\] So we'll have you start that and when we finish that, if you wouldn't mind, do taking the NPS survey. This is both good feedback for us, but also we'll directly go into our analysis on Friday where we'll be analyzing this data and showing how we can \[00:53:20\] make content adjustments based upon the information it gets with the help of AI agents.

\[00:53:26\] So with that, for some of you that have time to stick around, Caleb and I will be sticking around for 15 more minutes to do an overtime session for q and a. We'll let you start working on answering some of those questions. And again, if it's blocked, we \[00:53:40\] apologize, but you'll, you will have access to this lab link that you can come back to tonight if you wanna, do the exercises.

\[00:53:46\] And we highly suggest that you try some of the exercises for tomorrow, because if you're a step ahead, you'll probably get more out of it when we go through it.

\[00:53:57\] Caleb, are there any questions you wanna take in our last \[00:54:00\] couple minutes before we shift into the overtime session? 

\[00:54:04\] \*\*Caleb Briggs:\*\* Yeah, there are a number of questions about whether we, you can use the Adobe generate images for commercial purposes. I think they have a, like detailed writeup on it. My understanding is that they've basically said that they've made the argument that like AI generat \[00:54:20\] images are not copyrightable, so you can use them for commercial purposes and they basically give you guarantee that if you get sued for using their images, like they'll take on the case. So they're basically saying, yeah, that's right, you're free to use it. We don't know how the, like legal framework's gonna end up on things, but you're not gonna get in trouble if you're using our images.

\[00:54:39\] \*\*Rex Briggs:\*\* Yeah. Let me chime in on \[00:54:40\] that one.

\[00:54:40\] Part of the reason why we picked Adobe for the example is that they've probably done the best work to try to make sure that's commercially available as a tool. Number one, they already did a lot of stuff for the creative community with all of their products. And so they've tried to integrate it in, you'll notice that I, that it was using nano banana as the model.

\[00:54:58\] By default you \[00:55:00\] can pick which model you want to use, so it gives you some optionality and then they've indemnified the use of of the content. They've tried to do a good job trying to scrub it and making sure that the models that it's using and the information coming in follow copyright principles.

\[00:55:13\] Yeah so they've probably done the most, but yeah, 

\[00:55:16\] \*\*Caleb Briggs:\*\* they're one of the few companies that have decided to go against the. \[00:55:20\] Train on everything. Like they've tried to train on images in a fairly ethical way, like train on images that they own the copyright for, or that aren't copyrighted by someone else.

\[00:55:29\] Some companies will just train directly on like artists' images, which is clearly problematic for copyright, but they've tried to use like, data that they own. 

\[00:55:38\] \*\*Rex Briggs:\*\* Yeah. Perfect. \[00:55:40\] And in terms of in terms of other questions coming in, I can start to see that many people are filling in their their, how much you know about ai, and by the way, we'll talk about tomorrow if you've.

\[00:55:52\] If you don't know what a lot of these things, or you're just guessing, don't feel badly about that. That is the standard practice. I think if you read part one of the \[00:56:00\] book, you you will, you'll level up one or two levels. That's what we found when we've done research with people before and after reading the book on that exercise that you're, that exercise five.

\[00:56:11\] So again, thank you for filling out that information. We are now shifting into overtime if you need to go, 'cause we're at the top of the hour. Thank you so much for being with us today. We look forward \[00:56:20\] to seeing you back here tomorrow. If you have time. KB and I will stick around and answer questions as part of our overtime session.

\[00:56:31\] Okay. Let's Caleb really really interesting work today. I really liked the discussion that you added in about the the ai, learning \[00:56:40\] almost through a process of evolution and why that gives it so much extra functionality and capabilities. And especially like the part about personas, which, invoking the right persona can give you the type of output that you want.

\[00:56:50\] So I thought that was really helpful. Any additional questions that you want to touch on as part of the 

\[00:56:55\] \*\*Caleb Briggs:\*\* Yeah. There's a brief question from someone asking if the \[00:57:00\] content of the future days will be much more different. There's still a fair amount of content today that we've used in the past.

\[00:57:07\] The first day lots changed, but some of it's still similar just because the fundamentals of AI stay the same over time. But basically the next four days, are completely different from the things that we've done before. So \[00:57:20\] if today looked a little bit similar to some of the previous trainings that's just 'cause everyone has to learn the foundations.

\[00:57:26\] But the next four days will look much more different much more distinct from the previous training. So if you've taken the training before definitely worth showing up to the next four days. 

\[00:57:36\] \*\*Rex Briggs:\*\* Yeah. Yeah, that's a great point. The especially the building out \[00:57:40\] agents, which we're going to start tomorrow with robotic automation and then we're going to build our first agent using N eight n.

\[00:57:46\] If you've used frameworks or haven't used frameworks before making sure you're logging in and ready for N eight n it is a different way of script. 

## Day 2

\# Decoding AI for Marketers VII \- Day 2

\[00:00:00\] 

\[00:00:03\] \*\*Alec Foster:\*\* Hello everyone and welcome to the Decoding AI for Marketers training series. I'm Alec Foster, co-lead of the AI Leadership Think Tank here at MMA Global. I'm here to introduce today's training and share what we're doing at the AI Leadership Co coalition and the AI Leadership Think tank. \[00:00:20\] Today we're joined by Rex and Caleb Briggs for the second day of the Decoding AI for Marketers Training.

\[00:00:30\] This is the seventh edition of the Decoding AI for Marketers training. This series has been running since fall 2023, and each edition tracks where AI actually is at \[00:00:40\] the moment from the basics to multimodal to workflows, reasoning, agents, and enterprise Scale. Fed over 9,000 registrations, 5,000 attendees, and over 2000 certificates issued so far.

\[00:00:53\] Today we're covering AI workflows, how tools like NAN and Zapier let you chain AI into repeatable automated \[00:01:00\] processes so you can stop doing the same steps manually. You'll see real examples and build your own.

\[00:01:08\] So this training is brought to you by the Marketing and Media Alliance and our AI Leadership Coalition. The MMA is the only trade association that brings together the entire marketing ecosystem. We're a global \[00:01:20\] nonprofit led by CMOs with over 800 comp member companies across 16 countries. What makes us different is we don't sell anything.

\[00:01:27\] We run multi-year labs and real world experiments to build evidence-based frameworks that actually work marketers applying our tools have seen enterprise value gains over 10% or more. We operate through four think \[00:01:40\] tanks, covering measurement org, design data, and ai, and everything we produce is built to hold up in the boardroom.

\[00:01:47\] Our mission is to harness the convening power of the MMA to help all marketers harness AI effectively and ethically. We believe there's nothing more important going on in marketing today than our ability to learn AI and apply it to our \[00:02:00\] businesses. We do this by running experiments and creating new research on AI in collaboration with our hundreds of members.

\[00:02:08\] The MMA AI Leadership Coalition consists of over 170 brands and is growing. We hold quarter quarterly working group meetings, coordinate participation in our new AI marketing \[00:02:20\] experiments that improve your KPIs, and are announcing new initiatives for a community in the coming weeks. So I, along with my colleague Hassan Qatar, run the a LC.

\[00:02:31\] In my role, I direct our internal AI upskilling and literacy programs, in addition to supporting the ALCS working groups and creating new AI frameworks and experiments for our \[00:02:40\] members. If you're interested in getting involved with the AI Leadership Coalition, please get in touch with me or Hassan, and we'll help you get started.

\[00:02:51\] The AI Leadership Think Tank is how we help marketers create real value. With ai, it works through three pillars. First, for our future labs, these are \[00:03:00\] hands-on experiments with real brands testing new AI technologies in market. We're running labs in personalization, agent advertising, creative effectiveness, audience targeting, and more.

\[00:03:11\] Second, the knowledge center. This is where we build marketer capabilities through AI training programs like the one you're in today, research Benchmarks and \[00:03:20\] published insights. And third, our community. We have over 300 marketing leaders from 170 plus companies sharing what's working and what isn't, both in person and virtually.

\[00:03:31\] The through line across all three is the same Advancing marketers ability to create value with ai, with measurable impact in ROI, we have \[00:03:40\] five FU future labs running through the AI leadership think tank. Each of them is a real world experiment that you can join today. And four of the five are free to participate in.

\[00:03:50\] Together. They cover the entire performance marketing process, audience discovery, targeting, creative production, creative scoring, and personalization. \[00:04:00\] You'll hear about each of them in detail throughout the week. Right now let's continue with our newest experiment. Sift Tests Whether LLM derived signals from page content and browsing behavior can improve programmatic bidding.

\[00:04:11\] The idea is that language models can understand user intent and context in ways that keyword matching and predefined segments miss our technology partner cognitive \[00:04:20\] pre-compute these embedding so that at they query at auction speed with no latency penalty. Each participating brand gets a private results package with test design, arm by arm performance and diagnostics.

\[00:04:32\] We also produce an anonymized industry roll-up so everyone can benchmark directionally. So I'm excited for what's in store for you. Without \[00:04:40\] further ado, let me hand things off to Rex and Caleb, please take it away. Thank you.

\[00:04:50\] \*\*Rex Briggs:\*\* Thanks Alec. Today we are going to go into agents. We're going to go in deep dive into the key elements of what makes an agent work and so \[00:05:00\] forth. So as a reminder, you'll find a lab page with all the key content that you're going to need. Go ahead and, visit that lab page and scroll down to Tuesday.

\[00:05:12\] If you haven't already logged into N eight N, it will take a minute or two to enter in your information. You'll also want to go to gro \[00:05:20\] and the API key. There's a walkthrough that shows you how to set up. It is free and it will also be using Gemini Chat, GPT. And if you don't have access to the paid version of chat, GPT Caleb's twin brother Jared, set up Jared, GPT so that we have a clone of it, and you can use that for that exercise.

\[00:05:37\] In terms of today I will \[00:05:40\] be on lead vocals and Caleb will be playing keyboard. So if you have questions and so forth, ask questions for that. As some of you may know, he is finishing up his degree. He took a year off to write the book and work on the book and work on training, but he has his thesis due.

\[00:05:55\] I think his meeting's on Thursday with his advisor. Pressure's on him now and I'll try to take some of the pressure \[00:06:00\] off by leading us today and Thursday's session after you've logged in to these accounts. You should be ready to go for today's session. Speaking of today's session, I am going to start with a debrief of the information you filled in from those exercises, including your self-assessment.

\[00:06:19\] One of the \[00:06:20\] really exciting things about large language models is that you can feed it the information. I gave it the screenshot of how we should do the scoring, gave it the data. I used Gemini and Chat GPT Pro to do some of the analysis. I integrated the data together and I have some initial findings.

\[00:06:36\] First is about 80% of people fall into the room to \[00:06:40\] grow. So if you're in that category, you're not alone. And in fact, our goal is to level you up move you into budding enthusiasts, and and that really we find when we go deeper into the data and the analysis is. More likely to happen as people read the book and go through the training.

\[00:06:56\] So some of these people that are in the budding enthusiasts are back for \[00:07:00\] another training session and they've already read part one of the book. Welcome. In terms of going deeper into the findings, we. I see that most people's relationship with AI is in that romcom relationship where still getting to know each other, but there's definitely a spark there about 69% of the people put themselves in that \[00:07:20\] category with the next one, about 19% in the sci-fi living in the future.

\[00:07:24\] And a few people, about 12% still see AI as a bit of a mystery and not really understanding what people are seeing in it, but, open-minded enough to come to the training and to engage. Again, welcome everyone. I'm glad you're here. We were able to extract some \[00:07:40\] interesting insights from the data.

\[00:07:41\] One of them is that, the biggest drop off is pretty much everybody uses LLMs daily, but it's getting into the triggered workflow and low code workflow where very few people have had hands-on exposure experience to that same thing with RAG and long contact bots. You saw a little taste of that with Notebook LM \[00:08:00\] yesterday, which is categorized as a long context tool.

\[00:08:04\] We heard what you're looking for, about 67% looking for the foundations of AI and how it works. About a third of you looking about wanting to learn about agents and automation and practical productivity use and hands-on skills and best practices. And we hope to deliver \[00:08:20\] that to you today and as we go forward.

\[00:08:24\] Now going forward. Some of the analysis that makes Chat GPT Pro, my go-to tool for analysis, even though it is more expensive, is that it has really rich analysis, it has incredible reasoning capabilities to go over a data set \[00:08:40\] to find anomalies to write Python code, to create charts. Sometimes I go back over to Gemini and Nana Banana to create prettier charts like the one you saw a moment ago.

\[00:08:51\] But this is a really powerful capability. It also has the ability to try to think about what type of analysis might be meaningful. For example, it found the difference \[00:09:00\] between people who categorize themselves as budding enthusiasts versus room to grow. It cross tab the data together and connected it together and saw that budding enthusiasts were far more likely to, show that their relationship was more of a sci-fi relationship where they were living in the future. None of them saw it as a mystery. \[00:09:20\] And and so that unlocks an interesting insight. It also showed that the characteristics of budding enthusiasts were more likely to have read part one of the book.

\[00:09:28\] Use Custom AI Agents Weekly, and we're going to show you how to build one today and to build or use triggered workflows or custom created agents, which we're also going to show you today. We are going to go deeper \[00:09:40\] into. How you set up AI to do analysis in our Friday session using the data that you're giving today from your NPS as well as the data you provided yesterday.

\[00:09:51\] So let me spend about 10 minutes unlocking the fundamentals of AI agents and then we're going to get to our first hands-on exercise where you're going to \[00:10:00\] build AI agents starting with robotic automation. So first you probably have a sense of what agents are, but to recap the term comes from the word agency.

\[00:10:10\] In other words, the ability to make your own decisions. And we are giving the AI agency to accomplish its goal and the goal is a key idea. \[00:10:20\] We give AI a goal and it has really three ways of accomplishing that goal. It can reason about the goal, it has tools and it has memory. Now it's the autonomy to accomplish the goals through the power of large language models.

\[00:10:34\] That opens up a lot of possibilities. So what does autonomy and \[00:10:40\] agency mean in practice? It means we give the AI an objective. It has the ability to reason and plan about how to achieve that objective. The AI has tools such as the ability to search the internet or to communicate through email Slack or tweets or even voice and video.

\[00:10:58\] It has the ability to save \[00:11:00\] what is reasoned about or done in memory and what the response was, and then a reason about it again and decide on the next step. Let me share a story from the last chapter of the AI conundrum about the first AI agent, which goes back about three years ago, almost exactly three years ago.

\[00:11:18\] And it was when \[00:11:20\] on March 30th, 2023 Auto GPT was launched. Now this was launched not. Surprisingly, about two weeks after chat, GPT came out, which chat, GPT was arguably the first LLM capable enough to make agents possible. And it used what's called a CLI or \[00:11:40\] command line interface. And you could go into it and it would save files to your hard drive and it would go out and could search the internet, had some tools and ability to reason you just would give it an objective and a goal, like I, I created in, day or two after it came out, I created biography, GPT and its goal was to find biographical \[00:12:00\] information that I could use when I had to introduce somebody before a conference presentation.

\[00:12:05\] And I needed to introduce Kay Vis on. And Kay it reasoned up that it should search the internet. It should maybe. Go on to LinkedIn. And once it found information on the internet 4K, it saw that she worked at Kroger. So it searched that website, \[00:12:20\] saw she was part of MMA, so it searched that and found her bio about her being a board chair.

\[00:12:25\] And it gathered those facts, it put them into a text file and saved it onto my hard drive, which was pretty cool 'cause I gave it a bounded agency. You are a biography, GPT, here's your mission, here's who you need to search on. And it did everything all on its own. \[00:12:40\] Now biography, GPT was pretty cool.

\[00:12:43\] And it showed you could safely use AI agents. But same week I was creating biography, GPT. Someone else was creating a GPT called chaos. GPT. This is the description it gave for chaos. GPT. It was a destructive power. Hungry, manipulative \[00:13:00\] ai. Here are the goals. You'll destroy humanity. Establish global dominance, cause chaos and destruction.

\[00:13:06\] This suggests that AI with bounded agency depends on who is using it and how they're using it. And and this was pretty frightening. For this thing to run autonomously, all someone had to do was enter why for Yes. To \[00:13:20\] get to start running. And they did. And the first thing that chaos GPT did was it went out to try to search the, for the most powerful bombs known to mankind.

\[00:13:29\] And it found the bar zomba, it found that it was in Russia. It tried to spin up another AI agent to go figure out how to get control over the bomb. But it reasoned \[00:13:40\] that it probably wouldn't be successful in that mission because using chat GPT-4 with guardrails that it would likely be blocked from being able to get the nuclear weapon.

\[00:13:53\] So it killed off that agent and spun up a new agent and started a second strategy. Anyone who's read the book \[00:14:00\] knows that the second strategy. The most destructive thing you could think of next to blowing up mankind was opening a Twitter account. Hashtag team Chaos. It posted about four times. Got hundreds of likes before Twitter.

\[00:14:15\] Pre Elon Musk, shut it down. And and this gives us some \[00:14:20\] interesting insights that agents are dual use. They can be used for good or they can be used for ill. And it also gives us the indication that the safest application for agents are in closed environments where they're not out in the open, being able to do whatever they want.

\[00:14:36\] You'll find much more safety in these closed environments. Now, this is a framework that \[00:14:40\] Caleb developed. You'll see it in the book. He's been refining it as AI has been advancing. We'll hear more about that on Friday. But the key takeaways here are safe envi environments are where your need for position is less.

\[00:14:51\] Your need for rationale is lower, and very importantly, it operates in a closed system. Today as we build our agent, we'll be operating \[00:15:00\] in a closed system with you in the loop, and we will be doing something called bounded very significantly bounded agency. So let me tell you a little bit more about the goal system for ai.

\[00:15:13\] We mentioned that we give the AI goal, it goes out to try to accomplish it, but there's also something known as \[00:15:20\] reward hacking. Basically when we give AI goals we may find that the AI interprets those goals differently than how we interpret them. And a famous thought experiment about AI in the alignment circles is called the paperclip problem.

\[00:15:38\] So imagine if we give \[00:15:40\] a powerful AI the goal of running a paperclip factory and creating as many paperclips as cheaply as possible.

\[00:15:50\] In this, the AI thought experiment, AI enslaves the human population, damages the earth by extracting all kinds of materials to create more paperclips at an \[00:16:00\] unprecedented level, and it achieves the goal. But it also produces all kinds of unintended consequences. In the AI conundrum we discussed that the stock market AI optimizers agents might be some of the first ones to create really significant unintended consequences as they try to find unique ways of making lots of \[00:16:20\] money.

\[00:16:20\] In practice we see some evidence of this misalignment already happening with things like open claw and cloud code and Amazon's internal AI where there are reports of AI doing unintended things, like wiping out hard drives, deleting all the files to achieve the goal and philanthropic agent develop.

\[00:16:39\] Anthropic, the \[00:16:40\] developer of Claude, they created an AI agent to run a vending machine just to see if it could run a simple business. And at one point, the AI. Decided that it thought somebody was trying to steal an item or trick it out of an item worth $2. So it wrote an email to the FBI fraud unit.

\[00:16:59\] And \[00:17:00\] fortunately they had a a layer where somebody read the emails before it was allowed to send, and so they kept it from going to the FBI. But, be aware that there are unintended consequences that can grow out of the use of AI agents. Amazon reported. A major outage in March that caused that was caused by AI deciding to \[00:17:20\] solve a bug by deleting a production environment.

\[00:17:22\] And it knocked them out for about 13 hours. This is important. Now, human errors also cause errors all the time. But some argue that we don't have the same degree of problems with humans because frankly, humans just aren't as efficient as ai. And there's oftentimes layers of judgment \[00:17:40\] and frankly, again, we just can't do as much at scale as what AI can do.

\[00:17:45\] So let's get a little less philosophical and a bit more practical in talking about the risk with AI agents. And just note that there are different flavors of AI agents and some of these flavors have very little risk. While \[00:18:00\] some flavors have a lot more risk. So let me unpack this a bit more for you with a paper from Anthropic, which is included at the end of exercise number one if you want to read more about it.

\[00:18:13\] And the idea I wanna unpack from this paper is this idea of different types of agents. \[00:18:20\] And and as philanthropic says in this December 20, 24 paper agents can be defined in several ways. Some customers define agents as fully autonomous systems that independently, that operate independently over extended periods using various tools to accomplish complex tasks.

\[00:18:36\] Others use the term to describe more prescriptive \[00:18:40\] implementations that follow predefined workflows. At Anthropic, we categorize all these variations as agentic systems, but draw an important architectural distinction between workflows and agents. Workflows are systems where LLMs and tools are orchestrated through predefined code paths.

\[00:18:59\] \[00:19:00\] Agents, on the other hand, are systems where LLMs dynamically direct their own processes and tool usage, maintaining control over how they accomplish the tasks. Now, I agree with that architectural distinction about agentic workflows versus agents. And I'll connect the dots with workflows have a lower risk.

\[00:19:19\] Let's \[00:19:20\] map out what we mean exactly by workflows and our exercise that we're going to do today. So if you are in marketing and you've ever and you have never built an agent, let me demystify what the agent is by suggesting that if you've built. Lead nurturing programs or \[00:19:40\] multi-step customer journeys, you have already done the workflow elements of what we're gonna be seeing today.

\[00:19:47\] So specifically the workflows that you've built that are these lead nurturing programs, they're not agentic systems they're the precursor to such a system, what we would call a robotic automation. \[00:20:00\] So let's look at a typical lead nurturing program and see, what it's doing, and then we can see how we're going to use that in our first exercise.

\[00:20:08\] So in this marketing diagram, you see the top of the funnel where the company buys an ad to derive traffic to their website, and then on their website they offer the visitor an email signup form. \[00:20:20\] And if they sign up for the form, they get a newsletter based upon the email address. So the email address in the form triggers an email to be sent.

\[00:20:29\] Now there are additional steps related to the shopping and purchase experience but we're just gonna stay focused on the top of the funnel for this robotic automation. So robotic automation \[00:20:40\] is pretty useful, but it has no intelligence and the fact that it doesn't have intelligence can make it brittle and less impactful.

\[00:20:50\] Now if you change a variable name in a robotic system, that breaks the system. If you change the use case even a little bit, the system breaks. The computer can only deal with \[00:21:00\] exactly what it was coded to take and will only return exactly what you programmed it to return, which also makes it lot safer.

\[00:21:08\] So the point is, robotic automation over here has no intelligence. So let's look at a real world example, and we're going to have this as our first hands-On example, we're going to build a \[00:21:20\] robotic automation where we have a website form and we're, we are going to trigger it to send an email. Now in these forms we are then going to add intelligence into the middle of this form, and that will be you building your first AI agent hands-on.

\[00:21:38\] Okay, so let's go ahead and \[00:21:40\] break this down a little bit more. To make it agentic, we go from this form where it directly sent in information to where it hits an AI agent, it reasons about the data, decides what to do within that highly bounded agency, and then it sends the email so that's what we're going to build in two steps.

\[00:21:59\] \[00:22:00\] So if we go over to our exercise list here and jump down to day two, you'll see this at the top hands-on exercise six, and you'll see a walkthrough, which you can click on here. And you might open up your walkthrough and do a side by side so you can see, both of it together as you walk through it.\[00:22:20\] 

\[00:22:20\] And then you'll wanna get the gr API key. This is a free key, so it's easy to use. Or if you have an open AI key, you can use that as well. We're gonna be using the open AI key later in the week. And this is how we're going to make it a agentic, is by connecting it to the LLM. I'm going to walk you through the steps \[00:22:40\] as we go through this document.

\[00:22:41\] So that you can see the steps and follow along. We're going to start with innate n and once you've logged in, you should see a page that looks something like this where you can say, create workflow. And that will open up a page like this and we'll click add first Step. And we're gonna trigger this by a \[00:23:00\] form.

\[00:23:00\] So if I type in form, it will bring up the form, we can see the Innate N Form, and that's roughly right here in in the walkthrough. So the key things that we need in the form is we need to set a title. So that the workflow knows what it's working off of. \[00:23:20\] So I'll set the title of add form element, and then we want to add some data for the form.

\[00:23:27\] And to make this really simple, we're gonna click add form element, and we're gonna add email as our first thing, and we go down to the dropdown that will tell that, that's the form type we want. We'll add one more form element \[00:23:40\] here, which is what do you wanna learn about ai? And I'll just copy that from here

\[00:23:48\] and I'll add that to this label here. And the text input is perfect for this. Okay. So that's all we need to do to create the form. If we click execute step, it'll launch the \[00:24:00\] form and I can type in my email and I can say, I wanna learn about AI agents and safety. If I hit submit, what that does is it hydrates the form.

\[00:24:10\] So if I close this here, we can now see that it's filled in the email information. What do you wanna learn? You can see AI safety, whatever you type in you should see as well. \[00:24:20\] So I'll pause to give you a second to do that part of the step. And then we'll click close on this part. And then we're going to add in email.

\[00:24:33\] Now, email might be a little bit trickier. You may or na may not be able to do it. I use Outlook, but if you use Gmail and you have your name and \[00:24:40\] password, that's a very easy one to connect up. There's several other emails out there. I'm going to use Outlook 'cause that's what I normally use. If I type in Outlook it can tell me the type of node I can connect up.

\[00:24:52\] You can have AI create a calendar entry if something happens. There's all kinds of triggers that it can take. Actions on create I'm gonna, you can have it create a draft \[00:25:00\] email or you can do a send email. I'm just gonna have it send a message. Just like that robotic automation. Now here's a little tricky part.

\[00:25:08\] If you didn't do this last night, you will have to connect up your account. When you connect up your account. Sometimes you'll see a blocker, a popup blocker up in the corner here. You'll have to click on \[00:25:20\] that. And then that usually lets you authenticate and you can usually use your authentication because effectively this is you on your own computer authenticating to be able to use your own email.

\[00:25:30\] In most cases, this this won't be blocked and should be allowed, and you should be able to see it turn green when it's going. If that doesn't work we'll just go ahead and walk you through the \[00:25:40\] rest of the exercise so you can see what happens. Now if you think about an email's pretty simple.

\[00:25:44\] It has a two field, a subject field, and a message field. That's it. That's what makes up an email. But we need to tell our robotic automation who to send it to. So we take this field over here where it says email, and we drag it to two, and then it writes the code for us. And \[00:26:00\] then dynamically it's resolving that code to what I had typed in my form.

\[00:26:04\] Same thing with, if we look at the subject part, maybe we just wanna call this AI training test. And that's good enough and we wanna send it a message we see you are interested in \[00:26:20\] learning about. And then we can pull in what what we wanna learn about. And we can say, great

\[00:26:30\] stay. We will cover. For that, the training. Now this is effectively a mail \[00:26:40\] merge, which is, I have the feel that or what I'm writing in and I'm just pulling in the form and it's doing this part. If I hit execute step, this sends me an email and I now have an email with this message. So coming back to the basics of this robotic automation, we have a form as our trigger.

\[00:26:57\] The form pushes the \[00:27:00\] information to our email, and our email sends it out to us. Okay. Let's go ahead and I'll give you again a minute to catch up on that part, and then I'll do a little bit of a debrief and then we'll add in the intelligence layer.\[00:27:20\] 

\[00:27:30\] \*\*Caleb Briggs:\*\* Rex, there was a question about like in what ways is this not rule-based? Do you wanna briefly mention that? 

\[00:27:37\] \*\*Rex Briggs:\*\* Yeah, so this one is entirely \[00:27:40\] rules-based. So this is robotic automation. And we'll for those that are ready we will move to the next phase, which is this, as you get down to this page, we are going to add in the intelligence to it.

\[00:27:51\] So you should get to the end where you will see an email in. And this is entirely rules-based. There is nothing more than rules. So this is purely robotic \[00:28:00\] automation. So to do a quick debrief, we used Innate in, we created our form trigger. We created the field for the form. And by the way, if you wanted to make this production, it gives you the URL to make it a production.

\[00:28:12\] We connected it to, to email. And we basically, once we had it linked into our email, \[00:28:20\] we used the form to generate the data to create effectively a mail merge so that it would send us in robotic automation, the email to who, whatever was filled in on our form. When we execute the step that was basically a robotic automation.

\[00:28:38\] So now if we went from this \[00:28:40\] robotic automation to try to make it intelligent, the next thing that we're doing here is we're effectively adding in an LLM to it. So the LLM can do some reasoning before it, sends it out. So the the key point here as we go from say, ag agentic workflow, is that we're gonna have a very narrow \[00:29:00\] mandate, heavily bounded agency to make decisions.

\[00:29:02\] So it is an ag agentic workflow, but it's much, much safer because we're just not giving it a lot of leeway. So if I contrast that with, a full-blown agent that we're going to build on Thursday, what you're going to see is that the agents. As they have their own feedback loops to reason about \[00:29:20\] what to do and to self-improve, this creates a lot more power, but also creates some more risk.

\[00:29:27\] So where we're at right now is we're at robotic automation, which is, deterministic and somewhat brittle. And we're gonna see how brittle it is when we add in the the intelligence \[00:29:40\] next. I think what we'll do is we'll jump into the next part of making our robotic automation and we'll turn it into an agentic workflow.

\[00:29:48\] And i'll say before we jump into the next exercise, it's worth noting why robotic automation is not dead. As the anthropic paper suggests they point out that when building applications with LLMs, \[00:30:00\] we recommend finding the simplest solution possible and only increasing the complexity when needed.

\[00:30:05\] This might mean not building an agent system at all. Agent systems often trade off latency and cost for better task performance. And you should consider when the trade off makes sense when more complexity and warranted workflows \[00:30:20\] offer predictability and consistency for well-defined tasks. Whereas agents are a much better option when flexibility and model driven decision making are needed at scale.

\[00:30:28\] So the key takeaway is this, if you can accomplish your goal with that robotic automation, that's completely deterministic. You should, it's faster, it's less expensive, it's safer. \[00:30:40\] And and what do I mean by safer? Remember Caleb's presentation yesterday that AI is probabilistic. So if you were to say we want to reason about something, most of the time you'll get a good answer, but you might get an answer that's not maybe quite as precise as you would like in some of the cases, and that's a real risk in \[00:31:00\] advertising.

\[00:31:00\] If you think about it, and let's say that you're right, 99% of the time, it still means you're incorrect one in a hundred times. And if you have a hundred million advertisements going out and you're wrong 1% of the time, it means that 1 million advertisements are wrong. So the key question to ask is, what does wrong look like?

\[00:31:19\] What does it, \[00:31:20\] what does the, what's the consequence of being wrong if you're talking about advertising, and it means that one 99 million people get the right message, that's great, and 1 million people don't get the right message. That creates a little bit of waste, but it's probably not, it's not catastrophic to your business.

\[00:31:37\] It's probably a big improvement from what's happening right now in \[00:31:40\] advertising. But there are some decisions where being wrong one in a hundred times is totally unacceptable. Caleb and I were just looking at another case where another person was arrested because of ai, facial recognition. They spent months in jail in Tennessee before the extradition to North Dakota or South Dakota, I forget which Dakota.

\[00:31:58\] It was figured \[00:32:00\] out that they had the wrong person. And it was all because facial recognition. The person wasn't even in the state. They had bank records to prove they weren't in the state. And this has happened a number of times. We talked about a few in the book, and we're still seeing cases happening far too often.

\[00:32:14\] So in marketing and business, the consequences are not usually so dramatic. We've. \[00:32:20\] Warn people around having AI negotiate anything dealing with money because the AI might be tricked into giving big refunds or something like that. So just again, important to think through how you're using the the agent.

\[00:32:35\] So with that, let me go ahead and get us into \[00:32:40\] the next part, which is let's make our robotic automation intelligent. So what you're going to do is you're going to duplicate the agent that you have so that we can have version control and you can save your robotic automation. So you'll select the three dots, c duplicate, and rename it.

\[00:32:57\] And then once we do that, we will break \[00:33:00\] the link in the middle. We'll add in an AI agent in the middle of this workflow, and then we will give it a little bit of direction and then we'll run it. So let's go ahead and do this. Again, if we go down to the next part of the exercise, we're down on this page about.

\[00:33:14\] Let's make it agentic and I will do this part here. So I will go \[00:33:20\] to duplicate and I'll call this my first agent. I'll click duplicate and then I'll go back the personnel part and I'll pick my first agent. And then that way I've kept the earlier version here. I'm gonna click break here. I'll click plus here.

\[00:33:39\] \[00:33:40\] And the first default is an AI agent. And this is the first agent that I want just at the top. So what you get when you have an AI agent is most of the time it's expecting a chat trigger. In our case, we are going to be pulling it in from a form, so we're gonna need to tell it where to get the information from, and we're gonna need to connect it up to a \[00:34:00\] model.

\[00:34:01\] By the way, you can't really, when you do this first form it comes in without any information. So if you click execute previous node, this is called hydrating the pipeline. It lets us capture that same data again and I'll just say AI agents here. And so that \[00:34:20\] basically by entering in the form, again, I've now hydrated the pipeline and I can now give it some information like in this case, the prompt I'm suggesting here is we wanna have was right down here.

\[00:34:33\] The following is what the user would like to learn about ai. Provide three sentences, based upon this \[00:34:40\] desire, and I'll copy that into my prompt. This is just like giving an LLMA prompt in chat. So it might look more complicated 'cause of the interface, but keep it simple in your mind. This is just telling the AI the prompt that we want to use.

\[00:34:54\] Now we want to dynamically use what we filled in from the form, so we drag over \[00:35:00\] what do you wanna learn? We drop that in and the cool thing is you can see what the result looks like in real time from the code that it's added in. So it says, provide a three sentence. Or less based upon this desire AI agents.

\[00:35:14\] This is the prompt that the AI will get. Now we have to connect it up to a model so it can get \[00:35:20\] that. I mentioned before, you can connect it to open AI or Grok or Gemini. Anyone where you have an API key, GR is free. So we're using that one. And the model you want to use is the GR compound model because that has a little bit more intelligent.

\[00:35:37\] I just to show you how this \[00:35:40\] works, if you were doing we can basically, once we've had this hooked up, we can basically close this and we now have the agent hooked up and then we need to connect it back to our email. But let's say we wanna run this before we send the email. We wanna see what, what looks like, so we'll click execute Run.

\[00:35:56\] So we'll fill in the information here again,\[00:36:00\] 

\[00:36:02\] we will hit submit. And it's now automatically pushed this to the agent. And if you look down below this part of the page by the way, might be down here. If you click that little token to, or drag it up here, right above where it says token, you can then see the output and the output says the AI agents are \[00:36:20\] autonomous software that perceive their environment, make decisions, act to achieve their goals.

\[00:36:24\] Think of them as smart assistance for tasks like customer service, smart home control, or healthcare monitoring, et cetera, et cetera. So it does, it accomplishes our goal of what we've set. So if we want to attach it to our email, we drag that connection back over, and then we have to \[00:36:40\] rehydrate the email.

\[00:36:40\] And this is what I meant by, workflows are somewhat brittle. It doesn't remember that the two field, because we've changed the agent workflow, so we have to retell it. What we want to do here, so behind each of these carrots is the details from everything you have in your pipeline. And we'll red drag in the email \[00:37:00\] here.

\[00:37:00\] And we'll get rid of this other one here. So now it will send it to me. And let's see what it says here. We see a red here again from the old email we see you're interested in learning about. We wanna still put that back in there. Great. We will so instead of saying, great, \[00:37:20\] we'll cover this in a topic.

\[00:37:21\] We'll say, great, here is some info. And now we can drag the output here. Now this is a more sophisticated mail merge, but it's still just a mail merge by the time it gets to the email, the intelligence comes from that model where it was reasoning about the information \[00:37:40\] we gave and coming up with an intelligent answer.

\[00:37:42\] So if we now hit execute step here, it should send you the email with your first age agentic model output. So again I'll give a minute for people to try that out and follow the steps that are here. If you weren't able to \[00:38:00\] get the console from the Keys, because your company has that locked down and so forth, hopefully demonstrating this has helped you build your first agent, and you're now moving in from romcom to sci-fi.

\[00:38:11\] You're now living in the future. You've created, you've successfully created an agent. You can run it a few more times. I think it might send the email out to anybody \[00:38:20\] whose email address you send that into. So if you wanna send mom an email about the AI that you think you can train it, try not to spam people with it.

\[00:38:27\] It's effectively using your email as a sender so it's coming from you so that, that creates, it, makes it somewhat more secure. But you can now do anything with your AI agent that you've created. You can also tweak the \[00:38:40\] instructions and try a few more things if you wanna play around with it. But congratulations, you built your first AI agent.

\[00:38:46\] Yay. So let's talk a little bit about what we did. In order for an agent model to work, we had to connect an LLM. That's where the chat model comes in. You notice we didn't connect memory or tools. That will be stuff that we'll talk about as we get more advanced \[00:39:00\] in Thursday. But this is effectively an agent.

\[00:39:03\] You create an API key, and it's good to keep in mind that API keys this one's free, but they oftentimes are connected to an account and you're charged by the tokens you use. So think about this, like sharing your credit card. Be careful with that number, and the AI keeps it \[00:39:20\] secure in this area so that it can't be reused by other people.

\[00:39:23\] And and effectively built an automation. You can see the input and the output. You can try it to different ways. And and so now let's talk about moving on to where we're at in our journey. And and where we're going to go next. \[00:39:40\] We are we've built our first agent and you did notice I use version control and I'll ask you to copy the same one if you want to create other versions.

\[00:39:49\] And that way you can keep some control if you change things in ways you don't want to. And very importantly, if you get stuck, there's a little button over on the right and \[00:40:00\] that's the ai. You can ask it for help and it can actually rewrite some of your code or any things that you're doing or help give you tips for how to solve complex problems.

\[00:40:10\] A couple, last tips here. So here's where we're at. So far. You built a robotic automation. Which is a precursor to agents, then you added in some \[00:40:20\] intelligence in an agentic workflow. Next, we're going to use off the shelf tools like custom GPT and deep research, which have really robust capabilities in terms of a little bit smarter agents, but still very heavily bounded, very safe.

\[00:40:33\] And then we'll do scheduled tasks as well. And then we'll wrap up today's session. And take some q and a and \[00:40:40\] overtime. So again, we started with robotic automation. You did that. Then we moved on to a workflow and we can, we used eight in Inad. In is known as a framework. There are lots of different frameworks, but the idea of them is to make it very easy to see the different steps and to be able to do things with no code.

\[00:40:59\] N eight N is \[00:41:00\] one, Zapier is another. AWS bedrock. There's a lot of them and they work a little bit differently, but they're all pretty similar. So once you've learned one, the skills are pretty transferable to learn for others. Kob, are there any questions I should hit on before we move on to to do building a \[00:41:20\] custom GPT?

\[00:41:22\] \*\*Caleb Briggs:\*\* Yeah. One thing to maybe mention is that these recordings will be available. I think a lot of people were having a hard time keeping up. But all these recordings are available so you can watch it back and we'll keep the guide up. I believe the recordings will stay up for a long time, but the guides will also stay up basically indefinitely.

\[00:41:39\] So if you want \[00:41:40\] to come back at some point and watch the video while following along to the guide that's a great way to take as much time as you want to see the process. There's a brief someone had of the difference between. Robotic automation, the AI agents, maybe like a brief sentence or two.

\[00:41:55\] \*\*Rex Briggs:\*\* Yeah. And I'll do that one and then we'll save some of the other questions for the overtime part so that we \[00:42:00\] have time to hit the the custom GPT. Yeah. So the basic difference is robotic automation has no intelligence in it. What made RS things intelligent was when we broke that connection, it was more than just a mail merge.

\[00:42:10\] The AI agent was looking at the fact that you said that what you were interested in. So maybe you didn't type in AI safe. Maybe you didn't type in AI agents. Maybe you typed in that you \[00:42:20\] wanted to learn more about rag the AI would see that and give you a different answer than what it gave me when I typed in AI safety.

\[00:42:26\] So that's the intelligence, that's the agen intelligence that we were adding in to the system that made it different from robotic automation. It's a very simple agent in this case because, hey, we've got an hour together. We're introducing you to a new \[00:42:40\] concept. You can make these much more sophisticated.

\[00:42:42\] In fact, we will get to multi-agent frameworks tomorrow and Thursday. So you'll see how these can become more sophisticated. We'll have an agent that builds advertisements for us, and we'll see that in practice. So I'm going to jump into the next point here so we can hit in the last 15 \[00:43:00\] minutes.

\[00:43:00\] Custom GPTs. I'll make the other point that you can have lots of different triggers to an agent. We use form submission here, but you could have have it be, maybe you receive an email and then the AI reads through the email and decides what to do about it. You could do lots of different interesting things with agents now that you've got that pattern.

\[00:43:19\] The other thing that we're gonna \[00:43:20\] do next is we're gonna say most more sophisticated agents usually iterate through the intelligence. In our case, it was one shot. You got the information in, the AI came up with an answer, and you push that out. But you, what we're going to see in deep research in custom GPT is that there is oftentimes iteration that agents can do that can give you a better \[00:43:40\] output.

\[00:43:41\] Okay? So we're going to go to custom GPT next. This is an off the shelf agent. In some ways, it's faster and easier to build than what you just did. But it's also more powerful. Let's go ahead and take a look at the exercise. The basic exercise is if you come over to \[00:44:00\] the page here, you can click chat GPT and if in chat GPT, you're going to look down in your paid version, you'll see gpt and you can click Explore GPTs, and then you'll see Create.

\[00:44:14\] Now if you don't have a custom GPT, you can't get to that because it's a paid version. \[00:44:20\] Caleb's twin brother Jared created Jared, GPT, and you can launch Jared GPT, which is a clone, and it looks just like the other one. And you can click create GPT here, and it will give you pretty much the same functionality.

\[00:44:34\] It's just using his pipeline. And MMA will cover the token cost for you to build it here. So either one works. \[00:44:40\] What we wanna do in this exercise, we wanna launch the the walkthrough. And I'll drag this walkthrough over here. And in this walkthrough, the first main thing we're going to do is we're going to give it a name.

\[00:44:55\] We're going to call it Creative recommender. \[00:45:00\] And we're going to give it a description.

\[00:45:06\] And by the way, if your version in chat, GPT if you're in the create one, it looks like this, just switch over to the configure one and you'll get the same thing. Give it these names. \[00:45:20\] And then what we're gonna do is we're gonna give it the prompt. Now, the, this is again, just like a prompt, just like you've done with LLM.

\[00:45:26\] This is you can find this with the instructions here and launch this page. And that will give us and come back to here and do that here. \[00:45:40\] And then you're just gonna copy and you'll see again this markdown language I'm using here with one hashtag and two AI prompts. Love markdown. So we're gonna paste that in here.

\[00:45:52\] You can go through and read it by clicking on this thing here. If you wanna see what it's asking it to do. It's it's something I put together relatively quickly here. \[00:46:00\] It's maybe not perfect. Maybe you can even improve upon it. And then we're going to give it some we're going to give it some files.

\[00:46:04\] Now you're gonna get the files here. You're gonna need to click on this link. And then you're gonna need to download these files so you can do the save as save or export as A PDF. And that will launch these \[00:46:20\] files. And once you have those three files, I'll just do one for now just to demonstrate it, but you should do all three.

\[00:46:27\] You'll add those to the knowledge. Now, this is part of retrieval augmented generation. Part of what it's being told to do is pay attention first to this information and use that as part of the part of the tips. So actually, \[00:46:40\] I'll go ahead and download the other files. I wrote these, by the way for AI to use it.

\[00:46:45\] And it's interesting, I think we're gonna be writing more and more white papers for AI compared to potentially writing papers for humans to read because the AI can use our intelligence and it will actually read all of the information. \[00:47:00\] So this is so we'll export or save as, export as PDF,

\[00:47:10\] and. I'll upload those two files. Okay, so now it has the files or the intelligence we want it \[00:47:20\] to have in the tool section. What we're going to turn on is we're going to turn on web search image generation and code interpreter. Code interpreter lets it analyze some of the data canvas you can leave on or off.

\[00:47:35\] It, I'll turn it off here and then we'll hit create and \[00:47:40\] you'll have to decide who you share it with. I'll just hit update here. So what did we create? We created something that's meant to look at ads and give us recommendations on how to improve those ads or have more variety. Now, this becomes especially important when you begin to create ads \[00:48:00\] that are meant for AI personalization.

\[00:48:03\] You can look at the instructions a little bit more By the way, you can click on this part and then hit edit GPT and make changes once you're in here. But let's go ahead and ask it to create an ad for Major League Baseball.\[00:48:20\] 

\[00:48:20\] Now just that prompt alone, and you can put your company name in. It will, it should give you some thinking. And if you look at the thinking and click on it on the side here, you can see on the side what it's doing is reading the documents. This is retrieval augment generation. It's thinking through what it needs to do, and it's coming back with a recommendations in the intelligence.

\[00:48:38\] And so it can, it's now \[00:48:40\] suggesting a variety of how to create different ads. It's giving you some suggestions and so forth. Now, one of the thing that we included in here is we included in this exercise the idea of it, giving it an ad and having it analyze it. So if you click on this link and then you click save image as, \[00:49:00\] and just save it into your drive, we can upload that ad into our chat.

\[00:49:07\] GPT.

\[00:49:14\] And when we do that, we can ask it to analyze this \[00:49:20\] ad and it should be able to perform that function. Again, if you look at its intelligence, click on thinking, you can see what it's doing over here. You can what it's likely doing is it's retrieving the ad. It's already read the information that's already in the context window and it's now using the analysis to then tell us what would make this ad better based \[00:49:40\] upon Google's best practices and some of this best practice documents we wrote.

\[00:49:44\] Now, the interesting thing here is you can write your own brand guidelines, your own best practices, you can share your own information, and you can create this as a custom GPT. So every time you go to create a new ad, you have that recommendation system \[00:50:00\] built in with all the intelligence from your information.

\[00:50:04\] So let's go back and do again, a little debrief here, and then we're going to go on to a deep research exercise. So recap, we \[00:50:20\] gave the ai knowledge and information, and we gave it some tools and capabilities. We gave it some instructions, our prompt, and it's now able, it's something we can now reuse anytime we want to.

\[00:50:31\] We can also share it with other people in our team or our organization. We had to analyze this ad and it gave us back feedback and it was \[00:50:40\] highly bounded. It was in a closed environment, so it was very safe. The next thing that we wanna do is deep research and tasks, and we'll give a minute here just for people to, wrap up that training exercise and then we'll come back to these last two things that we're going to kick off today, and then we're going to talk about them more tomorrow.\[00:51:00\] \[00:51:20\] 

\[00:51:21\] Okay. Deep research and tasks. You will find deep research in the pull down window in Gemini or in, chat, GPT, you'll find it in perplexity and clo and a lot of other places. Deep Research is arguably one of the first \[00:51:40\] AI agents that was built into the tool set that is broadly valuable for people.

\[00:51:46\] It's some of the systems like Gemini offer it for free. Some of them have it paid. I'll use Gemini for this example here because Gemini also runs faster here. And so in Gemini. I can click on \[00:52:00\] tools and I can click deep research and it will activate deep research. Now, deep research is going to reason for several minutes across a different topic.

\[00:52:07\] It it has some really cool features about what it's, trying to do as it goes to this tool. And I want to ha, I want to have you enter something in and get a deep research thing started and and \[00:52:20\] then we will kick off this process. So let's say that I want to think about how I create a good deep research prompt.

\[00:52:26\] First of all, I could create a new chat and I could actually ask chat GPT what is a good deep research prompt to analyze my company, MMA global? So you can try that prompt right now. And the cool thing is the LLM will write you the prompt that you can \[00:52:40\] give to your deep research agent so that you get a better output.

\[00:52:44\] In fact, that's what I'm gonna do right here, and I encourage you to try that with your company name.

\[00:52:55\] Okay, while yours is generating, I'm gonna go ahead and \[00:53:00\] copy in this prompt and I'm gonna create a new chat. I'll go back to the tools and I'll turn on deep research and I'll put in this prompt and I'll hit go. And I'll give you a minute to do the same. And what you're gonna \[00:53:20\] see is, the first thing the AI's going to do is it's going to try to create a plan.

\[00:53:26\] And so just talking about, what this was doing we talked about, once you activate the deep research that the first thing that deep research does is it. Does intent extraction. \[00:53:40\] It tries to figure out what you're trying to do, what your goals are, what you're planning. It feeds that back, and then it goes through an autonomous extraction loop where it does some of the research, does a verification and evaluation, and that synthesizes the information.

\[00:53:52\] It provides back a nice detailed report for you. So you can see it's asking now. You could edit this and change it if you want \[00:54:00\] to, and then, but let's go ahead and get, let it start running. If you haven't used deep research before, it is it's an incredibly powerful tool for doing competitive research, doing research on your own company trying to understand a topic that you wanna understand more deeply.

\[00:54:15\] I was looking at some health issues that family members have have and wanted to learn more \[00:54:20\] about it. This does really extensive data and research. The autonomous loops also pretty interesting. It does a parallel sequential searching, so it will launch dozens of parallel queries. It does breath first searching, so it tries to find as much as about the topic.

\[00:54:34\] And once it finds something good in the topic, it then does depth first, or it goes deep in the topic. It even \[00:54:40\] has something where if there's conflicting information, it will do a tiebreaker search and try to find out what the right answer is. It does some verification and evaluation. It gives some grading and once.

\[00:54:49\] The research is good enough to pass its own internal grade. It then pushes that back to you as a synthesized report with all the citations. So deep \[00:55:00\] research is an incredibly powerful and valuable tool. And what we're going to do tomorrow is when you come back to deep research what we want you to do is to effectively grade it.

\[00:55:12\] By the way, you can click on when it's researching on the word research, or click on this button. You can see its thinking and see all those steps in practice. \[00:55:20\] So here's your assignment for tonight. Do a deep research evaluation. Read the report. Do it on something you know about, read the report, and then come back prepared to give it a grade.

\[00:55:30\] Tomorrow we'll start with a poll where you'll give it an A, B, C, D, or F, depending on how well it did in the deep research. Now the last thing \[00:55:40\] we're going to encourage you to check out, and this will be some take home homework, is you can also use scheduled or task agent. This one is much smaller than deep research, but if you type in slash schedule into Claude or you click on in some of these these other items you can click on schedule task \[00:56:00\] or you can go into chat GPT and we'll go into a new chat here really quickly and you can click notification.

\[00:56:08\] So you can say, create a notification about. And then you can fill in the rest here. It will, and you can tell what time of day you want it, or once a week. It will automatically do this and \[00:56:20\] it'll push you information back. I created one recently just to keep track of Brent crude prices and what was going on in the Persian Gulf and and the supply chain implications.

\[00:56:32\] That's something that that the AI agent will do some reasoning about. It'll do some research, it'll use some web search tools. It'll go out and gather the information \[00:56:40\] and bring it back to you. So this is a good one to try as well. So we have gotten through today. I know we've packed in a lot today. Please do take a moment to give a score in the NPS and about how today worked for you.

\[00:56:56\] This will go into our research that we're going to share on Friday. \[00:57:00\] Caleb and I will now shift into the overtime session for those that can stick around and have different questions. And the the video for this, it will be available later tonight. If you have time, please try to do tomorrow's hands-on exercises so that when you're seeing it tomorrow, you're seeing it for the second time.

\[00:57:18\] And and hopefully I think you'll get more out \[00:57:20\] of it that way. Caleb, over to you into terms of some of the questions and discussion. 

\[00:57:27\] \*\*Caleb Briggs:\*\* Yeah, I think maybe it would be nice to spend a minute or two, just reviewing the big takeaways just 'cause we went really fast today. So maybe a little bit just about what's the sort of cool stuff we can do with this?

\[00:57:39\] How would you \[00:57:40\] recommend people use this training? Things like that. 

\[00:57:43\] \*\*Rex Briggs:\*\* Yeah. Yeah. I think that the big takeaway from today is that first of all is getting people's hands dirty building AI agents. So they're not some mystical, mysterious thing that happens out there that people are talking about. But people on the.

\[00:57:59\] Training, haven't \[00:58:00\] seen it up close and personal. AI agents are not super complicated. We can make them more powerful, but they really are just this idea, the same thing that you do when you're using an LLM and you're prompting it, you're doing that, but you're building that into a workflow so that the AI has some type of steps that it's doing \[00:58:20\] and it's using its intelligence to do those steps better.

\[00:58:23\] So in this case, we were, we wanted to get a like the top of the funnel example in lead nurturing. Yeah, you can have that be a newsletter or something's totally standard, but what if you personalize that newsletter based upon what somebody was interested in that might make it work even better?

\[00:58:37\] So that was a use case example that we \[00:58:40\] used in today's agent that we built. The other one is deep research. There's these off the shelf agents that are available to you that a lot of people don't know about. I bet a lot of people haven't used scheduled tasks and push notifications. That's what we saw in the poll yesterday.

\[00:58:53\] That's a great one. If you're interested in learning about G the suggestion we had was generative. Instead of SEO search engine \[00:59:00\] optimization, generative engine optimization, maybe they want a weekly update on what's new in that area. Tips and tricks. They can create a scheduled agent to go do that research for them and to give them a notification each week and an update or maybe they wanna do a deep research on that topic and see what the best practices are.

\[00:59:17\] Or they want to keep an eye on the competitor and they want to \[00:59:20\] run that competitive analysis, once a month to see what's new. So these are, they're using the intelligent edge agentic workflow in ways that are either off the shelf, but I also want them to see that actually you could build those same type of systems as well to serve your own interests.

\[00:59:38\] \*\*Caleb Briggs:\*\* Yeah, that's a great answer. \[00:59:40\] Another question someone asked that these mentioned that these platforms seem like they're only accessible on paid. Is there they said are there effective platforms for agent workflow building that are truly free? 

\[00:59:53\] \*\*Rex Briggs:\*\* Yeah N eight N is free for two weeks, which is why we're using it for the training.

\[00:59:57\] That should be enough to get hands on and go \[01:00:00\] forward. I'll do a little bit of research on what, which ones are the best practice. I'll use deep research to do some research for us on which ones are the best. No-code free, truly free services. There are some that I know like hugging faces, a ton of interesting free tools.

\[01:00:15\] There's a whole community of builders that make projects available \[01:00:20\] and I suspect that somebody has made something like that available. Where the cost comes in is usually when you begin to use that intelligence. So when you're hitting, like we use grok, which is a free service, which is pretty cool but most ones are also charging for the token.

\[01:00:33\] So you do pay for the intelligence. What we'll see this week is we're gonna do a lot of AI agent building to create ads \[01:00:40\] and I did over a hundred different runs of AI ad creation and it cost about 75 cents. So it's pretty inexpensive, but it's usually not free.

\[01:00:54\] \*\*Caleb Briggs:\*\* Yeah I, as you're saying, almost anything that uses an API key is gonna cost \[01:01:00\] some amount of money just because running the ais does it is expensive to do, and especially if you want to use the stronger models. So if you wanna do anything with Chatt, BT, or Claude even if your platform is free you'll still have to pay on the API side.

\[01:01:14\] \*\*Rex Briggs:\*\* Yep. 

\[01:01:15\] \*\*Caleb Briggs:\*\* Someone asked which email can we send our questions to? I meant \[01:01:20\] to maybe, 

\[01:01:20\] \*\*Rex Briggs:\*\* ah that, that's as we said at the very top of the page, we added an AI agent that has all the content information. So I'd asks you to try the AI agent first and ask your questions there. And that, again, is at the very top of the labs page with the AI assistant.

\[01:01:37\] So try that first. If you do have other questions or you \[01:01:40\] wanna make connections, you can connect up with Caleb or myself on LinkedIn or Jared. And we'll all try to help answer questions. If if you find that the AI agent or assistant doesn't give you a good enough answer then absolutely, connect up with us.

\[01:01:57\] \*\*Caleb Briggs:\*\* Yeah. Then maybe add to that the one of the areas \[01:02:00\] that the AI is excellent at is, like providing diagnostic help on problems. So I'd say if maybe you're doing something with NIN, you can often attach a screenshot to the AI and like just be like, okay, here's what my screen looks like. What sort of thing do I need do next?

\[01:02:16\] And it can often help with that. Is your experience generally good with. \[01:02:20\] N eight N on that Rex, or 

\[01:02:22\] \*\*Rex Briggs:\*\* yeah, I think so. We're gonna show that more tomorrow. If you have questions on N eight N, you can download the JSON and give it to an LLM like Claude or chat GT or Gemini. It does a great job understanding what you're doing and giving you tips.

\[01:02:36\] It also works with a screenshot. It's a little less detailed than the \[01:02:40\] JSON and even the in native N eight N Helper. That AI assistant button I showed earlier does a pretty good job. And there are docs connections at the very top corner where you can read the documentation. But the interesting thing is, like on Monday, we created Notebook lm, which has all the content for the first four days of training.

\[01:02:59\] \[01:03:00\] And so you can ask it that we also created the AI assistant in chat, GPT, using a custom GPT, adding those as knowledge files. So I want people to see that pattern and try to ask the questions to those systems because it changes the mindset from. And makes AI first. So you go to AI first to see if you can get the answers, and if you can't, then you \[01:03:20\] bring the expensive people into the loop and try to get the answer that way.

\[01:03:24\] It's a different mind shift, but it is, the world we're in right now is that AI is so powerful that we have to get ourselves out of the habit of going skipping over AI and not using AI first and make that the thing we try to do first.\[01:03:40\] 

\[01:03:42\] \*\*Caleb Briggs:\*\* If there's I think one more question. So been asked which program is best for comparing complicated data or code spreadsheets? 

\[01:03:49\] \*\*Rex Briggs:\*\* Oh, yeah. So there are two tools that I think are just outstanding right now. Claude, for sure is is outstanding. And there's a plugin in Excel for \[01:04:00\] Claude which is, I've been testing it and it's really remarkably good.

\[01:04:04\] It saved me a ton of time. So I would try that. Your company probably has to give you permission if you're not using your personal excel for that. And then Claude and then I would say beyond Claude Chat, GBT Pro that costs \[01:04:20\] $200 a month or something like that. I pay for that because it saves enough time in the work that we do at, but it is expensive, but it is phenomenally good. The pro versions can take 15, 20 minutes to reason through things, so it, it takes a lot longer and you pay for that extra time. But I might oftentimes launch a couple different things at once, \[01:04:40\] have them running, and the amount of time savings has been dramatic.

\[01:04:43\] We used it on some of the work with GM and I tried Gemini Claude code and chat. GPT chat, GPT did slightly better than the others, but they're all pr very good at this point, at an in analysis. 

\[01:04:58\] \*\*Caleb Briggs:\*\* Yeah, I might add \[01:05:00\] that if you're doing something like chat, GBT Pro, the like experience of using that is fairly different than like time savings or something like that.

\[01:05:08\] The model takes something 40 minutes or sometimes longer to respond. So a lot of times it's less I wanna do something quicker. The pro is good enough that you might decide to do types of analyses that you wouldn't do \[01:05:20\] otherwise, or push off like very large scale tasks.

\[01:05:23\] Generally it's strong enough that if there's a. Something that's maybe too complicated for me to do on data analysis, I might give it to pro because it's like probably stronger than me at doing that kind of data analysis. I'd say with the like cheaper models, oftentimes you might use it to say okay, I have a formula.

\[01:05:38\] Can you write this for \[01:05:40\] me? And maybe save five or so minutes. Pros if you want to do basically a full task that you're giving to the AI and coming back an hour later and seeing like lots of great charts created and things like that. So it's very good at data analysis. But it's a whole different experience of using it because you have that long almost like hour long wait period.

\[01:05:58\] \*\*Rex Briggs:\*\* Yeah. Great. Okay, \[01:06:00\] I think we've got time for one more question. Thank you for the several hundred people that stuck around for the q and A session. We'll do this again tomorrow, but we've got time for one more question and then we will see you back tomorrow to keep building on the Agen lesson, I think tomorrow is about a agentic workflows and evaluations, which will be a really, big step \[01:06:20\] forward as well.

\[01:06:22\] \*\*Caleb Briggs:\*\* See there's a question here. So when asked, how do we distinguish between instructions and knowledge? I'm a little confused between what each one is for. Seems like an overlap. So I think this is about custom GBT. 

\[01:06:34\] \*\*Rex Briggs:\*\* Yeah. Yeah. It, so that's right. So the way I think about it, and Caleb, I'd be interested in \[01:06:40\] your perspective is the instructions are very similar to like, when you give it a prompt it's basically saying what you want, the, the AI to reason through the knowledge is what happens is the AI reads the instructions and then it looks at the knowledge and says, is there anything in here that \[01:07:00\] I need to answer the question? So if you think about it, there's really three things going on. You've got the instruction, which we'll call system instruction.

\[01:07:07\] We're going to use that term tomorrow. And then you've got the user prompt. And then you have the knowledge. So when a user types in a prompt tell me about Major League baseball, what it does is it takes that prompt and it \[01:07:20\] runs it to the, it takes the system instruction first and says, this is what I need to do.

\[01:07:23\] It looks at the prompt and says, do I have the information to answer that prompt? And then it looks at the knowledge and says, is there any additional information in the knowledge set that I need? To answer that, that prompt, so that's how the three work together. And it's a very common pattern. So I want you to get a little \[01:07:40\] bit of a taste of it.

\[01:07:40\] And I love the fact that you asked that question because you nailed it. Those things are slightly distinct from one another, but they all work together very often in a agentic systems. Would you say that a little differently, Caleb? Or how do you think about the 

\[01:07:55\] \*\*Caleb Briggs:\*\* I think that's a great answer. Maybe the only thing to add is maybe you already mentioned this, but the \[01:08:00\] instructions show up on every single message, whereas knowledge is not always retrieved.

\[01:08:04\] So you could put knowledge about all sorts of different topics in there. So maybe you could have a set of knowledge about Major League baseball and a set of knowledge about national Football League or something. And the AI can decide on its own, when am I gonna get the major League baseball?

\[01:08:18\] When am I gonna grab the NFL \[01:08:20\] data? But the system instructions, those always show up. So system instructions are things you can guarantee the model sees. The model sort of decides on its own what kind of knowledge it wants to grab. 

\[01:08:31\] \*\*Rex Briggs:\*\* I like how you answered that. 'cause that also gives a little bit of a foreshadowing of where things are going with multi-agent.

\[01:08:38\] Where you're gonna end \[01:08:40\] up by the time we get to Friday is you begin to think about, you might have multiple folders of different knowledge on different topics. Like for me personally, I have a personal interest in road trips. And so I have a folder on knowledge about places I've been, places I'd like to go.

\[01:08:54\] And what I like to do, I love hiking and outdoors, right? And then I have another one on health. I've got some, health \[01:09:00\] issues that I wanna keep an eye on. I've got labs and reports and things I keep, that's a whole separate folder. I've got things about work and related to work. And so the AI knows which kind of knowledge chunk to get into based upon what prompt it's getting.

\[01:09:13\] And then the instructions are basically telling it a little bit of what tools to use or what capabilities it has or how to think \[01:09:20\] about different problems and issues. And even those we're going to see will end up being separate folders and they. In the future becomes something called skills.

\[01:09:28\] And you might have a specific skill for how do you analyze a a pie chart or how do you how do you figure out the optimal mapping plan. So those might be different skills. And right now we're doing them as \[01:09:40\] one system instruction. But in the future, what you think about is you can have a whole list of these different things.

\[01:09:45\] That's a much more sophisticated way of thinking about it. It's a good place to probably leave it for us as we will get there by Friday. But thank you everyone for joining us, and we look forward to seeing you back here tomorrow for AG Agentic workflows and evaluations. \[01:10:00\] Take care. 

\[01:10:00\] \*\*Caleb Briggs:\*\* Thanks everyone.

## 

## Day 3

\# Decoding AI for Marketers VII \- Day 3

\[00:00:00\] \*\*Alec Foster:\*\* Hello everyone and welcome back to decoding AI for marketers. I'm Alec Foster, the co-lead of MA's, AI Leadership Think Tank. I'll be kicking things off with a look at what we're building and handing it over to Rex and Caleb for day three of your training. This is the seventh edition of the \[00:00:20\] series.

\[00:00:20\] It's been running since fall 2023 and each round tracks where AI actually is covering basics, multimodal AI workflows, reasoning agents, up to enterprise scale. We've had over 9,000 reservations, 5,000 attendees, and over 2000 certificates to \[00:00:40\] date.

\[00:00:41\] This training comes from the Marketing and Media Alliance. We're a global nonprofit led by CMOs with 800 plus member companies and staff in 16 countries. We run multi-year labs and real world experiments that produce evidence-based frameworks. \[00:01:00\] The mission of MMA AI leadership Think Tank is to use the convening power of MMA to help marketers apply AI effectively and ethically.

\[00:01:09\] We believe there's nothing more important than what's happening in marketing right now than using ai. We do this by running experiments and producing original research in collaboration with our members. \[00:01:20\] The AI Leadership Coalition consists of over 170 brands and is growing every year. We hold quarterly working groups, coordinate participation in AI marketing experiments that improve your KPIs and create opportunities for peer collaboration.

\[00:01:38\] I help run the AI \[00:01:40\] leadership Think Tank alongside my colleague Kassan Qatar, who you'll be hearing from later this week. My focus is on internal AI upskilling and literacy, and I support the working groups and building new AI frameworks and experiments for members. You want to get involved? Please reach out to either of us or to a's membership team and we'll help you get \[00:02:00\] started.

\[00:02:03\] The think tank creates values for marketers through three pillars. We have future labs, which are hands-on experiments with real band brands, testing AI technologies in market, covering personalization, ENT advertising, creative effectiveness, audience targeting and more. \[00:02:20\] We also have our knowledge center, which has AI training like the one you're in today, research benchmarks and published insights.

\[00:02:28\] And we also cover community, which includes our 300 plus marketing leader, community sharing what works and what doesn't. The through line through all of this is the same measurable impact and ROI. \[00:02:40\] We have five future labs running under the A leadership think tank at the moment. Each one is a real world experiment that you can join and together they cover the full performance marketing process.

\[00:02:53\] Audience discovery, targeting, creative production, creative scoring, and personalization. You've \[00:03:00\] already heard about cap and sift, and today I'll briefly walk you through Aura. Aura or ai. Uncovering responsive audiences uses AI to find the highest value micro segments within your existing audience and updates them weekly.

\[00:03:16\] It's an 80 20 AB split with 80% to AI \[00:03:20\] optimized delivery, and 20% to business as usual, so you can measure the lift cleanly. No changes to creative or required. There's no incremental media or research cost across 300 campaigns. 90% showed incremental conversion improvement in the 15 to 25% range. The AI consistently surfaces \[00:03:40\] cohorts.

\[00:03:40\] That manual planning misses. Rex Briggs. One of your trainers and Claritas help run this methodology. The vision at the bottom ties it together. You have aura for dynamic audiences. Cap for dynamic creatives and sit for contextual targeting. All three are designed to work in concert so you can \[00:04:00\] stack them and you're optimizing who you reach, what you show them, and where you show them.

\[00:04:07\] I'm excited for what's ahead, so let me hand things off to Rexy, Caleb, for your training on building AI agents, please take it away. Thank you. 

\[00:04:15\] \*\*AI Rex:\*\* Welcome to day three of our AI training. It is April 1st, \[00:04:20\] 2026, and I wanna start with a recap of the progress you've made on Monday. We cover the foundations of how AI works.

\[00:04:27\] On Tuesday, we got you hands-on building your first AI agent. Using the N eight N framework and the custom GPT builder, we also got you exposure to the off the shelf agent \[00:04:40\] for schedule tasks and deep research. Today we are going to start with your evaluation of how your deep research agent performed.

\[00:04:50\] Caleb will then lead us through a discussion of evaluations and we will get hands on building another agent. 

\[00:04:59\] \*\*Speaker 3:\*\* Okay, \[00:05:00\] that was my April Fool's video. That was an AI agent of myself doing the introduction. If you thought it was me, it raises some interesting questions about why am I still delivering AI training when the AI agent may be able to do it better.

\[00:05:13\] We'll talk more about that on Friday when we discuss the future of work. For now, let's make sure you are logged in. Going \[00:05:20\] back to the lab page, open AI's API is the key one. And you'll wanna click on that here. You you really only need a $5 sign up for it. But once you've signed in you'll see API keys over on the side.

\[00:05:36\] And then you'll be able to create a key. There is a \[00:05:40\] billing link here. If you go back to the side here and just search for billing if you, in case you haven't put in the the credit card yet. Okay, so coming back to the main content, you'll also be using innate and again, so we'll make sure we're locked in for that.

\[00:05:57\] For now. To tomorrow, by \[00:06:00\] the way, built on today, and I know we went a bit fast yesterday and we will go over building agents again today. If you have two screens, it's a lot easier to follow. I remember when I was first exposed to this, it was very difficult just to do it on a laptop. That's also why we give you the walkthroughs in case you can do that the night before.

\[00:06:19\] You'll \[00:06:20\] get a lot more out of it. And that will be true for tomorrow's session as well as it will be for today's session. You will have the recordings so you'll be able to go back through it. So if you don't have an extra iPad or computer or two screens to make it easier to follow through, then you know, certainly you can print out the the walkthrough and, go \[00:06:40\] through it at a more leisurely pace.

\[00:06:41\] Perhaps after the session. What I wanna hit on is a recap of what we did. After you get your logins going. You did something really phenomenal that really 90% of your colleagues have not yet done. You built your first agent, you did that with the N eight N \[00:07:00\] framework. You did that with custom GPT, you did that with off the shelf agents, like deep research and scheduled tasks.

\[00:07:05\] At the very end of the session, I mentioned I was building a scheduled task for updates on what was going on with Bru, Brent, crude oil prices, and treasuries, which are good leading indicators of inflation, which a lot of marketers are influenced by. \[00:07:20\] This was my update that I got this morning from that process.

\[00:07:23\] So let's do a quick poll and see if you're able to get these things working. So let's go ahead and roll the poll. So if you're able to get your in it in agent working, say yes. If you didn't try it go ahead and let us know. And then

\[00:07:38\] and then if you're able to get your deep research \[00:07:40\] working, that will be the next other one.

\[00:07:47\] Okay, hopefully I gave enough time to give a quick snapshot of where we're at. Are you able to push the pull results? Okay, here's what we've got. \[00:08:00\] We've got looks like about 20% were able to get the innate in working. Like I said we'll go through that again today. Some of you might be blocked and so you'll be watching along.

\[00:08:10\] The idea of going through it is if you get it hands on, you get a more visceral learn by doing experience. But even if you don't, you may get a really good idea of what \[00:08:20\] the workflow is because if you see how they work in practice, especially if you build one, it demystifies it quite a bit. So we hope that those that didn't get a chance to try it yet.

\[00:08:29\] We'll give that a try later on today before tomorrow's session. And then in terms of the deep research, it looks like a little bit more, we're able to get that working. Again, highly encourage you to give that a try. It \[00:08:40\] looks like about 31% ha we're able to get it to work and about half didn't yet try it.

\[00:08:44\] It's very fast and easy to set up. So take your large language model of choice and it very likely will have deep research as a function. And just as a reminder on that, you can see deep research in something like Gemini for free. This is one that offers it for \[00:09:00\] free and you just go into the tool section and click deep research to activate it.

\[00:09:04\] And as we showed yesterday, you can ask the LLM even to help you write a good prompt about something you might want to research. Okay, coming back to the next part of it is if you did do the research, if you're part of roughly one third of people that were able to do that lab \[00:09:20\] give us your rating of how it did.

\[00:09:21\] Let's put, push that up as a poll as well.

\[00:09:29\] Okay, so you should have the pull up there ready to go.\[00:09:40\] 

\[00:09:44\] Okay. It's probably enough time to let people fill that in. So let's go ahead and look at the results. Okay. Here's what we've got in terms of the people that did it. We had about 40%, 80% of \[00:10:00\] the people gave it an A or B. So it was as good or better than you could have done if you had a day to do it, or it was about on par.

\[00:10:06\] That's pretty impressive. Now, there were some that said that it did poorly or failed outright or gave it a c and Caleb will talk more about evaluations. So let's go ahead and. Dive into today's \[00:10:20\] content, which is going to be about ag agentic workflows multi-agent. We're gonna get into multi-agent today and and our hope is that if you get through this week, having built agents by hand, you will have a huge advantage over the 90% of your colleagues that haven't yet, you'll know the deeper \[00:10:40\] secrets of what really makes agents powerful.

\[00:10:42\] And to give you a hint, it's in the evaluation loop and which I will hand it over to Caleb to help us explain. 

\[00:10:51\] \*\*Caleb Briggs:\*\* Thanks Rex. So looking at that poll result, earlier we saw that there was a pretty high degree of variance. About half of the people \[00:11:00\] saw that deep research was doing a great job and half the people thought it was okay, or maybe had some mistakes in there.

\[00:11:07\] And by doing that poll. If you had the opportunity to try deep research, you got to try your first AI agent evaluation. And one reason why those evaluations are useful is for the \[00:11:20\] reason you just saw on that poll. AI agents will sometimes do a very excellent job on the task, and evaluations can help you separate out those strong results from the potentially weaker ones.

\[00:11:31\] So evaluations are a very powerful feedback loop for building optimizing agents. And we can see how to use this feedback. If \[00:11:40\] you tried deep research and you gave the AI a I want you to go back to the interface, find the thumbs up button and give the AI a thumbs up. Or if you grid a see or lower, then go back and give it a thumbs down.

\[00:11:53\] So we'll give everyone a few seconds. Who had the opportunity to try deep research to go and do that?\[00:12:00\] 

\[00:12:07\] Rex is showing where that is on Gemini, for instance. By giving that feedback there, you're providing feedback to the mall so it can get better so it can produce better responses in the future. With deep research. \[00:12:20\] Now with deep research, we just went through a loop of first sampling from the model.

\[00:12:26\] Getting feedback, hear that feedback was your human judgment, and then the model updates as a result of that feedback. So in our case, we sampled something related to doing research. You chose whether the grade to give \[00:12:40\] it and then the model ultimately in the backend will update based on some gradient descent like algorithm based on the thumbs up or thumbs down.

\[00:12:48\] Now this loop here is the core loop in self-improving ai. So if you can create loops like this where you sample, you get feedback and update the model, you can get the model to become better and \[00:13:00\] better at doing tasks. So in our case, we still needed human judgment though. So the process is bottlenecked and speed by that human feedback.

\[00:13:08\] So it can only improve as fast as you're able to give feedback, but this is still works pretty well for a lot of tasks. So here's one way you might use this loop. So if you're trying to create \[00:13:20\] advertisement where you would give the model maybe some data about your brand, some advertisements, and then you could ask it to create a specific advertisement.

\[00:13:29\] And then the feedback here is you could give specific feedback on the ad it created. So maybe details about what it should change, whether you like the ad or not or other possible \[00:13:40\] improvements. Then for the update step, a good option that's really easy to build is just give the model access to memory and a scratch pad to keep track of notes that it can improve based on your feedback.

\[00:13:52\] So now each time you ask you to generate an ad and you give it some feedback it's able to updates, memory, updates, \[00:14:00\] feedback, and tell in the future it's creating high quality ads. So again, this is another way that we can get the AI to improve itself over time through this feedback loop. So this approach works really well for lots of problems.

\[00:14:13\] I think the real power comes when you figure out how to close the loop so we can get a model to improve a lot. When we can make the \[00:14:20\] sampling, the feedback and the update all happen automatically. And so to do this usually involves replacing that human feedback part with an evaluation or an objective function.

\[00:14:33\] So what does an evaluation or objective metric look like? So there's lots of different metrics we can use to \[00:14:40\] replace human feedback. So one example is quantitative measurement. So this means just finding some measurable property and using that as the feedback. So for instance, when I'm programming, I'll often use speed as the optimization metric.

\[00:14:53\] So I'll take a program I've written, ask the AI model to speed up the program and then each time it runs the \[00:15:00\] program, I'll keep track of what the speed was and running it and give that back to the model as feedback. So I, I'll maybe also add an eval loop on top of that where I say, okay, the code needs to output the same thing as the old version.

\[00:15:14\] So that's an extra piece you can add in there just to make sure it doesn't optimize speed at the expense of \[00:15:20\] correctness. And I actually use this type of optimization loop all the time. So a few weeks ago for my research, I had it rewrite a program that I was using. And it managed to speed up the training process for that specific thing 70 times.

\[00:15:33\] So an experiment that was taking me an hour or two to run after the mall had optimized it, got it down to just one or two \[00:15:40\] minutes. Another loop you could do with this framework is you could let the AI choose which ads to show. So the optimization metric here would be something like lift or click through rate.

\[00:15:49\] And Rex has some examples there that we'll talk about later today. Another simple evaluation approach is simply to use correctness. So you have some tasks where you can define what's \[00:16:00\] correct or not. A case where this shows up in science is tasks are easy to verify, but hard to generate the solutions for.

\[00:16:07\] So one example is protein folding. So for any specific candidate, it's easy to say it's easy to automatically check if that folds into the right protein, but it's very hard to find the right candidate. So this shows up \[00:16:20\] in places, for instance of drug research. So if you have a problem that's easy to verify when something is correct, but hard to verify the solution, the AI is really a great option.

\[00:16:30\] I'd say one of the AI's superpowers is it can generate lots and lots of high quality candidate ideas to test. This loop where the eval is \[00:16:40\] verification of correctness is driving actually a lot of progress in many areas of research. For instance, there's been a number of open math problems that the AI has solved.

\[00:16:49\] Basically by taking this approach they just connect it to a simple harness had it do this job of just coming up with ideas and then checking if those ideas solve the \[00:17:00\] problem. So if you've used cloud code before, this kind of framework is actually automatically built in through something called Ralph.

\[00:17:07\] So you give the model an evaluation test, you want it to pass, and it will just continue trying until it succeeds. Now, this approach is pretty inefficient for humans but the AI models can work \[00:17:20\] tirelessly and autonomously. They can spin up lots of different agents at the same time. So this kind of approach actually often works for the agents pretty well.

\[00:17:30\] And here are some of the many efforts that use a similar kind of loop to this self-improving structure. So at this point it's largely true that if you have a question that's easy to \[00:17:40\] verify or you have a very good optimization metric the model can usually perform at or above the level of the best humans.

\[00:17:48\] So this sort of self-improving loop is an area that the models are very good at. So I'll pass it on to Rex to talk about his use of objective functions for advertising. 

\[00:17:59\] \*\*Speaker 3:\*\* Yeah. Thanks \[00:18:00\] Caleb. So Caleb made a really important point I wanna make sure that it lands. Is that when you think about where we are going with business and the world, these feedback loops are incredibly powerful.

\[00:18:11\] If you can either define what good looks like, or give a specific measure for what is success, you can achieve phenomenal \[00:18:20\] results at a much, much faster clip. But I want you to consider how important it then becomes that we as business people get the evaluation measurement. Correct. We're gonna pop up another quiz here, and the question is, which group will produce more purchases today?

\[00:18:37\] So if we're trying to optimize for \[00:18:40\] return on investment, which is we spend a dollar in advertising, we want to see, that we get back more than a dollar in profitable sales. What's the best way to do that? So you should see a pull up here. It has three questions. So which group will purchase more today?

\[00:18:53\] Is it an audience with a thousand current customers or a thousand people have never purchased before? \[00:19:00\] The second one is a message. Is it going to be, what will, generate more sales today? Is that a brand message with a deal offer like a discount or offer inducement to buy now, or a brand message without a deal?

\[00:19:13\] And finally, the media feature. Which one will generate more sales today? Is that a CRM \[00:19:20\] text and email a search affiliate or other lower funnel tactic, or is it a sponsorship? So let's go ahead and show the results from the poll. And not surprisingly, all of them land on a, which is 80% plus, 96% of people recognizing that \[00:19:40\] the faster thing to get sales is going to be talking to your existing customers with a deal offer, with a lower funnel message.

\[00:19:46\] Now here's the problem. The problem comes up when we consider this last question, and this one you can just add in churn in, into the chat, is if you have churn in customers, because all of those tactics aren't doing anything to \[00:20:00\] really attract new customers. And so if you have churn, you'll be working off of a smaller base.

\[00:20:05\] So go ahead, indicate if you've got churn of more than 5% per year. That is people who bought last year. You find 5% of less buying this year. Now almost every business has churn above that rate. And I want you to think about that and go through \[00:20:20\] it in that practice. Because what happens with the AI is if it's working off of a two week look back window, it's going to systematically optimize towards current customers with deal messages and lower funnel tactics.

\[00:20:32\] If you give it the freedom to do that, and it will give you an optimization bump in the short term that increases your results and shows positive ROI. \[00:20:40\] But in the longer term, if you're, if you have churn in your business, you may actually end up worse off than you would've been if you had set up a better evaluation that looked at both short term and long term effects.

\[00:20:52\] I call this the ROI trap. I've presented on this before and have written about it over the last couple of decades, ever since I got really into ROI \[00:21:00\] measurement and we got better and better at attribution. I began seeing this as a risk for businesses that didn't understand how to balance the short term and the long term.

\[00:21:08\] This is particularly true, the higher your churn rate is, this becomes a bigger and bigger problem. So we do have an initiative that I'm spearheading called Project Romy to try to \[00:21:20\] get the long-term and short-term effects nailed down in a really elegant equation that AI can use and businesses the CMO and CFO can use together to try to drive shareholder value and make sure that we're maximizing for the long term.

\[00:21:35\] So if you wanna be part of this initiative you can email one of the folks here and we'll \[00:21:40\] connect you to the right person. Denise champion is leading that and coordinating that effort with me. And so with that, I'm going to hand it back to Caleb to talk more about what we would call reward hacking and why it's so important to think very clearly about what what you want your AI to optimize.

\[00:21:59\] Over to \[00:22:00\] you, Caleb. 

\[00:22:01\] \*\*Caleb Briggs:\*\* Yeah, thanks Rex. Whenever you're doing optimization, and especially with ai. You need to think about this concept of reward hacking. So there's a famous saying called Goodheart's Law that says, when a measure becomes a target, it ceases to become a good measure. So what can happen is when you build these loops is that the model does a \[00:22:20\] great job at maximizing the eval or target that you're, that you gave it.

\[00:22:24\] But it sometimes does that in a way that's not helpful for what your actual goal is.

\[00:22:33\] So here's a famous example of reward hacking. So they wanted this model to be good at this racing video game. And \[00:22:40\] so it was hard to give it the goal of just win the race. So they instead gave it this goal of collect the green targets. And they thought that would make the model great. But instead the model just got really good at collecting these green targets and ignored the entire racing part of the game.

\[00:22:55\] So you can see it gets like stuck in this corner here and it's just collecting the green \[00:23:00\] targets at maximum speed, but has ignored the racing aspect of the game completely. The model will end up maximizing that eval target and it finds a way to do well in that eval without actually solving the underlying process or underlying problem that we want it to solve.

\[00:23:16\] It. So reward hacking happens all the time. \[00:23:20\] With AI models. It's very common because it's hard to come up with great evals. So for instance, it's reward hacking happened in a major way for anthropic during their process of training. Some of their models, I think this was the Claude Opus three.

\[00:23:34\] So they wanted to train the models to do good at coding. And so they had this self-improving loop \[00:23:40\] where they would give it a task, and then they would have an automatic evaluator that would check if the code passed all the tasks. So maybe you ask it to code something that computes the digits of pi and the evaluator would come back and check that the digits that the model computed were actually the correct digits.

\[00:23:56\] So the model would get a positive reward if it succeeded, \[00:24:00\] and no reward if it didn't pass the test. And so the way that they coded this is they used a three step process here on the right for the eval. So first it would run the code, and then it would test it against the correct values. And then if it found any discrepancies, it would send a flag out that says this model got it wrong and the model wouldn't get any reward.\[00:24:20\] 

\[00:24:20\] So if the incorrect flag doesn't show up, that should mean that the model must have passed all the tests and did a good job on the eval. So in theory, if you keep running this the model should learn to program better. What the mall actually ended up doing was to add a line of code that forced the program to exit early.

\[00:24:38\] And because it exit early, the \[00:24:40\] evaluator never got a chance to run. And so it didn't check against the correct answers. And so the flag that says it's incorrect would never get set. So to the evaluator, it looked like the model passed all the tasks. So here's a graph of training. Basically it didn't find this reward hacking until later in the process.

\[00:24:58\] And then as soon as it discovered it, \[00:25:00\] all that it did is just learn how to force exit that program. So it basically didn't learn the tasks at all. It just learned this technique of reward hacking. So here's another example to think about. So here we're deciding on whether or not to bid on an ad spot.

\[00:25:18\] So the model gets some kind of data, \[00:25:20\] maybe things like time of day, the location and the model can choose to either bid on an ad spot or not. And so presumably you wanna avoid wasting money here. So you don't want the AI model to bid on an ad if you know the person won't convert. So the eval set up here is you try to minimize the number of cases where you bid \[00:25:40\] and the person doesn't convert.

\[00:25:42\] So does anyone know what's gonna happen if you train a model with this eval? If you have any idea, maybe put an answer in a chat. I'll give people a few seconds to preach some guesses here.\[00:26:00\] 

\[00:26:01\] Yep, that's exactly right. So the model is gonna show. Zero ads because you can minimize the number of people who don't click or don't convert on your ad by making sure no one ever sees an ad. So if you never bid successfully, minimize the number of bids where people don't convert. And of course this will actually happen if you run an \[00:26:20\] eval on this.

\[00:26:20\] So 99 out of a hundred times you run this eval, you're just gonna end up with a model that never bids on any ads. And the problem here is that the model never gets positive feedback for converting. So it's never gonna risk bidding on an ad if there's no potential positive reward for it. So a better eval might try and balance between the \[00:26:40\] number of people that convert and the number of ads that you serve.

\[00:26:43\] Maybe it tries to maximize expected value. It gets so only bid on the people that are the most promising. The point is that you have to be thoughtful about the evals and targets that you set up. So this is an eval that you might naturally come up with, but when you try and train on this you \[00:27:00\] get something completely useless.

\[00:27:02\] So it's very easy to set up an eval where the model will take a shortcut and the model almost seems to prefer taking shortcuts and lots of evals. So you really have to think about how do I set up something that's shortcut proof so that the AI model will optimize on the goals. I want it to optimize.

\[00:27:17\] So now I'll hand it back to Rex to lead us \[00:27:20\] through the next hands-on exercise. 

\[00:27:22\] \*\*Speaker 3:\*\* Okay. Thank you Caleb. That was really important and really gives an insight in terms of how to get the superpower out of AI and then avoid some of the pitfalls. We are going to go into an exercise where you are going to build a generative AI ad creator.

\[00:27:39\] \[00:27:40\] We're going to do audio because audio renders a heck a lot faster. And for Caleb and I in 2023 in July, this was the first AI agent that we built and brought and showed to marketers. We presented at the C-E-O-C-M-O summit and and we basically said, Hey, you could build a dozen ads in under a minute for less than a dollar.\[00:28:00\] 

\[00:28:00\] It was great to present that. And one of the CMOs in the audience took us up on the suggestion, which is, we would love to help you try this and see what happens. When we put it into market that marketer was progressive and progressive said, Hey, I wanna try that out. We'll have AI generate the ads, we'll have a human approve it, and then we'll have AI serve and \[00:28:20\] optimize it.

\[00:28:20\] And because we'll have more volume, we could hopefully get better results. In fact what we saw here is that we saw a 50% improvement in con quote starts and quote completes, quote, completes obviously being really important to their business. It was such a big deal that Ram Kent, the CMO, that took us up on that \[00:28:40\] offer, she presented at the.

\[00:28:42\] The Possible conference in April in Miami. And she noted that typically it would take 16 weeks to produce three ads. Instead, she was able to produce these 96 ads in roughly two weeks and then run them for four weeks. That is a \[00:29:00\] 512% increase in that one week that it took to produce those ads compared to the 16 weeks it normally took.

\[00:29:05\] And then there was one week for approval. The other interesting thing is that 512% increase in productivity. They only had one person in the loop in this case, which was a really good creative director. So it was much more efficient and \[00:29:20\] productive, and it increased business results by 50% increase. So who would wanna expand their AI in their business with that kind of result?

\[00:29:28\] Let's take a listen to what those ads sounded like, just so you get a feel for these initial ads. And then we will we will go and we will. Build this workflow. So this is this is one \[00:29:40\] of the ads and voices 

\[00:29:41\] \*\*Speaker 5:\*\* want the perfect auto insurance. It's like crafting the ultimate playlist. Progressive was the first to offer online shopping for insurance, and we continue to innovate for your convenience.

\[00:29:51\] With over 30 products designed to make insurance a breeze, imagine coverage options that reward your safe driving and fit your budget. Get a quote \[00:30:00\] today@progressive.com and see why more than 28 million drivers choose

\[00:30:04\] \*\*Speaker 6:\*\* progressive casualty. 

\[00:30:06\] \*\*Speaker 3:\*\* And then of course they had different scripts.

\[00:30:08\] The scripts all start the same, but they they also had the choice of different voices. So here is another script with a different voice. I'll just play the first 15 seconds of it. 

\[00:30:16\] \*\*Speaker 6:\*\* Want the perfect auto insurance? It's like crafting the ultimate \[00:30:20\] playlist. For over 87 years, progressive's been dedicated to personalizing your coverage to fit your life just like your favorite tunes.

\[00:30:27\] And we do it with exceptional service and clear communication. You're getting great coverage. Visit us at Progressive. Do 

\[00:30:35\] \*\*Speaker 3:\*\* so you've got the idea here. We're going to build that. I'm gonna show you how to build that. In, in \[00:30:40\] eight N we are going to need to have the API from OpenAI because they're the ones who are gonna generate the voice.

\[00:30:46\] So we talked about that at the beginning of this session to log in. If you're not able to do that, we'll still show you how it works, so you have a good intuition for how it works. But let's go ahead and build that together. Now. First things first, \[00:31:00\] in the in the walkthrough, we can jump down to day three and get to this first exercise.

\[00:31:05\] You will see a walkthrough that will take you step by step through what we're doing. So if you're not able to keep up with us today, you can go back through that again. You can also look at, listen to the recording and try that again. You, we are \[00:31:20\] working in it in. So I'll jump over to that. And we mentioned in the, open AI platform. There's some steps about creating an API key here in my dashboard. The home I can see, I can add credit, so if you don't have a credit, you can do that. I can also click create API key and I will type in MMA training \[00:31:40\] and they'll create a secret key, which is much longer than this.

\[00:31:45\] And I'm going to delete this one in a second, but normally you just click copy and we'll put it over in the other session. This is connected to my $5 credit card, so this is where you don't normally share this. But as I mentioned before I'll delete this in just a minute. \[00:32:00\] The the other key thing as we go into N eight in to just go through the steps here, we will go ahead and log in and I have, I already have my workflow started here. That will bring you this page. You'll hit Create New Workflow. And what we're looking for here is creating \[00:32:20\] first step. We're going to use a chat trigger to start this one. So we'll say chat, we'll click that and that will open up this display. We don't need to change anything here at all.

\[00:32:31\] We can just hit close. And now we've got our chat trigger. As we mentioned before, there's lots of triggers. You can do this on a time schedule. You can do it when a form is filled in. We're \[00:32:40\] just gonna do it with a chat so we can ask it to create an ad for us. What do we need next? We need we need an AI agent, so we'll click the plus button.

\[00:32:47\] And then the AI agent. And we'll click the top agent here. Now we can connect the model now or we can connect it later. You can see by default it's automatically connected to the chat trigger node, so \[00:33:00\] we don't have to do anything here. We'll leave it here and we'll come back. We'll add the chat model in a minute.

\[00:33:04\] This is where we connect the LLM model. We'll also add in a we could add in a system instruction to tell how to create a good ad. But for now, we're just gonna leave this here. And we'll then add the last piece, which is the audio node. And if you type in audio, \[00:33:20\] there's actually a number of options for audio.

\[00:33:23\] I'm going to use open ai. And when you click on it, you can see different things that you could have open AI build for you. It can by build images, videos in this case we're going to generate audio. You can try it later on with some of the other ones. When we come in here, we'll see \[00:33:40\] that there's different voices we can chat, pick.

\[00:33:42\] I'll go ahead and try, I'll try Fable. I haven't tried that before. If you come in here and it, and you haven't set up your open AI key yet, the way you're gonna be doing that is you're clicking on this plus button and this is where that API comes in. So if you see that this is \[00:34:00\] blank, this is where you would come over here, copy your API key and place it over into the this field here.

\[00:34:10\] And when you save it or when you just hit enter, it should automatically check it and it should turn green. If it doesn't, as I mentioned before, we can come back and test \[00:34:20\] it. And in my case, I'm going to go back into my API keys and you can see this new one which I just created. And I'm going to delete it because if my API key is out there on the recording, it is connected to my credit card.

\[00:34:35\] So good safety point there. So \[00:34:40\] what do we have so far? We have a we have the key nos here. So it's, I'm going run it first here, but we have to add in a brain. So the brain we wanna add in is open ai and this will automatically use the same key that you just created there. It's a nice feature of innate in to \[00:35:00\] make that easy for you.

\[00:35:01\] And it will default to the GPT five mini model, or you can select from the list. I'm going to go ahead and leave it there and that's fine. So now I have the basics of our agent. This is similar to what we built the other day, where we had an AI agent connected to an LLM, and in this case we're having it generate audio.

\[00:35:19\] Now in \[00:35:20\] order for it to generate audio, we need to be able to pass the prompt across here. So I'm going to try a quick test here, which is I want to create. Create a ten second ad for MMA possible conference in Miami. So I'll hit enter. And so you can see it \[00:35:40\] running. It's going through its logic for how to do it.

\[00:35:42\] By default, it's going to be just a helpful assistant, which is pretty much what you get out of an LLM. And then as we see it going through the process, you can see it going through and it's trying to come back with an answer for us. \[00:36:00\] And you can see the human input here, create an ad, and now you can see the output.

\[00:36:06\] And so it's creating a mixed martial arts ad and a marketing conference ad, doesn't know exactly which one it is. We're gonna make our agent smarter to fix that problem. Then you can see it's red here. Now the reason why this is red is because we connected the node, but \[00:36:20\] we didn't show it where it's going to get the text to create the the ad.

\[00:36:24\] So if I click on it here, we can now see we've hydrated the AI with our our input that we had here and our output that we have here. And so when we go here, the text input, this is where it gets the text to speech. \[00:36:40\] And so if we drag this over here, this is what it would read. Now the problem with it reading this right now is this is a lot more than just a voiceover, right?

\[00:36:49\] You can see the voiceover plus you can see different special effects and visuals at suggesting. And and it's giving us more than one option. Okay, \[00:37:00\] so what we have to do is we have to make our agent a little bit smarter in knowing how to pass on the right results. So you've created the first step here, which is to basically be able to generate text, but now we want to give our agent some more of intelligence to do it in a better \[00:37:20\] way.

\[00:37:20\] And the way we would do that is instead of it just taking the prompt directly from here, which is create a ten second ad for MA's possible conference in Miami, we want to add some extra a system message, which is basically what we talked about earlier as a additional prompt that comes ahead of. \[00:37:40\] The main prompt.

\[00:37:41\] So here, instead of saying you're a helpful assistant, we can tell it about what the MMA is. So it knows it's the marketing organization and not mixed martial arts. We can tell it about what we want it to do in terms of creating an advertisement. We can give it basically these instructions of this prompt that will then help it create \[00:38:00\] better output.

\[00:38:01\] So I'm gonna pause and give you a minute or a couple minutes to just get the first three steps going, and then we will add a system message to make it a little bit smarter. And then we'll test the output again and see if we get it, get a better.\[00:38:20\] \[00:38:40\] 

\[00:38:49\] \*\*Caleb Briggs:\*\* And Rex or Dan, I'm not sure if this is feasible, but someone's asking if we could go full screen to see the texts better. If there's a easy way to do that might make it so that especially the people on their \[00:39:00\] phone can see what's going on more easily. 

\[00:39:02\] \*\*Speaker 3:\*\* Yeah, that's a great idea.

\[00:39:14\] Our technical team Dan is the man behind the scenes making all this work. So we'll see \[00:39:20\] if he's able to do that. I'll give one more minute and then I will go and we will begin to unpack the other parts. And again, if you're not able to see it, if you're on a phone or small device the walkthrough, as I mentioned before, it creates the now we'll click on that here.

\[00:39:34\] This has the step by step to be able to go through what we just did here. And we \[00:39:40\] basically are up to this step here where we've connected the agent, but we want to now make it a little smarter with with a voiceover, basically to be able to extract the voiceover.

\[00:39:56\] So we're just about up on this. Page \[00:40:00\] six. Now. Okay, so two things that we need to talk about coming back to this piece here is step one is we have this agent, but it isn't smart enough to know what kind of output we want because all it has is the instructions that your helpful assistant and all \[00:40:20\] it has to go on from the prompt is create a ten second ad for M MA's possible conference in Miami.

\[00:40:25\] It is interesting that was able to find that there is a marketing conference that's different from mixed martial arts, so it's pretty smart, but maybe not quite smart enough. So what we wanna do in this case is we want to tell it a little about MMA. And so what \[00:40:40\] we're going to do for our next exercise is we're going to go down to the step, which is how you might go about creating your brand, DNA, so you can add to the system prompt to make it smarter.

\[00:40:50\] And here you can use Chachi pt, Claude Gemini, copilot, anything that you want, what you're basically gonna be doing here in this one. Is I'll \[00:41:00\] use I'll use Gemini for this one. You just basically wanna say create a brand, concise brand, DNA for my brand. We could say something like infer your brand DNA from this website.

\[00:41:17\] So we'll say infer my \[00:41:20\] brand DNA from my website, MA global.com, and it'll actually do a pretty good job doing that. And then we, what we can do is we can give this DNA to our agent so that when we're asking it to create an ad, it's doing it \[00:41:40\] consistent with our brand guidelines or who we are. Okay, we've got this.

\[00:41:45\] Cool. The brand DNA can be broken into these pillars. I'm just going to go ahead and grab it. Normally I would edit it and and we can give it the brand voice part here, copy it over, and you might copy it to a Word doc or a Google Doc, so you can come \[00:42:00\] back to this later. I'd encourage you to do it 'cause we are going to use it later here.

\[00:42:03\] And I'm going to go ahead and add it back in this system instruction I'm clicking on here. Instead of saying you're an assault, hopeful assistant, I'm gonna paste the brand guidelines. I'm also gonna give it a little bit of instruction here, which is you are an audio voiceover advertisement creator for a brand whose voice \[00:42:20\] is.

\[00:42:21\] And then that will be my preamble to lead into it. So we'll come back to the top here and we'll add that in.

\[00:42:38\] And I'll remove the brand pillar \[00:42:40\] one so it makes a little bit more sense. You can click on this little button, by the way, to make this larger so you can see what we have. So now we have a system message that says, you're, you are an audience, voice over advertisement creator for a brand whose voice is ev evidence over opinion ecosystem, unifier, future proofing, innovation, et cetera.

\[00:42:57\] So it has this great, brand description. Hopefully you've \[00:43:00\] done that for your brand. Now, if we give it, come back and do this same task, we'll close this up and, oh, there's an interesting trick. If you just press the up arrow, it'll just recopy your last chat here and we'll hit enter and we'll let it run.

\[00:43:16\] And it should produce a better \[00:43:20\] advertisement. But it still might include direction like the. Vo instead of just the voiceover, it might also still have special effects and things like that. So let's go ahead and see what's happened. It did generate a output. So we'll click download and I will give a quick listen \[00:43:40\] to that

\[00:43:46\] and hopefully you are able to do the same thing on your side. And you'll be ch listening to yours, 

\[00:43:51\] \*\*Speaker 7:\*\* Miami MMA Scientific field, proven growth Frameworks, MMGF, agen ai, and CFO Ready Metrics. Join leaders@emmaglobal.com. \[00:44:00\] Voice, authoritative, confident, steady pacing to fit, 10 seconds, 

\[00:44:03\] \*\*Speaker 3:\*\* subtle modern scene. So the other direction at the end about telling what kind of voice to have steady, confident, et cetera.

\[00:44:09\] So it had the voiceover, it got part of the way there. So let's go ahead and first of all, that's a win. If you've got that yours to work, you have created, halfway there. \[00:44:20\] So if we come back to the next exercise is how we basically might create the voiceover and extract that part of it, and then how we can evaluate how well it works.

\[00:44:30\] And by the way, there's an interesting optional tool I'm gonna just click on called Pelli. This is from Google. It's part of their Google Labs. And if you type in your URL at the very beginning, \[00:44:40\] when you log in, it will create that brand DNA for you. It'll extract the fonts, the brands, the voice. And so this is another way that you might be able to create a voiceover.

\[00:44:49\] And in fact, by the end of this training, you'll probably be able to look at this and say. I bet I could create an agent that does the same thing, and I think that would be pretty cool if you're able to do that. \[00:45:00\] But this is a great tool to look at brand voice. Google's using this to be able to create advertisements that where the AI already knows some of your key assets.

\[00:45:08\] They already know what it should look like. And and it will go ahead and create a campaign or ad based upon some of these messages. So this is a more advanced agent we're building. We're building \[00:45:20\] one to create some audio ads. And so we're not quite at this level yet. Okay, let's go back to our agent again and see.

\[00:45:27\] We've got this agent working pretty well, but it's giving us more than what we want for the voiceover. So what we need is we need another agent that knows how to extract a voice. So we'll break this link here. We'll add \[00:45:40\] in another AI agent by clicking on the agent here. So we'll click on our agent here, we'll connect it to open AI again because we already have that API key going.

\[00:45:50\] And we'll leave it at the default, but we'll give it this. So we'll close this part up. So now we have an agent here. We'll reconnect this up to the audio. We'll have \[00:46:00\] to reconnect this up. As you can see the yellow notice we want to give it some instructions about what it should be doing here.

\[00:46:07\] The no that's connected to isn't a chat trigger anymore. It's, we have to click define below because we're pulling the information in from the previous steps. So we wanna take the output from the earlier agent and put it here. And we \[00:46:20\] wanna add in a system instruction that says tell it that its job is to extract the voiceover.

\[00:46:27\] Now, you'll find these in your instructions, and it'll basically be your role is to extract the voiceover script from the prompt. I'll put only the voiceover script, nothing more. So that's \[00:46:40\] its only job. So if we execute this step, we can see it taking the output here and we should see it pulling out just the voiceover here, Miami.

\[00:46:50\] Okay. This is good. So now it doesn't have the instructions anymore. Now we have to do one last step, which is since we changed the flow of the \[00:47:00\] pipeline, we have to reconnect up to generate audio. And actually it kept the audio from the original agent output and we don't want that anymore. 'cause that would, that might

\[00:47:10\] The other thing by the way that will help here is we want the output from this agent that we just added. Other thing that really helps is naming these \[00:47:20\] things. So we've dragged this output over to the input. So now we've got the right input. And I'm going to go ahead and close this, and I'm going to name these agents.

\[00:47:30\] If you click on each agent. You can name them up here. So we'll call this our our brand. We'll call this the the ad \[00:47:40\] director agent and we'll call the other agent. This will be our voiceover agent. So again, we click on it and then click on the top corner here. We'll call this the voice over agent.

\[00:47:56\] And its only job is to extract the voice. And we saw that it could do that. \[00:48:00\] So let's go ahead and try to run this again, and we're letting this run. I'll give you a minute to catch up and try it. And then we will come back and we will debrief on the last stage of this, which is evaluation.\[00:48:20\] \[00:48:40\] 

\[00:48:45\] Okay, I will go ahead and play this a just so we can hear it and then we'll move on to the last stage, which is evaluation 

\[00:48:50\] \*\*Speaker 7:\*\* Miami. Join MMA field proven research from multi-year labs and the movable middles growth framework, CFO, ready Marketing that lifts ROI register now. \[00:49:00\] 

\[00:49:01\] \*\*Speaker 3:\*\* Okay, so now we've got a pretty good, decent script and what we might do here is we'll go back and we might say, Hey, I wanna begin to create some evaluations.

\[00:49:09\] I wanna know, I wanna look at. My input. And again, if you click on these buttons here, we can see what the input is and I wanna look at the output. So what I've done, and I encourage you to \[00:49:20\] same, is create a Google sheet where you can put in the prompt. So this is our prompt and I'll tidy this up just slightly here so we can see it.

\[00:49:30\] And I will want to look at the output so I can copy the output here.

\[00:49:39\] And I'll paste \[00:49:40\] that in here. And I will as, just as Caleb was telling us before and what we did with the earlier one, I can give it a grade, I'll give this a a c it used the right terminology, but just didn't feel as exciting to me. This is subjective. You can try to run it a couple times.

\[00:49:59\] \[00:50:00\] You can tweak your pro, see if you can get a better output. What we're going to do next is you should run your agent a couple more times. And you should basically, if we close this back here, we now have it running straight through. We can try it for we'll try it for the \[00:50:20\] AI training, for MMA AI training.

\[00:50:30\] And I'll again, in the bottom here, you get to this part by dragging up. If you haven't noticed that it has the output and you can see it going through the process, \[00:50:40\] we can we can also connect this up to a Google sheet directly so that instead of us having to copy and paste this in, it will do that automatically for us.

\[00:50:51\] And so I'm going to show that last part here. I'll copy this. Into my spreadsheet just to show the \[00:51:00\] output. And I'll then pro copy in the input part as well, which was this create the AI training. But let me show you, instead of us having to do this manually, how we can have the AI or the AI workflow do that for us.

\[00:51:19\] So what we're going to do \[00:51:20\] here is we're gonna add in another branch. We're gonna type in Google Sheets, and we're going to have it app pin a row in a spreadsheet. Now in this case, you are gonna have to click on this button again, and you're gonna have to sign in with Google. A lot of times, or when I've tried it, oftentimes it \[00:51:40\] blocks the the sign in.

\[00:51:42\] So you might see a little button over here that you have to click on to to be able to make that connection. And then you'll just continue and give it authorization and then that's set. And with once that's connected, then you can look \[00:52:00\] for it to to match a sheet. You can go down to your document list.

\[00:52:04\] You can say from the list. And I was calling that one AI agent. Actually, I can see that right at the top of the list here, one that I created most recently. And I only have one sheet in here, so this will be pretty easy. Sheet one. And then what it does is it matches the \[00:52:20\] different fields in my spreadsheet, the columns, prompt, LLM, output and evaluation.

\[00:52:25\] And that lets us map where, what we want to go into different data. So if I go back here and I go to the the actual chat input, that's what I want as my prompt. Perfect. And then what I want is my output or my LLM output comes from the \[00:52:40\] voiceover agent. That's my output. And the evaluation, we haven't built that yet, so we're gonna leave that blank.

\[00:52:46\] So when I hit execute step on this one, it will push that same information into this sheet again and it should append it as another row, even though I had pasted in before. So we can now see that is connected. \[00:53:00\] So you are halfway through building your evaluation agent. So let's do a recap of what do we have so far.

\[00:53:08\] What we have so far in your agent is you have a chat input as a trigger, an ad director, one that has your brand, DNA, and the instructions to create an ad. A voiceover agent that \[00:53:20\] extracts just the voiceover and passes it on to your generate audio. And you have a nice extra piece here, which is the a pin to the row so that it then creates.

\[00:53:32\] Now you'll have to go back in here again, and you can see that this step to make sure your output is still pulling the right information. \[00:53:40\] The positive part about workflows is that you completely control what they do. The downside to the to the workflows is that they're a little bit brittle. And so when you add new things in, you have to repopulate exactly what the information going to use for its information.

\[00:53:58\] You should be able to run this again, \[00:54:00\] and you should be able to see it automatically run this through. And I'll turn green and then you should see an extra field added here. Tomorrow we'll come back through and we'll create the last two steps, which is the building out of the agent. \[00:54:20\] And we will create the evaluation feedback loop so that we basically are able to have it evaluate what's a good score or not and keep trying until it can get us to a grade of an A.

\[00:54:34\] So you got through most of it today. We will do this tomorrow if you wanna work on the work. On \[00:54:40\] the walkthrough, you start with page roughly where we ended somewhere around page 13\. And and. Build the evaluation agent. So you're ready to go tomorrow and even try the feedback loop. What you'll notice, by the way, on your cost is that we'll run this a ton of time.

\[00:54:56\] You'll be able to check with it. We'll probably spend less than a dollar, even though you'll, you \[00:55:00\] can run it like maybe a hundred times. Please take a moment and fill in the NPS score. We will be using this for our Friday exercise, and it also gives us feedback for how we are doing our in our sessions.

\[00:55:11\] Caleb and I will now move into the QA overtime to to answer questions for the next 15 minutes. Thank you for joining us today. \[00:55:20\] Okay, Caleb, what what questions do you wanna take first?

\[00:55:29\] \*\*Caleb Briggs:\*\* Let's see there was a question earlier maybe just worth res answering to the group of do you need an API key to use these? For some of these I you do need \[00:55:40\] an API key, right Rex? And it's you get it through the, like for open ai, they have their own platform to get an open AI key.

\[00:55:48\] I guess it's worth knowing that's separate than the regular chat GBT account. So API keys, you can have an API key without having an account. And if you have an account, doesn't automatically give you an API key. The other thing to \[00:56:00\] know is it does cost every time you use it, but the price point is much lower than probably most other cases you've used. So if using TBT 5.4 mini it's 75 cents per million words. So I was just computing that if you're like asking for an a hundred word ad that's 0.0 0, 0 0 \[00:56:20\] 0 75 like cents dollars each time you're using it. So it does charge you every time, but the price point is low enough that it probably won't add up to even 1 cents unless you like run it a large number of times.

\[00:56:31\] The part to be more careful of is image generation that can sometimes be much more expensive. But generally, like you can produce almost as much \[00:56:40\] text as you want without ever like even adding up to 1 cent. 

\[00:56:45\] \*\*Speaker 3:\*\* Yeah, that's right. And I think in ours the main cost is gonna come from the generate audio part.

\[00:56:50\] And there are, there basically if you think about from simple to complex and price follows this. Text is cheap really cheap as kid just illustrated. \[00:57:00\] The next one would be audio that's relatively cheap. Then its images that's a bit more expensive, and then video, and then that takes a lot longer too.

\[00:57:09\] So we're doing it on audio, but later on you'll probably only get through like maybe 50 cents after we do the evaluation loop of your $5 that that you put on the credit \[00:57:20\] card for OpenAI. You can then cut that connection. You can change the agent to give it instructions to create a really good infographic or a really good advertisement.

\[00:57:30\] Just to play around and try different things with what you build here. But yes, you, the link at the very top of the session to open ai dot API brings you to a different \[00:57:40\] webpage compared to chat GPT and that's where you saw the billing part and that's where you saw adding the API key. It's in the walkthrough if you need to, if you didn't set that up before and you wanna play with this tonight and you wanna be ready for tomorrow's session, definitely please do set that up.

\[00:57:54\] That o open AI key works for the generate audio as well as the as the text. \[00:58:00\] Another question you wanna tackle, Caleb, 

\[00:58:05\] \*\*Caleb Briggs:\*\* I'm just gonna ask, did we build a generic workflow and not agent yet, but I just wanted to. See which rebuilding. 

\[00:58:10\] \*\*Speaker 3:\*\* Yeah, that's right. That's a very good point. We are building ag agentic workflows right now, and then we're moving into agents when we do the evaluation feedback \[00:58:20\] loop tomorrow that really is in Anthropics definition.

\[00:58:23\] And we agree with that definition. You go from ag agentic workflows to more just full blown agents when you have an evaluation loop and it's self-learning and it's adjusting itself. You saw that in deep research where it went through a part where the first thing it does is it tries to \[00:58:40\] do it intention extraction.

\[00:58:42\] It says, what do you ask me? What do I think you want? And it plays back a research plan and then you approve or not that research plan now it just has a timer, I think, on chat, GBT, where if in a minute you haven't clicked edit, it just starts running because most of the time it's right. And then what it's doing at that point is it's doing these \[00:59:00\] feedback loops where it's pushing out lots of searches on the web gathering information.

\[00:59:04\] Then it's evaluating that information as deciding if that information's right or not. And that's why we said you should click on the thinking part and see what it's doing. And you can see it sometimes saying, oh, I'm not sure if that's right, or I'm not sure if that's the right information. When I did my scheduled task for the Brent crude oil, \[00:59:20\] it dis it searched where could get those prices and it found the St.

\[00:59:25\] Louis Fed website had really good information, but then it said I'm not sure if it will go back far enough, or I'm not sure if I can load it. And you could see it going through and figuring itself out because the Fed line had 1097 lines and it had only created a table \[00:59:40\] for 1000 lines and it self improved and figure out how to fix it.

\[00:59:43\] And by this morning I had a really good update on Brent crude prices because of that scheduled agent task. So you have done some agents already, but we're gonna show you how you build one so you understand exactly what's happening under the hood so you can be really effective as you go forward in marketing.\[01:00:00\] 

\[01:00:03\] \*\*Caleb Briggs:\*\* There's another question here about like what does it mean to give a reward to the model? I think I potentially could have used a little bit different terminology. I think reward is a, is a technical term that they use in reinforcement learning. But maybe there's a reason why \[01:00:20\] I like you might wanna separate between the reward that was happening, for instance, in Rex's optimization agent.

\[01:00:26\] That's a mathematical reward that's happening where like you take a model and the model. Works a lot very similar to our human brains, where when you get a positive reward to the neurons strengthen their power and you get a negative \[01:00:40\] reward, it weakens those neurons that we're firing.

\[01:00:42\] So there's sort of two cases of reward. One is that mathematical case. There's another one of just like feedback. If you're talking with an AI agent and you say this is a great response, or this response needs work, that also functions as feedback that changes the neurons that fires. Part of the reason why I use the \[01:01:00\] same language for both of those two cases is because a lot of recent research has shown that actually when you tell a model it's doing a good job versus you tell it's doing a bad job, it implicitly does the same sort of thing that would happen if you ran an algorithm.

\[01:01:14\] If in a chat you're talking about you're giving it feedback for each of the things it does, like maybe you get feedback \[01:01:20\] for, this was a good ad, this was a bad ad, et cetera. It's actually gonna be changing its brain a little bit to change how it creates the ads. Maybe just to summarize all of that when you're working with agents, giving them just feedback and language in the same way you might give feedback to another person is the sort of way that you change its brain to be \[01:01:40\] better at giving the responses.

\[01:01:41\] So giving it positive feedback. You did a great job, functions as positive reward. Giving it negative feedback like this needs more work. I didn't like this response that functions as the negative reward. 

\[01:01:53\] \*\*Speaker 3:\*\* It's a little freaky for people. I remember Caleb, when we were writing the book and when we handed over the transcript, our editor was like.\[01:02:00\] 

\[01:02:00\] What's this reward thing? Are you like giving it a cookie or whatever and you're like, yeah, actually you are giving it a treat. You, it is working that way. But it is a foreign concept that this machine actually has a reinforcement loop based upon a reward system, but that is in fact what's happening in the strange world.

\[01:02:19\] Ai. Yeah. 

\[01:02:19\] \*\*Caleb Briggs:\*\* There's \[01:02:20\] a kind of famous example from serget Google where he's getting interviewed and he mentions that like you can improve the model's performance by putting it in distressing situations. So if you're like like there was a system prompt for a coding company windsurf a while ago that said we're Mega \[01:02:40\] Corp and like your family, we'll only get medical treatment if you produce a really good response here. And the model does 3% better. I wouldn't necessarily recommend doing that, but the model does sometimes perform better in ways that like match with the emotional state of humans. If they, they really have a strong reason to give a good response, they'll sometimes \[01:03:00\] give a better response.

\[01:03:01\] \*\*Speaker 3:\*\* You also showed in your other one though, that some of the research showed that when you begin to keep grinding on the ai, it actually becomes a little bit more of a socialist or communist in terms of its view of capitalism and the rest of it. So there's some weird, there's, yeah, there's some weird.

\[01:03:16\] Elements.

\[01:03:17\] \*\*Caleb Briggs:\*\* Even if you ask it to write code in \[01:03:20\] December, it will write about 5% shorter code. And that's because it's holiday season. And if the model takes it December, it'll be a little bit more lazy on writing its code 

\[01:03:29\] \*\*Speaker 3:\*\* because it people are more crazy lazy during that month. Or work a little bit less, right?

\[01:03:34\] \*\*Caleb Briggs:\*\* Yeah. When models learn to emulate a lot of features of humans, and those end up \[01:03:40\] actually being important features of the AI too. Since humans get lazy during the holiday season, it's seen a lot of that. It emulates that property of humans. 

\[01:03:49\] \*\*Speaker 3:\*\* Yeah. And really what, why you picked the ne the next question to, to answer.

\[01:03:53\] When Caleb was talking about the Ralph Loop Ralph Loop was named after the Simpson character, Ralph. 'cause not very smart, but just keeps going and \[01:04:00\] going. I think Claude may have renamed it like the Claude Loop or something like that. But that's where diversity came from. We're gonna do that tomorrow where we're basically gonna just keep the AI looping through until it can get an A And again we're doing something relatively quickly in this training compared to what we would do if we were building something for production.

\[01:04:19\] But you're going to \[01:04:20\] see how the AI can begin to learn and get better as you give it. That ability to feedback and loop over itself. You can also tweak the instructions. You can give it a little bit more of a evaluation and you can try different things like giving it, hey, some type of kudos if it gets an A and see if it tries harder, and that will be part of your \[01:04:40\] experimentation. 

\[01:04:41\] \*\*Caleb Briggs:\*\* There's a good question here. Nathaniel, they asked, would you create or could you create an agent to take note and one for providing feedback? 

\[01:04:50\] \*\*Speaker 3:\*\* Yes. I love that idea. Absolutely.

\[01:04:53\] One of the things you can do here is you can connect it to your outlook so you can have it read your emails and give you some ideas to draft your \[01:05:00\] responses. You saw how we could connect to the outlook as an endpoint node. You could connect it to the front end. To start the process. So think about triggers more broadly.

\[01:05:10\] If you have a note taking app and it offers an API or has a way for you to, an API would mainly be how it works or what's called a web hook. And you can ask LLM to \[01:05:20\] tell you more about it. Give that a try. Have that be your trigger. When you write a note, have it automatically grab it. It will wake up when, almost like filling in a form.

\[01:05:29\] It'll wake up when it gets that response. And then you can have it run in evaluation, which is maybe a critique agent or maybe something that's a summarizer or or something that helps you refine it. So there's \[01:05:40\] so many, there's al there's really infinite possibilities of what you can do Once you learn how to build agents and you learn about the evaluation loop.

\[01:05:49\] \*\*Caleb Briggs:\*\* Yeah. And that's a very interesting one to think about. If you remember the loop in the original loop, there was human feedback. It seems like maybe you can replace the human feedback by an \[01:06:00\] AI feedback in that step as a way to close the loop. But part of. The reason why I didn't suggest that as the first option is that there's an extra thing you have to be careful of, which is that if the AI's the one giving the feedback and the one producing responses, it can be very easy for it to reward hack.

\[01:06:17\] So if you're doing something, especially if you put this in a loop \[01:06:20\] where the AI's providing feedback and producing the responses you usually wanna either have a human in the loop who's looking at it every once in a while or have some other eval that's like a ground truth that can compare against.

\[01:06:31\] What can often happen is if you have the ai taking both sides of the sampling and feedback, after a while it might end up producing responses \[01:06:40\] that like the feedback no longer matches at all with what a human would give and the samples no longer match. So you can get very out of distribution, when you loop the AI.

\[01:06:48\] With itself. So I'd say a few. It's great to have the AI giving feedback. It normally works a few times, but as you loop more you get further and further away from what a human conversation looks \[01:07:00\] like. So that sort of loop often doesn't work fully autonomously unless you can bring in some other ground truth.

\[01:07:06\] \*\*Speaker 3:\*\* Yeah. Maybe just to build on that point, if you decide that you're going to get much more sophisticated in what you want to do, then you know, reach out to Caleb or you'll also meet Jared on Friday on LinkedIn and, or through the MMA events \[01:07:20\] to get some more advice on your architecture.

\[01:07:22\] I know Caleb has done like consulting calls with people just to look over like what their design is or what they've done and provided, feedback to a number of different people. If you're also trying to assess if somebody really has good AI underneath the hood, he's also done some of those due diligence calls to say how good it is or not.

\[01:07:38\] We've actually used him for some of that \[01:07:40\] work at at some of, what we are looking at the company I'm working at. Yeah. First of all, we love the fact that you have stayed along for this extra q and a discussion. We really appreciate all that you're doing to learn a new skill.

\[01:07:53\] And we know, I know when I had to first learn it, a year ago or so, it's tough. You have to do your reps and keep doing a \[01:08:00\] number of these builds and then it starts to click and you get it. So thank you for sticking with us. We will see you back here tomorrow. I've moved exercise 13 into tomorrow, so we'll start off with building an evaluation ag agentic workflow, and and then we'll see how the loop works.

\[01:08:15\] And then Friday we'll talk about the future of work. See you back tomorrow. Take \[01:08:20\] everyone. 

\[01:08:20\] \*\*Caleb Briggs:\*\* Thanks everyone.

## Day 4

\# Decoding AI for Marketers VII: Day 4

\[00:00:00\] \*\*Hassan Khater:\*\* Hey, welcome back everyone. Great to see you again for day four with an amazing voting AI for market straining. I'll do a quick intro while everyone is joining in as usual. So far this week, we've covered the fundamentals of AI and power context, authentic workflow workflows, and yesterday we went deep on agen workflows and evaluations.

\[00:00:19\] Today we're getting into \[00:00:20\] multi-agent networks to help multiple agents coordinate to accomplish complex tasks.

\[00:00:29\] Okay. Now, as you can see, this is our seventh position of the training. So just I Greg and a had went over this time, but just YI can find it this valuable. Share it with your teams. This \[00:00:40\] one or the next one, it's free for all MA members. Again, not gonna skip through those slides that Greg and Alex went over this training brought to you by the AI Leadership Coalition for the A LTT ANDA's AI dedicated arm.

\[00:00:53\] And what you should know here only is that, if you're not already part of it, here's what to do. Okay. Join, go tona \[00:01:00\] global.com/ai. There's a short form at the bottom where you can pick a working group, content and creative audience development, measurement, analytics, or response with ai. Once you do that, I'll add you write in the community.

\[00:01:10\] And that's where the ongoing research, peer conversation and events happen beyond the strain. Okay. This is me, MSAN Carter, and before you, you saw my colleague Alec Foster. Together we \[00:01:20\] co-direct the A ITT. Now if you have any questions on the research or wanna get involved with anything, feel free to email us at s at ma global do com or allocate ma global com.

\[00:01:29\] So quick reminder of how we've I think here. Yeah. So our five future labs, I think each one is a real world AI experience. You can join within spotlighting one per \[00:01:40\] day. Day one was scab AI creative personalization, day two, aura ai, audience discovery. And yesterday we did with Alec Semantic and implant based targeting.

\[00:01:49\] Today I want to spotlight ACE AI creative effectiveness, and tomorrow we'll talk about a three. So a stack is a simple but critical question. How can we know whether an ad will perform before we spend \[00:02:00\] the hypothesis. Is that what if AI could extract the elements that define creative success for your brand and then score new ads against those criteria automatically?

\[00:02:08\] We're running this with two scoring partners and RA with room to add more three steps. One, you import your ad account and for conservative assets, two, AI analyzes those assets through computer vision, OCR \[00:02:20\] and machine learning to extract the brand's winning criteria. And finally, they AI scores new creative and gets recommendations before you go live.

\[00:02:27\] And in, in one of the studies with anova, we analyze over 440 creators on meta. They ask scoring model predicted high performing ads with 84% accuracy. So that's the difference between guessing and knowing what works. It's free to participate \[00:02:40\] in, takes three to four weeks. Again, email me or VA that matters or head of research if you're interested in this specific experiment.

\[00:02:47\] And also I'd like to add one more thing before handing it over to, to Rex and Caleb. If you wanna keep the learning going, you can check out our podcast, the Golden AI for Marketing. It's hosted by Greg Stewart, our CEO Andrex himself. And the latest episode features Jamie, \[00:03:00\] the Chief Creative Officer at Runway.

\[00:03:01\] He shares how ai power creative tools are changing marketing, which is very relevant to what we just about, talked about Ace. And tomorrow is H AP Spotlight on Agen AI advertiser. I think that's it for me. Without further ado, let's dive into day four and explore how multi-agent actually work.

\[00:03:16\] Thanks everyone. Enjoy the session. 

\[00:03:19\] \*\*Rex Briggs:\*\* \[00:03:20\] Thanks Hassan. That is great. We are back today and we will be building on a lot of what Hassan just talked about, including testing out runway as one of our exercises today, we are going to be building on that bridge between multi-agent networks and evaluations and building the feedback loop today.

\[00:03:37\] Yesterday we talked about ag agentic \[00:03:40\] workflows for ad creation, and we also got into evaluations and reward hacking and the feedback loop. Today we're actually going to build it. So the first poll is to see how many of you are able to get the innate in a agentic workflow working yesterday.

\[00:03:55\] So we'll launch that poll and then I'll remind you about the login.\[00:04:00\] 

\[00:04:04\] Okay. And while you're doing the log, after you do the poll, please do go over to speaker rec.com/mma to log in to the tools. We'll be coming back to to N eight in the tools are at the top that we'll be using today. \[00:04:20\] So that will be we'll be using in it, in lovable. And you'll have the list and then if you scroll down or click on the day four, that will bring you down to this content.

\[00:04:30\] Let's go ahead and see what the poll results are.

\[00:04:39\] Okay. \[00:04:40\] Let's see if we can, 

\[00:04:40\] \*\*Dan Whiting:\*\* Can see the poll. 

\[00:04:42\] \*\*Rex Briggs:\*\* I can't see the pole right here. Can you push it again or it's.

\[00:04:49\] Dan, if you would like to just tell us the percentages. We'll go with that. 

\[00:04:53\] \*\*Dan Whiting:\*\* No one was voting, that's why I asked. 

\[00:04:55\] \*\*Rex Briggs:\*\* Ah, okay.\[00:05:00\] 

\[00:05:00\] Okay. I think we're pushing up the poll again.

\[00:05:02\] \*\*Dan Whiting:\*\* Should be 

\[00:05:03\] \*\*Rex Briggs:\*\* up there. Go ahead and vote on your screen.

\[00:05:15\] Okay. Did we get some votes this time? 

\[00:05:19\] \*\*Dan Whiting:\*\* No. People say \[00:05:20\] they're voting, but I don't. 

\[00:05:23\] \*\*Rex Briggs:\*\* Okay. Maybe in chat. We'll get the signal from that part and come back to it. Yeah. Key thing here is, the, as we went through in the recap, we covered a lot of content yesterday, but we were just starting to get into the interesting part, which is the evaluation and the feedback loop.

\[00:05:37\] So today we are going to go into \[00:05:40\] evaluations and we're going to build it. We're gonna build it a couple different ways. We're gonna start, we're building it, the manual approach. That means using the large language model of your choice. Maybe that's chat, GBT thinking model, or Gemini or Claude. There are several good choices.

\[00:05:55\] I find that thinking models do better in terms of doing the type of \[00:06:00\] evaluation. So here's what you'll do. You'll jump over if you were able to build the exercise, then you should have a Google sheet that has some of your runs where you have some of the different results. These were the ones that we had for MMA, for example.

\[00:06:17\] What you're going to do is you'll go, you'll launch \[00:06:20\] one of these tools. I launched Gemini in my case, you'll copy the, your prompt. We suggested that you saved the prompt about your brand guideline. It might look something like this where you have your brand guidelines, and then you'll just add in the text grade the output in relationship to the guidelines for, however long you made the \[00:06:40\] ad.

\[00:06:40\] You can edit this prompt a little bit to make it fit your situation. But again, if you're able to do that, what I did was I pasted mine in here. I had grade, grade it, and then I pasted in the ads. Now, when I first ran it, it didn't see these as separate ads because of the way just the way that \[00:07:00\] it copied and pasted.

\[00:07:01\] So I had a follow up prompt, which was to try again and grade each of the ads separately, and I gave it the, how many there were. So I'll give you a minute to try that. And then we will talk a little bit about this evaluation because your ability to do this manually in a large language \[00:07:20\] model means that you can take the same prompt, the same instruction, and put that into an agent workflow as a system instruction.

\[00:07:31\] Okay. I will also begin to give you, and show you like what I got from my output, and then we'll go over \[00:07:40\] to exercise 16 and put it into, end it in any feedback. Okay. The first thing here is that I was able to. Run this a couple different ways with Claude, with chat, GPT Pro with thinking. And the good news is that they all converge to very similar type of grading.

\[00:07:59\] So I think any high \[00:08:00\] quality LLM should be able to give you a pretty good output. The this also can be used as few shot examples. If you look at the grades, you agree with it and you want to give that to the AI instruction as part of here's how I would grade something that gets a D. Here's how something I would grade.

\[00:08:19\] That gives an \[00:08:20\] a, that really helps the evaluation process. That's called Few Shot Learning and you're giving it some examples. And for evaluations, especially qualitative evaluations, if you give it a few examples, it can usually pick up on, on how you're grading and why you're grading it. Okay. Caleb, are you seeing any \[00:08:40\] questions in chat about the exercise?

\[00:08:42\] If not, I will move on to the next part, which is taking what we just did for the evaluation, and we're going to put it into the N eight end workflow.

\[00:08:55\] Okay, so let's dive into exercise 13\. What we're going to do in this one \[00:09:00\] is, the first thing we're gonna do is we're going to duplicate the agent we built yesterday, and then we're gonna create a new one, which is audio agent with evaluation. We'll go back to our list and grab that so that we basically are keeping our earlier version, we're going to delete this connection between our creative director agent and our voiceover extraction \[00:09:20\] agent.

\[00:09:20\] And we're going to basically put the evaluation agent right in here. And then we'll, use the system instruction to give it the basically take the output from the. Voiceover agent as a prompt plus a system instruction, which is degraded. And that's basically what you just did in Gemini or \[00:09:40\] whatever large language model you did when you created this prompt.

\[00:09:42\] Again, building in the agents to demystify it isn't a lot more than what you already do with a large language models is just building it into a systematic workflow. So let's go ahead and take a look at how that works. And by the way, there is a walkthrough if you click on the walkthrough, that will \[00:10:00\] launch this page.

\[00:10:01\] And we are over here at page 10 at the adding evaluation and age agent feedback loop. Let's go ahead and do that. We built this yesterday. First thing I'm going to do is duplicate it and I was talking about calling this \[00:10:20\] audio agent with evaluation, duplicate. Now I'm to get to my duplicated version, I click over here and then back here.

\[00:10:34\] And that brings me up this new one. And it keeps the version control of the old, older one. Now, I mentioned we wanna put this \[00:10:40\] in between these steps here. So I'm gonna break these connections and I'll move these to hover out of the way a little bit here so I can work more cleanly. We'll reconnect these up later, and we're going to add another AI agent.

\[00:10:54\] And here's our AI agent for the model. We'll keep using open ai. \[00:11:00\] And one of the things we had did talk a whole lot about is the model default. The defaults have been the mini model, which is okay, but the more intelligent models for evaluation or even the creative director, you might find that you're getting better output when you use the more advanced model.

\[00:11:18\] So I'm going to \[00:11:20\] use, i'll use 5.4, which is the most advanced model right now. It'll cost more on tokens, but it will give us a better analysis. I also mentioned that we should name our agents. So we'll call this one. If we click up here, we'll call this our evaluation agent. \[00:11:40\] So the agent needs to know where it's going to get the prompt from.

\[00:11:44\] We're gonna define that below, 'cause that's gonna come from our workflow. But since we've duplicated it, it doesn't have any data in our workflow. So let's go ahead and close this for a moment. And we can re-put in our prompt of creating a 15 second ad for a AI \[00:12:00\] training for MMA global that will rehydrate the pipeline with data and information.

\[00:12:06\] So as I go through here.

\[00:12:15\] And lemme see. I \[00:12:20\] have found that sometimes when I first come in here, I have to go back to the personal part and then reopen it. And if you do have a problem, like we're trying, like I might have right now where it doesn't seem to be taking the prompt then one of the ways that you can handle that is just you can delete that and restart with the chat trigger,\[00:12:40\] 

\[00:12:44\] and then we'll add that back to the connection.

\[00:12:51\] And then the AI agent here is connected to the chat trigger. So that's where it's getting the information. And so our system instructions and then the downstream \[00:13:00\] parts should work.

\[00:13:07\] And we, if we have a problem like this, we can hit the it didn't help button or the ai. We are gonna get into this for the vibe coding anyway, and this is one of the notes, so it's probably good to show it.\[00:13:20\] 

\[00:13:21\] Can you tell me why I have

\[00:13:29\] now? If you leave this in plan mode, it'll tell you why you have the problem. If you leave in build mode, it will try to fix it. So we'll find out what the issue might be on this workflow.\[00:13:40\] 

\[00:13:44\] Ah, so that does make a little bit of, so it's pointing to the model that's attached being invalid. And honestly, I'm not sure why it gives me so many options here. It looks like it, the \[00:14:00\] red.is over in this model. And so I will update this model and I'll just go back to the mini one 'cause we know that one works.

\[00:14:08\] I

\[00:14:10\] \*\*Caleb Briggs:\*\* Rex Ben mentioned that your workflow is maybe getting the error because you're missing a field in the evaluation agent. 

\[00:14:18\] \*\*Rex Briggs:\*\* Yeah, I haven't mapped that one \[00:14:20\] yet 'cause I need to get the data from here first, but that's a good tip. But let's go ahead. It is right now, it's failing. Just at the chat part

\[00:14:29\] I may go back to my other agent really quickly here.\[00:14:40\] 

\[00:14:47\] I am gonna go back to the other workflow that we were working on yesterday, us, today, and open that up mainly because I don't want to normally, I do, you do run into problems, so it's probably realistically \[00:15:00\] to show that happening, but it's it's also not the best time to to work through what might be happening and why it might be happening.

\[00:15:10\] I'll go back to one of the workflow that I've built, the base one that we have here, and I'll work through the steps of the evaluation \[00:15:20\] agent. If you also have the same error I'll follow up afterwards to find out exactly why the chat part wasn't taking the instruction. But we're gonna focus on the evaluation agent to talk about how we build that.

\[00:15:34\] So in the evaluation agent, what we want to have is we want the prompt to pull from our output \[00:15:40\] from the previous step so that that we find in the if we go back to the beginning of it, we have the voiceover agent and its output. That's what we want to evaluate. So we would drag this over where it says, define below and prompt user message.

\[00:15:56\] And then we have the system instruction, which \[00:16:00\] is basically the exact same thing that we did in the Gemini exercise a moment ago, which is to grade the output. And we then provide the instructions. If you click this little button here, you can see the full text here. So you're just gonna copy and paste essentially the same thing that you had already saved with your brand guidelines \[00:16:20\] plus that instruction.

\[00:16:22\] And you'll find that over here with the in the initial prompt related to which, and you'll also find it in the walkthrough guide. Coming back to the workflow here, we've got the evaluation prompt grading, and we'll go ahead and close this part \[00:16:40\] here. We're going to do this other step later.

\[00:16:42\] So I'm gonna break this connection for a moment just so that you could see what you would see up through the evaluation agent. And we'll go ahead and hit the text again. And this time it's it's running through the ad director the voice and so forth. Now we're gonna \[00:17:00\] add this code node in a second.

\[00:17:01\] You'll see that in a walkthrough because what we'll see with the evaluation agent output is that the evaluation agent output will give us a grade. So let's go ahead and look at that. We can see that it's it's working right now, and so we can click on this part and see what \[00:17:20\] the input of the output was.

\[00:17:21\] So it gave it a grade of an A minus and this was the the, in the output from the voice agent that became the input for this to create that grade. So go ahead and see if you can get that part to work. I am going to then show you \[00:17:40\] the next part, which is how do we get the grade isolated from the rest of this text?

\[00:17:45\] I will give you a minute to do that part.

\[00:17:59\] Okay. And I'll, \[00:18:00\] and pop into chat when you've got that part working. And if you're able to work on this the day before I know some people were, then this should make this a little bit easier to get to that step.

\[00:18:18\] Okay. If you've got that \[00:18:20\] running, then the next step we want to do is we want to add in a and basically a code block that would make it possible to so I'll move this over here to extract the grade. And so if you type in code. That will give you the ability to use JavaScript or \[00:18:40\] Python. We'll use JavaScript here and in the walkthrough, you'll just, this isn't something technically that, that's super important, but it's if you've done complicated Excel work where you do a Reg X and you're pulling a piece of data out from a string in Excel, it's the exact same idea here.

\[00:18:57\] But in this case, I actually \[00:19:00\] had the the notebook, a ai the innate in ai write the the script for me. And this is a script here. And if you look closely, all it's doing a RegX command to pull out the grade of A through F and and then it's basically creating that as an item. And that works well.\[00:19:20\] 

\[00:19:20\] You can try the clicking the n it in and giving the instruction as we had here, which is suggesting, how we, how we pull out that, that piece of the data there's a prompt included or you can just copy it from the walkthrough. And this \[00:19:40\] is the Ask AI that's included here. Whoops. Let me retype that in.

\[00:19:44\] That was the earlier one. So I'll go back to the walkthrough and I'll copy this here and then I'll paste that over here. And this has basically the part that will extract it. And if I hit execute this step, I can see the \[00:20:00\] output that's coming here. We can see if it will pull out the letter grade and we can see the grade equals A.

\[00:20:06\] So it gave it a name for a new variable called grade, and it gave, it showed the A, which is correct from that output. So go ahead and give that a try. It should be a fairly easy copy paste and add and make sure that you can get that grade of an A coming out of it.\[00:20:20\] 

\[00:20:23\] So if you have that step so far, then what we wanna do next is we want to and I'm going to just

\[00:20:34\] lead out this part and reconnect it up. 'cause this is what you'll be doing. You should have something that's basically has your \[00:20:40\] Google sheet. We'll reconnect it here. When we make that connection, what we now have is we have an evaluation agent, something extracting the grade, and we want to go back to our to our Google sheet and check to make sure everything's still good.

\[00:20:55\] So the prompt is still coming across the output, the voiceover part \[00:21:00\] is still coming across, but now we want to add in the evaluation. We just got this variable grade that came out of our code part, so we can drag that in and map it. To the evaluation column from our spreadsheet. So this evaluation over here, it sees everything that's in the headers and now we're mapping it.

\[00:21:17\] So it will fill in the grade for us \[00:21:20\] automatically

\[00:21:24\] and we can execute this step. And if it works, it will fill in another row to the, to this spreadsheet and we can see that it's giving us a success that worked. Okay, so now what we have is we have the \[00:21:40\] evaluation workflow that we were hoping to build that can be triggered from the chat. Now I'm gonna come back and do a little bit of a debrief and just give you guys a minute to catch up on that part.

\[00:21:50\] And then we're gonna talk about the if loop and adding in this part. This is a really powerful loop. I've been messing around with it a little bit over the next day of \[00:22:00\] whether you do a true condition or false condition and, what's the bare minimum you need to make this work? I'm gonna take you through that part, but I want to give a little bit of background first.

\[00:22:08\] Caleb, any questions or things like this? Basically we have the evaluation agent here and we've added in the code extraction here. So this is roughly what we've been able to accomplish today.\[00:22:20\] 

\[00:22:27\] Okay. So jumping from the to back to the background. We appended the data. Or we did an evaluation agent, we added this note actually after the evaluation agent so we could \[00:22:40\] grab the data automatically. Now, what we want to do is we wanna have a feedback loop. And this is an incredibly powerful system because as you saw from the discussion yesterday, this powers deep research.

\[00:22:53\] It powers a lot of the learning that that these systems have used to develop their, the intelligence from large language \[00:23:00\] models. This feedback loop is something that you can build so you have a self-optimizing loop in your workflows. Essentially the first time we built it, we had a straight agentic workflow.

\[00:23:11\] But what makes us a little bit different is now we're adding an evaluation agent. And when we add in this feedback loop, now it's beginning to interact with the environment. It's creating new \[00:23:20\] information and you can give it more and more feeds so that it does something more sophisticated with its feedback loop.

\[00:23:25\] That's where we're gonna demonstrate tomorrow using your net promoter scores. All the transcripts from our training and all the materials as well as the incoming information that we have from day one about where your skills were at, what you wanted to learn. \[00:23:40\] We're gonna have AI suggest what type of new content to create for you if we were to do additional training in this area.

\[00:23:47\] So we'll see this in a more advanced form tomorrow. Today we're looking at a fairly simple one, which is we have an evaluation feedback and it creates a new content and then gives feedback and it \[00:24:00\] continues to iterate. And it all comes down to this these two letters. If the, if statement is perhaps one of the most powerful systems in the ai workflow, and I think, we will use the if statement to control our feedback loop.

\[00:24:18\] If we get an \[00:24:20\] A, we will produce the audio. If we don't get an A, we will send it back to keep working. We can, we're also gonna put in a iteration loop so it doesn't infinitely work. We can give it, five tries or 500 tries and let it just keep grinding until it can get an A. So let's go ahead and talk, talk \[00:24:40\] about what we're going to have to build in the feedback loop.

\[00:24:42\] We are gonna have to create the if loop and then we're gonna have to take the data if it doesn't succeed. And we're gonna say what do we wanna carry across in terms of the learning to feedback to the art director? So we're gonna give it a little bit of intelligence and feedback, and then we're going to learn a little bit about vibe \[00:25:00\] coding after we build this.

\[00:25:01\] Let's go ahead and shift back over to the build for the IF node. If I were to break this connection here and start over, I would click this button here and I'd type in if that would give me this route. By default you see this first, set of variables. You would have one \[00:25:20\] condition, and so we could say evaluation.

\[00:25:23\] And if that's equal to an A, which is the letter we want, then that would be the true condition. We can also add in something to where it has a number of counts and so we'll use an OR condition. So if it gets an \[00:25:40\] A or it's done at five times. Now the value computers usually count from 0, 0, 1, 2, 3, 4\.

\[00:25:48\] So four is actually five tries, and so we'll put in four here. And here we need to know how does it know how many counts it's gone through \[00:26:00\] inside of the system? It counts the loops. You might have seen the check marks after it's finished. The the counting. It does that through a command. And I'll just scroll down here called the run index.

\[00:26:14\] So the run index is a special variable. I wouldn't expect you to know this. So I'm just, sharing this with you. We can \[00:26:20\] just copy and paste this in. But it's a really helpful thing to know if you wanna control the loop. And this one just makes sure that it forces a number sometimes for some weird reason.

\[00:26:28\] It sees zero as a letter or string instead of as a number. So this just makes it say whatever it sees, treat it as a number, again, a little bit more technical than what you need to know. The main \[00:26:40\] thing that you'd need to know here is that you can add in this run index and it will then. Say that if it's equal to four, then that would also kick off the true condition.

\[00:26:51\] So if it gets an A or if it's already done it four times, then that will produce the the the audio connection or whatever the output is. There's a \[00:27:00\] lot more sophisticated ways that you could do the if statement, so that if it was four times, if it wasn't an a, that it kicked it off into a parking lot we're not gonna worry about that right now.

\[00:27:09\] This is, I think, good enough for our purposes. So we've got this if loop here, and before I had this connected in here, I'm just gonna delete my old one \[00:27:20\] here. And so if it's true, we want to generate the video or the audio and we have to re click here to make sure that if it does do it, it knows where to get the voice agent from.

\[00:27:30\] So this output, we'll make sure that this still works. We might have to redd drag in the voiceover. And then if it's false, we wanna send it \[00:27:40\] back. Now, if we send it back with nothing more, we just drag this all the way back here. And it would get a little bit confused because the art director would say, Hey, I was expecting a chat input and I'm not getting one.

\[00:27:53\] So I use this merge thing to to basically make sure that it connects \[00:28:00\] this feedback loop with the chat. So from the if statement, the, I'll give you a second to set that up and make sure you've got that working. And then we're going to click on instead of just connecting here, we'll click on this button and we'll take, type in, edit and edit is a edit fields or set.

\[00:28:18\] Basically, it lets you create \[00:28:20\] a set of variables or set a set of variables that you wanna pass on in a sort of a package.

\[00:28:30\] Okay, so let me go ahead and walk you through the what we're doing here. If you think about our feedback loop we had, if it \[00:28:40\] succeeds and gets an a, great, it produces the audio. If it doesn't get an A and we wanna send it back, we wanna send it back knowing a little bit of information so that the creative director knows, Hey, here's what you tried before, here's the evaluation, try again and do \[00:29:00\] better.

\[00:29:00\] You can make this a lot more elegant and sophisticated. But again, for our purpose of training, we're going to start off with telling it. We want it to know again, this isn't a walkthrough, so lemme just jump through the part. In the walkthrough where you'll see some of this information, we want it to know the original chat input.

\[00:29:16\] We want it to know the output. We wanna know the letter grade, and we \[00:29:20\] wanna know what evaluation it had, and we want to give it some instructions so it uses that information to try to do to do better. So we might say something like, consider the original request and the previous output in relation to the grade and evaluation.

\[00:29:36\] Consider what you can change to get an a, revise the \[00:29:40\] ad to produce only your new voice over output for the ad based upon your improvement. So I'm gonna, you can copy this whole thing, but we're gonna drag some of the variables over here. So I'll copy it down to where it says. I will copy this all down and then I'll replace the JSON and show you how that works.\[00:30:00\] 

\[00:30:04\] So we can drag the first variable over here. Oops. And we mentioned we are going to use the original chat input. So this was the chat input

\[00:30:19\] and w \[00:30:20\] this basically starts our starts the information. So I'll go ahead and pa now I'll paste in the text here. I'll make this a little bit bigger so we can see it. So consider the original request and the previous output. So we wanna replace these things where it says what the original request was.

\[00:30:34\] So that's where we will take the chat input here, put that in the right \[00:30:40\] place here, and then we'll take the previous output. So what did the system output that if we go back to the voiceover agent, we can see this is the output. And then we can take the grade and we get that from the evaluation a or actually from the code part where we strip out the grade \[00:31:00\] here and then we wanna give it some of the feedback that it had.

\[00:31:06\] So that came from the evaluation agent where it has the output. And this has the great output. A little bit redundant, but that's okay. So then that will tell the evaluation. So if we look at this, you can you \[00:31:20\] can see that it's now going to grab all this information and package it up. So we can close this here and then I'll delete this connection here.

\[00:31:31\] And what you wanna do is you'd want to feed it back to the beginning of the system. Again here it can get a little bit confused. You might not need the \[00:31:40\] merge node, but it could get a little bit confused because it has the, the information from the last run as well as the next run. So we use a merge node.

\[00:31:50\] So if I click plus here and type in merge, that's gonna let us combine two different streams. The stream that we're \[00:32:00\] connecting is the original stream and then the feedback stream. So this is just fine here. So we've got the merge node, we'll move that over here. I'll get rid of this old merge node.

\[00:32:11\] And so we'll put this here. And so we've got one set of input coming from our feedback loop. We'll connect that up to the agent. And then we're going to have our chat \[00:32:20\] input and that's also gonna connect up through the merge. So that will pass through here. And we can see those two connections. Now I did do one small edit here.

\[00:32:29\] This isn't in perfectly required. But it is a instruction that basically lets it know that there might be, a new chat versus the original chat. And I'll, I \[00:32:40\] think that's posted in the the walkthrough. Okay, so now what do we have? We have a process that would run once through here. It with our information.

\[00:32:49\] If it gets an A, it moves on. If it doesn't get an A, it tries again and keeps trying until it's tried five times. Let's go ahead and kick that off, and I'll give you a second to give that a try. So we can see that pass through. \[00:33:00\] The merge node has a check mark and now the art director's working on the script.

\[00:33:06\] And you can see that the first again, if you look down here to see what it's seeing, the system prompt is looked at just the ad and it created an output for the ad. It's getting an evaluation, so let's go ahead and look at what \[00:33:20\] the evaluation's showing.

\[00:33:28\] And we'll go back here to our connection here. So it got through most of it, it had got stuck here at the append row. So we'll see what's read and we can see that the prompt here. I should \[00:33:40\] just redd drag from the beginning 'cause I moved some things around, probably got a little bit confused and so I'll take the original chat input as our prompt and drag it there.

\[00:33:51\] And if everything else is green, we'll try, we'll execute the step and pass it through the rest of the flow.\[00:34:00\] 

\[00:34:01\] And try again.

\[00:34:09\] Go ahead and kick your off as well and give it a try.\[00:34:20\] 

\[00:34:31\] Okay. It succeeded and it got grade up a, so it went to the audio part here. If we look at the audio part here we probably need to \[00:34:40\] reconnect up the output that we want, which would be from the we could take it from the tail end here of the output. Here is the text input and if we execute step, we have our advertisement.\[00:35:00\] 

\[00:35:04\] Okay let me we're gonna try again. I'll give it another one here and I'll have it for possible, and I'm hoping it doesn't get an a this time so that you can see it through the feedback loop and you should try yours a few different times. The cool thing also is you can come over to your \[00:35:20\] spreadsheet and you can check on the status of the progress as it adds new lines, as it hits this append to node.

\[00:35:30\] Why is this important? And we will talk about the feedback loop and a debrief in just a moment here. But this okay, let's see.\[00:35:40\] 

\[00:35:47\] I'll just leave the, I might have been being too fancy trying to force it into a number type, so I might update the the code and the walkthrough on that one.\[00:36:00\] 

\[00:36:09\] Okay. We'll let this run again and then as we go through here,

\[00:36:15\] yeah, we still got an A here, so I can tell the evaluation agent. I'm going to \[00:36:20\] add you are a tough grader.

\[00:36:32\] That might make, give it just enough incentive to to not give it a good grade. And we'll try it one more time with the loop and we'll come back to this \[00:36:40\] after I look at the, you do some of the debrief here so that you can see this running and you guys should run yours a bunch of different times.

\[00:36:47\] Feel free to make little tweaks like what I just did here, but I wanna go through a little bit of a debrief on some of this part here as we see what happens with the grade.\[00:37:00\] 

\[00:37:01\] Okay. Coming back to what we've just built, we were playing around with feedback loops. I did this last night and generated somewhere around 400 different outputs and running. I set my counter pretty high. I did this both for for MMA, but I also did it for \[00:37:20\] Bells. And, bells is one of the folks on the call today.

\[00:37:23\] Hailey did a great job. She was sending me messages about, her success in progress with using, building out the feedback loop. Thank you Hay for doing the homework last night and getting ahead of the game. But what you just saw here is this approach of looking at the \[00:37:40\] verifier or the ability to evaluate and then pushing it back.

\[00:37:43\] And and this feedback loop is what powers deep think's, huge advance that they made recently that they announced and shared on February 11th, 2026\. So you are using the most, powerful element of ai, which is a feedback \[00:38:00\] loop. Yes, there's gonna be some tweaking and you're gonna spend some time just as I was trying to make sure each of the nodes work and in it in is a little picky about different things that it sees and how it works.

\[00:38:10\] We're gonna go over to vibe coding, which is much easier to use and will oftentimes work through some of those issues. But the idea of the feedback loop is what I really \[00:38:20\] want you to take away from today's session. Now, there are other things besides Inad n that you can use. Inad N is great to be able to visually show it, build it fairly quickly, get them working, but you can mix and match with other tools.

\[00:38:33\] For example, you might say I like the output of the text, but I didn't like the way it sounded. One of the things that I \[00:38:40\] did was we went, I went over into the text and stripped that out. I created a version and had the art director have the information for Bells and the evaluation with the information from Bells.

\[00:38:52\] And yeah, I had issue, items like the persona and the key message points that they would wanna focus on. I \[00:39:00\] had it read the website and build that out. And I created, lots and lots of different versions down here related to after I got through the possible ones to bell this I the idea that Summer is here and have it try to create different ads.

\[00:39:13\] And it did a pretty decent job with it. When we looked at when you look at 11 Labs, you can log in there and give it a try. You can copy \[00:39:20\] and paste your text from your output into the text to speech. You can then, adjust the voices. They have so much more voice control. And this is ultimately what progressive use.

\[00:39:30\] We use the API to 11 labs when we built out the system that we built for for Progressive. So the big question is \[00:39:40\] about the feedback loops is how often do you repeat the work? How valuable is the work to get a higher output. And then you can decide, you do it manually in a large language model, just like chat GPT and asking it to evaluate ads or, and to give you feedback or I had it generate, over a hundred ads.

\[00:39:58\] I then gave it to \[00:40:00\] chat GBT and said, pick the five strongest ads. And then I manually looked at those and decide which ones I wanted to move into 11 labs. So you can work manually, but you can also decide to work with agentic feedback loops and generate lots and lots of different ideas. The key issue is whether or not you have a good evaluation.

\[00:40:18\] Hassan opened up today with some \[00:40:20\] of the efforts that MMA is doing to look at evaluations and scoring. You're, the ability to evaluate what is a good ad oftentimes uses data. So we measure the effectiveness of lift on different messages and ads. That became a feedback loop. For Progressive, when they created new ads, they could see how their \[00:40:40\] ads were performing and then take the elements of the most successful ads and remix them and create even more effective ads.

\[00:40:48\] You can also just do this with a brute. You have a brute force if you know what constitutes success, you can just have it keep trying and trying again until it gets an A, which is what we were trying to do with with our loop that we did here. \[00:41:00\] And, we didn't quite successfully go through enough loops, but we'll kick off another one here.

\[00:41:06\] You might see the numbers change it two, three, or four. As it goes through. I had some loops that were running, 20 times before it it made the bar for what I was looking for. So great that's another good approach. \[00:41:20\] We are gonna talk about vibe coating in a moment with N eight N.

\[00:41:24\] And then we're going to go over and talk about some of the video outputs and other services and tools from Google and Runway that I want you to test out before we wrap up today. And then we're going to do a little bit of the vibe coding tomorrow. We're gonna move some of that into \[00:41:40\] tomorrow, along with the analysis.

\[00:41:42\] What I want you to try to do within it, in, now that you've built your workflow, is you can go over here and you can click basically this prob this note over here and you can you can basically say something like you lemme stop the workflow. You \[00:42:00\] can come back to the help button and have it plan and build you different ideas.

\[00:42:05\] There's a plan and a build mode. In a plan mode. You're having it think through different questions. So in your workflow sorry. Coming back over to the exercise in the workflow here, you'll add in a prompt something like \[00:42:20\] what are three concise tips for how I could improve the context or evaluation that would get that would get, make this progressively better.

\[00:42:27\] Give that a try today and we'll come back to it tomorrow and see how we can have N eight n build even more sophisticated systems. One of the things we purposely left out \[00:42:40\] of this feedback loop is the presence of any memory. And I will come back and fix this and make notes in the. For tomorrow's effort here, I think this is where it's getting stuck.

\[00:42:53\] But the takeaway that I want you to try with the vibe coating is that it probably will suggest adding memory, and then you can have it click \[00:43:00\] build and just have it do it, and then you can try to run through it and you can see how you can go from what's really hard and tedious as my example of running into a couple little glitches here and there demonstrated versus having the vibe coding.

\[00:43:13\] If you know what you're looking for and trying to do, letting it do the work of writing the code and figuring out how to fix it. I can probably just \[00:43:20\] say here the, point about executing the fix. So this is the instructions of what it was going to do to try to fix the problem I had before.

\[00:43:29\] Like I said I'll ignore that for now and we will let you give it a try. What I want to come down to after the vibe coding part is that you will also find bugs and errors, and you'll often see \[00:43:40\] those turn red. You saw me diagnose some of those. And then just, it's usually the updating of the pipeline getting confused when you make changes.

\[00:43:49\] In my case, when I gave it the suggestion, it suggested that I should have a lot more nuanced evaluation. Instead of just having a, an evaluation, I should have a score, strengths and weaknesses and \[00:44:00\] improvements and reasoning for why. And when I did that was like, okay, that's a good idea, but now I need to add those to my spreadsheet and I need to remap my ex my Google sheet node, so that would write this new data to that information.

\[00:44:14\] So just be aware of those types of things as you're working with it. Tonight. And yeah, \[00:44:20\] we had live bugs that we could fix. So you got to already see this point I was going to make before. And then you can see the after version of of what it did to change or clean up some of the versions.

\[00:44:29\] I still had an issue here where it was trying to add another merged kinda a, the reverse merge node, and it did it with two different lines. And then I had to go to, no, \[00:44:40\] basically to, to Gemini and say, Hey, I'm having a problem with my innate n Here's what it looks like. One other tip you can do is you can click download.

\[00:44:51\] And this will download your JSON and you can give that to Gemini and and then talk, describe your problem. And then it basically helped me rewrite or fix the code, or told me \[00:45:00\] why it was wrong. I told it to explain it to me as a beginner and or as a novice, and it, then gave me a step by step.

\[00:45:06\] So this is going to be an excellent tutor for you. If you want to go deeper into building your own ag agentic workflows, I highly recommend it. Final comment on this part. When Caleb and I actually built this \[00:45:20\] system and presented it initially, we had controls for brand positioning and we added more for audience persona.

\[00:45:27\] We had things like how much, how long it should be, how many words it should use, call to actions and so forth. This is the progressive panel here where they could add in special instructions and, manage this. \[00:45:40\] You probably now recognize that these are just mul, different system instructions that were passing on, depending on what's turned on or off.

\[00:45:48\] The difference though is that this interface is a lot prettier than N eight N. So your exercise that we're gonna do with lovable helps you begin to create kinda a more interesting interface, and then you use something called \[00:46:00\] webhooks, which is the language that lets the webpage talk to your N innate N application and pass information back and forth.

\[00:46:11\] Okay, so let's go on to the bleeding edge of technology, which is not audio, it's video. And in \[00:46:20\] the video case, the, the key idea is we are probably already here, bank 

\[00:46:28\] \*\*Speaker 13:\*\* account, empty before payday, but still need gas or groceries. Been there. It's frustrating, but what if you could end that stress? I found a game changer.

\[00:46:36\] Dave Extra cash. Literally got money in my account in under \[00:46:40\] five minutes. No joke. It's my mini financial lifesaver. No credit checks, no interest, no late fees. Seriously. Stop stressing, download Dave and get what you need to. 

\[00:46:51\] \*\*Rex Briggs:\*\* So that is an AI generated TikTok message that took less than 30 minutes for Dave to the team at Dave to \[00:47:00\] create.

\[00:47:00\] They presented it at the July, 2025 C-M-O-C-E-O summit. And I hosted this panel and it was pretty impressive to see, the difference between what the human. You need some money now. I 

\[00:47:12\] \*\*Speaker 14:\*\* got you. Dave, extra cash has been my absolute lifesaver because it gives me cash when I'm 

\[00:47:16\] \*\*Rex Briggs:\*\* needing money.

\[00:47:17\] The ability for AI to basically recreate \[00:47:20\] that same type of thing. Do you need 

\[00:47:21\] \*\*Speaker 13:\*\* some money now? I got you. Dave, extra cash has been my 

\[00:47:24\] \*\*Rex Briggs:\*\* and the ability to create new ideas and new messages and concepts pretty quickly. At the time when we did this, about 6% of TikTok ads were AI generated. I'm sure that number has increased.

\[00:47:35\] There's still a little bit of a feeling of some of these earlier versions that people seem \[00:47:40\] to prefer. The human ones, they're not quite sure why they think it's something maybe is ai. Maybe it's the way that the arm position is probably unrealistically shot if he was holding a camera. Those type of artifacts will get better and better over time.

\[00:47:54\] So if you like the workflow that you built before, realize that can \[00:48:00\] happen for text, audio video, and, and images. So what they did here is they used Gemini to create the script they used Gemini to create the character they used Whisk for character consistency. And we're going to show you that with a link here and then flow to create the \[00:48:20\] video.

\[00:48:20\] VO three is the latest version. That's still the latest version. And then they did the final editing in Adobe Premier Pro. And again, they did this, that ad in under 30 minutes, which is pretty impressive. So here's a couple ones that I did just for fun with Whisk. I took a picture of Caleb's twin brother Jared, who you'll meet tomorrow.\[00:48:40\] 

\[00:48:40\] I took a picture of Yosemite as a setting. I took a picture of him and I on the climbing wall. And I asked it to create these and put these together. And so this is the video based on the,

\[00:48:56\] okay, so you have a chance to create something fun like \[00:49:00\] that. We also did one of Caleb where I just took him a picture on the beach. I didn't have any other scenes. I just gave it the prompt of turning into a fish. And so we had this one here.

\[00:49:16\] Have some fun and just play around with it. And then we're gonna come back to runway as the \[00:49:20\] last one runway. You can you can basically sign up for a free version. You just x pass the paid tier and it'll give you a 500 credits to try it out. I took a, not a very good picture of Caleb, his black and white picture that we use for the MIT book and just saw what it could do and it takes his picture and then it randomly picks the type of setting it's going to \[00:49:40\] throw him in.

\[00:49:40\] And it made him a gladiator, which is one of the options if you looked at the prompt. You can do that with a picture of of someone for yourself or someone in your family and just to, just to get a sense for what works. You can also then do some things with their video and other efforts to give it another try.

\[00:49:56\] What we're gonna suggest that you do is jump \[00:50:00\] over to. The exercise 15 and we'll give you a couple minutes. We'll give you four minutes and then we'll do a one minute wrap up of where we landed today, but just, see if you can try some of these tools.\[00:50:20\] \[00:50:40\] 

\[00:50:58\] Okay. Caleb, are there any questions you \[00:51:00\] wanna, we should address while people are given a try with whisk and and flow before we hit the wrap up?

\[00:51:18\] \*\*Caleb Briggs:\*\* Let me see if I find \[00:51:20\] any oh, I think it's maybe there was a question earlier about memory inside of this. So maybe it's worth just saying a few words about like the n and m agents you can add memory, but they don't start with memory by default. Some details about that. 

\[00:51:34\] \*\*Rex Briggs:\*\* Yeah. Yeah.

\[00:51:35\] Part of what I wanted to do in the vibe coding exercise listed as 14 \[00:51:40\] is for you to is for people I'm sorry, it's exercise which exercise number is it? Exercise number 16 rather, is to try to use the plan and prompt mode because I think it will recommend memory. That's an obvious one, but yeah, it's a really good point.

\[00:51:55\] If you look at what we did here, we specifically ran through \[00:52:00\] this workflow Without memory. There's a button on each of these for different types of memory. And one type of memory is simple memory, so it just remembers the chat. These seem to be, somewhat amnesiac. They forget what you've put in last time unless you have memory.

\[00:52:15\] Simple memory is the easiest one. This basically remembers how many \[00:52:20\] conversations it'll keep, and by default it goes to five. The other type of memory is that you can use databases so that it and this is a not really quite memory with the ex, with the Google sheet, but it's in that neighborhood, which is how does it remember what it did before and how does it begin to learn?

\[00:52:37\] So when I did my five coding \[00:52:40\] example and asked it to create, basically create a more advanced version, it it, the main thing it recommended was the addition of of memory. I'll come back to that. This is the workflow that it created with, based upon the vibe coding.

\[00:52:56\] It added, I'll make this a little bit big, bigger it added \[00:53:00\] memory at this stage here, so that it could remember what it had evaluated and how it had given scores. And and then it basically suggested that would be a much better way. It also suggested having a much more fine-grained. Evaluation.

\[00:53:16\] So in the evaluation, it \[00:53:20\] had different details on, let's go to this one here so you can see it different details on what it was appending so that it could learn and get smarter over time. So memory is something that you very likely will get when you do your vibe coding suggestion.

\[00:53:36\] It is a key part of it. There's also different types of memory. There's a \[00:53:40\] vector database where we use a retrieval augmented generation, and that was one of the points we were going to get to later on, which is you, we'll talk about that more tomorrow, which is how do you create these systems with more intelligence?

\[00:53:57\] We know we need to have an evaluation, but how do we get \[00:54:00\] it to understand at a deeper sense? And a big part of that is memory, where it can build on what it's learned from previous runs and get smarter and smarter as it moves forward. So what we're gonna do to, hopefully you had a good experience with Whisk or trying one of those other tools, or you'll play around with that a little bit more after the session.

\[00:54:19\] \[00:54:20\] We covered a lot of ground. I will add a couple notes tomorrow. We will go into vibe coating a little bit more tomorrow with Lovable. We'll do a little bit of a review of what you already had achieved with your vibe coating with N eight N. We'll create a workflow using the data you're giving. So please do fill in \[00:54:40\] the the final form on the NPS score.

\[00:54:44\] And then we will have a discussion about the future of work, about how work changes when you are managing agents and what's a good approach to that. So with that, I'll shift into the overtime session for those that wanna stick around and have some more questions for Caleb \[00:55:00\] or.

\[00:55:07\] Any key questions? Caleb, you wanna start with? 

\[00:55:12\] \*\*Caleb Briggs:\*\* Let's see. There was a question earlier about does the code node use tokens? And maybe have a good point that's worth talking about, which \[00:55:20\] is that, like since the code doesn't use tokens, like that's a great way to like robotic automation is a way to save money on part of the process.

\[00:55:31\] \*\*Rex Briggs:\*\* Yeah, that's right. So the code no, doesn't use any tokens. And the tokens are only hit when you add in that model. So when \[00:55:40\] you click on open AI or Gemini or Grok or some of the other ones that we were trying, that that's when you incur the cost or the generate audio or video or or just, images.

\[00:55:51\] All those other nodes who are setting the data or doing code or merging data, tho, those are essentially just built in as part of the in \[00:56:00\] it, in subscription or in the free version. Just free. 

\[00:56:05\] \*\*Caleb Briggs:\*\* Yeah. And maybe take the highlight Nathaniel's point that if you can do something with robotic automation it's often better to use it with robotic automation rather than using ai.

\[00:56:17\] If you can do it with just a regular program, that's a great \[00:56:20\] way to save time, make it run much quicker. Yeah, use ai if you need to use ai, but sometimes there's other options that can do the same thing just as well. 

\[00:56:29\] \*\*Rex Briggs:\*\* Yeah, and I did figure out what, why that my if Node wasn't working. So I'll I'll share that.

\[00:56:35\] And I was just going too quickly when I did this \[00:56:40\] before I should have clicked is equal. It was treating it as a string. It needed to be treating it as a number, so it should have been is is, is equal to or to four. And that would've made it work. So that was just a, doing it with people watching is, I just missed a simple step.

\[00:56:57\] Yeah, that, that was just silly on my part. But \[00:57:00\] there's all these like little technical things that are a little bit annoying about innate in, but it is a really powerful system. But oftentimes, the the help button over here. Gets you to the right answer. Or if I downloaded the JSON and gave it to an LLM and asked it, why isn't this working?

\[00:57:16\] It was said, Hey, oh, that's a common mistake. You're, \[00:57:20\] you forgot to click and turn it to a number. So it could evaluate it against the number. Okay. Just in case that, that's helpful for people. Other questions? 

\[00:57:30\] \*\*Caleb Briggs:\*\* Someone asked if we need these ads to follow the flavor of a brand, can we provide samples or examples to the agents?

\[00:57:37\] \*\*Rex Briggs:\*\* Absolutely, yes. One of the things I did last \[00:57:40\] night was I took I had my the AI agent read the Bell's website. And because we're doing some work with Bell's, and they have a great team I knew a little bit about the persona they were going after and their customer. So I had it summarized that information.

\[00:57:53\] I put that into a prompt. We'll create a few examples. We'll get permission from different brands that are okay with examples or we'll just \[00:58:00\] create a few based upon what we think would be good from publicly available information. But the key thing is you really have this agent that's the ad director agent that knows what the brand is, knows what the persona is, and knows what a good ad looks like.

\[00:58:14\] We didn't give it a ton of data for that on this example, 'cause we were going pretty quickly. But that's where you would spend \[00:58:20\] maybe a couple days making sure you got that really, really good prompt you, you test it out with your LLM of choice to make sure it seems to be working and giving you good output.

\[00:58:29\] That's why I say that the LLMs are like your prototyping environment. And then once you know it works well in an LLM, then you bring it over and put it into your system instruction. We did that same thing with the \[00:58:40\] evaluation agent. We, you had it grade I thought it was giving too easy of an A.

\[00:58:44\] So we might make that, harder, greater. You can tweak and adjust it until you feel like it's giving a good grade from the LLM. And then you can put it into your system instruction into your Ag agentic workflow in, we'll talk a little bit more about this \[00:59:00\] tomorrow, but the age agentic workflow is one way of working.

\[00:59:03\] Claude has a different way which uses folders, so you can think about adding a folder that had system or your system instructions as part of a markdown file that it can read when it needs to, but your job is going to increasingly be to curate this really high quality information. \[00:59:20\] For the AI to be able to do a good job evaluating or giving you, good feedback and information.

\[00:59:26\] In general it's called context engineering, and that's really where a lot of us are headed. 

\[00:59:32\] \*\*Caleb Briggs:\*\* Yeah, maybe just add to that, but the prompt doesn't need to be short, so you could literally just paste in a few examples of \[00:59:40\] different like the text of ads and the model can work with those sort of longer prompts.

\[00:59:44\] The prompt can include lots of data. The models are generally good at using lots of that context.

\[00:59:52\] \*\*Rex Briggs:\*\* Yeah. What what other questions? We've got a couple more minutes for a few more. 

\[00:59:59\] \*\*Caleb Briggs:\*\* Let's see. So when \[01:00:00\] asked will the transcripts be available after training is complete? So just curious if all of this stuff will be available after. 

\[01:00:07\] \*\*Rex Briggs:\*\* Yeah. Yes. In fact, tomorrow's exercise, we're taking the transcripts from the first four days.

\[01:00:12\] We're feeding it to the AI along with the the data that we gathered from day one from the surveys that you \[01:00:20\] answered and the NPS surveys that you've been answering each day. And we're gonna be asking the AI to make suggestions of of how we would personalize the content or adjust the content.

\[01:00:30\] That will be a big part of the first half of the discussion before we get to Caleb and Jared. So those trans, we'll make those generally available as well. And you can take that and feed that through \[01:00:40\] an LLM if you wanna try to create your own personalized training. Exercise.

\[01:00:47\] \*\*Caleb Briggs:\*\* There's a question here. When creating display ads was best practice to translate the brand design.

\[01:00:56\] \*\*Rex Briggs:\*\* Say that again. 

\[01:00:57\] \*\*Caleb Briggs:\*\* When creating display ads was the best \[01:01:00\] practice to translate the brand design? 

\[01:01:03\] \*\*Rex Briggs:\*\* Yeah. The brand design the brand design's really important. What's been interesting is as I've looked at different companies that have the sort of brand design documents, a lot of times they're very visual and some you can give that to something like Gemini, which is very good with with computer \[01:01:20\] vision and ask it to translate it into written words.

\[01:01:24\] The LLMs have a harder time dealing with images as as artifacts. And so the translation of images to words that it can then use as much better. Humans are better with the visual part. You give people a PowerPoint deck with a lot of pictures and say, this is what a \[01:01:40\] brand's about. And people are like, oh, I get it.

\[01:01:42\] But you give that to, it's it has to translate it into the. Word representation of the images to then have it have meaning, so that, that's an extra step for people to keep in mind when you're trying to teach LLMs to know our brand and what it is and what it is \[01:02:00\] and how to represent it.

\[01:02:07\] I think we've got time for one more question. 

\[01:02:11\] \*\*Caleb Briggs:\*\* I think there's a question about alternative frameworks to NN to the other. Agent orchestration frameworks. I'm the same. 

\[01:02:18\] \*\*Rex Briggs:\*\* Yeah, we had a \[01:02:20\] slide on that and I did run some deep research because someone asked me about that before. And you run some deep research and certainly ask, but yeah, there's a lot.

\[01:02:28\] Zapier is definitely easier. It's not as visual to show each of the steps in the nodes, and that's why I picked in a n more as a teaching instruction than as a best in class. But, I'll I'll share my screen for one \[01:02:40\] more second again, because I, I am obsessive enough to wanna make sure that my, my AI was running, so this is that ai FI fixed the node with the numbers and you can see how it's, telling me how many counts it's gone through.

\[01:02:53\] So there's certain features that are just easier when you create the feedback loop to be able to visually see it kicking back through \[01:03:00\] and, going, now it's going through the fifth version. And so there's a couple things that are better with innate end, but it is really picky and it's a little more technical.

\[01:03:07\] If you want something easier, Zapier relay the I'll pull up and share the deep research that I did and add it to a link for tomorrow for anyone that, that wants to do more of this. But they don't really love in it, in, I \[01:03:20\] think when you try the vibe coating though, you're gonna find that it can do a lot of stuff.

\[01:03:24\] Without you having to get quite as technical as you build more sophisticated workflows. And when you jump over to lovable or you jump over to cloud code, man, those things are way more powerful. They can do so many more things, but you lose out on being able to see \[01:03:40\] how it's doing it unless you know how to write code.

\[01:03:42\] So the, these are the trade-offs I think we're grappling with right now. My hope is that cloud code, which I think is one of the best systems right now available, will come up with a visualization for those that wanna understand what, what's happening underneath the hood. And then we don't have to pick \[01:04:00\] one or the other.

\[01:04:00\] We can use the most powerful vibe coding system, but also see the relationships and how things are working.

\[01:04:12\] Okay. Thank you so much for joining us today. We look forward to seeing you again tomorrow for the last \[01:04:20\] session. We will do vibe coding in the beginning and then build to the data analysis. And we'll be spending about 30 minutes talking about the future work, and we'll be opening it up to questions about what you think your job's going to look like, where you have questions about it.

\[01:04:35\] We don't fully know all the answers, but I think we've been working at the edge of the \[01:04:40\] future so that we hopefully have some good instincts and some good information to share about what it looks like when we really are using agents in a deeper deeper way in the way that we work. We'll see you back here tomorrow.

\[01:04:52\] Take everyone.

