Hello, this is one of the sessions that I was so excited about having the ability to introduce, and that is three people that I consider real thought leaders in the industry talking about a topic that everyone is on the top of their head today, and that's how quickly will generative AI change the world, hype or, but that's not for me to answer, that's for M. R. Rangaswami, who's the founder of Sandhill, who's going to be moderating this session today, Pranay Ahlawat, who's a partner at Boston Consulting Group, who has done deep research on the state of AI and the adoption rate of AI, and Godard Abel, the founder and CEO of G2, probably has as much information, insight on what people are searching for in AI and SaaS. With that, M. R., please take the show. Thank you so much. Thanks Ray, and welcome everyone. This is going to be a very dynamic, hard-hitting, fast-moving, fast-paced session. I've got two amazing people here. Again, the backdrop for this session was our recent CEO enterprise retreat in Half Moon Bay, where we brought together 75 CEOs of companies in B2B SaaS, there are 25 million and above in ARR, along with great leaders like Pranay and other people from the ecosystem, and the whole conference, the whole retreat was all in, all about AI. So I went in as the founder of this event with thinking this might be 10x the size of cloud, how cloud computing was 10, 20 years ago. When I came out with all the hype, I felt it was like 100x, but is that real or not? We're going to find out, I think, by talking to these two experts as to, is this really hype and is it real? How big is this market? How things will change? Will it really change the world? So I want to start first with Pranay. Like I said, I came out thinking this is going to be 100x what cloud was, right? And so give us the, is this real? Give us the macro perspective. You guys at BCG presented a fantastic report at our event that showed the good, the bad, the ugly. Let's start with the macro perspective. What do you guys see? So look, I mean, clearly a lot of excitement and the promise is there to fundamentally open up new markets, change the way work gets done, transform industries in sort of a big way. And that's evidenced by just the amount of capital that's actually, despite all the talk of like the recession in software, there is still a ton of capital that's actually flowing into generative AI. You talk about the $10 billion investment from Microsoft, the $4 billion recent investment from AWS into Anthropic, or the $1.7 billion that actually went into generative AI just in the first quarter of 23. We have more than 500 startups. And also just, there is the commercial capital, but then there's also just a huge growth of open source, whether it be in foundation models with a growth of like Falcon and Lama, whether it's new techniques like PEFT or vector databases like Melvis. I think there is just such excitement about this technology in the community. And I think that the third thing that to me was very interesting is just the growth of multi-modalities of LNN. So far, it's only been text. And I think now with GPT-4 and the recently announced Gemini from Google, these models can actually extend to other modalities like voice and images and video. And of course, the alleged meeting of Sam Altman with Masayoshi-san and Johnny Ive. I mean, who knows, maybe we're at the cusp of a completely new transformative way users interact with technology. So lots of promise, lots of excitement, actually backed by big donors here. So that's the hype, the excitement, the enthusiasm. Billions of dollars have gone in and so it adds up to a trillion by the time it's all over. So let's get down into one level down into what Fortune 500 companies are doing. I know you guys did a great comprehensive survey. So what are you finding in the large IT buyers in that community? Yeah, I think three sort of big takeaways for those listening. First, we are in really early days here. I mean, we shared some numbers, MR, but I think when we last ran the survey, it was like less than 10% of the enterprises have any sort of at-scale deployment of generative AI. Lots of pilots, lots of companies, I think, greater than 90% experimenting, but very few that have actually taken it to scale. The second sort of key learning here is that it does vary by industry and geo. So obviously tech, a little bit further ahead from more regulated industries like healthcare and utilities. North America, a little bit further ahead than Asia back. And in our conversations and in our research, there are tons of things that are actually holding companies back. Obviously talent comes up as sort of a universal issue, availability of data. And look, we are still in the early innings of the technology itself. There are well-documented problems with things like hallucinations, cybersecurity and legal risk comes up as a big thing that's actually holding companies back. And then obviously there is the risk of change management. You know, the big barrier here outside of just the technology is the fact that enterprises need to actually change, fundamentally change in big ways to basically fully unlock the power of generative AI. And that takes time. That takes a lot of change management on the ground. Yeah. So you mentioned something, certain industries are ahead, like technology, media, telecom, some of those industries are ahead of some of the more regulated ones. You also said North America is ahead of some of the other geographies, but what kinds of use cases are companies experimenting with? Are these customer-facing ones, internal ones, development areas? What do you see there? It's a great question. It's a great question, MR. What we are seeing is, it's a lot of horizontal use cases that are still internally facing. I think those are the ones that are actually getting prioritized the most because of lower risk and lower exposure. Some of the customer-facing use cases, I mean, I think a very good example here is like using content generation for marketing. I think human-in-the-loop use cases for customer-facing use cases, we're seeing a lot of them actually being implemented. However, I think some of the really sort of what I would call transformative use cases, which really opens up new markets, and I think those are still very early in sort of the strategy and the ideation phases. The bulk of how companies are actually building AI, it's sort of the horizontal and the lower risk use cases with some dabbling in the more transformative use cases. So two other things, issues I want to double-click on. One is ROI, and the other one associated with that is cost of computing, right? Token costs and all these things that new models that are upon us. So thoughts on cost and ROI? Yeah, so obviously cost is a huge consideration, and I'll sort of land a very interesting insight. In our research, when we actually talk to customers on how they're actually choosing generative AI platforms and their KPCs and actually buying generative AI, funnily enough, nobody thinks about costs. However, cost becomes a huge parameter after the fact. So there was also in the same survey that we did, what we learned was a lot of customers are actually starting off with their generative AI use cases without actually building proper ROI in business cases. And interestingly, I think there are three things that I think every company needs to think about. First, and you mentioned it, Ammar, the underlying architecture actually has a huge impact on your business viability. Generative AI isn't like a one-size-fits-all. You could build your custom models, which is what Bloomberg did with Bloomberg AI or Bloomberg GPT. You could use models as a service, which is use something like an open AI or like Anthropic. Or you could actually use an open-source model and deploy it on your own machines. And the build, the run versus the one-time setup costs and the trade-offs are actually very different. And you have to think through those architectural complexities, not just initially, but at scale. I think the second thing I'll say is you need to actually think about secondary costs. I think everybody models their AI costs. You mentioned ROI. What most customers don't think about are the secondary costs. It's the organizational change management. It's the increased legal costs and so on and so forth. And those are what I would almost, customers almost like discovering these costs as they start to scale their use cases. And then I think the last point I'll say is, look, a lot of people make the mistake of not understanding that the cost for pilots is not the same as actually like running AI at scale. So I think companies are sort of advised to actually think through those economic considerations and be very, very sort of deliberate about how much AI is going to cost and even when they would build versus buy, because for a lot of these use cases, maybe the underlying costs make the use case completely not viable. Got it. Got it. Hey, Godard, let me turn over to you. You got this dual perspective, right? As CEO of a SaaS company serving customers, you got to look at internal use, how do you reach your customers and so forth. On the other hand, the customers you serve are all looking for software to buy, right? And you were at a retreat and you heard from all the big guys, right? Microsoft to Google to Nvidia to OpenAI and Amazon and all these players, right? So you're putting it all together. So first let's walk through what companies are seeing just to follow up on Pranay. What are your customers looking for? What are they seeing? What are they asking? And then we'll come into what you're going to do internally. Yeah. I mean, G2, we do serve software buyers. This could be a knowledge worker. Ultimately, I hope all billion, the world's knowledge workers, we all need software. And a lot of them come to G2 to research record options. And what's been interesting the past year, obviously AI software overall. And actually one cool thing, if you Google AI software, G2 is the number one result. So as a result, a lot of people looking for come to G2 and then we track at what they're researching. And I think overall, it's up about 3x in the past year, no AI software, but I guess we all know AI software has been around for a long time. But I think what's more interesting within AI, generative AI is up 84x in the past year, both on G2 and on Google. And so that's the thing to explore again. I think we all saw it in the media with the chat, TPT 3.5 launch last November, all of a sudden it just exploded. And now we'd also all software buyer to kind of figure out how do they take advantage of it, how do we bring it to our enterprise? And one interesting thing I did also mention that we're preparing in the last three months, if I could pull back a bit, so we're down about 20% in the last few months, although still up 84x year over year. But two months ago, the undernext, maybe we're getting a little bit at that point where there was so much hype, there is so much hype. Now businesses are starting to say, hey, I'm not seeing the ROI yet. The other cool thing we're seeing on G2, every software vendor now trying to immigrate it. Companies like Salesforce, they just launched Einstein 1 at Dreamforce, where obviously, as we all know, the number one traditional CRM vendor, and now they're trying to bring it to their sales cloud, their service cloud. And so categories also that really certainly within AI software on G2 are things like AI sales assistance, AI support assistance, because those seem to be some of the first use cases. I think we're almost all enterprises are seeing value because the opportunity to automate a lot of sales specials, a lot of sales FAQs, as well as a lot of support questions. It does seem like AI fits right there for getting customers faster answers and really missing productivity. So I think those are some of the really exciting areas. But there's also a lot of AI infrastructure software now. We track things like vector databases. Now, a year ago, I have to admit, I didn't know what it was. It wasn't even a category yet on G2. And all of a sudden, we're seeing massive interest, because once you start studying AI, you realize, wow, it's really all about the data. And then the vector database, that's the new tool to get your data ready for AI. And so overall, it's been explosive and exciting. But like I said, I think we're still at the point where we also ask people inspired, have you seen the ROI yet? And the reality is that most people are more experimenting still. They're not getting a full ROI yet, which is why people are down a little bit from peak height. Great. And so how are you leveraging this internally to both service your customers, but for internal use? What are the types of projects? I know when we had our CEO cohort meeting, you demonstrated how a customer of yours on G2 can go use an assistant or co-pilot, whatever you're calling it, to actually evaluate or find solutions. So talk us through some of the ways your customers are interacting with you. Yeah. And we have launched our, we call it our Keats and Avanti. And that's our AI software buying assistant. And the idea, you know, you can almost get somebody as smart as Pranay in a box. You can't always hire BCD or Accenture. But the idea is, you know, a lot of your basic questions about technology can now be answered by Avanti. And Avanti, we built in partnership with OpenAI. And we had Lisa Rosenthal, another head of sales. She also has a partnership. So working with them. So we take the basic chat computing that's been trained by Altenapo on the internet. And then on top of that, you know, we created that plugin where we inject G2 unique data on software buying trend and software buying use cases. And all the data we have are millions of reviews of the new software buyers. And then we train Avanti to make recommendations. And what I'm really excited about traditional software buying, you have to know what category you're in. For example, you have to know you're shopping for CRM software or for customer success software or for AI, sales, et cetera. But the reality, most of those people, they're just trying to solve a problem. They're trying to say, hey, how do I increase my sales productivity? And then Avanti can ask more questions like what industry and a bigger company. And then Avanti can make more and less recommendations. And so we're really excited about that potential. We have to help every software buyer use that AI bot to more quickly discover the best apps for them. And how are you leveraging AI internally within the company? Because we talked earlier, there's so much marketing content to generate that you used a lot of people or sales emails that go out or support things. I mean, are you using this internally as well to help your staff? For sure. And I think we've shown they want to beat you to get at least 10% better, more productive at their job. And one great example, I think most of us are aware of our AI code generators. Yeah, GitHub to man and dollar force code. And also our co-founder, CTO Mike Wheeler was building Avanti. He also said, yeah, they can be a 10x better programmer and they're using co-pilot. And so now I think we're encouraging and almost forcing all our engineers, for example, to use co-pilot because we have both code speed, code quality gets much better. And so I think that's just one exciting use case. The other one I was just talking with Sarah Rothfield, our key partner for today is translation. We have this G2 today being with language only. And by next year, we're going to launch G2 in all the European languages. And we're using Chats and Qt because now translation, I think is one of the best use cases. And I think the transformer developed by Google, you know, the TPT at a European newspaper was translatable. And so by next year, we're going to have E2 not just in English, but now we have a German, French, Portuguese, Spanish. And forgetting, our chief product officer said before the templation cost would have been about a million dollars. Now we can get it all done for less than $15,000 and the date. So I think there's just a 20x savings on translation to localize a site like E2. So I think that's one I'm probably most excited about. Oh, that's fantastic. So Pranay, coming back to you, I mean, you're seeing, you know, you were at the retreat and you saw what B2B SaaS companies are doing, what customers are doing. When do you see these two guys meet up in terms of just off the shelf kind of products? You know, as Godard was actually like going through his own product strategy, what's interesting, you know, one interesting trend that at least we're observing is that a lot of these use cases are actually, and Godard, you mentioned like the Salesforce example, but there's like ServiceNow and there is like Adobe and there is like Microsoft. What's interesting is I do think a lot of the traditional systems of record will actually continue to continue to actually service some of these horizontal and sometimes even vertical use cases. And I think that's the reality today. I think the second thing that's super interesting to me is the modality of how end users are actually interacting with enterprise tech. I think that's changing and it's changing. You know, I think we saw sort of one movement from like the web browser to the mobile. And, you know, I do think, you know, this will open up sort of a completely new modality to interface with technology, which is, you know, through language. And I think that's happening today as well. So MR, you asked a very difficult question, which is when that is happening. I think a lot of this is actually happening today. Now, the really interesting thing is when some of the more transformative use cases will start to actually happen and can organizations actually change quickly enough to adapt this technology? Because I do think there is a lot of fatigue. I agree with Godard. I do think we're at the top of the hype cycle. There is a lot of fatigue. There is also a lot of customer confusion. And because this is an area that's actually moving very quickly. So I do think we'll see a lot of innovation, whether enterprises will be able to adapt quickly or not. That's a DVD. Right, right. And Godard, coming back to you, I mean, part of your job of your product, your solution is to help customers through that process, right? So it's not just evaluating software, but reviewing these helpful hints. So are you looking at kind of easing the friction, if you will, that Pranay referred to? And we are. One of the things we've done in every software category now in G2, we do ask our viewers, real business users, about their experience using software. That's really the key source of data and insight for G2. That's really why we started the company, because people used to rely on Gartner and Forrester, and they do some quality work, but it's like one human publishing report every two years. And we thought, well, if we can get real-time users talking about software in real time, that'll be much more real-time. And this is very relevant now with AI, because one of the things we did earlier this year, we've added AI-related findings to every software product. And this is what Pranay was mentioning, right? It's not the Salesforce service now, every software vendor is incorporating AI. And so we're also asking users about how good are those AI features with these traditional software products, and asking them, hey, which of those is already generating ROI for you, real value? Because we do think the best way to trust something is from peers. And yeah, we haven't seen that tipping point yet, but I do hope. And long-term, I'm very bullish on AI. We'll go all in at G2. But I think most of those that aren't yet seeing the ROI, and I think it'll help enterprise adoption, once they can see peers validating, yes, I'm using, Einstein 1 wasn't just a beautiful keynote of people, which it was. Yeah, but once they hear thousands of Salesforce customers saying, hey, you've deployed it, we're getting value, our sales are more productive. And so we're building that in every deep team review now to get feedback on, yeah, how good are the AI features, and how much value are we getting from that? Well, that's good to know. And I think we keep track of that, Godir, because that's going to help or hurt the adoption. Pranik, coming back to you, obviously, you have all these off-the-shelf kind of apps, but then a lot of companies have proprietary data and proprietary apps. So what areas do you see customers, the large customers saying, hey, I want to keep my own system, I want to develop my own applications? What do you see in that area, realm of rolling your own apps? Yeah, it's actually a very pertinent question, Amar, because, so I think as a general rule of thumb, we are not seeing a lot of customers actually build their own models. I think that is the realm of hyperscalers and big tech. I don't think it's necessary or even advisable to actually build your own model. So I think let's put that to the side. After that, I think you basically have like four options. You can buy a solution off the shelf. Godir, for example, they decided to buy GitHub Copilot. You could actually use models in the service, right? So you consume, you don't modify, you just use it out of the box and you use like a vector database and use a technique like retrieval augmented generation. That's like 80% of how most companies are building AI. And then you could actually tune either a commercial model or an open source model. For the horizontal use cases, I think a lot of companies are preferring to buy rather than build. I think that's a more pragmatic solution. It gets them value faster. It honestly removes a lot of the risk and execution and that's what companies are doing. So Copilot is like a perfect example of that. I think for some of the more value-added or differentiating use cases where either you're building a new product or you're actually building a new customer experience, that's when companies are actually choosing to build their own solutions. Like I said, most of what we are seeing, I think 85% plus are just using open commercial models without actually tuning or modifying them. But what we are starting to observe is that, and by the way, this is evidenced by OpenAI recently announcing that you can tune their models. What we are starting to see is that the barrier to actually tune models is actually continuing to come down. And this is both through commercial vendors as well as open source vendors. So even though a lot of customers are not tuning models right now, we do expect all of this innovation to open up a new wave of innovation for more regulated industries or companies that actually have data residency requirements, for example. So it's a mix right now, but the whole area is moving so quickly. I think the more interesting question is how will this evolve and what value will this unlock in the future? Right. Go to, I mean, yeah, go ahead. Interesting news. Yesterday, Sequoia Capital said, and they aren't going to invest anymore in startups building foundation models. I think that is because even for them, and they're one of the biggest, most well-known VCs, I think we all know, but even for them, it's just too big a bet. You have to incentive. And so I think what Pranay was saying, I think, and that's where kind of you have to be too, you know, we partnered also with OpenAI, Tech2BT on our AI Monty, and then, you know, we're injecting unique data, we're adding unique training, but we use a foundation model we have around that OpenAI has already created. And it does seem like the foundation model looks like a, you know, $1,100 billion arms race that very few people can play. It's also, I think the lines have already been drawn as well, Gautam, because I think if you take a look at at least the commercial model vendors, each one of the hyperscalers basically has anchored, I think AWS on Anthropic, Microsoft with OpenAI, and Google recently with Gemini and Palm. So I do think... Google also say that they have like kind of a model, like marketplace on... Exactly. I think with Vertex, they do support, and by the way, that's the same approach that Amazon is taking as well. I think they're saying, hey, listen, they understand different customers would actually need different models for different use cases. So both of them want to be open about what... So you're absolutely right. And frankly, I do think, I completely agree with you, Robert. I think the winning strategy in AI is actually going to be picking the right model for the right use case at the right economics. And I think that'll be a big sort of art of how you basically continue to build your successful AI strategies. So Gautam, one last question. I know my images are being annotated now. There's a version coming out for that. Voice has got to be another big part of interacting with AI. What are you seeing in those areas? Yeah. Now, especially I take voice, you know, potentially being a whole new interface to most legacy software. And because I think CRM software would be a great example, right? Sales reps in general re-typing. You know, they're feeding data into... And most CRM software are writing about rigid structure forms, you know, kind of like most certified software, like forms, you have to fill in required fields and form, and it goes through some required workflow and approvals. And frankly, it's the way most, frankly, most software workers don't like working that way. They're sort of forced into rigid structure. So I do think voice is immensely exciting. You know, now there's also all these conversational intelligence tools like Gong, they've been around for a while, but they also apply AI, and they can just take the conversation and then pull all the structured data out. And so I think the future of the knowledge worker, hopefully most of them won't be structured, riveted form workflow systems anymore. We'll just do our job, right? We can just be talking to customers. You can record it, AI can parse it and fill in all the structured data for us. So I think it could be very freeing. You know, we'll probably have less of these traditional type and form interfaces, you know, in most software and more voice, or even, you know, on a recording like this, it can record our facial expressions, and it can store our motions. So pull even more information out than we could ever, you know, tap traditional systems world. Gentlemen, I know we can go on and on and on, but we're out of time. So some closing thoughts. Where are we in the hype cycle as you see it today? And where will we be a year or two from now? Rane? I think the problem is with all this innovation, the hype cycle keeps shifting. I wish the innovation stopped so, you know, we could make that. But I do think we have to be measured. I do think the physics of organizational change takes time. So I do think we are, you know, we're still very hyped overall. Right. Go ahead. And I would agree. We're in that early hype cycle, like the most new tech. But also one of my favorite quotes of Kim Bill Gates, you know, where he said, you always overestimate what you can do in a year. Yeah, but estimate what you can do in 10. And I think in 10 years, that's a super goal. I think in 10 years, AI will transform all software, all systems, all knowledge workers. Yeah, but it's probably going to be right. And we're probably year one into generative AI. And that's at least the blue curve. And some currently some ups and downs, but I'm sure in 10 years, it will transform almost everything we do. Fantastic. On that note, we'll end the program. Godard and Pranay, thank you so much. Thank you, Rane. Back to you. MR, Pranay, Godard, thank you so much. I don't want to put you on the spot, but a year from now, when we do SaaS Metrics Palooza 24, I want to revisit this conversation and see if we hit that trough of disillusionment or if we're still on the up hype cycle. Thank you so much. And I will have your contact information so everyone can actually reach out and follow you on LinkedIn, follow your report. But thank you so much, everyone. Thanks. Thanks, Ray.