GPT Changed Everything. And Nothing Really Changed.
I joke often that my startup, Coseer.ai, was Open AI — just a decade too early and $10 billion too poor!
Since 2012 I have been working in the field of getting a computer to understand the human language in all its glory. With AI and NLP if you want jargon. In its unusually long life of seven years as a revenue funded startup, Coseer took many forms. The most successful was a tool that people could use to ask questions while writing long complicated documents and get one sentence or one phrase answers from their entire information base. Kind of an Enterprise Search, if you insist on jargon. In each of these forms we had to solve some fundamental problems about Product, Engineering and Science. Still, in early 2020 Coseer developed Covid-like symptoms that turned out to be fatal and the world moved on.
ChatGPT was launched more than two years later on November 30, 2022. It’s from a category of software called Large Language Models. As the name suggests, these are computer models meant to mimic human language and they are very large — think trillions of data points. The Generally Pre-trained Transformers are the LLMs from Open AI, known today for their sheer brilliance. Transformers are a category of AI models that transform real world inputs like the words humans use to an etherial plane where it’s easier for computers to make sense of them. GPT is not trained for specific applications — it’s general purpose — and Open AI has already trained the model; you don’t have to do much to use it.
Since its launch, ChatGPT has captured the world’s imagination like none other. It is GPT version 3.5. Open AI had attempted, with stellar but relatively bleak success in hindsight, something similar with GPT version 3 in 2020 when the computer wrote op-eds in The Guardian. With ChatGPT (and also with its younger sibling GPT version 4) it’s simply different. The model got 100 million users within two months — fastest ever for any software in human history. It’s just relatable to everyone. It does not come with cognitive load to use it. It works. It’s cheap, mostly free.
And to a business person or an enterprise executive, it’s largely useless in its base form. (This article was not written by GPT or any of its ilk).
Once ChatGPT took off, an obvious thought sparked in my mind — how easy would it now be to build enterprise software like Coseer. It’s while peeling the layers of this onion that I came to a simple realization — while GPT has changed everything, nothing has really changed.
Take another instance. My current employer helps our clients conduct millions of conversations with their customers every day. Haptik’s AI products are responding automatically to users, reaching out to them over messaging channels and working for our clients wherever people are speaking to them — web, WhatsApp, SMS, social media, etc. Needless to say, at first blush it appears that the whole thing is now redundant. I would be lying if I said that panic did not set in for a moment. Then the same realization dawned again. The reality of successfully using a Large Language Model for businesses is far more nuanced.
Why? It basically comes down to Product, Engineering and Science.
Product Innovation
As its name suggests, ChatGPT is a general purpose tool to chat with you about any topic — science, programming, law, medicine, current events, life, universe and everything. It can compose texts like letters, blogs, essays. It can reason with you, tell jokes and compose songs. For a business, though, it’s too uncontrollable. Any application of an LLM for a business needs very sophisticated product innovation to control it without thwarting its brilliance. There is a lot of innovation necessary to bring a version of the AI out to the end user and get it to advocate for the business — imagine your naive chatbot telling your customers about how your competitors’ product may actually be better. Innovation is also necessary to set the right modes of interaction with the users. These are just a few among hundreds of product problems we face when we try to use LLMs for businesses.
Hallucination
One particular problem with the current generation of LLMs is a phenomenon called hallucination. Data scientists train these models in a way very similar to games people play with five year old kids: Take some text, hide a few words, and ask it to predict the hidden words. It’s gross over-simplification, of course, but the fundamental nature of these models is to generate text. They just don’t yet know when to stop. In my experiments with these LLMs they have hallucinated about products in a catalog, legal precedents, academic papers published in the Nature, python libraries, names of Mahatma Gandhi’s kids and many such things. It’s a huge problem for a business. It takes a lot of product and engineering innovation to make sure that it’s not selling your customers shampoos that don’t exist. How we control this is perhaps a whole other article if not a book (DM me if you are interested).
Real World Action
It also takes a lot to convert a conversation over an LLM to a real world action like shipping a product or cancelling an order. The response formats are unpredictable, and you have to process them to convert to structures like JSON reliably. Ideally, you can innovate on the product itself so that the user selects the right options without any ambiguity. It’s not that simple: I have seen teams start with GPT-first ambition but then give up on its uses one by one to make things work in the real world, till they are back to traditional rules engines. (They seldom give up the GPT label, though!)
Scalability, Cost and Latency
Next you need very deep thinking about the core engineering problems of scalability, cost and latency like any other system. How do you make this work for thousands and millions of your customers together? How do you control costs? At the surface it appears very cheap — hundreds of words processed for a penny. However, to make workable products takes multiple calls to the base models, something called chain prompting. It also takes a large amount of context and instructions. All this quickly adds up. Then there is latency. If you are making multiple calls, each call will take it’s own sweet time. How can we accelerate this? What is your customer doing in the mean time? Some people trade to a lower accuracy models to help with these things. Stanford researchers recently published how you can have your own LLM for barely $600. Sure, but then who trains and maintains that?
Technological Differentiation
I was speaking with an investor recently who focuses solely on companies at the elite startup-incubator YCombinator. He quipped that three to six months ago, any idea where the powerpoint mentioned GPT or LLM was getting funded. Already investors are looking beyond it. They are focusing on how thick is the technological layer on top of LLMs and how deep of a differentiation does it create. There is a lot of rationality to it. This technology is available to everyone and anyone now. Currently there is a rush of investment and startups rearing to capitalize on this. However, true winners in the end are developing a moat on top of it. For example they are making student models for their own application areas, writing their own LLMs, or using differentiated embedding vectors (don’t worry about what that means).
In other words, to win in today’s post-GPT world takes the same focus that it did on Product, Engineering and Science. If we were to think about Coseer as an enterprise search tool again today, we would still have to solve many problems that were relevant the first time around. For example, how to make sure not a single byte of data leaves the enterprise firewall. Or how to make sure that the results are credible by attributing and linking it to the source of the information. In fact while on the NLP dimension GPT or another LLM is going to be very useful, it will create a whole new set of problems that we must solve for before it can be useful to enterprises.
Similarly at Haptik we have come around full circle from GPT being a competitor to it being another wind in our sails. This was once we realized that it still takes a village working on product, engineering and science to deliver value to businesses.
