May 17, 2023
Understanding Generative AI Implementations for Enterprises
How to generate the Highest Accuracy Answers at the Lowest Cost of Ownership
As enterprises continue to embrace Generative AI, there is a lot of excitement about its promises, but also a lot of confusion and misgivings on how to implement enterprise level solutions without it becoming prohibitively expensive. You are also probably wondering whether to start a project in-house, use a SaaS platform specialized for your specific use cases, or to outsource it.
I have had people even say “What’s the big deal? My IT guys can do it in a couple of days.” Maybe. It all depends on the complexity of your use case and your reference data. If you are an executive in an enterprise high tech, industrial, banking or similar enterprise with reference documentation comprised of tens of thousands or millions of documents in PDF, HTML, Word, PPT, and relational databases siloed in different data sources – I can assure you the right solution is complex, and probably going to be very expensive unless you adopt the right strategy.
Use Case and Goals
Let’s start with your use case.
Are you planning to use it for customer facing applications, say a Chatbot or Voicebot for your Customer Support?
Or is this a tool that you want to give to your Sales Staff? Or, perhaps to your Field Support staff?
Whether it is for Self-Service or for your employees, you need to consider the following:
Accuracy of the Generated Answers – your generative AI implementation needs to have very high accuracy, otherwise you may be misleading your user, whether they are customers or your employees. And, if there is no answer available, your generative AI solution should be able to indicate that to the user and generate a report that tells you that there is missing documentation for the specific query the user (customer or employee) asked.
Traceability of the Generated Answers – for a user query, generating the answer is not enough. You need to be able to present the user with the Answer Sources, i.e., point to the specific sections of different documents that were used to generate the answer. Otherwise, there will not be much confidence with your implementation.
No Hallucination – make sure that generative AI does not make up an answer for you. For enterprise implementations, a made-up answer could have serious repercussions.
Generative AI and Implementation Options
Let’s start with a brief overview of Generative AI. You can use GPT 3.5, GPT 4, Anthropic Claude, Google PaLM, or even ChatGPT. If your reference documentation is all publicly available, ChatGPT may be an option. Otherwise, for security reasons, I would recommend GPT 3.5, GPT 4, Anthropic Claude or Claude Instant, or Google PaLM. Let me briefly discuss a solution based on GPT 3.5 or GPT 4.
The way it works is that you can input a prompt which includes the user query and a document, along with other attributes such as instructions to GPT. If the document has the answer to the query, GPT will generate a well-formed answer in English and can even give you details on what content was used in generating the answer.
With GPT 3.5 the prompt can be a maximum of 4,096 tokens. In the case of GPT 4, the prompt can be about 32,000 tokens. When using Anthropic, the prompt can be 9,000 tokens. If you want to use GPT 3.5, you obviously have limited space to include the necessary content in the document for GPT 3.5 to generate the right answer. With GPT 4, there is more room but it also has its challenges.
Now let’s look at your reference documentation for your enterprise use case. Your enterprise documentation is your core IP – it possibly includes thousands to millions of unstructured documents in PDF, HTML, Word, PPT, Google docs, Google sheets, and more. You also likely have structured data in a relational database. Your data probably resides at different web locations and in applications like SharePoint, Google Cloud, Salesforce, ServiceNow, SAP. Your data is not static, but gets updated continuously. Lastly, your documents are not just text – they include images (drawings, flowcharts, etc.), tables (with many rows and columns) and lists, sometimes even a table within a list.
So, given a query, how do you create this input document with just the right content given the size constraint of 4096 tokens for GPT 3.5?
If you want to use GPT 4, the prompt can be up to 32,000 tokens, which is more room – but consider if you have a 500-page PDF document which has the answer. Firstly, you can’t input a raw PDF, and even if you extract the text, the entire content may not fit in 1 prompt. So, if you cut up the document and make multiple requests to GPT 4 each time with a different part of the document, it is quite possible that the section of the document which has the answer to the user’s query gets cut up such that none of the requests to GPT 4 has the entire answer content. As a result, the answer generated by GPT 4 will be incomplete.
Now look at the cost. GPT 3.5 costs $0.002 for 1000 tokens, whereas GPT 4 costs $0.03 per 1000 tokens – 15x of GPT 3.5. If you rely on a search-based solution and use GPT 4, the cost can soon become prohibitive.
The main thing to remember is that the quality of your answer depends on the quality of the document in the prompt that you input to GPT – only if it has all the necessary content to answer the user query will you get high accuracy, not otherwise. Also, if you use search to get a list of documents that you input to GPT in pieces, you run the risk of “partial answers” and loss of traceability. It is important to accompany the generated answer with the exact sections of content from the documents with the answer – include images, tables, lists, videos etc. in that answer section. Otherwise, you will not have the confidence of your users, particularly if they are customers. No customer is going to spend the time to read through a document if you add that as the source with the generated answer. You need to provide them with the specific Answer Sections that they can reference quickly.
What makes the implementations expensive
Your choice of GPT -- GPT 3.5 or GPT 4, makes a big difference in cost. Depending on your use case, the users, and the traffic volume, this is a serious consideration. Make sure the cost doesn’t become prohibitive – and that you are getting ROI.
The other point to think about is – do you need to do custom training of the models for your enterprise implementation? This can also get very expensive.
Security Issues
I will write a separate article on this, but wanted to mention in this article that security is also a big consideration. Make sure your data is not used for training of the LLMs, nor should the data be used for any purpose by generative AI. It is also important to make sure that the platform you are using has enterprise level security.
What you need
There are many more complexities and challenges that would be hard for me to include all in one article. Overall, you need a platform that can provide you with very high accuracy, 100% traceability, 0% hallucination, and all at a low cost of ownership. The key is the prompt document that is input to Generative AI – if it doesn’t include the correct complete content to answer the user’s query then you run the risk of partial answers, feel good answers and hallucination.