GenAI: Overview of Business and Technical Safety
For almost the past 2 years, since GPT-3.5 started to become popular in the mainstream, the way we utilize technology has changed a lot. Why? Because once again (similar to how we adopted the internet as a norm), technology is starting to solve our problems. Let's be honest, who isn't using Generative Artificial Intelligence in their daily routine right now?
If this kind of technology has started resolving problems in our non-business lives, it's not difficult to guess that businesses will be the first to try to utilize it as much as possible. There are many questions about ethics, consequences, data privacy, or justifications for using this technology to replace humans for general tasks.
This will be a great story for the next article. Today, I want to focus more on safety in the commercial usage of this relatively old yet new technology that is now overwhelming us from every side of our lives.
Concentrate on solutions
As always, whenever new technology starts knocking on our doors, technology companies (both small and big) take part in a race to cross the finish line first, offering their perfect solutions for our problems. That's why today, when we try to get new information about AI products, we are overwhelmed by new approaches to RAG (Retrieval-Augmented Generation), Agents, Multi-Agents (Grr… writing about agents, I've started putting my complaints down on paper about how many people are trying to abuse what they really are... okay, I'm back), and new products that supposedly save our lives in one way or another but are clones of related products.
As is typical, it is always easy to offer something new while speaking only about the pros of one's own solutions and avoiding discussion of important issues.
It's difficult to convert concerns into money.
Business perspective
Understanding of technology
As an AI Solution Architect, I strive to deeply comprehend our clients' challenges and business landscapes by gaining comprehensive insights into their unique needs and industry context. Because honesty is one of my top 5 values in life, I spend a lot of time explaining to them that GenAI technology will not magically solve every problem that previously required extensive development work. Using GenAI can be cheaper, but still requires integrations.
To give you a better perspective about what I want to say, let's talk about automation. Let's be clear — none of the current popular LLMs are created to automate anything. They are trained to answer questions.
In the market, there are many good low-code/no-code platforms for automating processes, like Make, Zapier, or n8n, and GenAI is just another block that can be inserted into the chain of a full process. This block can replace some parts of other processes or merge them, making the full process easier to understand and maintain.
What I want to say is that, regardless of which LLM you would like to use to solve complex problems in your products or processes in your company, you need additional work to be done.
Working with company data
At the beginning of GenAI popularity, many products were related to chatbots. This is not a surprise. It was too easy to transform any company knowledge into vectors and inject the corresponding parts (regarding the question) into the LLM prompt to get a satisfactory answer. And all of this before what we now call RAG. Currently, this is even easier!
Chatbots and knowledge bases are now dependent on each other, and it's much easier to ask a company bot for information than to search through thousands of company Confluence pages. However, what is good can sometimes work to our disadvantage.
This part could be written in another article (and I know it will be), but for now, let's focus on simplifying this process. Imagine we give our company chatbot full access to our company knowledge from Confluence. In our software platform (in this case, Confluence), we always have an access layer for certain types of documents. Access can be based on email or group permissions. If we don't think about this during the integration process, we might bypass those security checks because the process that injects data for LLM use fetches everything from the API. Once again, we're talking about a very simplified scenario. Experienced integrators will be aware of this issue and try to address it in the business process.
To show an example, as a developer and architect, I've seen in analytics that employees often try to ask the bot about company payrolls. This is standard human curiosity, and we need to remember that.
J.A.R.V.I.S.
Imagine having your own J.A.R.V.I.S. (the AI from Iron Man) – an intelligent entity capable of handling numerous tasks and potentially saving lives. While it remains a captivating dream, today's Large Language Models (LLMs) like GPT-4, Claude 3.5 Sonnet, Gemini, or LLAMA are making significant strides in multitasking capabilities.
These LLMs excel at various tasks, yet they aren't quite as versatile as we might hope. With a basic grasp of prompt engineering, we can reliably obtain good answers to simple questions, generate text summaries, or produce translations. However, challenges arise when we attempt to chain multiple complex tasks together.
For instance, summarizing an article, translating it, and then crafting an email with our thoughts on the content is possible but not always seamless. The success of such multistep processes hinges on several factors:
The quality and specificity of our prompts
The model's context window limitations
The inherent capabilities of the chosen LLM
While we're not quite at J.A.R.V.I.S. levels of AI assistance, the rapid advancements in LLM technology continue to narrow the gap between science fiction and reality. So whether you're an integrator or a company seeking a solution, it's good to keep in mind that LLMs work best when the task is well-described, definite, and has a single responsibility. If not, we can always run two separate LLM tasks and merge the results with a third one. Or we can use agents… wait, that's essentially the same thing, but as I said before, we'll speak about that later.
I need to write an article about the teleportation paradox… don't ask why I've been thinking about it for the last 35 minutes while writing this article…
Technology perspective
Chat is a developer consent
A crucial distinction exists between deploying an LLM service with chat functionality and one without open-ended user input. Chat-enabled services introduce an element of unpredictability, as we cannot anticipate or control the wide range of prompts users might submit. This variability in input poses unique challenges and considerations for system design, content moderation, and user experience management.
In contrast, LLM services without chat features operate within more defined parameters, allowing for greater control over input and output. This controlled environment can be advantageous for specific applications, ensuring consistency and reducing potential risks associated with unexpected or inappropriate user inputs.
However, both solutions will be open to uncontrolled usage if we do not take care to implement additional steps to improve the safety and security of our solution.
Output is important no less than input
When implementing Large Language Models as a service, establishing clear guardriles and evaluation criteria is paramount. This approach ensures optimal usage, maintains quality, and maximizes the value derived from LLM integration.
While input design is crucial, it's equally important—if not more so—to scrutinize the output. Prompt Engineering plays a vital role in creating a persona for the LLM, defining how it processes our input, determining what can and cannot be consumed, and shaping the structure of the output. For simple solutions, this might suffice. However, as your product gains popularity, robust safety measures become increasingly significant, as potential attacks become inevitable.
Prioritizing the verification of LLM responses before they reach the end user can prevent numerous issues. Various patterns exist for implementing this safeguard.
First, utilizing an additional LLM invocation with a Chain of Thought prompt. Second, implementing rule-based filtering systems. Third, employing sentiment analysis to flag potentially inappropriate content. Fourth, using content classifiers to categorize and validate outputs. Fifth, implementing human-in-the-loop review for critical applications.
By adopting a comprehensive approach that addresses both input and output, you can create a more secure, reliable, and effective LLM-based service. This strategy enhances the user experience and protects your product's integrity as it scales.
Missing Evaluation
A good thought to keep in mind is that we cannot trust the input, and even more, we cannot trust the output. This is sad but true, especially in chat features. That's why we should create a mechanism to verify the output. I've already touched on this topic, but there are still some tasks that can be done before you decide to release your solution to the world.
Evaluations can be carefully prepared scenarios about the input and output. In these scenarios, we can check everything, from verifying if the correct words exist in the output to ensuring the schema of the output looks as we want it to. They shouldn't focus only on expected scenarios, but also simulate numerous versions of attacks. In the end, we should guarantee that every scenario passes.
It's good to remember that with the power of GenAI, we can generate as many scenarios as we need. How they will be integrated into our development process depends on us.
Missing Guardriles
Clear rules for your AI system help reduce risks and keep it on track. These rules should cover things like how it talks, what content is okay, and ethical issues.
When making these rules, it's good to work with people from different teams, like legal and experts in the field. This helps make sure the rules are complete and cover all important points. It's also significant to check and update these rules often, as AI and society change quickly.
Arch Enemy
Let me finish this with one more thesis.
Our greatest challenge lies in the unpredictability of human users. The vast diversity of individuals interacting with our product, each with their own unique intentions and backgrounds, creates a complex landscape of potential scenarios.
This human element introduces an unparalleled level of uncertainty, making it crucial for us to design robust, adaptable systems that can effectively navigate this unpredictability while maintaining security, ethical standards, and user satisfaction.
That's why at the start of this article I was writing about understanding of technology. GenAI is a powerful solution, but with this power comes many safety and security issues that everyone (business and technology) should keep in mind before, during, and after software development.