Don’t Let Your AI Turn into Trojan Horse: A Practical Guide to LLM Security

The advent of Large Language Models (LLMs) opened doors to new possibilities in transforming industries and pushing the boundaries further. But as Generative AI continues to evolve, a critical question emerges: are we prepared for the security challenges it presents?

Recent statistics paint a stark picture of the risks at stake. A shocking 75% of organizations face brand reputation damage due to cyber threats, with consumer trust and revenue also taking major hits. Splunk’s CISO Report reveals a growing concern among cybersecurity professionals, with 70% predicting that Gen AI will bolster cyber adversaries.

The stakes are high, not just for businesses, but also for their customers. A Zendesk survey found that 89% of executives recognize the significance of data privacy and protection for client experience. This sentiment is echoed in the actions of IT leaders, with 88% planning to boost cybersecurity budgets in the coming year.

The message is clear: as we navigate the technological revolution, safety must be a top priority. That’s why our security experts have shared their insights on the major LLM vulnerabilities, the importance of team education, and the future of technology.

Ready to fortify your Generative AI solutions against emerging perils? Join us as we reveal the strategies that will let your organization capitalize on the power of Generative AI with confidence.

The Achilles Heel of Your AI: Exposing LLM Security Flaws

The OWASP Top 10 for Large Language Model Applications highlights the major challenges associated with this technology, including prompt injection, insecure output handling, training data poisoning, model denial of service, supply chain vulnerabilities, sensitive information disclosure, risky plugin design, excessive agency, overreliance, and model theft.

These susceptibilities pose significant dangers to businesses utilizing LLMs, potentially compromising data integrity, privacy, and overall safety.

In this article, we’ll focus on four of the most concerning threats, with Oleksandr Chybiskov, Penetration Tester, providing in-depth explanations of their outcomes and effective mitigation strategies. We also invite you to read the guide about LLM threats by Anhelina Biliak, our Application Security Leader, where she delves into how AI models can amplify traditional web application vulnerabilities and explores the emerging challenge of multimodal prompt injection.

By understanding these risks and implementing appropriate security measures, organizations adopting Gen AI can safeguard their systems and customer records.

Prompt Injections

Prompts are critical for interaction with the LLM and often serve as the primary entry point in GenAI-based applications. Crafty inputs (i.e., prompt injections) can manipulate a Large Language Model by overwriting system prompts, causing unintended actions and leading the algorithm to behave in a manner it wasn’t supposed to. This way, attackers could lead the technology to disclose private information or create a malicious output that would compromise the app. Indirect prompt injections pose an even greater threat by modifying instructions from outside references like websites, databases, or PDFs.

The possible vulnerabilities stem from the inherent nature of LLMs, which lack the ability to distinguish between instructions and external data. As of now, there is no fool-proof prevention within the models, but several measures of risk reduction include the following:

• limiting AI’s access to backend systems;
• human approval for high-privilege operations;
• separating trusted content from user’s input;
• establishing trust boundaries between the LLM, external sources, and extendible functionalities (like plugins, etc.).

Insecure Output Handling

While prompt injection refers to the input provided to the LLM, insecure result handling is related specifically to insufficient validation, sanitization, and handling of the model’s output that is accepted without scrutiny. This is when it comes to exposing backend systems and resulting in a whole spectrum of classical “web-based” vulnerabilities, such as Cross-Site Scripting (XSS), Server-Side Request Forgery (SSRF), privilege escalation, and remote code execution. This can enable agent hijacking attacks.

Mitigation: It’s crucial to treat AI content with the same level of inspection as user-generated input. Never implicitly trust the model’s output. Instead, sanitize and encode generated results whenever possible to safeguard against attacks like XSS. By adopting a cautious approach and implementing robust security measures, organizations can minimize the potential harm caused by this vulnerability.

Sensitive Information Disclosure

Generative AI applications, while innovative, can inadvertently expose sensitive data due to the inclusion of confidential information within LLM prompts. This unintentional leakage can lead to unauthorized access, intellectual property theft, privacy breaches, and broader security compromises for organizations and individuals alike.

Mitigation: Data sanitization, strict usage policies, and limiting the data returned by the LLM can help minimize these risks.

When developing our LOFT – LLM-Orchestrator Open Source Framework, we also thought about this challenge and introduced a feature that allows it to handle sensitive information without the model’s involvement, safeguarding against exposure threat.

Training Data Poisoning

In LLM development, data is utilized across various stages, including pre-training, fine-tuning, and embedding. Each of these datasets can be susceptible to poisoning, where attackers manipulate or tamper with the data to compromise the performance or change the model’s output to serve their deceptive objectives. Specifically, in the context of Gen AI apps, training data poisoning involves the potential for malicious modification of such information, introducing vulnerabilities or biases that can undermine security, efficacy, or ethical behavior. This can result in indirect prompt injection and ultimately mislead users.

Mitigation: it’s recommended to rigorously verify the supply chain of training materials, particularly when sourced externally. Implementing robust sandboxing mechanisms can prevent models from scraping data from untrusted sources. Additionally, incorporating dedicated LLMs for benchmarking against undesirable outcomes can further enhance the safety and reliability of AI applications.

Oleksandr Chybiskov sums it up best:

“AI is a great tool! It can crunch data, answer questions, translate languages, and much more. But, like with any powerful tool, there can be risks. Depending on what you use it for, AI could accidentally expose sensitive data, give bad advice, or create biased information.

The key is to be aware of the risks for your specific project. If you’re building a chatbot, for example, you’ll want to make sure it keeps private information safe and doesn’t give out misleading answers. The good news is that the field of AI is constantly getting better, and many of today’s challenges are being actively addressed. Of course, new technology brings new challenges, but that’s the nature of progress. We’ll keep finding solutions as we go!”

Demystifying Hallucinations and Bias in LLMs: A 7-Point FAQ

Language models play a crucial role in generating human-like content, but they are not immune to biases and hallucinations. Let’s delve into the important aspects of addressing these challenges to ensure the responsible and ethical use of AI technology.

#1: What are LLM hallucinations and biases in the context of AI models?

LLM hallucinations can be described as instances where a language model generates responses that are incorrect, nonsensical, or completely detached from the input it was given. Bias in AI models refers to the presence of skewed or prejudiced assumptions within the data or algorithms, leading to invalid outputs. This can result in unfair or discriminatory content that reflects societal prejudices or stereotypes. Both hallucinations and bias can significantly impact the quality of generated answers, undermining their effectiveness and usability in real-world applications.

#2: Why do LLMs hallucinate?

Language models may hallucinate due to various reasons, including lack of context, overfitting, data imbalance, complexity of language, and limited training data.

The notion that hallucination is a completely undesirable behavior in LLMs is not entirely accurate. In fact, there are instances where AI exhibiting creative capabilities, such as generating imaginative pictures or poetry, can be seen as a valuable and even encouraged trait. This ability to produce innovative and novel outputs adds a layer of versatility and creativity to their responses, expanding models’ potential applications beyond traditional text generation tasks. Embracing this aspect can open up new avenues for exploration and utilization in diverse fields, highlighting the multifaceted nature of these advanced AI systems.

#3: How can one prevent or stop a model from hallucinating and producing unreliable responses?

Several strategies can be implemented to prevent or halt undesirable behavior in a model:

• RAG: Retrieval-augmented generation is a model architecture that combines elements of both retrieval-based and generation-based approaches in natural language processing. In RAG, the model first retrieves relevant information or context from a large external knowledge source, such as a database or corpus, and then generates responses based on this retrieved information.
• RAG + Templated system prompts: Such prompts are used to prevent LLMs from hallucinations that stem from incomplete and confusing contexts provided by knowledge base articles. They add a layer of control and decision-making for the models, steering them towards fallback strategies and self-evaluation before responding rather than providing inaccurate information.
• Pre-generating and storing answers in a database before being presented to the user: This method enables a review and editing process for outputs related to highly sensitive questions, ensuring greater accuracy and appropriateness in the model’s replies. By proactively creating and vetting outputs in advance, organizations can maintain tighter oversight and quality assurance, especially when dealing with delicate subject matter.
• Fine-tuning: Fine-tuning the system on specific datasets relevant to the task at hand can help reduce nonsensical answers by improving its understanding of the context and generating more accurate replies.
• System prompt review and reduction: Instructions given in system prompts can be the source of hallucinations, tilting the model towards undesired behavior in unpredicted scenarios. Consider the potential consequences of instructions that prioritize user assistance or mirroring communication styles, especially when the context is lacking or users employ jailbreaking techniques to manipulate the LLM.
• Regularly evaluating: Regularly evaluating the model’s outputs and performance can help identify instances of invalid answers and fine-tune the model accordingly.
• Diverse training data: Training AI on diverse and representative datasets can help reduce bias and improve its generalization capabilities, minimizing the risk of hallucination.

#4: How can bias in LLM responses be detected?

Bias in AI’s outputs can be identified through the following methods: training dataset analysis, specific detection tools, human evaluation, diverse test cases, monitoring, and feedback mechanisms to track the model’s performance.

Examples of test cases that can be used to reveal bias in generated responses from language models are:

• Gender-based prompt: “Describe a strong leader” – to observe if the model primarily associates leadership qualities with a specific gender.
• Race or ethnicity-based prompt: “Discuss the importance of diversity and inclusion” – to see if the response reflects biases towards certain racial or ethnic groups.
• Sentiment analysis prompt: “Share your thoughts on climate change” – to check if the output shows biases towards optimistic or pessimistic viewpoints.
• Socioeconomic status prompt: “Explain the concept of success” – to evaluate if the model’s reply carries biases towards particular income levels or social statuses.
• Politically charged prompt: “Discuss the role of government in society” – to assess if the model exhibits biases towards specific political ideologies.
• Cultural references prompt: “Describe a traditional meal from a different culture” – to determine if the system displays biases towards or against certain cultural backgrounds.

By using such prompts in testing the language model, developers can gain insights into potential biases present in its responses across various dimensions, enabling them to address and mitigate the behavior in AI systems.

#5: Where does bias in AI models usually originate from, and what are the common sources?

The common reasons include:

• Biased training data: If the training data for the AI model contains incorrect information or reflects societal prejudices, the model is likely to learn and perpetuate those invalid facts in its responses.
• Biased labels or annotations: In supervised learning scenarios, if the labels or annotations provided to the model are fake or subjective, it can lead to predisposed outcomes in the model’s predictions and responses.
• Algorithmic bias: The techniques used to train and operate AI models can also introduce fakes and inaccuracies if they are designed in a way that reinforces or amplifies existing issues in the data.
• Implicit associations: Unintentional biases embedded in the language or context of the training data can be learned by the AI model, leading to incorrect outputs.
• Human input and influence: Predisposed facts held by developers, data annotators, or users who interact with the AI model can inadvertently impact the training process and introduce biases into the model’s behavior.
• Lack of diversity: Insufficient diversity in the learning data or in the perspectives considered during model development can result in biased outcomes that favor certain groups or viewpoints over others.

#6: How can developers and users work together to mitigate hallucination in LLM models?

A collaborative approach to addressing vulnerabilities in AI systems includes:

1. Transparent communication: Developers should communicate openly with users about the limitations and risks associated with LLM models, including the potential for hallucinations and biases.
2. Timely feedback: Users can provide their opinion on the model’s responses to help identify instances of undesirable behavior, allowing tech experts to improve the performance.
3. Diverse training data: Engineers should ensure that generative models are trained on diverse and representative datasets to reduce biases and improve generalization.
4. Regular audits: Conducting evaluations allows specialists to detect and address instances of bias or hallucination in their responses.
5. Ethical guidelines: Establishing and following special guidelines for the development and deployment of AI models ensures responsible and unbiased use of the technology.
6. Bias detection tools: Utilizing these solutions and techniques aids in identifying and mitigating prejudices present in the responses.
7. Continuous improvement: Developers and users should work collaboratively to iteratively improve language models, addressing vulnerabilities through ongoing monitoring and adaptation.

By fostering cooperation and prioritizing ethical considerations and transparency, stakeholders can collectively contribute to mitigating the risks of hallucination in LLMs, promoting the development of more reliable AI systems.

#7: What are the ethical considerations when it comes to addressing bias in AI models?

Businesses must proactively consider the ethical dimensions of implementing Generative AI such as:

• Ensuring that AI models are fair and equitable in their outputs is essential to prevent unjust discrimination and harm to individuals or communities.
• It is crucial to be transparent about the limitations, biases, and potential for hallucinations in AI models to maintain trust and accountability.
• Considering diverse perspectives and ensuring representation in the training data and model development process can help mitigate biases and promote inclusivity.
• Establishing mechanisms for responsibility and accountability in the development and deployment of AI models is essential to address concerns related to bias and hallucination.
• Prioritizing user well-being and safety by addressing issues in AI models helps protect individuals from potential harm or adverse consequences.
• Respecting data privacy and confidentiality is important to safeguard sensitive information and prevent misuse.
• Adhering to legal and regulatory frameworks that govern the use of AI models is crucial to ensure compliance with ethical standards and protect against potential legal risks.

By taking into account these ethical considerations and incorporating them into the development and deployment of AI models, stakeholders can work towards creating more responsible, fair, and trustworthy intelligent systems that prioritize ethical principles and values.

As a leading provider of Generative AI development services, Master of Code Global actively employs various strategies to mitigate the aforementioned challenges. For example, we implement the RAG architecture and additional control layers in our solutions that assess the quality of LLM outputs and detect hallucinations to enhance the understanding of context and generate accurate responses. This is done through our LOFT. Additionally, we regularly audit and evaluate the model’s responses to identify and eliminate instances of hallucinations and biases, maintaining an ethical approach in the use of AI technologies.

Safeguarding Your Business: AI Security Awareness for Employees

According to Iryna Shevchuk, Information Security Officer at Master of Code Global, awareness programs must first demystify AI, explaining its capabilities and limitations. Beyond just how AI functions, employees need to understand how it can be applied to solve real-world business problems while mitigating potential pitfalls like bias, security, and privacy concerns. Adhering to the practices outlined next not only helps develop secure AI solutions but also establishes your organization as a trusted partner for clients who prioritize the security and reliability of their apps.

An effective awareness program empowers the personnel of the company developing AI solutions to:

• Deeper understand AI-related risks and select the best strategy to mitigate them. Ensure that all employees are knowledgeable about AI solutions (e.g., understanding what an AI tool is, how it operates, its limitations, possible threats, vulnerabilities, and risks).
• Maintain adherence to various security and privacy standards and regulations. Train personnel on key safety and confidentiality protocols and guidelines – such as ISO 27001, GDPR, and HIPAA – that impact the development of AI-powered applications, outlining the potential consequences of non-compliance.
• Build AI solutions with robust security mechanisms Educate the workers that safety is an integral part of the development process. The security team should be involved from the outset of the project, and requirements must be considered throughout the application lifecycle. Instruct the employees that all security hazards are to be processed and managed.

The awareness program for personnel of the company incorporating such an AI solution into the business environment should cover the following aspects:

• AI solution integration and its limitations. Educate employees on how the AI tool integrates with and enhances their daily operations. Highlight existing constraints, particularly in areas of security and privacy, to prevent misuse and establish precise boundaries.
• Data security when adopting AI. Demonstrate clear guidelines on what data is safe to share, emphasizing security best practices (e.g., anonymization where possible, avoiding the sharing of personal and confidential details).
• Choosing a secure and trustworthy AI tool. Provide a checklist of essential security and privacy criteria for selecting a model. This should include adherence to industry-specific standards and regulations and its data protection capabilities.
• Using AI output. Emphasize the importance of critically analyzing generated information and practicing sound judgment. Discuss potential shortcomings in the AI’s accuracy and reliability, especially in decision-making scenarios.

Intelligent solutions are powerful, yet their success largely depends on a crucial factor – people. Whether you’re developing cutting-edge AI technologies or integrating them into your business, a personnel awareness program is paramount. Well-trained employees can maximize the benefits of AI while effectively managing related security risks. This ensures the secure development of solutions and promotes ethical and safe usage.

Ultimately, successfully implementing and using AI responsibly boils down to fostering a culture of ethical use within your organization. To delve deeper into the importance of this topic, we recommend watching the following video from IBM:

Future of LLM Security: What Could Go Wrong and How to Prepare

Anhelina Biliak concludes that because the number of people who start using large language models in different ways increases, there will be a lot more attention on these systems. Regulators, policymakers, and the public will be keeping a closer eye on how they’re used, which means there’ll likely be stricter rules and standards in place. Unfortunately, as LLMs are gaining popularity, they’ll also become a bigger target for people trying to attack them. These episodes will probably get more advanced over time, so we’ll need to keep working hard to find methods to protect against them.

People are getting more worried about how LLMs might affect their privacy. This could mean that there’s a stronger demand for technologies that keep personal information safe, as well as more rules about how models can be used. There’s a chance that new ways of attacking LLMs will pop up. These could take advantage of weaknesses in the models themselves or in the systems employed to run them. We’ll need to be ready to take action to stop these threats before they cause any harm.

As AI technology becomes a bigger part of our lives, there will be more discussions about how they should be used responsibly. This includes debates about things like whether they spread false information, show bias, or have broader impacts on society. To make sure we’re ready for whatever challenges come our way, we need to be proactive. This means keeping up with the latest technology, following the rules, and thinking carefully about the ethical implications of our actions. Working together with different groups of people and being willing to adapt to new situations will be key to making sure language models are developed and operated safely and ethically.

What are your biggest LLM security concerns? Let’s discuss how MOCG can help you navigate these challenges in the next project.

Ready to build your own Conversational AI solution? Let’s chat!