Top Three Scenarios for PII Leakage in GenAI

Recent advancements in AI, particularly in the development of large language models (LLMs), are transforming human-machine interactions. However, these models are also susceptible to a slew of new threats and vulnerabilities, such as Prompt Injection, Jailbreaking, and Hallucinations. Among these, the leakage of Personally Identifiable Information (PII) has emerged as one of the most significant concerns. According to a recent Gartner survey on the need for data privacy solutions for GenAI, 42% of respondents ranked PII leakage as the top risk, significantly higher than other concerns.

Data leakage of enterprise PII is most common in the following three scenarios:

1.  LLM interaction with external users

This scenario typically occurs when an LLM is integrated into customer-facing applications, such as chatbots, virtual assistants, or automated support systems. In these interactions, the LLM may inadvertently expose confidential information, such as customer data, proprietary business details, or internal processes. A malicious user can exploit the LLM to extract sensitive information by crafting inputs that trick the LLM into revealing restricted data. That sensitive data can reside in multiple locations and may be embedded within the LLM itself, especially if the model has been fine-tuned on datasets containing PII. Additionally, private data can exist in Retrieval-Augmented Generation (RAG) systems, where external knowledge bases are queried to improve model responses. Other databases used to assist the LLM in completing its tasks can also hold sensitive information.

This risk is heightened when the LLM is fine-tuned or has access to sensitive databases or internal knowledge repositories to provide accurate, context-based responses. In cases like this, the model might disclose PII, intellectual property, or other confidential data, either due to prompt manipulation such as Prompt Injection or unintentionally.

2. Enterprise users relying on external LLMs for task execution

In this scenario, enterprise users depend on external LLMs to perform various tasks, such as text summarization, code generation, content creation, data analysis, or other activities requiring advanced natural language processing (NLP).

When enterprise users input sensitive information into an external LLM - such as proprietary business data, customer details, or internal communications - this information is transmitted to third-party servers, where it may be processed, stored, or even used for further training of the model. There is a risk that this sensitive data could be exposed, misused, or inadvertently shared with unauthorized parties if the external LLM provider lacks stringent data protection policies or privacy agreements. Furthermore, even if the external LLM provider has strong security policy, there is still a possibility that the data could be intercepted during transmission or retained in logs or caches without the enterprise knowing about it. This risk is particularly concerning when enterprise users engage with publicly accessible LLM APIs or platforms, which do not provide clear assurances about data handling practices. The following diagram illustrates this risk.

3. Internal LLM usage for employees with varying data access permissions

In this scenario, an enterprise uses an internal LLM to support various employees or teams, each with different levels of data access permissions. The LLM, which is trained on datasets containing mixed data from various departments, may be integrated into internal applications for tasks like generating reports, summarizing documents, automating processes, or assisting with decision-making.

While this offers significant benefits in terms of efficiency and productivity, it also presents a risk of data leakage between users or departments with different access levels. For example, if an employee from a department with lower clearance inadvertently inputs a query that triggers the LLM to access or generate information from a restricted data source, the model could reveal confidential data to unauthorized personnel, such as financial records, trade secrets, or personal information.

‍

DeepKeep’s Solution for PII Leakage

To safeguard enterprises from PII exposure and data leakage, DeepKeep recommends implementing security measures throughout the entire lifecycle of a model/application. This includes inspecting data used for training the LLM, conducting penetration testing before deployment, continuous monitoring of the LLM during production, and applying real-time protection mechanisms such as DeepKeep’s AI firewall.

DeepKeep provides a multi-layered approach to PII protection:

1. PII Scanning of data used for fine-tuning to ensure that all sensitive data is either removed or obfuscated.
2. Penetration Testing to assess the model’s susceptibility to data leakage.
3. Real-Time AI Firewall to block or redact PII data both from the prompts sent to the model and the responses generated by the model.
4. Administration Tools for monitoring events and controlling the AI firewall.

‍

Reach out for a demo of DeepKeep’s solution for dynamic PII Detection, combining pattern recognition, NER, and context interpretation/awareness.