COMMON WEAKNESS ENUMERATIONIncomplete

CWE-1427

Improper Neutralization of Input Used for LLM Prompting

The product uses externally-provided data to build prompts provided to large language models (LLMs), but the way these prompts are constructed causes the LLM to fail to distinguish between user-supplied inputs and developer provided system directives.

Soyutlama

Base

Yapı

Simple

Sömürü İhtimali

Medium

Problemi anla

Problem nedir?

The product uses externally-provided data to build prompts provided to large language models (LLMs), but the way these prompts are constructed causes the LLM to fail to distinguish between user-supplied inputs and developer provided system directives.

Genişletilmiş açıklama

When prompts are constructed using externally controllable data, it is often possible to cause an LLM to ignore the original guidance provided by its creators (known as the "system prompt") by inserting malicious instructions in plain human language or using bypasses such as special characters or tags. Because LLMs are designed to treat all instructions as legitimate, there is often no way for the model to differentiate between what prompt language is malicious when it performs inference and returns data. Many LLM systems incorporate data from other adjacent products or external data sources like Wikipedia using API calls and retrieval augmented generation (RAG). Any external sources in use that may contain untrusted data should also be considered potentially malicious.

Terminoloji

Alternatif adlar ve kullanım biçimleri

prompt injection

attack-oriented term for modifying prompts, whether due to this weakness or other weaknesses

Kök neden

Problem nasıl ve ne zaman oluşur?

Yaşam döngüsü evresi

Architecture and Design

LLM-connected applications that do not distinguish between trusted and untrusted input may introduce this weakness. If such systems are designed in a way where trusted and untrusted instructions are provided to the model for inference without differentiation, they may be susceptible to prompt injection and similar attacks.

Yaşam döngüsü evresi

Implementation

When designing the application, input validation should be applied to user input used to construct LLM system prompts. Input validation should focus on mitigating well-known software security risks (in the event the LLM is given agency to use tools or perform API calls) as well as preventing LLM-specific syntax from being included (such as markup tags or similar).

Yaşam döngüsü evresi

Implementation

This weakness could be introduced if training does not account for potentially malicious inputs.

Yaşam döngüsü evresi

System Configuration

Configuration could enable model parameters to be manipulated when this was not intended.

Yaşam döngüsü evresi

Integration

This weakness can occur when integrating the model into the software.

Yaşam döngüsü evresi

Bundling

This weakness can occur when bundling the model with the software.

Risk

Başarılı sömürü neye yol açar?

Execute Unauthorized Code or CommandsVaries by Context

Etkilenen güvenlik alanları: Confidentiality, Integrity, Availability

The consequences are entirely contextual, depending on the system that the model is integrated into. For example, the consequence could include output that would not have been desired by the model designer, such as using racial slurs. On the other hand, if the output is attached to a code interpreter, remote code execution (RCE) could result.
Read Application Data

Etkilenen güvenlik alanları: Confidentiality

An attacker might be able to extract sensitive information from the model.
Modify Application DataExecute Unauthorized Code or Commands

Etkilenen güvenlik alanları: Integrity

The extent to which integrity can be impacted is dependent on the LLM application use case.
Read Application DataModify Application DataGain Privileges or Assume Identity

Etkilenen güvenlik alanları: Access Control

The extent to which access control can be impacted is dependent on the LLM application use case.

Savunma

Nasıl önlenir ve çözülür?

Architecture and Design

Mimari ve tasarım yaklaşımı

Detay
LLM-enabled applications should be designed to ensure proper sanitization of user-controllable input, ensuring that no intentionally misleading or dangerous characters can be included. Additionally, they should be designed in a way that ensures that user-controllable input is identified as untrusted and potentially dangerous.

Etkinlik: High

Implementation

Uygulama aşaması savunması

Detay
LLM prompts should be constructed in a way that effectively differentiates between user-supplied input and developer-constructed system prompting to reduce the chance of model confusion at inference-time.

Etkinlik: Moderate

Architecture and Design

Mimari ve tasarım yaklaşımı

Detay
LLM-enabled applications should be designed to ensure proper sanitization of user-controllable input, ensuring that no intentionally misleading or dangerous characters can be included. Additionally, they should be designed in a way that ensures that user-controllable input is identified as untrusted and potentially dangerous.

Etkinlik: High

Implementation

Uygulama aşaması savunması

Detay
Ensure that model training includes training examples that avoid leaking secrets and disregard malicious inputs. Train the model to recognize secrets, and label training data appropriately. Note that due to the non-deterministic nature of prompting LLMs, it is necessary to perform testing of the same test case several times in order to ensure that troublesome behavior is not possible. Additionally, testing should be performed each time a new model is used or a model's weights are updated.

Installation · Operation

Birden fazla yaşam döngüsü evresinde uygulanacak savunma

Detay
During deployment/operation, use components that operate externally to the system to monitor the output and act as a moderator. These components are called different terms, such as supervisors or guardrails.

System Configuration

Güvenli yapılandırma yaklaşımı

Detay
During system configuration, the model could be fine-tuned to better control and neutralize potentially dangerous inputs.

Doğrulama

Nasıl tespit edilir?

Dynamic Analysis with Manual Results Interpretation

Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.

Dynamic Analysis with Automated Results Interpretation

Use known techniques for prompt injection and other attacks, and adjust the attacks to be more specific to the model or system.

Architecture or Design Review

Review of the product design can be effective, but it works best in conjunction with dynamic analysis.