COMMON WEAKNESS ENUMERATIONDraft

CWE-1434

Insecure Setting of Generative AI/ML Model Inference Parameters

The product has a component that relies on a generative AI/ML model configured with inference parameters that produce an unacceptably high rate of erroneous or unexpected outputs.

Soyutlama

Base

Yapı

Simple

Sömürü İhtimali

Medium

Problemi anla

Problem nedir?

The product has a component that relies on a generative AI/ML model configured with inference parameters that produce an unacceptably high rate of erroneous or unexpected outputs.

Genişletilmiş açıklama

Generative AI/ML models, such as those used for text generation, image synthesis, and other creative tasks, rely on inference parameters that control model behavior, such as temperature, Top P, and Top K. These parameters affect the model's internal decision-making processes, learning rate, and probability distributions. Incorrect settings can lead to unusual behavior such as text "hallucinations," unrealistic images, or failure to converge during training. The impact of such misconfigurations can compromise the integrity of the application. If the results are used in security-critical operations or decisions, then this could violate the intended security policy, i.e., introduce a vulnerability.

Kök neden

Problem nasıl ve ne zaman oluşur?

Yaşam döngüsü evresi

Build and Compilation

During model training, hyperparameters may be set without adequate validation or understanding of their impact.

Yaşam döngüsü evresi

Installation

During deployment, model parameters may be adjusted to optimize performance without comprehensive testing.

Yaşam döngüsü evresi

Patching and Maintenance

Updates or modifications may be made to the model that alter its behavior without thorough re-evaluation.

Risk

Başarılı sömürü neye yol açar?

Varies by ContextUnexpected State

Etkilenen güvenlik alanları: Integrity, Other

The product can generate inaccurate, misleading, or nonsensical information.
Alter Execution LogicUnexpected StateVaries by Context

Etkilenen güvenlik alanları: Other

If outputs are used in critical decision-making processes, errors could be propagated to other systems or components.

Savunma

Nasıl önlenir ve çözülür?

Implementation · System Configuration · Operation

Birden fazla yaşam döngüsü evresinde uygulanacak savunma

Detay
Develop and adhere to robust parameter tuning processes that include extensive testing and validation.

Implementation · System Configuration · Operation

Birden fazla yaşam döngüsü evresinde uygulanacak savunma

Detay
Implement feedback mechanisms to continuously assess and adjust model performance.

Documentation

Documentation aşaması savunması

Detay
Provide comprehensive documentation and guidelines for parameter settings to ensure consistent and accurate model behavior.

Doğrulama

Nasıl tespit edilir?

Automated Dynamic Analysis

Moderate
Manipulate inference parameters and perform comparative evaluation to assess the impact of selected values. Build a suite of systems using targeted tools that detect problems such as prompt injection (CWE-1427) and other problems. Consider statistically measuring token distribution to see if it is consistent with expected results.

Given the large variety of outcomes, it can be difficult to design testing to be comprehensive enough, and there is still a risk of unpredictable behavior.

Manual Dynamic Analysis

Moderate
Manipulate inference parameters and perform comparative evaluation to assess the impact of selected values. Build a suite of systems using targeted tools that detect problems such as prompt injection (CWE-1427) and other problems. Consider statistically measuring token distribution to see if it is consistent with expected results.

Given the large variety of outcomes, it can be difficult to design testing to be comprehensive enough, and there is still a risk of unpredictable behavior.

CWE-1434: Common Weakness Enumeration Entry CWE-1434 · DayBreach Sözlük · DayBreach