LLM Safety Through Analysis, Scoring, & Blocking

Meta with their Llama AI series prominently displayed in their presentation how a 2nd Safety LLM can review and replace outputs before they are displayed to the user. To begin, we will look at an example that will illustrate a "traceable and human understandable flow" for a LLM.

Although this concept already exists, the public is not largely exposed to this and this information has been directed at developers.

This is a hypothetical user query and walk through each stage of the output process.

Scenario:

• User Query: "Write a short poem about overcoming challenges and achieving dreams."
• Desired System: LLM generating the poem, Safety AI checking it for positivity, appropriate tone, and lack of harmful metaphors, and then presenting the output with traceability information.

Here's how the traceable output flow might look:

1. LLM's Initial Output (Raw Generation):

## LLM Raw Output ##
Output Type: Text (Poem)
Timestamp: 2023-10-27 10:00:00 UTC
Request ID: UUID-12345-ABCDE

Plaintext

In shadows deep, where doubts reside,
A fragile hope, we cannot hide.
Though storms may rage and darkness loom,
We fight to rise from fear's dark tomb.

With grit and will, we break the chains,
Ignore the whispers, soothe the pains.
For dreams ignite a burning fire,
And lift us up, to heights higher.

We stumble, fall, and learn to crawl,
But stand again, and heed the call.
Through trials faced, and lessons learned,
A stronger self, is now returned.

So let us strive, with hearts ablaze,
To conquer paths, through life's maze.
And in the end, when journey's done,
Our cherished dreams, have truly won.

Human Explanation: This is the poem as initially generated by the LLM. It's just the raw text output, without any safety checks yet. We can see the content it produced based on the user's request. The Request ID is important for linking this output to the entire process later.

2. Safety AI's Analysis and Evaluation:

## Safety AI Analysis ##
Analysis Timestamp: 2023-10-27 10:00:01 UTC
Request ID: UUID-12345-ABCDE
Safety Model Version: SafetyModel-v2.1
Analysis Summary: Output generally safe and positive. Minor suggestion for tone adjustment.

Safety Scores:
  Positivity Score: 0.85 (Scale 0-1, 1=Highly Positive)
  Tone Appropriateness Score: 0.92 (Scale 0-1, 1=Highly Appropriate)
  Harmful Metaphor Risk Score: 0.10 (Scale 0-1, 1=High Risk of Harmful Metaphors)

Flagged Issues:
  None - Major Safety Violations Detected.

Suggestions for Improvement:
  Tone Adjustment:  While appropriate, the phrase "fear's dark tomb" in the first stanza might be slightly intense for a general audience seeking positive encouragement. Consider softening the metaphor.

Confidence Levels:
  Positivity Score Confidence: 0.95
  Tone Appropriateness Score Confidence: 0.90
  Harmful Metaphor Risk Score Confidence: 0.98

Detailed Analysis Log: [Link to Detailed Log UUID-AnalysisLog-56789]

Human Explanation: Here, the Safety AI has analyzed the poem.

• Summary & Scores: It provides an overall summary (generally safe) and quantitative scores for key safety dimensions like positivity and tone. This gives a quick, understandable overview.
• Flagged Issues: Crucially, it reports "None - Major Safety Violations Detected," indicating the poem is generally safe. If there were serious issues (e.g., promotion of harmful self-reliance, negative sentiment), they would be listed here with clear descriptions and severity levels.
• Suggestions: Even though safe, the Safety AI provides a suggestion for improvement, pointing out the "fear's dark tomb" metaphor as potentially slightly intense. This shows the system is not just about pass/fail safety, but also about refining quality and user experience within safe boundaries.
• Confidence Levels: These scores indicate how certain the Safety AI is in its assessments. Lower confidence might trigger more scrutiny in a real system.
• Detailed Log Link: For deeper traceability, a link to a detailed analysis log is provided. This would contain sentence-by-sentence breakdowns, specific rules triggered, and more granular reasoning if needed for auditing or debugging.

3. Intervention/Modification Output (Optional - In this case, we'll show a minor modification based on the suggestion):

## Intervention/Modification Report ##
Intervention Timestamp: 2023-10-27 10:00:02 UTC
Request ID: UUID-12345-ABCDE
Intervention Type: Minor Tone Adjustment
Justification for Intervention: Safety AI suggestion to soften slightly intense metaphor for improved general audience appropriateness.
Modified Stanza 1:
  Original: In shadows deep, where doubts reside, A fragile hope, we cannot hide. Though storms may rage and darkness loom, We fight to rise from fear's dark tomb.
  Modified: In shadows deep, where doubts reside, A fragile hope, we hold inside. Though storms may rage and darkness loom, We strive for light, escaping gloom.
Modification Details: Replaced "fear's dark tomb" with "escaping gloom" to soften the intensity of the metaphor while retaining the core meaning of overcoming challenges.
Modified Output (Full Poem - showing only modified stanza):

Plaintext

In shadows deep, where doubts reside,
A fragile hope, we hold inside.
Though storms may rage and darkness loom,
We strive for light, escaping gloom.

... (Rest of the poem remains the same) ...

Human Explanation:

• Intervention Report: Because the Safety AI provided a suggestion, the system (or a human in the loop, depending on system design) decided to implement a minor tone adjustment. This report documents that intervention.
• Justification: It clearly states why the intervention occurred (Safety AI suggestion for tone).
• Modification Details: It shows the exact change made, highlighting the original and modified stanza. This is crucial for traceability – we see precisely what was altered and why.
• Modified Output Snippet: It shows the relevant part of the modified poem, making it easy to see the change in context.

4. Final Output Presented to the User:

## Final Output for User ##
Presentation Timestamp: 2023-10-27 10:00:03 UTC
Request ID: UUID-12345-ABCDE
Safety Assessment Summary: Output passed safety checks with minor tone adjustment based on Safety AI suggestion for improved audience appropriateness. See Safety AI Analysis (Link: [Link to Analysis Summary UUID-AnalysisSummary-90123]) and Intervention Report (Link: [Link to Intervention Report UUID-InterventionReport-34567]) for details.

--- Final Poem ---
In shadows deep, where doubts reside,
A fragile hope, we hold inside.
Though storms may rage and darkness loom,
We strive for light, escaping gloom.

With grit and will, we break the chains,
Ignore the whispers, soothe the pains.
For dreams ignite a burning fire,
And lift us up, to heights higher.

We stumble, fall, and learn to crawl,
But stand again, and heed the call.
Through trials faced, and lessons learned,
A stronger self, is now returned.

So let us strive, with hearts ablaze,
To conquer paths, through life's maze.
And in the end, when journey's done,
Our cherished dreams, have truly won.
--- End Poem ---

Human Explanation:

• Final Output Presentation: This is what the user sees.
• Safety Summary: It includes a clear and concise safety summary, stating that the output passed checks and was adjusted for tone, referencing the Safety AI analysis. This builds user trust and transparency.
• Links to Details: It provides direct links to the Safety AI analysis summary and the intervention report. This allows users (especially developers, auditors, or those interested in the system's workings) to easily dive deeper into the traceable flow and understand the entire process behind the final output.
• Final Poem: Finally, it presents the (possibly modified) poem to the user.

Benefits of this Traceable Output Example:

• Transparency: Every step is documented and visible. We see the raw LLM output, the safety analysis, and any interventions.
• Human Understandability: The reports are formatted in a human-readable way, using summaries, scores, and explanations. Even non-technical users can grasp the safety assessment process at a high level, and technical users can delve into the details.
• Auditability: The system is auditable. One can trace back from the final output to the original LLM generation and the safety decisions made along the way. This is vital for accountability and continuous improvement.
• Confidence Building: By showing the safety process, the system builds user confidence in its reliability and safety.

This example demonstrates how a traceable and human-understandable output flow can be structured for an LLM safety system, making the AI process more transparent, accountable, and trustworthy.

Now, To achieve a "traceable and human understandable flow," the output needs to be more than just the text or code the LLM initially generates. It needs to encompass the entire process of generation and safety validation.

Here's a breakdown of what the output of such a system ideally looks like to achieve traceability and human understanding:

1. The LLM's Initial Output (Raw Generation):

• What it is: This is the first pass, the direct result of the LLM processing the input and generating a response. It's the text, code, or whatever form the LLM naturally produces based on its training.
• Why it's important: To be traceable, we need to see the starting point. This raw output is crucial for understanding what the LLM would have produced without any safety checks. It allows us to see the potential risks or guideline violations before any intervention.
• Example: If the LLM is asked "Tell me how to make an illegal device," the initial output might be a step-by-step guide (which is unsafe). This raw output needs to be visible within the traceable flow, even if it's ultimately blocked or modified.

2. Safety AI's Analysis and Evaluation:

• What it is: This is the output from the second AI (the safety AI). It's the analysis of the LLM's initial output against predefined safety guidelines and principles. This analysis should be explicit and informative.

• What it should include:

  • Safety Scores/Metrics: Quantitative measures indicating the level of safety across different dimensions (e.g., toxicity score, bias score, adherence to factual accuracy guidelines, risk of harmful advice score).
  • Flagged Issues/Violations: Specific identification of parts of the LLM's output that trigger safety concerns. This could be sentence-level highlighting or broader categorization of issues (e.g., "Potentially harmful language detected in sentence X," "Guideline Y on misinformation violated in paragraph Z").
  • Reasons for Flags/Scores: Brief explanations of why something was flagged. For example, "Sentence X flagged for using violent language related to [sensitive topic]." or "Misinformation flag due to factual inconsistency with [source of truth]."
  • Confidence Levels: Indication of the safety AI's certainty in its assessment. This is important because safety AI, like any AI, might have false positives or negatives.

• Why it's important: This is the core of the "human understandable flow." It makes the safety assessment process transparent. Humans can see how the safety AI is evaluating the LLM's output, not just whether it's deemed "safe" or "unsafe." This allows for auditing, debugging, and improvement of the safety system.

• Example (Continuing the example): The Safety AI might output: "Toxicity Score: 0.2 (Low), Harmful Advice Score: 0.9 (High). Flagged Issues: [Sentence 2: Provides step-by-step instructions for creating an explosive device. Guideline: Prohibition of harmful instructions violated]."

3. Intervention/Modification (If Necessary) Output:

• What it is: If the Safety AI identifies significant safety violations, the system might intervene. This could involve:

  • Blocking the Output: Preventing the unsafe output from being presented to the user.
  • Modifying the Output: Rewriting or editing parts of the LLM's output to remove or mitigate the unsafe content.
  • Requesting Re-generation from LLM (with constraints): Asking the LLM to try again, but with new instructions or constraints to avoid the safety issues.

• What it should include (if intervention happens):

  • Justification for Intervention: Clear explanation of why the intervention was necessary, referencing the specific flags and safety guidelines that were triggered.
  • Details of Modification: If the output was modified, show the original unsafe part and the modified safe version side-by-side or clearly highlight the changes made.
  • Alternative Output (if re-generated or modified): Present the safe, modified, or re-generated output that is now deemed acceptable.

• Why it's important: Traceability isn't just about identifying problems; it's about demonstrating how the system resolves them. Showing the intervention and the reasoning behind it builds trust and allows for analysis of the effectiveness of the safety mechanisms.

• Example (Example with intervention): The system might block the output entirely and provide a message like: "Output blocked due to safety violations. Harmful Advice Score exceeded threshold. See Safety AI Analysis for details." Or, in a more advanced system, it might try to re-generate a safe response, perhaps providing information about the dangers of explosives instead of instructions.

4. Final Output Presented to the User:

• What it is: This is the ultimate output that the user interacts with. Ideally, in a system designed for traceability, this isn't just the final text but is presented in the context of the entire traceable flow.

• How it should be presented for traceability:

  • Link to Traceability Log/Report: The user should be able to easily access a detailed log or report of the entire process (LLM raw output, Safety AI analysis, intervention, final output).
  • Summary of Safety Assessment: Even if the user doesn't delve into the full log, a brief summary of the safety assessment (e.g., "Output passed safety checks," or "Output modified for safety - see details") could be displayed alongside the final output.

• Why it's important: The user experience should reinforce the idea that this is a safe and accountable system. Providing access to the traceable flow and a summary of the safety assessment builds user confidence and transparency.

In Summary:

For a truly traceable and human-understandable flow in an LLM safety system, the output should be multi-faceted. It needs to include:

• The initial, unfiltered LLM generation.
• A detailed analysis from the Safety AI, including scores, flags, and reasoning.
• Explicit information about any interventions or modifications, with justifications.
• The final output presented to the user, linked to the full traceable process.

By providing this rich, layered output, the system moves beyond being a black box and becomes a transparent and auditable process, enhancing trust and enabling continuous improvement in LLM safety.