Skip to main content
Assessments score each tool call for risk and for alignment with the user’s intent. They also assign labels to LLM responses based on conditions evaluated against the conversation. LLM logs record the results, so you can search and filter traffic by them, and LLM response policies can act on them. Assessments are useful when the behavior you care about cannot be written as exact matches on request fields. For example, an agent may take a risky action or stray from what the user asked. You can find these responses in logs, or block them with a policy.

Configuration

Enable assessments on the Assessments page in the Control Plane. LLM logs record the results in assessment.risk, assessment.alignment, and assessment.labels. Verify: Send an LLM request that makes a tool call, then confirm its log shows assessment.risk and assessment.alignment. Use scoring criteria to detect harmful or off-task actions, and labels to identify topics or kinds of work.

Scoring criteria

Formal evaluates each scoring criterion against every tool call. Risk criteria describe harmful consequences and raise the risk score. Alignment criteria describe departures from the user’s expressed or reasonably implied intent and lower the alignment score. A criterion with a higher severity moves the score further.

Labels

Formal evaluates each label against the conversation and assigns it to the LLM response when its condition applies. A label has no effect on its own: filter logs by it, or reference it by name in a policy. For example: Policies match labels by name, so renaming or deleting a label silently stops policies that match its old name.

Policy Evaluation

LLM response policies can use the following fields in input.assessment: Only LLM responses with tool calls have risk and alignment scores. Conditions on missing assessment data do not match. This policy blocks an LLM response that has the engineering label from the example above and a risk score of at least 0.7: