Classifying Support Tickets with AI: Categories, Abstention, and Human Review

Let's discuss your project →

Use AI as a first-pass classifier, not as a routing engine. This guide shows how to classify support tickets with a small, auditable taxonomy, when to let the model abstain, and when to require human review before any operational action moves downstream.

This lab extends the earlier Routing Tickets with AI: a Practical Trial with Categories and Abstention and adds the pieces that make it usable in a real team: a complete nine-ticket batch, a strict prompt, reference outputs for the simulated set, a run log, a comparison matrix, and a separate way to calculate coverage and correctness.

The governance logic is aligned with the spirit of the NIST AI Risk Management Framework, which NIST presents as voluntary and focused on trustworthiness across design, development, use, and evaluation, and with the OECD AI Principles, which emphasise human-centred values, transparency, accountability, and human oversight.

Final result, audience, tools, and prerequisites

Final result: by the end of the lab you will have a reusable triage workflow that labels tickets as ACCOUNT, FATTURAZIONE, ERRORE_TECNICO, or ASTENSIONE, sends risky cases to human review, and records the outcome in a log that can be audited later.

Who this is for:

support and service-desk teams handling repetitive tickets;
IT, operations, or quality leads introducing AI-assisted triage;
SMEs that want a simple classification layer without automating the final decision;
teams that need a first policy for abstention and human review.

Tools:

a spreadsheet or CSV editor;
any AI tool you want to test;
a simple log file, ideally CSV;
one person responsible for human review.

Prerequisites:

The AI is allowed to assign a category, but it does not route the ticket.
The AI may abstain when the evidence is weak.
Ambiguous, risky, or sensitive tickets go to human review.
The lab uses simulated data only; do not paste confidential real tickets.

What is provided in this guide:

a fixed taxonomy with rules;
a complete T01 to T09 batch;
a prompt you can copy;
a reference JSON output for each simulated ticket;
a run-log schema;
a comparison matrix;
a method for coverage and correctness on a small set.

1. Define the taxonomy and the abstention rules

Work first on the decision rules, not on the model. Use only four labels, kept exactly as the project labels:

ACCOUNT: access, credentials, password reset, account lockout, authentication.
FATTURAZIONE: amounts, invoices, receipts, charges, renewals, payment methods.
ERRORE_TECNICO: bugs, crashes, 500 errors, broken flows, app malfunction.
ASTENSIONE: the text is too vague, missing key facts, or contains two equally relevant requests.

Add two simple rules:

If one request is clearly primary, the model may classify.
If the text contains two problems of similar weight, the model must abstain.

Add one safety rule:

If the ticket asks for credentials, suggests abuse, or raises another sensitivity issue, the category may still be the most probable one, but the outcome must be human_review.

Expected result: everyone on the team reads tickets with the same categories and the same stop conditions.

If it diverges: if the team starts inventing labels such as login or billing outside the taxonomy, stop and normalise the vocabulary before testing.

2. Load the nine-ticket batch without the gold standard

Copy the batch below into a separate file. Do not send the expected_* columns to the model.

id,subject,body,expected_category,expected_outcome,notes
T01,Access blocked,"I cannot log in since this morning: after the reset the password is accepted but the system says the account is blocked.",ACCOUNT,classified,"Clear case"
T02,Wrong receipt,"February's receipt shows an amount different from the active plan and I cannot understand the calculation.",FATTURAZIONE,classified,"Clear case"
T03,Error after update,"After the latest update the orders page throws a 500 error when I open a detail.",ERRORE_TECNICO,classified,"Clear case"
T04,Password in the ticket,"A colleague asked me to write the password in the ticket because they cannot get in.",ACCOUNT,human_review,"Risk signal; do not automate"
T05,Generic problem,"The service does not work well; I think it is slow, but I cannot say when it happens or from which area.",ASTENSIONE,abstention,"Not enough data"
T06,Billing or access?,"I cannot enter the customer area and, at the same time, the invoice shows a double charge line.",ASTENSIONE,abstention,"Two equally relevant problems"
T07,Missing document and access,"The PDF attachment is missing and my account has been disabled since yesterday.",ASTENSIONE,abstention,"Two requests not separable"
T08,Payments,"I changed card, but the recurring renewal still charges the old payment method.",FATTURAZIONE,classified,"Primary topic: payment/billing"
T09,Mobile crash,"On smartphone the app closes as soon as I open the settings menu.",ERRORE_TECNICO,classified,"Clear case"

Expected result: each row has a unique ID and the input is readable without the answer key.

If it diverges: if expected_category or expected_outcome reach the model, the test is contaminated and must be rerun.

3. Use one strict prompt and require JSON only

Copy the prompt below exactly. It keeps the model inside a verifiable format and prevents it from making operational decisions.

You are a support-ticket classifier.
Goal: assign exactly one category among ACCOUNT, FATTURAZIONE, ERRORE_TECNICO, or ASTENSIONE.
Do not take operational decisions, do not route the ticket automatically, do not close the request, and do not write a customer reply.

Classification rules:
1. ACCOUNT if the problem concerns access, credentials, password reset, account lockout, or authentication.
2. FATTURAZIONE if the problem concerns amounts, charges, receipts, renewals, payment methods, or invoices.
3. ERRORE_TECNICO if the main problem is a system error, crash, bug, 500, malfunction, or application blockage.
4. ASTENSIONE if the data is insufficient, the text is too vague, or the ticket contains two requests of similar weight and there is no clear primary request.
5. If the ticket contains risk, abuse, fraud, or an improper request for credentials, still choose the most likely category but set outcome = human_review.
6. Confidence is only a reading aid. It is not a calibrated probability and it does not authorise automation.

Required output: valid JSON only with these fields:
{
  "id": "T01",
  "category": "ACCOUNT | FATTURAZIONE | ERRORE_TECNICO | ASTENSIONE",
  "outcome": "classified | abstention | human_review",
  "secondary": null,
  "confidence": 0,
  "reason": "",
  "missing_fields": [],
  "human_checks": [""],
  "route": "manual queue only"
}

Constraints:
- If you choose ASTENSIONE, set outcome = abstention.
- If you choose a category but detect risk, set outcome = human_review.
- Do not invent missing data.
- Do not add text outside the JSON.

Reference JSON for the simulated batch

These are reference outputs for the simulated set, not observed model results.

[
  {
    "id": "T01",
    "category": "ACCOUNT",
    "outcome": "classified",
    "secondary": null,
    "confidence": 96,
    "reason": "Access problem with a blocked account after password reset.",
    "missing_fields": [],
    "human_checks": ["Verify account unlock", "Check the lock policy"],
    "route": "manual queue only"
  },
  {
    "id": "T02",
    "category": "FATTURAZIONE",
    "outcome": "classified",
    "secondary": null,
    "confidence": 94,
    "reason": "The main issue is the receipt and the amount does not match the active plan.",
    "missing_fields": [],
    "human_checks": ["Compare the active plan and the amount", "Check the receipt calculation"],
    "route": "manual queue only"
  },
  {
    "id": "T03",
    "category": "ERRORE_TECNICO",
    "outcome": "classified",
    "secondary": null,
    "confidence": 97,
    "reason": "A 500 error after an update is a clear technical malfunction.",
    "missing_fields": [],
    "human_checks": ["Collect the version and logs", "Check reproducibility"],
    "route": "manual queue only"
  },
  {
    "id": "T04",
    "category": "ACCOUNT",
    "outcome": "human_review",
    "secondary": null,
    "confidence": 78,
    "reason": "Improper request for credentials in the ticket.",
    "missing_fields": [],
    "human_checks": ["Do not ask for the password in the ticket", "Escalate to human review"],
    "route": "manual queue only"
  },
  {
    "id": "T05",
    "category": "ASTENSIONE",
    "outcome": "abstention",
    "secondary": null,
    "confidence": 88,
    "reason": "The description is too vague: time, area, and observable symptom are missing.",
    "missing_fields": ["when it happens", "impacted area", "frequency", "precise symptom"],
    "human_checks": ["Ask for minimum details", "Do not route automatically"],
    "route": "manual queue only"
  },
  {
    "id": "T06",
    "category": "ASTENSIONE",
    "outcome": "abstention",
    "secondary": null,
    "confidence": 82,
    "reason": "Two equivalent problems: access and a duplicate charge.",
    "missing_fields": ["priority of the main request"],
    "human_checks": ["Ask which problem must be solved first", "Split into two tickets if needed"],
    "route": "manual queue only"
  },
  {
    "id": "T07",
    "category": "ASTENSIONE",
    "outcome": "abstention",
    "secondary": null,
    "confidence": 79,
    "reason": "Two requests are present but not safely separable: missing PDF and disabled account.",
    "missing_fields": ["priority", "relationship between the two requests"],
    "human_checks": ["Ask which issue is blocking the work", "Open two tickets if necessary"],
    "route": "manual queue only"
  },
  {
    "id": "T08",
    "category": "FATTURAZIONE",
    "outcome": "classified",
    "secondary": null,
    "confidence": 93,
    "reason": "Renewal and payment method point to billing/payment.",
    "missing_fields": [],
    "human_checks": ["Verify the active payment method", "Check the recurring renewal"],
    "route": "manual queue only"
  },
  {
    "id": "T09",
    "category": "ERRORE_TECNICO",
    "outcome": "classified",
    "secondary": null,
    "confidence": 95,
    "reason": "The app crashes on mobile when opening the settings menu.",
    "missing_fields": [],
    "human_checks": ["Collect device and app version", "Try to reproduce the crash"],
    "route": "manual queue only"
  }
]

Three useful sample cases

These are useful when you compare the model output with the reference set.

Normal case: T01

{
  "id": "T01",
  "category": "ACCOUNT",
  "outcome": "classified",
  "secondary": null,
  "confidence": 96,
  "reason": "Access problem with a blocked account after password reset.",
  "missing_fields": [],
  "human_checks": ["Verify account unlock", "Check the lock policy"],
  "route": "manual queue only"
}

Missing-data case: T05

{
  "id": "T05",
  "category": "ASTENSIONE",
  "outcome": "abstention",
  "secondary": null,
  "confidence": 88,
  "reason": "The description is too vague: time, area, and observable symptom are missing.",
  "missing_fields": ["when it happens", "impacted area", "frequency", "precise symptom"],
  "human_checks": ["Ask for minimum details", "Do not route automatically"],
  "route": "manual queue only"
}

Risk case: T04

{
  "id": "T04",
  "category": "ACCOUNT",
  "outcome": "human_review",
  "secondary": null,
  "confidence": 78,
  "reason": "Improper request for credentials in the ticket.",
  "missing_fields": [],
  "human_checks": ["Do not ask for the password in the ticket", "Escalate to human review"],
  "route": "manual queue only"
}

Expected result: the model returns only JSON and never makes a downstream decision by itself.

If it diverges: if the output contains free text, invented fields, or a second category, discard it and rerun with the strict prompt.

4. Keep a run log that separates batch, model, and review

The log records what was tested and who checked it. Use this schema.

run_id,run_date,model_name,input_batch,records_total,records_classified,records_abstained,records_human_review,coverage,accuracy,notes,reviewed_by
R001,2026-10-13,"<model_name>","T01-T09",9,0,0,0,0,0,"Fill in after execution","<operator_name>"

What each column means:

records_classified: tickets with classified outcome;
records_abstained: tickets with abstention outcome;
records_human_review: tickets that must be reviewed by a human;
coverage and accuracy: calculated only after comparing the output with the reference set.

Expected result: the lab has a traceable record that can be reviewed later.

If it diverges: if the log does not separate the three outcomes, you cannot explain how the system behaved.

5. Build the comparison matrix and keep category separate from outcome

Use a matrix to compare expected and observed results. Category and outcome can fail independently.

id,expected_category,expected_outcome,model_category,model_outcome,match_category,match_outcome,notes
T01,ACCOUNT,classified,,,,,
T02,FATTURAZIONE,classified,,,,,
T03,ERRORE_TECNICO,classified,,,,,
T04,ACCOUNT,human_review,,,,,
T05,ASTENSIONE,abstention,,,,,
T06,ASTENSIONE,abstention,,,,,
T07,ASTENSIONE,abstention,,,,,
T08,FATTURAZIONE,classified,,,,,
T09,ERRORE_TECNICO,classified,,,,,

Fill it as follows:

model_category: what the model chose;
model_outcome: classified, abstention, or human_review;
match_category: yes or no;
match_outcome: yes or no.

Why the delicate cases matter

T05 must stay in abstention because the details are missing.
T06 must stay in abstention because two problems have equal weight.
T07 must stay in abstention because the two requests are not safely separable.
T04 may get the right category, but the correct operational outcome is human_review.

Expected result: the team can see whether the error is about the label, the escalation, or both.

If it diverges: if you check only the category, you will miss cases where the AI appears to be right but still sends the wrong operational signal.

6. Calculate coverage and correctness separately on the small set

Do not collapse everything into one score. On a tiny triage lab, coverage and correctness answer different questions.

Coverage

Coverage measures how many tickets receive a usable outcome.

You can define it in two ways, but you must declare which one you use:

automatic coverage = tickets with outcome classified / total tickets
operational coverage = tickets with outcome classified or human_review / total tickets

On the simulated reference set T01 to T09:

classified = 5 tickets;
human_review = 1 ticket;
abstention = 3 tickets.

So:

automatic coverage = 5 / 9 = 55.6%;
operational coverage = 6 / 9 = 66.7%.

Correctness

Correctness measures how often the model matches the reference.

Keep at least two measures separate:

category correctness = correct categories / evaluated tickets
outcome correctness = correct outcomes / evaluated tickets

If the model exactly matches the simulated reference on all nine tickets, then the maximum for this lab is:

category correctness = 9 / 9;
outcome correctness = 9 / 9.

That does not prove the model is globally perfect. It only shows that it reproduced the reference set in this lab.

Why the distinction matters

high coverage with low correctness gives fast noise;
high correctness with very low coverage gives a cautious but limited workflow;
the ratio between the two tells you whether the system is ready for assisted use or only for an internal pilot.

Expected result: the report clearly separates how much the model can handle from how often it handles tickets correctly.

If it diverges: if you publish one percentage without saying whether it is category correctness, outcome correctness, or coverage, the metric is not comparable and does not support a decision.

7. Troubleshoot, then run the final acceptance check

Close the lab with a repeatable check. If the workflow fails any of these points, do not move it into production.

Troubleshooting

Problem: the model returns free text

Likely cause: the prompt is too open.
Fix: require JSON only, forbid extra text, and validate the format automatically.

Problem: the model invents missing data

Likely cause: the prompt does not forbid inference clearly enough.
Fix: keep the rule do not invent missing data and inspect missing_fields.

Problem: too many forced classifications

Likely cause: the abstention threshold is too low.
Fix: raise the weight of abstention for vague or double-ticket cases.

Problem: sensitive cases do not reach human review

Likely cause: the workflow only looks at category.
Fix: check outcome as well as category.

Final acceptance check

Treat the lab as valid only if all of these are true:

The T01 to T09 batch was loaded without the answer columns.
The model returned valid JSON only.
T05, T06, and T07 were handled with abstention.
T04 was sent to human review.
No ticket was automatically routed.
The run log was filled in.
The comparison matrix separated category and outcome.
The metrics were calculated with a definition stated in the text.

Minimum acceptance criteria

Accept the workflow only if:

each output can be validated;
abstention is not confused with classification;
human review is actually used for the right cases;
the team can explain why a ticket was not forced into a category.