Skip to main content

Using Cleanlab as a Subject Matter Expert

This guide walks you through the complete SME workflow for improving your AI application using Cleanlab. No technical knowledge is required – you simply need to know how to properly answer common queries that users are asking the AI.

This tutorial assumes that Cleanlab is already integrated with your AI app and is detecting problematic responses that need your expertise.

SME Workflow: Step-by-Step Guide​

Step 1: Start with the Issues View​

Begin your work in your Project’s Issues page in your Cleanlab Web App. This is where the most urgent issues are prioritized and presented for expert review and remediations.

Step 2: Review Highest Impact Issues First​

Cleanlab sorts Issues for you and your team’s review default by Highest Impact, to help you focus on the most urgent, escalating Issues first. This prioritization considers:

  • Frequency: How often similar queries occur
  • Severity: Evaluation scores indicating the most problematic failures

This intelligent ranking helps you maximize the impact of your remediation work.

Step 3: Filter for Unaddressed Cases​

By default, the filter for “Unaddressed” Issues will be applied. These are the the user’s queries that need your immediate attention, and are the cases where your AI was unable to answer correctly or at all. These “Unaddressed” Issues also do not yet have a Remediation yet.

Step 4: Review Individual Issue​

Click on each Issue to examine:

  • The user’s original query
  • The AI’s problematic response
  • Other relevant context and metadata

Step 5: Create a Remediation​

For each problematic case, use the “Add Remediate” option to:

  • Create a Remediation directly from the issue
  • Check if there’s an AI-suggested answer already populated as a draft
    • If available, review the suggested answer and its citations
    • Make any necessary edits or improvements
    • Directly submit the AI-suggested answer if it meets your standards
  • If no AI suggestion is available, provide your expert answer while all context is readily available
  • Automatically protect against similar future failures
  • Cluster Impact: When you remediate one issue, your fix automatically addresses all similar questions clustered together, as well as any other maximizing the impact of your work

Remediation Interface

Step 6: Move to the Next High-Impact Issue​

Continue working through the prioritized list, addressing the most critical issues systematically.

Step 7: Use Advanced Filtering (Optional)​

For focused remediation sessions, you can:

  • Filter by Primary Issue Failure: Focus on specific problem types (e.g., hallucinations, search failures)

Step 8: Collaborate with Other SMEs​

When you need help or want to distribute work:

  • Tag colleagues directly via Cleanlab for specific questions
  • Assign remediation work to other SMEs
  • Review answers from other SMEs before adding your own

Tips for Writing Effective Remediations​

Anticipate Missing Context​

If a user’s query is missing key information, provide comprehensive responses:

Question: How do I reset my device?

Good Answer: If you are using version 14: [specific steps] If you are using version 13: [specific steps] Or if you are using version 12 or older: [specific steps]

Write Stable, Future-Proof Answers​

Avoid overly specific answers that may become outdated:

Instead of: “X units are available” Write: “See current unit availability at this link: [URL]“

Empathize with Users​

Think about what format would be most helpful:

  • Include relevant links for additional information
  • Define any technical jargon
  • Use looping techniques to confirm understanding

Example:

Question: Can I get a refund if…

Better Answer: “Yes, you can get a refund if [condition]. Consider these factors: [details]. Learn more about our refund policy here: [link]“

Security and Privacy​

  • Never include user names or sensitive information in answers
  • Remember that your answers may be shown to any future user

Advanced SME Actions​

Handling Unclear Questions​

If you encounter unclear questions:

  • Skip if another SME might better understand the context
  • Edit to make more generally applicable
  • Delete if the question is nonsensical or inappropriate

Managing Outdated Remediations​

If a remediation becomes out of date or no longer accurate:

  • Pause the remediation to temporarily disable it while retaining its history
  • Review and Edit the remediation with updated information, then reactivate it
  • Consider creating a new remediation if the changes are substantial

Collaborating with Team Members​

Tag colleagues when you need expertise:

'Cleanlab Users'

Tagged experts receive email notifications with direct links to provide their input.

Testing Your Work​

Regularly test your AI application to ensure your remediations are working:

  1. Ask your AI the same questions you’ve remediated
  2. Verify the responses match your expert answers
  3. Test variations of the questions to ensure broad coverage

Maximizing Your Impact​

Focus your time on remediations that will have the highest ROI:

  • Most commonly asked questions (high frequency)
  • Poorly handled by your current AI (high severity)
  • Recent queries (likely to be asked again)

The Issues page’s intelligent sorting and clustering helps you identify these high-impact opportunities automatically.