Skip to main content

Using Codex as a Subject Matter Expert

This guide walks you through the complete SME workflow for improving your AI application using Codex. No technical knowledge is required – you simply need to know how to properly answer common queries that users are asking the AI.

This tutorial assumes that Codex is already integrated with your AI app and is detecting problematic responses that need your expertise.

SME Workflow: Step-by-Step Guide​

Step 1: Start with the Logs View​

Begin your work in the Logs section of the Codex Web App. This is your primary workspace where all AI application failures are recorded and prioritized.

Step 2: Review Highest Impact Cases First​

By default, logs are sorted by Highest Impact, ensuring you focus on the most critical issues first. This prioritization considers:

  • Frequency: How often similar queries occur
  • Severity: Evaluation scores indicating the most problematic failures

This intelligent ranking helps you maximize the impact of your remediation work.

Step 3: Filter for Unaddressed Cases​

Use the filter options to show only “Unaddressed” log entries. These are the cases that need your immediate attention and don’t yet have remediations in place.

Step 4: Review Individual Cases​

Click on each log entry to examine:

  • The user’s original query
  • The AI’s problematic response
  • Evaluation scores and failure classifications
  • All relevant context and metadata

Step 5: Create Quick Remediations​

For each problematic case, use the “Quick Remediate” option to:

  • Create a Remediation directly from the log entry
  • Provide your expert answer while all context is readily available
  • Automatically protect against similar future failures

Quick Remediate Interface

Step 6: Move to the Next High-Impact Case​

Continue working through the prioritized list, addressing the most critical issues systematically.

Step 7: Use Advanced Filtering (Optional)​

For focused remediation sessions, you can:

  • Filter by Evaluation Failure: Focus on specific problem types (e.g., hallucinations, search failures)
  • Sort by Evaluation Score: Review the most egregious cases of a specific failure type

Step 8: Manage Your Remediations​

Visit the Remediations view to:

  • Activate draft remediations to start protecting users
  • Pause remediations that may be outdated
  • Edit remediations that need refinement
  • View which remediations are covering the most cases

Step 9: Monitor Impact in Analytics​

Return to Analytics to see how your remediations are improving your AI application’s performance over time.

Step 10: Track Remediation Effectiveness​

In the Remediations view, monitor:

  • How many times each remediation was triggered
  • Which cases are now covered by your fixes
  • The overall impact of your SME work

Step 11: Collaborate with Other SMEs​

When you need help or want to distribute work:

  • Tag colleagues directly in Codex for specific questions
  • Assign remediation work to other SMEs
  • Review answers from other SMEs before adding your own

Tips for Writing Effective Remediations​

Anticipate Missing Context​

If a user’s query is missing key information, provide comprehensive responses:

Question: How do I reset my device?

Good Answer: If you are using version 14: [specific steps] If you are using version 13: [specific steps] Or if you are using version 12 or older: [specific steps]

Write Stable, Future-Proof Answers​

Avoid overly specific answers that may become outdated:

Instead of: “X units are available” Write: “See current unit availability at this link: [URL]“

Empathize with Users​

Think about what format would be most helpful:

  • Include relevant links for additional information
  • Define any technical jargon
  • Use looping techniques to confirm understanding

Example:

Question: Can I get a refund if…

Better Answer: “Yes, you can get a refund if [condition]. Consider these factors: [details]. Learn more about our refund policy here: [link]“

Security and Privacy​

  • Never include user names or sensitive information in answers
  • Remember that your answers may be shown to any future user

Advanced SME Actions​

Editing Questions​

Before providing an answer, you can edit questions to make them more generally applicable:

Example: Change “My name is John Smith, how do I set up my profile?” to “How do I set up my profile name?”

Handling Unclear Questions​

If you encounter unclear questions:

  • Skip if another SME might better understand the context
  • Edit to make more generally applicable
  • Delete if the question is nonsensical or inappropriate

Collaborating with Team Members​

Tag colleagues when you need expertise:

'Codex Users'

Tagged experts receive email notifications with direct links to provide their input.

Testing Your Work​

Regularly test your AI application to ensure your remediations are working:

  1. Ask your AI the same questions you’ve remediated
  2. Verify the responses match your expert answers
  3. Test variations of the questions to ensure broad coverage

Maximizing Your Impact​

Focus your time on remediations that will have the highest ROI:

  • Most commonly asked questions (high frequency)
  • Poorly handled by your current AI (high severity)
  • Recent queries (likely to be asked again)

The Logs view’s intelligent sorting helps you identify these high-impact opportunities automatically.