Skip to main content

Using Codex as a Subject Matter Expert

This tutorial demonstrates how SMEs can use the Codex Web App to directly improve any AI (RAG/Agents) application. No technical knowledge is required, you simply need to know how to properly answer common queries that users are asking the AI.

This tutorial assumes that Codex is already integrated with your AI app. Once integrated, Codex automatically detects user queries that are poorly handled by your AI and logs these as entries in a Codex Project.

Your task as a SME is to provide desired answers to these queries, specifically focusing on the highest impact queries. After you’ve provided an answer, your AI application will respond with this answer whenever users ask a similar question in the future.

Answering Questions​

Find your Codex Project under the Projects tab and click Review Entries to begin answering questions.

Once you see the entries in your Project (which should be unanswered questions), select a question and click the Expand button to type in an answer. After you click Save, this question/answer will be instantly loaded into the ‘brain’ of your AI, such that it will be able to respond with this answer whenever future users ask a similar question.

Note: Answered questions will disappear from the initial table whose default View is: “Unanswered Questions Only”. You can toggle different views to see already-answered questions. If possible, we recommend first reviewing answers from other SMEs before entering your own answers to new questions.

Testing the AI application​

To check Codex is working, ask your AI application the same question (or a similar variant) that you just answered in the Codex Project. Your AI application should now properly handle this question.

If not, contact us for help with integrating Codex into your AI application: support@cleanlab.ai

When first getting used to Codex, we recommend regularly testing that your AI application is improving as you provide more answers. Once you feel more proficient, feel free to space out your testing.

Unclear Questions​

If you encounter a question that isn’t straightforward to answer, then consider the options below rather than providing a potentially wrong answer.

Perhaps it’s unclear what the user is asking or why, or you’re worried that key context/information may be missing. For multi-turn conversational AI applications: some questions in Codex may have been part of a longer chat between the user and AI, which did not get entirely captured in the Codex Project view.

If you are unsure how to answer a question but it seems good if your AI could answer such questions, then skip over this question (leaving it to be answered by another SME). Also consider manually populating Codex with a related question, which you’ve confidently answered and you believe future users would ask. Your related question does not have to match the original question’s intent, so you can rephrase unclear questions based on your own interpretation, as long as your version is a self-contained question which you’ve confidently answered.

If you encounter a question that seems too specific to one user (perhaps it contains user-specific information in it), then edit the question such that your subsequent answer will help other users as well.

Otherwise if a question seems irrelevant/nonsensical or undesirable for your AI to respond to with a crafted answer, you can delete it to reduce clutter.

Example handling of questions

Suppose you see a question:

My name is John Smith, how do I set up my profile name?

You might choose to edit this question in the Codex Project to instead read:

How do I set up my profile name?

Because the rewritten question is more generally applicable across users.

You might also manually add another question with the same answer to this Codex Project, such as:

Change my profile settings

If you see a question that appears nonsensical, such as:

147131923

Then if your AI already handles such situations appropriately, you might delete this from the Codex Project. If you are unsure, then skip this question.

Edit Questions​

Before editing a question, consider whether it would instead be better to leave the question as is and manually populate Codex with a related question that you’ve written.

On occasion, you may spot a question that could be phrased with more clarity or generality (such that it applies to more users rather than only a specific one). By clicking Expand, you can edit the question before providing an answer.

Note: This is only recomended if your manual improvement to the question is clear and does not make the question unlikely to resemble any future queries. Tips:

  • Remove user-specific information or irrelevant context that does not affect the answer.
  • Rephrase questions such that their answer is more likely to remain stable for the foreseeable future.

After making edits in Codex, you can test that your AI is now able to answer both the edited and original question (if not, try smaller edits).

By toggling the appropriate View on the Project Entries page, you can similarly edit existing answers to questions.

Delete Questions​

Remove a question from Codex by clicking the Delete button.

Caution: Deleting an unanswered question will prevent other SMEs from knowing it was ever asked by a user. Deleting an answered question will cause your AI to no longer respond with that answer.

Tips on writing effective answers​

Do not write overly specific answers that may become outdated in the near future.

Instead for questions whose specific answer may evolve: write more generic tips on how/where to find the answer (e.g. providing link to the answer rather than the answer itself, if the answer may change but the link won’t).

For instance, if X may soon change in the answer:

X units are available

Then a better answer might be:

See how many units are available at this link: …

Instead of providing links, you can alternatively provide stable contact information where future users will still be able to get an answer for questions like this.

Anticipate missing context in questions.

If a user’s query is missing key information that will affect the answer, you can still provide a helpful response through if-then statements. For instance, a question like:

How do I reset my device?

Could be answered like this:

If you are using version 14: … If you are using version 13: … Or if you are using version 12 or older: …

Alternatively if your AI application supports multi-turn dialogues, then you could request the missing information:

We will first need to know which version you are using. Can you find that information from … ?

Do not include user names or sensitive information in your answers.

Your answers may be returned to any future users of the AI application.

Empathize with users of your AI application.

Think about what type/format of answer they would find most helpful, even if their question was not perfectly phrased. Include links where the user can learn more about the topic mentioned in your answer, and define any jargon.

Employ looping techniques where you include key parts of the user’s question in your answer, just in case you misunderstood some aspect of their question or they misspoke. For instance, a question like:

Can I get a refund if …

Might be hastily answered like this:

Yes

But would be better answered like this:

Yes you can get a refund if … Consider the following factors: … Learn more about our refund policy here: …

Which questions to answer first​

Codex can help you prioritize your time, such that you maximize the ROI of your work. This is achieved by having you first provide answers for questions that are:

  • Most commonly asked by users and recent (likely to be asked in the future).
  • Handled suboptimally by your AI application, where there exists a much better answer.