Skip to main content

Tlm Calibrated

Warning: The utility methods in utils are not guaranteed to be stable between different versions of the cleanlab-studio API.

module cleanlab_studio.utils.tlm_calibrated

TLM Calibrated is a variant of the Trustworthy Language Model (TLM) that facilitates the calibration of trustworthiness scores using existing ratings for prompt-response pairs, which allows for better alignment of the TLM scores in specialized-use cases.

This module is not meant to be imported and used directly. Instead, use Studio.TLMCalibrated() to instantiate a TLMCalibrated object, and then you can use the methods like get_trustworthiness_score() documented on this page.

class TLMCalibrated​

method fit​

fit(tlm_scores: 'List[TLMScore]', ratings: 'Sequence[float]') → None

Callibrate the model using TLM scores obtained from a previous TLM.get_trustworthiness_score() call using the provided numeric ratings.

Args:

  • tlm_scores (List[TLMScore]): list of TLMScore object obtained from a previous TLM.get_trustworthiness_score() call
  • ratings (Sequence[float]): sequence of numeric ratings corresponding to each prompt-response pair, the length of this sequence must match the length of the tlm_scores.

method get_trustworthiness_score​

get_trustworthiness_score(
    prompt: 'Union[str, Sequence[str]]',
    response: 'Union[str, Sequence[str]]'
) → Union[TLMScoreWithCalibration, List[TLMScoreWithCalibration]]

Computes the calibrated trustworthiness score for arbitrary given prompt-response pairs, make sure that the model has been calibrated by calling the .fit() method before using this method.

Similar to TLM.get_trustworthiness_score(), view documentation there for expected input arguments and outputs.

method prompt​

prompt(
    prompt: 'Union[str, Sequence[str]]'
) → Union[TLMResponseWithCalibration, List[TLMResponseWithCalibration]]

Gets response and a calibrated trustworthiness score for the given prompts, make sure that the model has been calibrated by calling the .fit() method before using this method.

Similar to TLM.prompt(), view documentation there for expected input arguments and outputs.

class TLMResponseWithCalibration​

A typed dict similar to TLMResponse but containing an extra key calibrated_score. View TLMResponse for the description of the other keys in this dict.

Attributes:

  • calibrated_score (float, optional): score between 0 and 1 that has been calibrated to the provided ratings. A higher score indicates a higher confidence that the response is correct/trustworthy.

class TLMScoreWithCalibration​

A typed dict similar to TLMScore but containing an extra key calibrated_score. View TLMScore for the description of the other keys in this dict.

Attributes:

  • calibrated_score (float, optional): score between 0 and 1 that has been calibrated to the provided ratings. A higher score indicates a higher confidence that the response is correct/trustworthy.