Skip to main content

module cleanlab_tlm.utils.tlm_calibrated

TLM Calibrated is a variant of the Trustworthy Language Model (TLM) that facilitates the calibration of trustworthiness scores using existing ratings for prompt-response pairs, which allows for better alignment of the TLM scores in specialized-use cases.

function save_tlm_calibrated_state​

save_tlm_calibrated_state(model: 'TLMCalibrated', filename: 'str') → None

Save fitted TLMCalibrated model state to file.

Args:

  • model (TLMCalibrated): A fitted TLMCalibrated model instance
  • filename (str): Path where the model state will be saved

Raises:

  • sklearn.exceptions.NotFittedError: If the model has not been fitted
  • ImportError: If skops or sklearn package is not installed

function load_tlm_calibrated_state​

load_tlm_calibrated_state(filename: 'str') → TLMCalibrated

Load and reconstruct TLMCalibrated model from file.

Args:

  • filename (str): Path to the saved model state file

Returns:

  • TLMCalibrated: A reconstructed TLMCalibrated model with the saved state

Raises:

  • FileNotFoundError: If the specified file does not exist
  • ImportError: If skops package is not installed

class TLMCalibrated​

method fit​

fit(tlm_scores: 'list[TLMScore]', ratings: 'Sequence[float]') → None

Callibrate the model using TLM scores obtained from a previous TLM.get_trustworthiness_score() call using the provided numeric ratings.

Args:

  • tlm_scores (list[TLMScore]): list of TLMScore object obtained from a previous TLM.get_trustworthiness_score() call
  • ratings (Sequence[float]): sequence of numeric ratings corresponding to each prompt-response pair, the length of this sequence must match the length of the tlm_scores.

method get_trustworthiness_score​

get_trustworthiness_score(
    prompt: 'Union[str, Sequence[str]]',
    response: 'Union[str, Sequence[str]]'
) → Union[TLMScoreWithCalibration, list[TLMScoreWithCalibration]]

Computes the calibrated trustworthiness score for arbitrary given prompt-response pairs, make sure that the model has been calibrated by calling the .fit() method before using this method.

Similar to TLM.get_trustworthiness_score(), view documentation there for expected input arguments and outputs.

method prompt​

prompt(
    prompt: 'Union[str, Sequence[str]]'
) → Union[TLMResponseWithCalibration, list[TLMResponseWithCalibration]]

Gets response and a calibrated trustworthiness score for the given prompts, make sure that the model has been calibrated by calling the .fit() method before using this method.

Similar to TLM.prompt(), view documentation there for expected input arguments and outputs.

class TLMResponseWithCalibration​

A typed dict similar to TLMResponse but containing an extra key calibrated_score. View TLMResponse for the description of the other keys in this dict.

Attributes:

  • calibrated_score (float, optional): score between 0 and 1 that has been calibrated to the provided ratings. A higher score indicates a higher confidence that the response is correct/trustworthy.

class TLMScoreWithCalibration​

A typed dict similar to TLMScore but containing an extra key calibrated_score. View TLMScore for the description of the other keys in this dict.

Attributes:

  • calibrated_score (float, optional): score between 0 and 1 that has been calibrated to the provided ratings. A higher score indicates a higher confidence that the response is correct/trustworthy.