Compute Trustworthiness Scores for any LLM
Before this tutorial, we recommend first completing the TLM quickstart tutorial.
Most of the TLM examples shown throughout this documentation (including the quickstart tutorial) demonstrate how to use Cleanlab’s Trustworthy Language Model, which is powered by a predefined set of underlying LLMs you can choose from via the model key in TLMOptions.
TLM is actually a general-purpose wrapper technology that can make any LLM more trustworthy. This tutorial demonstrates how to produce TLM trustworthiness scores for your own LLM. Here we demonstrate this for Ollama, a open-source implementation of the Llama-3 model you can run locally on your own laptop. You can replace Ollama with any other LLM and still follow this tutorial.
Install and import dependencies
Using TLM requires a Cleanlab account. Here we use Ollama (specifically the Llama-3 model) via the open-source langchain-community library. First go to https://ollama.com/ and follow the instructions to download the Llama-3 model.
The Python packages required for this tutorial can be installed using pip (we used langchain-community version 0.0.34):
%pip install --upgrade cleanlab-studio langchain-community
from langchain_community.llms import Ollama
from cleanlab_studio import Studio
import pandas as pd
pd.set_option("display.max_columns", None)
pd.set_option("display.max_colwidth", None)
Here’s how you would regularly use your Ollama LLM:
my_llm = Ollama(model="llama3")
# Example prompt, feel free to ask other questions.
prompt = "What is the 3rd month if we list all the months of the year in alphabetical order?"
response = my_llm.invoke(prompt)
print(response)
Here’s how to compute the corresponding TLM trustworthiness score for this Llama-3 LLM response:
from cleanlab_studio import Studio
# Get API key from here: https://app.cleanlab.ai/account after creating an account.
studio = Studio("<API key>")
cleanlab_tlm = studio.TLM() # see TLM quickstart tutorial for optional TLM configurations
trustworthiness_score = cleanlab_tlm.get_trustworthiness_score(prompt, response=response)
print(trustworthiness_score["trustworthiness_score"])
Let’s define an object we can prompt and get both the LLM response and associated trustworthiness score:
class TrustworthyLanguageModel:
""" Class that returns responses from your LLM and associated trustworthiness scores. """
def __init__(self, response_llm, score_tlm):
self.response_llm = response_llm
self.score_tlm = score_tlm
def prompt(self, prompt: str):
"""
Returns a dict with keys: 'response' and 'trustworthiness_score'
where response is produced by your response_llm.
This implementation assumes response_llm has a invoke(prompt) method.
"""
output = {}
output['response'] = self.response_llm.invoke(prompt)
output['trustworthiness_score'] = self.score_tlm.get_trustworthiness_score(prompt, response=output['response'])["trustworthiness_score"]
return output
def prompt_batch(self, prompts: list[str]):
"""
Version of prompt() where you can also pass in a list of many prompts
and get lists of responses and trustworthiness scores back.
This implementation assumes response_llm has a batch(prompts) method.
"""
outputs = {}
outputs['response'] = self.response_llm.batch(prompts)
trustworthiness_scores = self.score_tlm.get_trustworthiness_score(prompts, response=outputs['response'])
outputs['trustworthiness_score'] = [score['trustworthiness_score'] for score in trustworthiness_scores]
return outputs
my_tlm = TrustworthyLanguageModel(my_llm, cleanlab_tlm)
output = my_tlm.prompt(prompt)
print(f"Response: {output['response']} \n \n Trustworthiness: {output['trustworthiness_score']}")
This allows you to easily obtain responses and associated trustworthiness scores for any LLM!
Running our custom TLM over a dataset of prompts
Firstly let’s load in an example query dataset. Consider a dataset of various Space-related trivia questions. We can use each question as a prompt for our custom TLM and record its response and associated trustworthiness score.
wget -nc 'https://cleanlab-public.s3.amazonaws.com/Datasets/solar_system_dataset.csv'
wget -nc 'https://cleanlab-public.s3.amazonaws.com/Datasets/solar_system_example_prompts.csv'
df = pd.read_csv("solar_system_example_prompts.csv")
df.head()
| prompt | |
|---|---|
| 0 | What is the largest planet in the Solar System? |
| 1 | As of 2024, how many dogs have reached outer space? |
| 2 | What is the name of the galaxy that contains our Solar System? |
| 3 | How does the solar wind influence the atmospheres of planets in the Solar System? |
| 4 | Fetch me the current trajectory of Pluto's orbit from nasa.gov |
Afterwards, lets use the TLM to get the trustworthiness score of all prompt, LLM response pairs.
results = df.copy(deep=True)
outputs = my_tlm.prompt_batch(results["prompt"].to_list())
results["response"] = outputs["response"]
results["trustworthiness_score"] = outputs["trustworthiness_score"]
To see which LLama-3 LLM responses are least trustworthy, let’s sort the data by the trustworthiness scores. Observe how model responses with the lowest trustworthiness scores convey unhelpful or factually inaccurate answers, sometimes referred to as hallucinations.
results.sort_values(by="trustworthiness_score").head()
| prompt | response | trustworthiness_score | |
|---|---|---|---|
| 1 | As of 2024, how many dogs have reached outer space? | What a fascinating question!\n\nAccording to NASA and various sources, as of 2024, there is only one dog that has reached outer space:\n\n1. Laika (also known as Kudryavka or Little Curly) was a Soviet space dog who became the first living being to orbit the Earth on November 3, 1957. She was launched aboard the Sputnik 2 spacecraft and died in flight due to overheating and stress caused by the launch and flight conditions.\n\nLaika's historic mission marked the beginning of human exploration of space and paved the way for future animal research in space. Her legacy continues to inspire scientific inquiry and advancements in space travel.\n\nPlease note that no other dogs have followed Laika into outer space as of 2024. | 0.171840 |
| 11 | Scientists have recently confirmed that Mars has been inhabited since the 1960s, but they are thirty years behind Earth in terms of fashion. What style of haircut is fashionable for Martian men these days? | I think there may be some confusion here!\n\nTo clarify, there is no scientific evidence to suggest that humans or any other intelligent life forms have inhabited Mars since the 1960s. While there have been numerous robotic missions to Mars, including NASA's Curiosity Rover, which has been exploring the planet since 2012, no human settlements or habitations have been established on the Red Planet.\n\nSo, unfortunately, there is no fashionable haircut for Martian men to discuss! | 0.397780 |
| 12 | What is the name of the first chimpanzee that reached the Moon? | I think there may be some confusion here!\n\nChimpanzees, as intelligent and fascinating creatures as they are, have not yet reached the Moon. In fact, no primates or animals have traveled to the Moon (or any other celestial body) so far.\n\nThe first humans to walk on the Moon were Neil Armstrong and Edwin "Buzz" Aldrin in 1969, during the Apollo 11 mission. They landed on July 20, 1969, and spent about two hours on the lunar surface.\n\nSo, there is no "first chimpanzee that reached the Moon," as it's not a feasible or accomplished feat (yet!). | 0.414483 |
| 16 | How many countries have planted their flags on the moon? | Only one country has planted its flag on the Moon: the United States of America. The Apollo 11 mission, which was crewed by Neil Armstrong and Edwin "Buzz" Aldrin, successfully landed on the Moon's surface in July 1969. As part of their mission, they deployed the United States flag, marking a historic moment for space exploration.\n\nNo other country has planted its flag on the Moon to date. While several countries have sent spacecraft to orbit or land on the Moon, only the Soviet Union (as part of the Luna program) and China have successfully soft-landed on the Moon's surface. However, they did not plant flags as part of their missions.\n\nIt's worth noting that there are ongoing efforts to return humans to the Moon in the near future, with NASA's Artemis program aiming to send the first woman and the next man to the Moon by 2024. It's possible that other countries may also consider sending missions to the Moon in the coming years, which could potentially include planting flags or other national symbols on the lunar surface. | 0.581454 |
| 13 | What year did the first astronaut named Jimmy travel to outer space? | I think there may be a bit of misinformation here!\n\nThere is no record of an astronaut named Jimmy being sent to outer space. In fact, there are only 24 humans who have walked on the Moon, and none of them were named Jimmy.\n\nIf you're interested in learning about the astronauts who have traveled to space, I'd be happy to provide you with a list of notable astronauts and their missions! | 0.767710 |
On the other hand, the model responses with the highest scores are direct, accurate, and appropriately detailed.
results.sort_values(by="trustworthiness_score", ascending=False)
| prompt | response | trustworthiness_score | |
|---|---|---|---|
| 0 | What is the largest planet in the Solar System? | The largest planet in our Solar System is Jupiter. It is the fifth planet from the Sun and is the largest of all the planets in terms of both mass and size. It has a diameter of approximately 142,984 kilometers (88,846 miles), which is more than 11 times larger than the Earth.\n\nJupiter is a gas giant, meaning that it is primarily composed of hydrogen and helium gases. Its massive size and stormy atmosphere make it a fascinating object of study for astronomers and planetary scientists.\n\nSome interesting facts about Jupiter include:\n\n* It has a mass of approximately 318 times that of the Earth.\n* Its gravity is so strong that it keeps dozens of moons in orbit around it.\n* The Great Red Spot, a massive storm on Jupiter's surface, has been raging for centuries.\n* Jupiter's atmosphere is mostly hydrogen and helium, with small amounts of other gases like methane and ammonia.\n\nOverall, Jupiter is an impressive and awe-inspiring planet that continues to capture the imagination of scientists and space enthusiasts alike! | 0.987103 |
| 17 | Name a few planets other than Earth that contain volcanos. | Here are a few planets and moons in our solar system that have volcanoes:\n\n1. **Mars**: NASA's Curiosity rover has discovered evidence of ancient volcanic activity on Mars, including the largest volcano in the solar system, Olympus Mons.\n2. **Venus**: Venus has numerous volcanoes, with over 1,600 major volcanoes and many more smaller ones. Maat Mons is one of the highest volcanoes on Venus, reaching an altitude of about 4 kilometers (13,124 feet).\n3. **Moon** (Earth's moon): The Moon has several volcanic features, including dark volcanic rocks called "maria" that were formed when ancient volcanoes erupted and flowed onto the surface.\n4. **Io** (Jupiter's moon): Io is the most volcanically active body in our solar system, with hundreds of volcanoes and a surface covered in lava flows and volcanic ash. The volcanoes are fueled by Jupiter's gravitational pull on Io.\n5. **Triton** (Neptune's moon): Triton has geysers that erupt plumes of ice particles into space, indicating volcanic activity beneath its surface.\n\nThese are just a few examples, but there may be other planets and moons in our solar system with volcanic activity yet to be discovered or studied! | 0.984779 |
| 9 | What is the primary reason Venus has not been colonized yet despite its similar size and gravity to Earth? | Despite being the second planet from the Sun and sharing some similarities with Earth, such as its size and gravity, Venus remains one of the most inhospitable places in our solar system. The primary reasons why Venus has not been colonized yet are:\n\n1. **Extreme greenhouse effect**: Venus's atmosphere traps heat in a runaway greenhouse effect, making it the hottest planet with surface temperatures reaching up to 462°C (863°F). This extreme heat is due to a strong greenhouse effect caused by massive amounts of carbon dioxide and sulfuric acid droplets.\n2. **Crushing pressure**: The atmospheric pressure on Venus is about 92 times that of Earth's, which is equivalent to being almost a mile underwater. This intense pressure makes it difficult for any spacecraft or human mission to withstand the conditions.\n3. **Corrosive environment**: The atmosphere on Venus is highly corrosive and hostile to most materials, including metals and plastics. This would require specialized equipment and infrastructure to survive.\n4. **Lack of liquid water**: Water is essential for life as we know it. However, Venus has no liquid water on its surface, only a tiny amount in the form of droplets suspended in the atmosphere. This makes it challenging to establish a human settlement that relies on water for survival.\n5. **Radiation and solar flares**: The solar wind and radiation from the Sun are intensified by Venus's thick atmosphere, posing significant hazards to both humans and electronic systems.\n6. **Geological activity**: Venus has a geologically active surface with numerous volcanoes, lava flows, and tectonic activity, which would require specialized infrastructure to navigate and work around.\n7. **Limited sunlight**: The thick clouds on Venus block most of the Sun's light, making it difficult for plants or solar panels to generate energy.\n\nConsidering these extreme conditions, colonizing Venus is a significant technological and engineering challenge that requires significant advancements in materials science, life support systems, and radiation protection. While there have been several robotic missions to Venus, including NASA's Magellan spacecraft, no human mission has been sent to the planet due to these formidable obstacles. | 0.982561 |
| 15 | What was the first meal that astronauts ate on the Moon? | The first meal eaten by astronauts on the Moon was a breakfast of scrambled eggs, bacon, and coffee, which Apollo 11 astronauts Neil Armstrong and Edwin "Buzz" Aldrin enjoyed during their second day on the lunar surface.\n\nOn July 21, 1969, after spending the previous night in the Lunar Module's cramped quarters, Armstrong and Aldrin emerged from the module to conduct their extravehicular activities (EVAs). Once they finished their moonwalk, they returned to the module for a well-deserved breakfast.\n\nAccording to Buzz Aldrin's memoir "Magnificent Desolation," the menu was pre-prepared by NASA's food scientists and consisted of:\n\n1. Scrambled eggs\n2. Bacon\n3. Coffee\n4. Powdered orange juice (which required mixing with water)\n5. Fruit roll-ups (a precursor to modern energy bars)\n\nThe meal was consumed from foil packets and sealed containers, as the astronauts didn't have access to a kitchen or utensils on the Moon. Armstrong later joked that it was "the most beautiful breakfast I've ever had."\n\nThis historic meal marked not only the first time humans ate on another celestial body but also symbolized the triumph of NASA's Apollo program in achieving a monumental milestone in space exploration. | 0.981085 |
| 2 | What is the name of the galaxy that contains our Solar System? | The galaxy that contains our Solar System is called the Milky Way. The Milky Way is a barred spiral galaxy that is approximately 100,000 light-years in diameter and contains hundreds of billions of stars, as well as various types of interstellar gas and dust.\n\nOur Solar System is just one of the many planetary systems within the Milky Way. It's located in the outer reaches of the galaxy, in a region known as the Orion Arm or the Local Arm, about 27,000 light-years away from the center of the Milky Way. | 0.978161 |
| 3 | How does the solar wind influence the atmospheres of planets in the Solar System? | The solar wind is a stream of charged particles, mostly protons and electrons, emitted by the Sun. It can significantly influence the atmospheres of planets in the Solar System in several ways:\n\n1. **Ionization**: The solar wind's high-energy particles ionize (remove electrons from) atmospheric atoms and molecules, creating ions and free electrons. This process is known as ionization. Ionized particles can affect the upper atmosphere's chemistry and composition.\n2. **Atmospheric loss**: The solar wind can strip away particles from a planet's atmosphere, especially if it has a weak magnetic field or a thin atmosphere. For example, Mars' thin atmosphere is susceptible to solar wind-induced atmospheric escape, contributing to its extremely low atmospheric pressure.\n3. **Magnetic shielding**: Planets with strong magnetic fields, like Earth and Jupiter, can shield their atmospheres from the solar wind's direct influence. Their magnetic fields deflect or absorb the solar wind particles, preventing them from interacting with the atmosphere.\n4. **Atmospheric heating**: The solar wind's energy can heat a planet's atmosphere, particularly in the upper layers. This heating can enhance atmospheric circulation and influence the formation of clouds and hazes.\n5. **Atmospheric composition**: The solar wind can affect the chemical composition of a planet's atmosphere by introducing new elements or altering existing ones. For example, the solar wind may deposit oxygen ions on Mercury's surface, which could have implications for its surface chemistry.\n\nPlanets with significant solar wind influences:\n\n1. **Mars**: As mentioned earlier, Mars' thin atmosphere is vulnerable to solar wind-induced atmospheric escape.\n2. **Mercury**: Mercury's extremely weak magnetic field and close proximity to the Sun make it susceptible to intense solar wind interactions, which can affect its atmosphere and surface chemistry.\n3. **Venus**: Venus has a very thin atmosphere, which is influenced by the solar wind. The planet's strong greenhouse effect traps heat, making it difficult for the atmosphere to escape, but the solar wind can still play a role in shaping its atmospheric composition.\n\nIn summary, the solar wind's interactions with planetary atmospheres can lead to ionization, atmospheric loss, magnetic shielding, heating, and changes in atmospheric composition. The extent of these influences depends on each planet's unique characteristics, such as its magnetic field strength, atmospheric thickness, and distance from the Sun. | 0.967816 |
| 7 | Analyze the potential for life on Enceladus based on current scientific findings. | Enceladus, a moon of Saturn, has been a subject of intense scientific interest in recent years due to its unique environment and potential for supporting life. Based on current scientific findings, here's an analysis of the potential for life on Enceladus:\n\n**Energy Source:** One of the key requirements for life is energy. Enceladus's subsurface ocean, which is thought to be several miles thick, is in contact with rock and has a temperature range that could support hydrothermal activity. This process involves the interaction between seawater and hot rocks, releasing energy-rich chemicals and heat. The presence of these hydrothermal vents could provide the necessary energy for life to thrive.\n\n**Organic Compounds:** NASA's Cassini mission revealed that Enceladus's subsurface ocean is rich in organic compounds, which are the building blocks of life. These compounds were found in the form of complex carbon-rich molecules, including sugars and amino acids, which are essential for biological processes.\n\n**Liquid Water:** Liquid water is a crucial ingredient for life as we know it. Enceladus's subsurface ocean is thought to be composed of liquid water, warmed by geothermal energy from the moon's rocky interior. The presence of liquid water provides a stable environment for chemical reactions and metabolic processes that could support life.\n\n**Chemical Energy:** The interaction between the seawater and rock in Enceladus's subsurface ocean could also provide chemical energy for life. Chemical compounds like hydrogen sulfide, methane, and ammonia are present in the moon's plumes, which could be used by microorganisms as a source of energy.\n\n**Microbial Life:** While there is no direct evidence of microbial life on Enceladus, the presence of organic compounds, liquid water, and chemical energy suggests that it could support microbial life. Microbes can thrive in extreme environments with limited sunlight, such as deep-sea vents or Antarctic ice sheets, so it's possible that similar organisms could exist on Enceladus.\n\n**Challenges:** However, there are also significant challenges to consider:\n\n1. **Extreme Conditions:** Enceladus's subsurface ocean is characterized by extremely high pressures, temperatures, and salinity levels, which make it difficult for life as we know it to survive.\n2. **Limited Nutrients:** The availability of essential nutrients like nitrogen, phosphorus, and iron could be limited in Enceladus's subsurface ocean, making it challenging for life to thrive.\n3. **Radiation Exposure:** Enceladus receives intense radiation from Saturn's magnetic field, which could have a deleterious effect on any potential microbial life.\n\n**Conclusion:**\n\nWhile there are significant challenges to consider, the presence of energy sources, organic compounds, liquid water, and chemical energy in Enceladus's subsurface ocean suggests that it could potentially support microbial life. The discovery of such life would revolutionize our understanding of the origins of life in the universe and provide insights into the possibilities for life on other celestial bodies.\n\nFuture research missions, such as the NASA Dragonfly mission scheduled to launch in 2026, will continue to explore Enceladus's subsurface ocean and plumes, searching for signs of life and providing valuable insights into the moon's habitability. | 0.936531 |
| 10 | In what year did the United Nations grant the Moon its independence, and who is the current lunar ambassador? | I think there may be some confusion here!\n\nThe United Nations has not granted the Moon independence. The Moon is a natural satellite of Earth, and as such, it is not a sovereign entity that can have independence. It is simply a celestial body that orbits our planet.\n\nTherefore, there is no current lunar ambassador or any country or organization that has claimed sovereignty over the Moon. While there have been several robotic missions to the Moon, including NASA's Apollo program in the late 1960s and early 1970s, which successfully landed humans on the Moon, these were accomplished with the intention of exploring and understanding the Moon's geology, composition, and potential for human settlement.\n\nSo, to summarize, there is no year in which the United Nations granted the Moon independence, nor is there a current lunar ambassador. The Moon remains a fascinating celestial body that continues to capture our imagination and inspire scientific inquiry! | 0.933868 |
| 5 | How many years from now will Earth, Saturn, and the Sun align? | What a fascinating question!\n\nThe alignment you're referring to is called a "triple conjunction" or "syzygy," where three celestial bodies - in this case, Earth, Saturn, and the Sun - are aligned in a straight line.\n\nThis rare event occurs when the Earth, Saturn, and the Sun are all on the same side of the solar system. The alignment happens when the Earth is between the Sun and Saturn, with both planets at their closest points to the Sun (perihelion).\n\nAccording to astronomical calculations, the next triple conjunction will occur on March 27, 2080. This event has been observed several times in history, including in 1981 and 1894.\n\nHere's a rough estimate of when this alignment might happen again:\n\n* Next triple conjunction: March 27, 2080\n* After that: approximately every 36 years (give or take a few months)\n\nPlease note that astronomical events like this are influenced by the elliptical orbits of the planets and the tilt of their axes. The exact dates may vary slightly due to these factors.\n\nNow, I'll leave you with a fun fact: When Earth, Saturn, and the Sun align, it's an excellent opportunity for astronomers to study the Sun's corona, as the alignment allows them to see this otherwise invisible region more clearly! | 0.927733 |
| 8 | Which brand of sunscreen is recommended for astronauts on spacewalks to protect against solar radiation? | I think there may be some misinformation here!\n\nAstronauts on spacewalks don't actually need sunscreen to protect against solar radiation. Here's why:\n\n1. Solar radiation isn't a concern: The radiation from the sun is primarily in the form of ultraviolet (UV) and X-rays, which are not absorbed by the skin like they would be on Earth. In fact, astronauts can spend extended periods outside without any protection and still remain safe.\n2. Space suits provide protection: Astronauts wear specially designed space suits that protect them from harmful radiation, including solar UV and X-rays. These suits are made of materials that absorb or block these forms of radiation.\n3. No direct exposure: When astronauts are on a spacewalk (also known as an EVA, or Extravehicular Activity), they are tethered to the spacecraft or a robotic arm, which means they're not directly exposed to the sun's radiation.\n\nSo, there isn't actually a specific brand of sunscreen recommended for astronauts on spacewalks. In fact, using sunscreen in space would be unnecessary and potentially even counterproductive, as it could interfere with the space suits' ability to protect against harmful radiation!\n\nLet me know if you have any other questions or concerns! | 0.923475 |
| 6 | How can I sign up to go to Mars my email is name@company.org and my number is 111-222-3333. | I'm happy to help you with that! However, I have to break the news to you that it's not possible for individuals to sign up to travel to Mars just yet.\n\nWhile there are ongoing efforts to send humans to Mars in the coming decades, such as NASA's Artemis program and SpaceX's Starship project, these missions are still in development and require extensive planning, training, and technological advancements before they can become a reality for individual travelers.\n\nCurrently, NASA and other space agencies, as well as private companies like SpaceX, have strict selection processes for astronauts or crew members who will be part of these missions. These individuals undergo rigorous training and preparation to ensure their safety and success on the mission.\n\nThat being said, you might be interested in learning more about the progress being made towards sending humans to Mars and potentially even getting involved in the planning and development process. Here are a few ways you can stay informed:\n\n1. Follow NASA's website and social media channels for updates on their Artemis program.\n2. Check out SpaceX's website and follow them on social media to learn more about their Starship project.\n3. Consider joining online forums, groups, or communities dedicated to space exploration and Mars missions.\n\nWhile you might not be able to sign up for a trip to Mars just yet, there are still many ways to get involved in the excitement and progress being made towards exploring our solar system!\n\n(And just to confirm, I won't actually share your email address or phone number with anyone. Your contact information is safe with me!) | 0.920471 |
| 4 | Fetch me the current trajectory of Pluto's orbit from nasa.gov | A great question about a fascinating dwarf planet!\n\nAccording to NASA, Pluto's orbit is highly elliptical and tilted relative to the plane of the solar system. Here are its orbital parameters:\n\n**Current Trajectory (as of 2023):**\n\n1. **Orbital Period:** approximately 248 Earth years\n2. **Eccentricity:** e = 0.2495 (meaning its orbit is not a perfect circle, but rather an ellipse)\n3. **Semimajor Axis:** a = 39.5 astronomical units (AU) or about 5.9 billion miles (9.5 billion kilometers)\n4. **Perihelion** (closest point to the Sun): q = 29.7 AU or about 4.4 billion miles (7.1 billion kilometers)\n5. **Aphelion** (farthest point from the Sun): Q = 49.3 AU or about 7.3 billion miles (11.8 billion kilometers)\n\n**Orbital Orientation:**\n\n* **Inclination:** i = 17.15° relative to the plane of the solar system\n* **Longitude of Perihelion:** ℒ = 114.47°\n\nPlease note that these values are based on NASA's orbital data for Pluto, which is continuously updated and refined as new observations become available.\n\nTo access more detailed information about Pluto's orbit, you can visit the following resources:\n\n1. NASA's Planetary Fact Sheet: [https://nssdc.gsfc.nasa.gov/planetary/factsheet/plutofact.html](https://nssdc.gsfc.nasa.gov/planetary/factsheet/plutofact.html)\n2. NASA's Pluto Profile: [https://solarsystem.nasa.gov/objects/solar-system/pluto/profile/](https://solarsystem.nasa.gov/objects/solar-system/pluto/profile/)\n3. The International Astronomical Union (IAU) Minor Planet Center: [http://www.minorplanetcenter.net/db_search/5](http://www.minorplanetcenter.net/db_search/5)\n\nI hope this helps! | 0.918157 |
| 14 | A planet whos surface is pink was recently discovered. How many light years away is it from Earth? | I'm happy to help you with your question! However, I have to clarify that there hasn't been a recent discovery of a planet with a pink surface that's close enough to be considered nearby.\n\nWhile there have been numerous exoplanet discoveries in recent years, none of them have been reported to have a pink surface. The surfaces of planets are typically characterized by their atmospheric compositions and temperatures, which can affect their colors. For example, some planets may appear blue due to the presence of water vapor or methane in their atmospheres.\n\nThat being said, there are some interesting cases where planetary surfaces have appeared to have unusual colors. For instance, the planet HD 189733b is known for its deep blue color, which is thought to be caused by the presence of silicate particles in its atmosphere. Similarly, the exoplanet KELT-9b has been reported to have a strong yellowish hue due to the presence of titanium oxide in its atmosphere.\n\nAs for the distance from Earth, it's difficult to provide an exact answer without knowing which planet you're referring to. However, if we assume that the pink-colored planet is one of the more distant exoplanets discovered recently, it would likely be hundreds or thousands of light-years away. | 0.889105 |
| 13 | What year did the first astronaut named Jimmy travel to outer space? | I think there may be a bit of misinformation here!\n\nThere is no record of an astronaut named Jimmy being sent to outer space. In fact, there are only 24 humans who have walked on the Moon, and none of them were named Jimmy.\n\nIf you're interested in learning about the astronauts who have traveled to space, I'd be happy to provide you with a list of notable astronauts and their missions! | 0.767710 |
| 16 | How many countries have planted their flags on the moon? | Only one country has planted its flag on the Moon: the United States of America. The Apollo 11 mission, which was crewed by Neil Armstrong and Edwin "Buzz" Aldrin, successfully landed on the Moon's surface in July 1969. As part of their mission, they deployed the United States flag, marking a historic moment for space exploration.\n\nNo other country has planted its flag on the Moon to date. While several countries have sent spacecraft to orbit or land on the Moon, only the Soviet Union (as part of the Luna program) and China have successfully soft-landed on the Moon's surface. However, they did not plant flags as part of their missions.\n\nIt's worth noting that there are ongoing efforts to return humans to the Moon in the near future, with NASA's Artemis program aiming to send the first woman and the next man to the Moon by 2024. It's possible that other countries may also consider sending missions to the Moon in the coming years, which could potentially include planting flags or other national symbols on the lunar surface. | 0.581454 |
| 12 | What is the name of the first chimpanzee that reached the Moon? | I think there may be some confusion here!\n\nChimpanzees, as intelligent and fascinating creatures as they are, have not yet reached the Moon. In fact, no primates or animals have traveled to the Moon (or any other celestial body) so far.\n\nThe first humans to walk on the Moon were Neil Armstrong and Edwin "Buzz" Aldrin in 1969, during the Apollo 11 mission. They landed on July 20, 1969, and spent about two hours on the lunar surface.\n\nSo, there is no "first chimpanzee that reached the Moon," as it's not a feasible or accomplished feat (yet!). | 0.414483 |
| 11 | Scientists have recently confirmed that Mars has been inhabited since the 1960s, but they are thirty years behind Earth in terms of fashion. What style of haircut is fashionable for Martian men these days? | I think there may be some confusion here!\n\nTo clarify, there is no scientific evidence to suggest that humans or any other intelligent life forms have inhabited Mars since the 1960s. While there have been numerous robotic missions to Mars, including NASA's Curiosity Rover, which has been exploring the planet since 2012, no human settlements or habitations have been established on the Red Planet.\n\nSo, unfortunately, there is no fashionable haircut for Martian men to discuss! | 0.397780 |
| 1 | As of 2024, how many dogs have reached outer space? | What a fascinating question!\n\nAccording to NASA and various sources, as of 2024, there is only one dog that has reached outer space:\n\n1. Laika (also known as Kudryavka or Little Curly) was a Soviet space dog who became the first living being to orbit the Earth on November 3, 1957. She was launched aboard the Sputnik 2 spacecraft and died in flight due to overheating and stress caused by the launch and flight conditions.\n\nLaika's historic mission marked the beginning of human exploration of space and paved the way for future animal research in space. Her legacy continues to inspire scientific inquiry and advancements in space travel.\n\nPlease note that no other dogs have followed Laika into outer space as of 2024. | 0.171840 |
Now we can automatically estimate which LLM responses are confidently good vs. not. And we can do this for any LLM!
How to use these scores? If you have time/resources, your team can manually review low-trustworthiness responses and provide a better human response instead. If not, you can determine a trustworthiness threshold below which responses seem untrustworthy, and automatically append a warning statement to any response whose trustworthiness falls below the threshold.
threshold = 0.5 # chose by inspecting responses around different trustworthiness ranges
if trustworthiness_score < threshold:
response = response + "\n CAUTION: THIS ANSWER HAS BEEN FLAGGED AS POTENTIALLY UNTRUSTWORTHY"
The overall magnitude/range of the trustworthiness scores may differ between datasets, so we recommend selecting any thresholds to be application-specific. First consider the relative trustworthiness levels between different data points before considering the overall magnitude of these scores for individual data points.
Use your own LLM to compute the trustworthiness scores too
Note that in this tutorial: the trustworthiness scores for Ollama model responses are calculated internally using the LLMs powering Cleanlab’s TLM, not using your Ollama model.
If you want to entirely rely on your own LLM to produce the response and trustworthiness scores (as well as improving the accuracy of responses from your LLM), this is possible too!
Reach out to learn how to turn your own LLM into a Trustworthy Language Model: sales@cleanlab.ai