Skip to Main Content

AI (Generative Artificial Intelligence)

Evaluating Generative AI

Critical thinking is key when we use generative AI: it can be difficult to know where generative AI tools get their information, and so fact-checking is crucial.

The video below addresses important points about human intervention in AI, and reminds us that AI does not understand bias, context, or nuance.


The ROBOT Test

Hervieux and Wheatley at The LibrAIry have created The ROBOT Test, which is a useful way to evaluate AI technology and AI-generated content. Learning more about AI technology while referring to The ROBOT Test can help you think critically about this technology.

 

Reliability

Objective

Bias

Ownership

Type

 

Reliability

  • How reliable is the information available about the AI technology?
  • If it's not produced by the party responsible for the AI, what are the author's credentials? Bias?
  • If it is produced by the party responsible for the AI, how much information are they making available?
    • Is information only partially available due to trade secrets?
    • How biased is the information that they produce?

Objective

  • What is the goal or objective of the use of AI?
  • What is the goal of sharing information about it?
    • To inform?
    • To convince?
    • To find financial support?

Bias

  • What could create bias in the AI technology?
  • Are there ethical issues associated with this?
  • Are bias or ethical issues acknowledged?
    • By the source of information?
    • By the party responsible for the AI?
    • By its users?

Owner

  • Who is the owner or developer of the AI technology?
  • Who is responsible for it?
    • Is it a private company?
    • The government?
    • A think tank or research group?
  • Who has access to it?
  • Who can use it?

Type

  • Which subtype of AI is it?
  • Is the technology theoretical or applied?
  • What kind of information system does it rely on?
  • Does it rely on human intervention?

 

Creative Commons License, Picture

This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.


Additional Resources

Rubric for Evaluating AI Tools (eCampus Ontario)