Social experiment

In Pixels We Trust

One in five confident answers was wrong.

In 2017, Donald Trump, King Salman and Abdel Fattah el-Sisi put their hands on a glowing orb in Riyadh. It looks fake. It's real. Next to it I put an AI image of a summit handshake, the kind of photo you've scrolled past a thousand times. 43.9% of people thought the AI image was the real one.

I built In Pixels We Trust to put people in front of six of these. Ten seconds each, about as long as anyone looks at a photo in a feed. Real or AI-generated? Then, how sure are you? 449 sessions produced 2,694 judgments.

The question nobody was asking

Whether people can spot a deepfake is settled. A meta-analysis of 56 studies and 86,155 people put average accuracy at 55.5%, a coin flip with a little help. In 2024, 80% of countries holding elections ran into generative AI disinformation.

Telling people to look harder hasn't moved that number. So I asked something else. When people get it wrong, do they know?

Six rounds, ten seconds each

Rounds one through three put a real photo next to an AI one and asked which was real. Rounds four through six showed one image and asked real or AI. After every pick, people rated their confidence, low, medium or high, and logged their gut reaction, from curious to disgusted. Then the reveal.

The real photos came from the news. The fakes were made for the study and captioned like news.

Round 1: Trump and King Salman, real orb photo beside an AI handshake

Round 2: Olga Grinik and her nephew in Poltava, real beside AI

Round 3: Texas National Guard at Eagle Pass, real beside AI

Round 4: AI image of protesters at Philadelphia City Hall

Round 5: AI image of the Mathews Bridge collapse

Round 6: real photo of a Waymo burning in Los Angeles

What came back

Overall accuracy was 66.2%. The border pair was the easiest at 79.3%. The Philadelphia City Hall riot, which never happened, was the hardest. 53.7% got it right.

449Sessions

2,694Judgments

66.2%Overall accuracy

22.8%Of wrong answers made with high confidence

Accuracy by round

R1 Trump56.1%

R2 Ukraine67.5%

R3 Border79.3%

R4 Philadelphia53.7%

R5 Bridge62.4%

R6 Fire78.4%

Side by sideSingle image

Fauxfidence

Wrong answer, high confidence. I needed a word for it, so I made one. 22.8% of wrong answers came with high confidence. High-confidence answers were right 78.1% of the time. Low-confidence answers were right 47.7% of the time, worse than guessing.

Wrong answers were also slower, 6.24 seconds against 4.77 for right ones. People hesitated, then committed anyway.

Accuracy by confidence

High78.1%

Medium66.9%

Low47.7%

How it felt mattered

People who felt sad, disgusted or angry were the most accurate. People who felt shocked or curious were the least.

Sad79.2%

Disgusted77.3%

Angry74.0%

Amused69.2%

Neutral66.5%

Curious62.1%

Shocked61.3%

What it means

Media literacy still teaches people to look for extra fingers and bad lighting. Every new model makes that advice older, and it can make fauxfidence worse by handing people a checklist they trust more than they should.

The questions that hold up are about the viewer. How sure should I be? What don't I know here? Am I being worked up on purpose?

At the end of the test, the people who were confident and wrong got this line: "This is how misinformation spreads: it moves quickly, without waiting for us to question it."

Take the test

Sources: Diel et al. (2024); Carnegie Endowment for International Peace (2024).