The Flagged Essay — Stand Up & Math

The Scenario

A school's academic integrity committee has deployed an AI-detection tool to analyse student essays. The tool uses two signals: whether the essay is flagged by the detector, and whether it scores low on vocabulary diversity.

Your group is on the committee. An essay has just come in — the detector flagged it, and it scored low on vocabulary diversity. Your job is to figure out: what is the actual probability this essay is AI-assisted?

Work through each phase on the whiteboard. Your teacher will unlock each phase with a code.

Reading the Evidence

Priors · Likelihoods · Conditional probability from a table

🔓

📋 School Database — 1000 essays reviewed last semester

	AI-assisted	Human-written	Total
Flagged by detector	153	82	235
Not flagged	27	738	765
Total	180	820	1000

Task 1a

Prior probabilities

From the table, find the probability that a randomly selected essay is AI-assisted, and the probability that it is human-written. Write these as decimals on your board. These are called the prior probabilities — our belief about the essay before any detector result is known.

Task 1b

Likelihoods — how the detector behaves

Using the table, find all four of the following conditional probabilities:

\( P(\text{Flagged} \mid \text{AI-assisted}) \)
\( P(\text{Flagged} \mid \text{Human-written}) \)
\( P(\text{Not flagged} \mid \text{AI-assisted}) \)
\( P(\text{Not flagged} \mid \text{Human-written}) \)

These are called likelihoods. They tell us how probable each detector outcome is, given the true origin of the essay.

Task 1c

A first instinct

The detector flags an essay. Your group's instinct: what is the probability it is AI-assisted? Write your estimate on the board — you'll revisit it after Phase 2.

🔐 Enter teacher code to unlock Phase 2

The Bayes Flip

Law of total probability · Posterior from one signal

🔒

Complete Phase 1 to unlock

Second Signal — Sequential Update

Posterior becomes prior · Bayes with three events

🔒

Complete Phase 2 to unlock

The Threshold — How Sensitive Is the Model?

Posterior as a function of the prior · Numerical solution

🔒

Complete Phase 3 to unlock

The committee's rule: They will act on a flagged essay (with low vocabulary diversity) only if the posterior probability of AI use exceeds 90%.

With the current prior of \( P(H_1) = 0.18 \), the posterior you found was approximately 85.5% — just below the threshold.

The question: How high would the prior need to be for the committee to act?

Tree hint: This requires two chained trees — a Tree 1 for the detector signal (giving an intermediate posterior), then a Tree 2 for vocabulary diversity where Tree 1's posterior becomes the new root. You'll run this chain for different values of \( p \) until the final posterior exceeds 90%.

Task 4a

Set up the two-tree chain — general form

Let \( p = P(H_1) \) be the unknown prior. On your board, draw the two-tree chain structure with \( p \) as a label at the root of Tree 1. All likelihoods stay the same as before.

Without substituting a number yet, write the expression for the intermediate posterior \( r(p) \) that comes out of Tree 1, and the expression for the final posterior that comes out of Tree 2 using \( r(p) \) as its root.

Task 4b

Find the critical prior — test values in your tree

Now substitute specific values of \( p \) into your chained tree. Work through the full two-tree calculation for each value and record the final posterior. Test at least:

\( p = 0.25 \)
\( p = 0.26 \)

Find the minimum value of \( p \) (to 2 decimal places) such that the final posterior exceeds 90%.

Task 4c

What does this mean?

The prior you found is the "tipping point" — the school's estimate of AI use at which the same detector output would justify acting. Currently the prior is 18%. How much would it need to increase? What does this tell you about the role of the prior in a Bayesian decision system?

🔐 Enter teacher code to complete the activity

✅

Activity Complete!

You've worked through all four phases of The Flagged Essay.
Well done — get ready for the class consolidation.