html template I use
Why I think LLM-generated quizzes are ~Good
If I have an LLM write a report for me, and then I only skim it...should I say so?
If I don't, a future session may assume I am aware of things I'm not aware of. But if I do (tell my computer I didn't it's little report), I'll need to explain why (to my computer), or it (my computer!) will generate even longer report on why it thinks I didn't read the first one.
Yuck!
Quizzes leverage the LLM's beyond-human persistence + domain knowledge while also leaving a trail (via answers) of my participation or lack thereof.
The best LLM dossier/report in the world still creates an artifact that says absolutely nothing about what the reader did/didn't take away from it. A quiz creates an unambiguous artifact of what I do and don't understand/care about.
How I use quizzes
(I got idea from Geoffrey Litt, btw)
Before starting a project:
atmosynth is so cool! It made me want to vibe up something. Previously, I would have just jumped in with something like:
"Make me something where with my MPK Mini I can interact with atmosynth. I don't know much about synth stuff. I know like what ADSR and LFO stand for but idk how to use them"
That would have totally worked. But this time, I started with:
"Make me a 10 question quiz to try to gauge my understanding of synths and stuff related to atmosynth, then we'll build something"
The quiz results then became a fantastic context object to jump off from.
Example question
after reading a blog post / a podcast
I listened to an Ezra Klein interview about the Bonds market and then threw the LLM a link to the transcript and asked for a comprehension quiz...and absolutely bombed it.
So then I threw the results of that quiz to a new session, and asked for an easier quiz + some widgets to try to think build intuition around the questions I missed.
(Widgets are great, too. I'll maybe write a blog post about them soon (but if I don't get around to it, the magic words are: "give me some interactive html widgets demonstrating this thing I am struggling to understand")).
Example widget
When I get a 70% on an LLM-generated quiz, I am delighted
For the first few LLM-generated quizzes, I found myself feeling that too familiar feeling of academic stress.
A teacher doesn't want their students to ace every test and quiz. But subjectively as a student? Every time, in my whole life, I've ever missed a quiz or test question I've felt bad about it.
Yuck!
Now, with the cheapness of generating LLM quizzes, I really do want to get about 60-80% correct! When I whiff on questions, that just means the quiz is well calibrated. And if the question is something I really don't even care to know about? I stop taking the quiz and ask for a new one!
LLM-generated quizzes, as bizarre as it feels to say this, have actually given me an appreciation what quizzes are/were supposed to be all along: a tool to gauge understanding.