Hacking One's Way Through
... towards teaching folks what p-hacking is all about
I came across a very cool tweet. Cool in and of itself, but also because I’m teaching introductory statistics this year, and am trying (very hard) to communicate why hypothesis testing is so important, and so very misunderstood.
P-hacking, if you haven’t come across the term, is the practice of massaging your data analysis until you find a statistically significant result — and it’s far more widespread than you’d think.
Now, I read this tweet on a Friday evening, after two rather large drinks of a most excellent feni (Tinto, well played. Really well played!). And when you chat with Claude in an elevated state, fun things can happen.
Those fun things resulted in a 6,700-word essay, but you will have to earn your way towards reading it, because my chat with Claude also resulted in a website. Click here to try it out, and have fun reading the essay, and learning more about this topic in NotebookLM (see the website for full details). If you would like to play around and improve the tool, here you go.
The idea was to help my students get a little bit of the “so what?” of p-values, and help them become better readers of academic literature. But anybody who is half-familiar with stats should also be able to follow along.
As always, it is the meta points that interest me:
For students and teachers alike, AI helps you spend your time better, it doesn’t necessarily save it.
If you are serious about learning this topic, there is no getting around reading the essay in full, and carefully. And probably more than once!
It becomes quite easy to do what I just did for this topic, if you choose to do this for every topic you choose to learn well (or teach well, for that matter). It will be hellish the first time around, but learning compounds, and AI workflows will become more intuitive as you do this more and more often.
You have a choice: use AI to reduce effort while learning (or while teaching). Or use AI to increase quality while learning (or while teaching). AI actively harms learning outcomes in the first case.
Given the topic I’m teaching via this post, I wouldn’t want to go so far as asserting that it definitely helps in the second case - but hey, it certainly does you no harm. And comes with a whole host of positive externalities (learning AI-first workflows, exploring advanced topics, sharing proof-of-work publicly, building out your portfolio, among others).
But I will say this much: if you think learning today is about memorizing a textbook and acing a written examination, you are playing the wrong game.
Please. Learn how to use these new tools. I’m begging you.
Where is the link to the essay?