How to measure changes in daily cognitive ability?
KPier wrote an article on LessWrong titled “Chess and cheap ways to check day to day variance in cognition”. After starting to play chess, they quickly noticed that bad chess performance is correlated with low sleep quality and a high stress level. Likewise, good performance seems to be predictive of doing well at cognitively demanding tasks. They propose using online chess games to quickly assess mental functioning.
This idea immediately captivated me, because I have been thinking about something like this for quite a while. Lots of lifestyle interventions (like meditation) promise to make you smarter or at least more focused. I would love to do experiments on some of them and check whether the claims are true. The main problem is that I have no reliable way to determine my level of cognitive performance on a given day. Sure, it is easy to count the number of hours worked or the lines of codes written. But these metrics are probably not really meaningful to assess cognitive ability and I can easily placebo myself. What I need is a quick and reliable method that measures the raw mental functioning and nothing more.
KPier’s method is playing 5-minute online chess games. This comes with a number of problems. Firstly, I would have to play several games a day to calculate some kind of average and I don’t want to spend this much time. Secondly, the website they use pairs the player with opponents at approximately the same skill level, so even if e.g. meditation improves my cognitive ability, the AI will match this improvement with improved partners. This might work to estimate fluctuations around a steady mean but could fail when applied to steady mental changes.
Has anyone else tried something like this? In this Astral Codex Ten post, Scott Alexander uses a letter-finding game called WordTwist to assess whether varying CO2 levels in his room change his cognitive performance. He found “absolutely no correlation” which could either mean that carbon dioxide has no effect on mental functioning, there was a problem with his study design or that this game is a bad surrogate metric for cognitive ability. Either way, it seems time intensive and boring.
Why not simply use IQ tests? After all, they are designed for this exact purpose. There are a couple of drawbacks. I probably can’t get tests that are reliable and short enough at the same time. Also, doing lots of IQ tests makes you get better at IQ tests and because my experiments have no control group, it would be impossible to correct for this effect.
In the comments of the LessWrong article, someone proposes simple reaction time as a metric. Because all this measures is how rapidly you can push a button, this is probably the quickest method by far. It also seems to correlate weakly to moderately with IQ [1]. That all sounds great, but reaction time as a metric is still worthless for me if it can’t be changed by lifestyle interventions.
A Spanish research team randomized 138 older adults into three groups. The first group took part in a resistance training exercise twice a week, the second group did the same exercise combined with some cognitive tasks and the third group served as controls. The experiment took twelve weeks and a battery of reaction time tests were performed before and after. Both interventions groups improved significantly on simple reaction time tests (p < .001) while the control group’s performance worsened non-significantly [2].
This graph shows the simple reaction times before and after the intervention for the three groups and is recreated from the paper. Because the absolute values were not explicitely given (only the differences), I had to guess them from the figure.
Another paper I found, Cranson et al., monitored two groups of students for two years and found that meditation improves reaction time [3].
Of course, two studies are not enough to give a high degree of certainty to show that reaction time is changeable and I couldn’t find a meta-analysis for this problem. Nevertheless, measuring reaction time is quick, simple and the best method I have found so I will just give it a try.