Can You Trust the 'Insights' Your Habit Tracker Shows You?

11 min read · Updated 2026-09-05

Every correlation feature in a tracking app runs the same kind of test: compare two piles of days and check whether the gap between them is bigger than noise. Run that test dozens of times and some gaps will look real by pure chance alone. We measured it on our own engine, then rebuilt five parts of it once we saw the number it came back with.

We fed our own correlation engine four hundred days of habits and medications logging nothing but random numbers, none of them connected to anything else, on purpose. It came back with a full list: 24 'patterns,' the most it will ever show at once. Several were labeled strong.

Every correlation feature in a habit or symptom tracker answers the same underlying question: on the days this happened, was that different? Split your days into two piles, compare the average outcome across them, check whether the gap is bigger than you'd expect from noise. It's a good method. Bearable popularized it for symptom tracking, and we've written before about why it beats staring at two wiggly line charts.

What that method doesn't protect you from, on its own, is asking the question too many times. One of the 24 was a habit rated strong for supposedly moving another habit, built on six days where the two happened to line up. Another was a medication rated strong for moving mood, compared against only four days where no dose had been logged at all. None of it was real, because there was nothing in the data to be real. The engine reported it anyway.

One test, run enough times, always finds something

Run one comparison at the usual 5% significance threshold and a coincidence has a 1-in-20 shot of looking real. Track ten habits and check each one against mood, sleep, and weight change, and you've run somewhere around sixty comparisons before lunch. At that threshold, you should expect about three of them to come back looking significant even if nothing in your life is connected to anything else.

That's not a flaw in the math. The math is doing exactly what you asked it to. The flaw is treating a p-value earned by asking sixty questions the same as one earned by asking a single question you actually cared about.

The fix has a name: false discovery rate control, and we've covered the mechanics of it elsewhere. What surprised us running it against random data is that the correction alone wasn't enough.

The correction wasn't the whole fix

Benjamini-Hochberg adjusts each finding's p-value into a q-value that accounts for how many comparisons ran alongside it. It was already in the engine before we ran this audit, and it's the right tool. It still let patterns through, because a q-value under the usual 0.05 cutoff isn't the only bar something can clear. Anything under about 0.25 got labeled a moderate pattern, and on a big enough batch of noise, some q-values land right in that gap by construction, not by mistake.

So the cap came down and the sample-size floor went up. Nothing gets compared with fewer than five days on each side now, up from four. And a finding needs both an uncorrected p-value under 0.05 and a corrected q-value under 0.25 before it shows up anywhere, not just a gap that looked big enough on its own.

A coin flip isn't a p-value of zero

There's a second kind of question buried in this list that a plain average-of-two-piles test can't answer honestly: did one habit happening also mean a second one got done that day? That's not a number with an average and a spread. It's a yes or no, counted twice, once per pile. Treating those counts as 0s and 1s and running the same test built for continuous numbers works fine at forty days a side. At five, it doesn't.

We had exactly that bug. Five days where a habit got done every single time, next to five days where a second habit never did, computed to a p-value of exactly zero. Certain. Except five wins in a row against five losses in a row is a real thing that happens by pure chance about 1 time in 126, which is nowhere near certain.

Fisher's exact test asks the right question directly: given how many total yeses there were, how many ways could they have landed the way they did? A concrete version, ten days logged, meditation happening on 8 of 10 gym days and 2 of 10 other days:

Fisher's exact test on that table comes back at about 2.3%, meaning a split this lopsided happens roughly 1 time in 43 by chance. Worth noticing. Run the same test on the five-wins-versus-five-losses case from earlier and it comes back at about 0.8%, not zero, which is the honest number and still enough to flag as a lead worth watching, not a certainty worth acting on.

A three-day gap between two habits doesn't mean anything

The engine can also test whether an effect shows up later rather than the same day: does today's workout move tomorrow's mood, does it show up three days later, a week later. Scanning every delay for every pair used to be standard here, checking eight different shifts, zero through seven days, for every single comparison.

On real habits and medications that's a reasonable thing to want. On random data it's a lottery. There's no mechanism by which one made-up habit should predict another three days later, so when the scan found one, that's what a lottery looks like: the winning number is uniformly random, and on the random data we tested, the winning lag really was spread evenly across zero through seven.

The fix bounds which delays get tested by what could plausibly carry an effect at all:

Behavior acts today or tomorrow. A skipped workout that's still quietly moving something three days later has already passed through the same-day and next-day comparisons on the days in between. A medication or a supplement is different: those can take weeks to show anything at all. We've written before about when a lag makes sense and when it doesn't; this is the part that now enforces it instead of asking a person to eyeball it. Fewer delays tested means fewer chances to get lucky, which is most of why the count of comparisons on the same random data dropped from around 3,800 to about 1,000.

The bug that was staring at us

The last one wasn't a statistics problem. It was a bookkeeping one, and we only found it by looking for the other ones.

Picture an account thirty days old, being analyzed over a sixty-day window, on a medication taken every single day since signup. At any of the longer delays the engine tests, from a week out to a month, there are no days anywhere in the window where the medication wasn't taken. The only days without a dose logged are the days before the account existed. Comparing those two piles reported the medication moving mood at one of those delays. What it had actually found was the difference between a month with an account and a month without one.

The fix is one sentence: a day with nothing logged at all can never count as evidence that something didn't happen. It just means nobody was there to write it down.

What changed, measured

We ran a fresh batch of pure noise after every fix landed: twenty habits and five medications logging random numbers across four hundred days, and separately a smaller batch of ten habits and three medications, to check the fix at a different scale. Strong patterns, the ones cleared to lead a sentence with real confidence, came back at zero in every run, at every window we checked. Moderate patterns, which are allowed a real error rate by design, up to one in four, still turned up once or twice in some runs and not at all in others. That's not a residual bug. That's what a one-in-four tolerance is supposed to look like on data with nothing in it.

So how much do you have to log before any of this works?

Tightening the bar is only half an answer. The other half is what it costs you, and we didn't have a number for that either, so we measured it the same way. Plant one habit with a genuine effect on mood, surround it with habits that do nothing, and count how often the engine catches the real one. A hundred and twenty runs per combination.

The effect we planted is the size of a good gym day, about 1.5 points of mood. Read this as the share of runs where the engine found it:

Two things fall out of that table. The first is that three weeks is not a slow start, it's nothing, at every width we tested. The second is the price of tracking more: at sixty days, three habits catches a real effect 88% of the time and twenty catches it 43%. Same person, same effect, same two months of logging, half the chance of hearing about it.

So the practical recommendation is three to five things you genuinely care about, if you want answers inside a couple of months. Track twenty and you are not getting more insight, you are getting a longer wait for the same one.

The cost is sensitivity, though, not lies. Across every run above, habits that did nothing were reported as moving mood between 0.00 and 0.19 times per run, and that number went down as the habit count went up, because the correction gets harsher as the list grows. Adding variables doesn't make the app invent more. It makes it go quiet.

Two caveats we would rather state than bury. A subtle effect, something like half a point of mood, stays out of reach on any normal timeline: 46% at a hundred and twenty days with three habits, 13% with twenty. And every number above assumes a habit done on about half of days, which is the easiest case there is. A habit you manage 85% of the time leaves a very small pile to compare against, and does worse than this.

Sixty days to measure a habit, sixty-six to build one

There is a number we deliberately refuse to give you, and it sits right next to this one. How long a habit takes to become automatic came out at an average of 66 days in Lally's field study, with individuals landing anywhere from 18 to 254. A fourteenfold spread. We won't promise you a formation date, because the research doesn't support promising anyone one.

The detection number is a different kind of claim, and that's why we'll put it in writing. How long your habit takes to form is biology. How long until there is enough data to test what it did is arithmetic, and arithmetic generalizes.

Worth noticing that they land in the same neighborhood. By roughly the time a habit stops being a decision, you have roughly enough history to ask what it changed. Those are two unrelated processes that happen to converge, not a law, but it does make the first two months a coherent stretch: build the thing, then find out.

It also means your first month of data on a brand-new habit is measuring a behavior that is still changing shape. That's another reason the early answers come back empty, on top of the sample size.

What none of this fixes

A pattern that survives all of this is still a pattern, not a cause. If your mood is higher on days you go to the gym, that's consistent with the gym helping and equally consistent with good days being the ones you make it there in the first place. Nothing in a p-value or a corrected q-value settles which. The honest next step is the boring one: change one thing on purpose, leave everything else alone, and watch what happens over the following weeks. We've written about running that experiment properly.

It also can't manufacture a finding out of three weeks of data. A brand-new account clears none of these bars for most pairs, and the honest answer is 'keep logging,' not a pattern dressed up to look like one. That's meant to feel disappointing on day four. It's the same discipline that kept the random data honest.

• It can't tell an antidepressant from a blood pressure medication. A dose is a dose. What it treats is something only you know, so a medication effect never gets ranked as more or less expected than any other one until you say so.
• It won't call a next-morning weight drop progress if the same drop is back within a week. That's water, not fat, and the engine checks the week before it colors anything green.

None of this is visible from the outside. Every one of these fixes shows up as a card that no longer appears, a lag that no longer gets offered, a 'strong' that quietly becomes nothing. The measure of whether it worked isn't a feature you can screenshot. It's a habit tracker that stops telling you things about yourself that a coin flip could have told you just as easily.

If you want to see this in the app, here is how HabitSync tracks habits.