His measure of how well rested he is is completely subjective. There's obviously no blinding (as he's subject and experimenter) which makes it almost certain that as he knows the hypothesis he's looking into and the measure is subjective, he's going to either subconsciously or consciously push the data towards proving or disproving it based on his preconceived ideas.

He talks about randomly choosing (possibly sloppy language but if you choose in any way it's not random, regardless it's not clear) and he's failed to adjust for or control any other variables that might influence sleep like, oooo, alcohol or caffeine or stress or a whole bunch of things that will make more of a difference than standing on one leg which given the relatively small data set is a pretty big deal (again some of this may depend on the method of randomisation).

And then the variation is within fractions of one percent (the scale showing 99 and 99.4) which given that the measure is subjective is essentially insignificant.

Please, please tell me that this is a wind up about how people believe anything scientific looking and trying to get people to stand on one leg for lulz rather than something serious?

Given that what he's doing is basically exercise and that linking sleep and exercise makes some sense, it's not a totally stupid hypothesis, but his experiment proves nothing.

An experiment like this one is entirely appropriate when trying to decide whether a hypothesis is worth exploring further; setting up a more elaborate study would have been premature.

So when you say "his experiment proves nothing" I think you are judging it by standards it was not, and should not have been, intended to meet. Don't focus only on the results of science; without the process, there would be no interesting results.

My issue is that he proceeds to draw very strongly worded conclusions (using the phrase "beyond doubt") based on very weak evidence and presenting the experiment in a way which both fails to highlight the inherent flaws and indeed hides them behind a veil of pseudo-scientific and statistical language.

Yes explore hypotheses, present anecdotal evidence, but make it clear that's what you're doing and explain why. The next step here isn't to try 5 and 6 leg stands, it's to try it with people who don't understand what is being tested and remove the worst of the flaws in the methodology.

As for being hung up on "proof", yes guilty as charged, but that's really not my issue which is don't claim proof when you don't have it. Which in claiming things "beyond any doubt" (which he does) this is what he is doing.

The statistical analysis of the study does not provide any additional weight to the statement. In fact, the statistical analysis is more of a distraction because he didn't control for an important influence in self-evaluating health. If a person expects that taking an action will improve their health, it's very likely that the person will think he feels better after taking that action.

And while this document cannot reflect this, I've been following Seth Roberts for a few years now -- he is the best kind of scientist, with a remarkable talent for objective measurement of oneself. While this cannot be generalized to other people easily, as far as it refers to himself, it's probably way better than most medical results published in the last 20 years.

Let's be realistic. The vast majority of evidence we use to update our beliefs is not of scientific caliber. (Just imagine how crippled a reasoner someone would be if they never changed their mind about anything unless they saw scientific evidence that indicated they should. Pivoting a startup would be very difficult, for instance.) There is a continuum of evidence quality, and we should reward, not punish, this guy for collecting higher-quality evidence even if he didn't get all the way to "scientific".

If you don't think he's intending this then you need to explain away why he uses the phrases "experiment" and "a randomized experiment" several times (and indeed the language throughout is clearly intended to convey scientific rigour), outlines an experimental methodology (which he contrasts with his mere anecdotal data), fills the pages with graphs showing the mean and error bars and uses language that imply statistical significant ("one-tailed p = 0.02"), uses the "results" to draw a conclusion beyond "any doubt" and then seeks to explain what has happened in scientific terms ("to allow the signaling molecules to be regenerated" - I'm struggling not to giggle at that incidentally).

Normal people don't talk like that unless they're trying to sound scientific.

The intention here is clearly to imply some level of authority in his results based on something which is either intended to be or intended to look like scientific method when it is, in scientific terms utter nonsense and proves nothing at all.

I have no issue with him suggesting his theories to people, I do have an issue with him, either as a result of ignorance or intentional deceit, dressing it up as in any way authoritative.

He's written some papers that talk about the methodological concerns. Here are two that Google churned up:

http://sethroberts.net/articles/2010%20The%20unreasonable%20...

http://escholarship.org/uc/item/2xc2h866

I can't honestly claim that he has successfully made n=1 experiments respectable, but I think it's safe to say he's very well aware of the criticism on these grounds and is not ignoring it.

I don't respect your skepticism, unless you are just as skeptical about advice you get from your Doctor or pharmacist.

Roberts looks for effective, simple treatments that you can try yourself without requiring a large budget or large lab, publishes everything without trying to make money of it -- including a diet-free diet that works remarkably well, that flies in the face of all "nutrition science" common sense. He is not the crackpot you think he is.

But if you read an article (as in this case) which has obvious flaws, not to point them out just because you can't research every drug or treatment a doctor gives me would clearly be absurd.

And I'm not saying he's a crackpot, I'm saying that in this instance he's carried out an experiment with a flawed methodology and drawn overly strong conclusions from it. The can be a (former) professor at Berkeley or a high school drop out, it doesn't change the weaknesses in what he's written in that article.

Not everything is physics. And I don't think it's intellectually optimal to flag everything that can't be physics as simple nonsense.

I'd rather acknowledge and talk about methodological weaknesses when we discuss how persuasive the experiment is. I mean, it's not like an experiment that does everything "right" is always perfectly persuasive either. Reality is complicated; sometimes you were asking the wrong question. Most things are neither ironclad nor nonsense, but fall somewhere in the middle.

So, two things about subjective measurement. First, just because something's subjective doesn't mean it's totally unreliable. You can fool yourself about small things; large things are harder. Suppose you're trying to demonstrate the presence of a material by the color of a flame. If you're trying to distinguish between pale green and pale pale green, better break out the spectrometer. But calling whether it's yellow or blue? Qualia will suffice.

Applied to this case, I think I would rely on someone to be able to answer the question, "Are you well-rested or not?" with a "yes", "no", or "sorta". On the other hand, a 99 vs a 98 percent measurement -- as done here -- is pretty darn iffy.

The second thing about subjective measurements is that, when you're after subjective results, that's the right sort of measure. For example, if someone says, "When I spend time with friends, I feel like I do better on math tests the next day," you might rightly respond, "We should be measuring the scores, not your feelings. It's possible your friends are making you feel competent when you really aren't." But if someone says, "When I spend time with friends, I feel happier," does it even make sense to say, "It's possible your friends are making you feel happier when you really aren't?" The best you can do is try to define degrees of happiness and measure statistically.

When the feeling is the object of the discussion, the subjective isn't an illusion. A pill which makes you feel competent when you aren't is producing an illusion, but a pill that makes you feel happy when you want to is a pill that works.

(This likewise reduces the point of blinding. If we're trying to figure out whether a chemical makes you happy, it makes sense to run it against a placebo. But if we're trying to figure out whether an activity makes you happy, being aware that you did it is appropriate. Whether it's the physical motion or the awareness that makes you happy is immaterial to whether the activity works.)

All of that is to say, if the guy is trying to prove he is well-rested in the sense that he performs better, he reacts more quickly, his judgement is more sound, all the various things we associate with being well-rested, he should be taking tests. But if he is trying to feel well-rested, he should be asking, "How well-rested do I feel?"

Now, it's hard to say exactly which he's after -- feeling well-rested but possibly performing like crap, or feeling like crap but possibly performing well. I'd think he probably wants both things. I certainly would. So I think he's only measuring half of what he should. The result could certainly be made more complete by doing something like the coffee guy did (http://quantifiedself.com/2009/10/the-false-god-of-coffee/).

It is promising that he measured how long he slept before first waking up -- that's pretty objective -- but I can't tell exactly what he's saying he did. (I can't tell what he means by baseline(3) and random(3), so I can't really critique the results).

It would be nice if he'd chosen the exercise randomly for the random portion of the experiment. By just picking one, there's a serious possibility he was picking based on how tired he already felt due to events of the day (it is exercise he's signing up for, after all.) I don't know that it reduces those results to nonsense, but it certainly does reduce them to possibly nonsense. I think the best you can get is, "Huh, that's interesting enough to investigate--do it again, and use a die this time."

And then we have the obvious criticism: he only tried it on himself. Whether it works only on him for psychological reasons relating to his life history, or whether it only works on some people for genetic reasons, or what . . . remains to be seen.

On the whole, I'd give it about a five out of ten for methodology. I do like that he tried the exercise over a period of months, kept notes, and looked for patterns. That's worth a lot of credit right there. I think it could be improved by stating more clearly what he's trying to prove, by incorporating objective measurements where possible and keeping subjective measurements as clear-cut as possible -- though that sort of thing is really more appropriate for the next iteration, anyway. There's investigating to find interesting patterns, and then there's setting out to demonstrate the truth of what you think you've found. An investigative experiment -- which this certainly was -- is inherently not going to be a demonstrative one.

I wouldn't mind clearer labeling of the charts, and I definitely wouldn't mind access to the raw data. That would bolster the case a lot.

And a repeat would be great -- someone else should try it and see if it works. Ideally several people.

Overall, though, it's far from nonsense, and I certainly wouldn't put it in the "trying to get people to stand on one leg to make a point about how gullible we are when we see graphs" category. I'd mark it investigative/interesting.

Wouldn't it be easier to just exercise? Regular physical activity leads to better sleep, and doesn't require you to stand on one leg.

http://en.wikipedia.org/wiki/Isometric_exercise

"While dynamic exercises are slightly better than isometric exercises at enhancing the twitch force of a muscle, isometrics are significantly better than dynamic exercises at increasing maximal strength at the joint angle. Flexibility may be increased when isometrics are performed at joint range of motion extremes. These isometric contractions recruit muscle fibers that are often neglected in some dynamic exercises. For example, gymnasts are extremely strong at great ranges of motion through the practice of isometric holds."

It would be interesting to find out if he would get better or worse results through cardio instead of working a muscle to exhaustion, and whether or not it is important that the legs be that muscle. Some people suffer from restless legs syndrome which keeps them from achieving deep sleep, and maybe exercising the legs is countering this.

(I wish there was some way he could experiment that wasn't subject to the placebo effect, though, which makes the results moot when applied to anyone else.)

Even if you stick to the legs, exercises like bootstrappers, squats, and lunges would be significantly better.

One legged standing could be just a good! I'll definitely try it out.