Wise Use of Null Hypothesis Tests: A Practitioner's Handbook

Front Cover
Wise Use of Null Hypothesis Tests
Copyright Page
Epigraph
Contents
About the author
What makes this book different?
This is not a mathematics book
Conventional books get it wrong
Ronald Fisher got it right, and his method is simple
What conventional books leave out
How to use this book
1. The conventional method is a flawed fusion
1.1 Three statisticians, two methods, and the mess that should be banned
1.2 Wise use and testing nulls that must be false
1.3 Null hypothesis testing in perspective
Notes
2. The point is to generalize beyond our results
2.1 Samples and populations
2.2 Real and hypothetical populations
2.3 Randomization
2.4 Know your population, and do not generalize beyond it
Notes
3. Null hypothesis testing explained
3.1 The effect of sampling error
3.2 The logic of testing a null hypothesis
3.3 We should know from the start that many null hypotheses cannot be correct
3.4 The traditional explanation of how to use p
3.5 What use of α accomplishes
3.6 The flawed hybrid in action
3.7 Criticisms of the flawed hybrid
3.8 We should test nulls in a way that answers the criticisms
3.9 How to use p and α
3.10 Mouse preference, done right this time
3.11 More p-values in action
3.12 What were the nulls and predictions?
3.13 What if p=0.05000?
3.14 A radical but wise way to use p
3.15 0.05 or .05? p or P?
Notes
4. How often do we get it wrong?
4.1 Distributions around means
4.2 Distributions of test statistics
4.3 Null hypothesis testing explained with distributions
4.4 Type I errors explained
4.5 Probabilities before and after collecting data
4.6 The null’s precision explained
4.7 The awkward definition of p explained
4.8 Errors in direction
4.9 Power and errors in direction
4.10 Manipulating power to lower p-values
4.11 Increasing power with one-tailed tests
4.12 Power and why we should we set α to 0.10 or higher
4.13 Power, estimated effect size, and type M errors
4.14 How can we know a population’s distribution?
Notes
5. Important things to know about null hypothesis testing
5.1 Examples of null hypotheses in proper statistics books and what they really mean
5.2 Categories of null hypotheses?
5.3 What if is important to accept the null?
5.4 Never do this
5.5 Null hypothesis testing as never explained before
5.6 Effect size: what is it and when is it important?
5.7 We should provide all results, even those not statistically “significant”
Notes
6. Common misconceptions
6.1 Null hypothesis testing is misunderstood by many
6.2 Statistical “significance” means a difference is large enough to be important—wrong!
6.3 p is the probability of a type I error—wrong!
6.4 If results are statistically “significant,” we should accept the alternative hypothesis that something other than the n...
6.5 If results are not statistically “significant,” we should accept the null hypothesis—wrong!
6.6 Based on p we should either reject or fail to reject the null hypothesis—often wrong!
6.7 Null hypothesis testing is so flawed that we should use confidence intervals instead—wrong!
6.8 Power can be used to justify accepting the null hypothesis—wrong!
6.9 The null hypothesis is a statement of no difference—not always
6.10 The null hypothesis is that there will be no significant difference between the expected and observed values—very, ver...
6.11 A null hypothesis should not be a negative statement—wrong!
Notes
7. The debate over null hypothesis testing and wise use as the solution
7.1 The debate over null hypothesis testing
7.2 Communicate to educate
7.3 Plan ahead
7.4 Test nulls when appropriate, not promiscuously
7.5 Strike the right balance between what is conventional and what is best
7.6 Think outside of the null hypothesis test
7.7 Encourage our audience to draw their own conclusions
7.8 Allow ourselves to draw our own conclusions
7.9 Strike the right balance when providing our results
7.10 Know the misconceptions and do not fall for them
7.11 Do not say that two groups “differ” or “do not differ”
7.12 Provide all results somehow
7.13 Other reformed methods of null hypothesis testing
Notes
8. Simple principles behind the mathematics and some essential concepts
8.1 Why different types of data require different types of tests
8.1.1 Simple principles behind the mathematics
8.1.2 Numerical data exhibit variation
8.1.3 Nominal data do not exhibit variation
8.1.4 How to tell the difference between nominal and numerical data
8.2 Simple principles behind the analysis of groups of measurements and discrete numerical data
8.2.1 Variance: a statistic of huge importance
8.2.2 Incorporating sample size and the difference between our prediction and our outcome
8.3 Drawing conclusions when we knew all along that the null must be false
8.4 Degrees of freedom explained
8.5 Other types of t tests
8.6 Analysis of variance and t tests have certain requirements
8.7 Do not test for equal variances unless …
8.8 Simple principles behind the analysis of counts of observations within categories
8.8.1 Counts of observations within categories
8.8.2 When the null hypothesis specifies the prediction
8.8.3 When there is only one degree of freedom
8.8.4 When the null hypothesis does not specify the prediction
8.9 Interpreting p when the null hypothesis cannot be correct
8.10 2×2 Designs and other variations
8.11 The problem with chi-squared tests
8.12 The reasoning behind the mathematics
8.13 Rules for chi-squared tests
Notes
nine The two-sample t test and the importance of pooled variance
10. Comparing more than two groups to each other
10.1 If we have three or more samples, most say we cannot use two-sample t tests to compare them two samples at a time
10.2 Analysis of variance
10.3 The price we pay is power
10.4 Comparing every group to every other group
10.5 Comparing multiple groups to a single reference, like a control
10.6 Is all of this a load of rubbish?
Notes
11. Assessing the combined effects of multiple independent variables
11.1 Independent variables alone and in combination
11.2 No, we may not use multiple t tests
11.3 We have a statistical main effect: now what?
11.4 We have a statistical interaction: things to consider
11.5 We have a statistical interaction and we want to keep testing nulls
11.6 Which is more important, the main effect or the interaction?
11.7 Designs with more than two independent variables
11.8 Use of analysis of variance to reduce variation and increase power
Notes
12. Comparing slopes: analysis of covariance
12.1 Analysis of covariance
12.2 Use of analysis of covariance to reduce variation and increase power
12.3 More on the use of analysis of covariance to reduce variation and increase power
12.4 Use of analysis of covariance to limit the effects of a confound
Note
13. When data do not meet the requirements of t tests and analysis of variance
13.1 When do we need to take action?
13.2 Floor effects and the square root transformation
13.3 Floor and ceiling effects and the arcsine transformation
13.4 Not as simple as a floor or ceiling effect—the rank transformation
13.5 Making analysis of variance sensitive to differences in proportion—the logarithmic transformation
13.6 Nonparametric tests
13.7 Transforming data changes the question being asked
Notes
14. Reducing variation and increasing power by comparing subjects to themselves
14.1 The simple principle behind the mathematics
14.2 Repeated measures analysis of variances
14.3 Multiple comparisons tests on repeated measures
14.4 When subjects are not organisms
14.5 When repeated does not mean repeated over time
14.6 Pretest-posttest designs illustrate the danger of measures repeated over time
14.7 Repeated measures analysis of variance versus t tests
14.8 The problem with repeated measures
14.8.1 The requirement for sphericity
14.8.2 Correcting for a lack of sphericity
14.8.3 Multiple comparisons tests when there is a lack of sphericity
14.8.4 The multivariate alternative to correction
Notes
15. What do those error bars mean?
15.1 Confidence intervals
15.2 Testing null hypotheses in our heads
15.3 Plotting confidence intervals
15.4 Error bars and repeated measures
15.5 Plot comparative confidence intervals to make the overlap myth a reality
Notes
Appendix A Philosophical objections
A.1 Decades of bitter debate
A.2 We want to know when we are wrong, not how often
A.3 Setting α to 0.05 does not mean that 5% of all null-based decisions are wrong
A.4 There are better ways to analyze and interpret data
A.5 The fallacy of affirming the consequent
A.6 Some say our method cannot be used to determine direction
A.6.1 The return of one-tailed tests
A.6.2 Kaiser’s absurd directional two-tailed tests
A.6.3 Invoking power to justify Kaiser’s directional two-tailed tests
A.6.4 Fisher did not follow Kaiser’s rules
A.6.5 Still not convinced?
Notes
Appendix B How Fisher used null hypothesis tests
B.1 Why follow my advice?
B.2 Fisher tested for direction
B.3 Others did too
B.4 Fisher believed α should vary according to the circumstances
B.5 Fisher came close to saying there should be no α at all
B.6 In practice, Fisher did not categorize outcomes
B.7 Fisher’s language answers many criticisms of null hypothesis testing
B.8 Except for Fisher’s use of “significant”
B.9 Fisher’s inconsistency explained
B.10 Fisher’s thinking expressed in one word
B.11 We have come a long way since Fisher, but the wrong way?
Notes
Appendix C The method attributed to Neyman and Pearson
C.1 Neyman and Pearson with Pearson
C.2 Neyman and Pearson without Pearson
C.3 An important limitation
C.4 Alternatives are always infinitely numerically precise
C.5 The method step-by-step
C.6 The method’s influence on the flawed hybrid
C.7 The method’s fate in the world of the flawed hybrid
C.8 Power spreads its wings
C.9 Neyman et al.’s method has no place in science
Notes
Index
Back Cover