Hypothesis Testing

Introduction

This is just a brain dump on mainstream state of the art statistical practices/procedures in scientific (and non-scientific) research.

The basic structure/pipeline/skeleton of research often start with some kind of assumption/interest/hypothesis in something humans want to test. The naive (bone-headed) procedure involves:

• Construct a null hypothesis (That you know before doing anything is false)
• Simulation test/power analysis (goes here but I am not going to talk about it. Probably not aware what this is until a reviewer tells you to do it after submitting the manuscript)
• Collect data (Not aware experiment design is a thing)
• Based on the collected data and accept/reject based on some statistics (with stars attached to values, without knowing what the real conclusions are).

(Excellent, you just completed an experiment and ready to submit a manuscript to a high profile journal for publication)

The more (correct) general term of this procedure is Statistical Inference. What are the properties and things you can infer and learn about the population by analysis the data. Statistical inference is very deep and majority of the humans are not aware of all the assumptions these test are making when they are performing statistical analysis.

Here, I am going to talk about two statistics of inference that people often mis-use and over-rate.

P-values

• talk about star values
• doing star-tistics

Information Criterion

• AIC, BIC, DIC, ICL, … likelihood based ICs
• I am more interested in talking about the plauge paper and delta BICs
• See panthers paper conclusion? (Li and Bolker 2017)? The more recent animal movement lit often talks about this problem.

Do we need to invent a all purpose (new) statistics?

• P-values and ICs are perfectly fine
• Why not get rid of it all as Gelman said?
• Effects, Confidence Interval, S+M baby!

Tips

• Keep a statistical journal
• Be Honest (so readers are aware you are doing stupid mistakes and not try to get away with clever writing)