The dataset consists of conversations between dyads that took place while they completed two different interactive tasks. The conversations were recorded, transcribed, and segmented into utterances according to some criteria. This means that there are repeated utterances from each participant, and from each dyad. Different research areas use different terms to refer to this kind of setup: repeated measures, panel data, clustered data, etc. The analysis is further complicated by the fact that the predictors and variables are all categorical. Some are binary, the presence or absence of something. The more interesting variables have more than two categories (in some cases, MANY more).

I am trying to estimate the strength with which each of a set of 15+ utterance goals is associated with one of three roles participants assumed as part of the study. To do this, I need to specify a mixed-effects multinomial logit model, with a set of fixed-effects categorical predictors and a hierarchical random effects control for participant within dyad. This involves choosing a reference category of the response variable, and then running a series of binomial logit models that compare all the other levels of the response variable in turn with the reference category.

Here is where I am running into a situation, again, where I am pushing up against what mainstream statistical software packages are reliably capable of, and even R does not seem to be able to do what I want without more programming than my meager statistical background has prepared me for. The problem as I understand it is, each one of the binomial logit models that makes up the multinomial results uses a different subset of the data, excluding those observations that are related to the levels of the response variable not included in the model. This means that the random effects are estimated differently for each binomial logit model, depending on which observations are included in the subset. The upshot of all of this is the overall multinomial model estimates come out differently, depending substantially on which category is chosen as the reference category.

So that’s the problem. However, I did not write this to whine about how I am stuck. I’ve been trying to figure out a solution that I can live with… do I bail completely? Hire a real statistician? How can I figure out how biased the results would be if I were to to do a purely fixed-effects model? (Without random effects controls, any results produced might in fact be due to some unique aspect of the conversation within a particular dyad in a particular role, rather than indicative of something that shows up across all of the dyads.)

Researchers in many fields work with categorical data, and at least some of them over the years must have encountered this problem, whether they knew it or not, and were faced with the same tradeoffs. In order to get the paper out the door they had to just pick a compromise and go with it. But, any results reached due to a compromise are biased in some way. Models like this are just now becoming possible for people like me, with just enough stats knowledge to be dangerous, to run using fairly standard statistical software packages. But what about all the research that has come before — how accurate are those models, and the results they produced? How much do people allow what is statistically feasible to determine their research design, vs. compromising on the analysis after the fact? We all stand on the shoulders of giants, but how often were the giants using naive or incorrect statistics?

My second reaction to the paper has to do with the distortion of the effect size as it moved from journal paper to book to popular press to blog entry. It is easy to dismiss this as just an instance of the “telephone game” —the message is not replicated exactly as it moves from one venue to the next, and the distortions are biased in the direction of making the results sound stronger and more controversial and therefore more interesting and worthy of attention. Nobody wants to read a blog entry that says, “Yep, in line with previous research, beautiful people are maybe 4.3% more likely to have female babies, but that result isn’t statistically significant so maybe there’s no noticeable effect at all.” Well, possibly nobody except people who are really interested in this type of research, I guess.

An interesting thing to me about this example is that the results were published and re-published not by some random blogger (like me!), but in reputable venues: the Journal of Theoretical Biology http://www.elsevier.com/locate/yjtbi, the Freakonomics blog on the New York Times website http://freakonomics.blogs.nytimes.com/, and Psychology Today http://www.psychologytoday.com/magazine. The author of the original paper, Satoshi Kanazawa (http://personal.lse.ac.uk/KANAZAWA/), even ended up writing a book about it, with Alan S. Miller, titled “Why Beautiful People Have More Daughters”. In each case, the quality filter was broken, but the reputation of the venue vouched for the results just the same.

It is easy to argue that these venues should just be more careful in the future about “quality control”; they should insist that reviewers consider statistical power and effect size, or do a better job of tracking down related work so they would be more likely to spot an effect size that was out of whack. However, it is more difficult to think about the role of the social structures in enhancing the validity of the information, and in communicating to readers that they don’t need to do any of the legwork themselves.

Thinking about the social structures that enabled and even encouraged this distortion to happen, it seems like similar reputation effects and influences exist in social media systems. If I’d posted a link to the Freakonomics story, for example, either to delicious.com or Facebook or Twitter, would anybody have thought twice about the excessively large effect size that was reported? How about the people who found it interesting enough to re-post? Would the people who found it interesting enough to re-post maybe have something in common, perhaps that gave them a more credible source for this kind of information than a random user? These contributions make up the information infrastructure of the so-called “social search” phenomenon, and yet we understand very little about the social forces that shape this infrastructure.

Another implication of this example just occurred to me while writing this post: the problem with playing around analyzing available datasets without knowing the related literature. I think the recent emphasis on more data as the answer to everything could mean that people who don’t know much about an area are suddenly immersed in a dataset fishing for results they aren’t necessarily equipped to interpret accurately. If Kanazawa truly didn’t know the sex differences literature, how might he be expected to know that an 8% effect is really incredibly large (aside from the multiple comparisons problem).

I recently read this short paper about statistical power, and how to interpret non-significant results:

Gelman, A., & Weakliem, D. (2009). Of Beauty, Sex and Power. American Scientist, 97(4), 310-316. [ PDF, on Gelman's blog ]

I’m having two very different reactions to the paper:

1. Why didn’t anyone talk much about statistical power and effect size in my intro stats classes?

2. There are implications for social media here, and I don’t just mean the “telephone game“.

The paper is only 7 pages long, so if you ever do any kind of statistical analysis, or even if you just read news stores about science results, you really should read this paper. It presents an example of what I think is a common problem in the reporting of statistical results in scientific journals—when analyzing a dataset, not enough consideration is given to what effect size one might expect for a given phenomenon. It is very important to think about (and explicitly report) statistical power, because as the authors write, “‘underpowered’ studies are unlikely to reach statistical significance and, perhaps more importantly, they drastically overestimate effect size estimates. Simply put, the noise is stronger than the signal.”

As an example of this, the paper critiques the interpretation of statistical results published by Satoshi Kanazawa, from which he drew the provocative conclusion that “beautiful people have more daughters.” Incidentally, according to his website he also has a new book coming out tentatively titled, “Escaping Biology: Why Intelligent People Are the Ultimate Losers in Life”.

Kanazawa took a publicly available dataset, the National Longitudinal Study of Adolescent Health, and did an analysis that consisted of multiple pairwise comparisons between 4 subjective “attractiveness” categories (ratings were made by the interviewers who collected the data). The results he reported came from a comparison of the group with the highest attractiveness vs. everyone else; this comparison yielded an 8% difference, or “…a 52 percent chance of girl births for the parents in the highest attractiveness category, compared to a 44 percent chance for the average of the four lower categories” (p311). However, Gelman & Weakliem did their OWN analysis of the SAME data—this time a more appropriate linear regression—and found a 4.7%, non-significant effect.

Gelman & Weakliem also did a review of other published research investigating factors that affect whether a baby will be born male or female, and found that effect sizes tend to be exceedingly small, “typically less than one percent” (p311). So they were surprised to see the following report of Kanazawa’s results in the Freakonomics blog: “…good-looking parents are 36% more likely to have a baby daughter as their first child than a baby son…”.

They argue that anytime you see an effect size that big, you should be skeptical, for two reasons. The first is that “…studies with insufficient statistical power will spit out random results that will occasionally be statistically significant and, even more often, be suggestive…” (p315). The second is that “…most of the low-hanging fruit in social science research has presumably been plucked, leaving researchers to study small effects” (also p315). So, in other words, even underpowered studies will produce statistically significant results from time to time, but these results should not be trusted—especially social science results!

Gelman & Weakliem also make an important point about the scholarly publishing process that allowed these results to be published, repeatedly and with increasing visibility and distortion: “…the papers managed to survive the review process because reviewers did not recognize that the power of the studies was such that only very large estimated effects could make it through the statistical-significance filter. The result is essentially a machine for producing exaggerated claims…”

So what we have here is a paper that presents unbelievably large effect sizes that got through peer review. For all the attention I pay to the statistics in papers I review, I can’t say that I’ve ever paid that much attention to statistical power. I definitely will be looking more closely at it in the future, and being more explicit about it in my own work. But I also continue to be disappointed in most intro stats education; from what I remember, the parts about probability and the logic of significance testing and practical vs. statistical significance take a back seat to the mechanics of it—which formula should be used where, etc. Maybe this is unavoidable, and students can’t understand the application and interpretation of statistics until one masters the arithmetic. But I think the application and interpretation parts are the most fun!

In my next post, I’ll write more about my second reaction to the paper.

Judging quality remains a difficult task for the initial reviewers, but also for the best paper award committee. Despite its honest efforts, the best paper award committee has not selected papers that are cited more often than other papers. In other words, the best paper award committee did not perform better than random chance…

We may speculate whether the task of selecting papers that will be highly cited in the future is simply too difficult, if not impossible. In any case, we do have to ask ourselves what the purpose of the award is if it does not correlate with the views of the HCI community. We speculate if it might be worthwhile considering whether the conference attendees should be allowed to vote for the best paper.

I’m not actually sure whether the controversy was due more to disagreement with the speculation that the best paper award might be worthless as a signal of quality (i.e, “did not perform better than random chance”), or because the statistical analysis was flawed. I remember both points being raised during the discussion of the paper at the conference, although at the time I was more focused on the statistical arguments. But I’ve since come across an interesting paper comparing various scientific impact measures, and so I’m revisiting both criticisms of the paper.

1. The stats are used incorrectly

The conclusion that the CHI best paper nominees and recipients are no better than a randomly selected set of papers was based the results of an ANOVA that compared three samples: the nominees, those nominees selected to receive the award, and a random sample of papers, all from 2004-2007. The dependent variable was number of citations each paper had received.

There’s little chance these data are normally distributed. In fact, this would be extremely bizarre if it were true. The paper includes evidence supporting this argument: “Figure 6 shows that there are not only one or two outliers, but a considerable number of them.” This is because these are count data, and distribution is likely power-law shaped: a couple of papers get a comparably large number of citations, and most are only cited a couple of times. The “considerable number” of outliers signals that ANOVA is probably not the best statistical choice. It just doesn’t make sense to use statistics based on a comparison of means and variances, when the mean does not represent the “central tendency” of the distribution.

Also, the sample sizes of the three groups in their ANOVA are extremely unequal (normal=76, nominee=64, winner=12). Their choice of n=76 for the “normal” paper group seems like they tried to create equal groups—but this would only have been true if they lumped together the nominee and winner groups (64+12=76) and compared those with the random set. This is another problem that causes ANOVA results to be biased. So it isn’t clear how accurately these results represent what’s going on in the data.

My favorite moment of the panel discussion was when one of the panelists suggested that statistics based on the Poisson distribution would be more appropriate. My memory (almost a year later) is that the presenter (the first author of the paper) responded with something like “the majority of the CHI audience probably doesn’t know what that is” and then asked the audience “how many of you have heard of the Poisson distribution?” From where I was sitting, it looked like maybe half the audience raised their hands. It is unlikely that the attendees of this session were a representative sample of the CHI community. But still, it gave me hope that CHI is becoming more statistically sophisticated.

2. The “quality” measure doesn’t measure quality

Most of the analyses reported in the paper use the H-Index, but this is an author- or organization-based indicator, not a paper-based indicator. For the best paper analysis, Bartneck & Hu used the simplest possible measure: “number of citations”. However, the paper does not include an argument for how this measure might be related to quality, or in what ways it might be a biased.

At the session last year, I wasn’t sure how I felt about using “number of citations” as a proxy for quality. It certainly is an operationalization that makes the data easy to collect. However, after reading Bollen et al. (2009), “A Principal Component Analysis of 39 Scientific Impact Measures” I really wish the Bartneck & Hu paper had given this operationalization more attention. Bollen et al. point out:

In fact, we do not even have a workable definition of the notion of “scientific impact” itself, unless we revert to the tautology of defining it as the number of citations received by a publication. As most abstract concepts “scientific impact” may be understood and measured in many different ways. The issue thus becomes which impact measures best express its various aspects and interpretations.

Just as the title says, Bollen et al. presented a PCA of a bunch of scientific impact measures, essentially illustrating how related several types of measures are to each other (see the paper if you want to read more about the different types of measures—it’s really interesting). They plotted the measures according to a solution that included just the top two components. Their qualitative interpretation of this graphical representation is that one component represents rapid vs. delayed impact, and the other represents popularity vs. prestige. Citation-based measures fall in the delayed popularity quadrant of the graph; “number of citations” is the simplest citation-based measure.

It makes intuitive sense to me that number of citations is a proxy for delayed popularity. As such, it seems unlikely that the best paper committee is making selections based on their projection of future popularity. Because nomination is part of the review process, it seems like more of a merit-based award to me.

What the Bartneck & Hu best paper critique may be indicating, if you accept the argument that the people who are making the nominations and selections are experts qualified to evaluate merit, is that popularity (measured by number of citations) and merit (as recognized by experts) are unrelated. Which to me, is actually a much more interesting interpretation of these results! Now I really want to redo the analysis using different statistics… anybody know someone who has already done this?

Finally, there are a couple of things I should point out. First, in the spirit of full disclosure, I have received a best paper nomination—at last year’s CHI in fact, for a Note. Regardless, I have no personal discomfort with criticism toward the nomination and selection process. My issue is with the operationalization of “quality” in the Bartneck & Hu paper, and with the statistics used to support the criticism. Second, the ‘best paper’ section was actually a small part of the paper; most of the analysis focused on presenting a historical account of participation in the conference (represented by accepted papers), broken down by organization and country (of the organization, not the authors).

The papers:

1. Bartneck, C., & Hu, J. (2009). Scientometric Analysis Of The CHI Proceedings. In CHI ’09 (pp. 699-708).

2. Bollen, J., Sompel, H. V., Hagberg, A., & Chute, R. (2009). A Principal Component Analysis of 39 Scientific Impact Measures. PLoS ONE, 4(6).

The researchers recruited 17 first- and second-grade teachers (all female) and assessed the math achievement of the students in their classrooms at the beginning and end of the year, as well as the teachers’ anxiety level. They also measured “students’ beliefs about gender and academic success in domains like math”. They found that higher teacher math anxiety was associated with an increase in girls’ tendency to adhere to “boys are good at math, girls are good at reading” gender stereotypes. They also found that girls who were more likely to hold “boys are good at math, girls are good at reading” stereotypes had lower end-of-year math achievement scores. Interestingly, when they put these predictors together in one regression (teacher anxiety and gender stereotypes predicting math achievement), teacher anxiety was “no longer a significant predictor” (and the coefficient decreased from -3.33 to -2.48). The paper presents a “mediation analysis” called “bias-corrected bootstrapping” that suggests math anxiety in female teachers affects girls’ gender stereotypes, which affects math achievement scores. I don’t know much about this analysis method, so I dug up a couple of papers so I can learn more about it. Yay, stats!

I have two issues with the way the results are presented in this paper. First, it took me way too long to figure out what they actually did. I didn’t notice the supplemental material initially, which is where the all of the analyses are described, and the text of the actual article is too vague about the statistics for me to believe the results from just that part of the text. I realize that PNAS limits submissions to 6 pages, but I feel that for this particular paper the supplemental material is not supplemental at all—it is essential. After reading the supplement, it is pretty clear that the analysis was adequate.

But, my second issue is that the interpretation of the result seems more concerned with sign and significance than with effect size. The paper doesn’t ground the numbers in real-world implications, nor does it present descriptive statistics on the instruments used (these are relegated to the online appendix as well). For example, it is impossible to interpret the coefficient in this statement without having some idea what the units mean for both teacher anxiety and math achievement: “In addition, the more girls at the end of the year endorsed the notion that boys are good at math and girls are good at reading, the lower was their math achievement (r = −0.28, P = 0.025).” This oversight surprises me. So what if gender stereotype belief is a significant predictor of math achievement for girls, if this is only associated with very small differences in test scores? Yes, it is still interesting that the effect was present in girls and not in boys, but if the magnitude of the effect is small, in my mind the implications of this particular study are more about gender stereotypes and behavior modeling, and less about figuring out how to help girls do better in math. (There’s a brief acknowledgement of effect size in the third to last paragraph: “It is important to note that the effects reported in the current work, although significant, are small.”)

Finally, it’s interesting to me, and a bit depressing, that in a paper about math anxiety and achievement, the complicated statistics are relegated to an appendix. There is no way to know if the authors expected the statistical analyses to be transparent/obvious enough they didn’t need to include the details in the paper, or if they felt the paper would be more understandable for readers without the stats. This is something I struggle with—how to appropriately describe complicated quantitative analyses for a multi-disciplinary audience that may or may not understand what I’m talking about, or even want to learn. I’m not sure I like the stats appendix solution, but I like it a lot more than two other alternatives I’ve seen: 1) the “sink or swim” approach—describing the analyses as if to an expert, and less experienced readers are left to flounder; and 2) only using stats one believes most members of the community should be familiar with.

The purpose of the book is to convince readers that Null Hypothesis Significance Testing (NHST) should no longer be practiced, and to suggest alternatives like using confidence intervals and always reporting effect sizes. I think my favorite quote from the book so far is this one, in a paragraph devoted to the suggestion that one way to fix the NHST problem is just to use more careful, less overreaching language in talking about p-values and significance tests (like phasing out the word “significant”):

You can put candles in a cow pie, but that does not
make it a birthday cake.

You can tell what the author thinks of that idea. Ouch.

Part I of the book also makes an interesting argument that NHST is not only bad social science, it is bad FOR social science. The idea is that because p-values are colloquially understood to mean something they actually do not, researchers believe the findings of a single study are more robust and reliable than they actually are. For example, a p-value represents the conditional probability of the data given the null hypothesis, NOT the probability that the null hypothesis is true given the data. According to the book, this and other misinterpretations about the logic of significance testing cause the literature to be biased toward research results “about fad topics that clutter the research literature but have little scientific value”, that are never replicated:

…if one believes that p < .01 implies that the result is likely to be repeated more than 99 times out of 100, why bother to replicate? A related cognitive error is the belief that statistically significant findings should be replicated, but not ones for which [the null hypothesis] was not rejected (F. Schmidt & Hunter, 1997).

This bias perpetuates research for which the practical, meaningful significance of the results is not clear.

A lot of these arguments make sense to me—it seems like the process of NHST hides a lot of the error and uncertainty that is part of doing science, making it seem like the results of individual studies are more definitive and certain than they actually are. I’m looking forward to making it through the rest of the book and starting to practice the alternatives it suggests.

The latest topic of interest is model selection. I won’t go into the details here — I need to spend my writing energy getting a draft out the door. But I’m reading a great little book right now that not only covers the statistical concepts I wanted to learn about, but also has an interesting “philosophy of science”-type discussion. I have a feeling the author of this book might have a few problems with The End of Theory…

Here’s the book:  Model Based Inference in the Life Sciences: A Primer on Evidence, by David R. Anderson

i also have to say, what the heck did people do before the internet? i mean, here i am using open-source statistical software, doing fairly nonstandard analyses for my field. and yet, it seems like no matter what problem i encountered either with the tool or in trying to specify the model correctly, somebody else had already figured it out or written a paper on it. for example, i will definitely be citing this really fabulous journal article on generalized linear mixed models (Bolker et. al, 2009) with not one, but TWO online supplements that are equally fabulous. who knows what i would have done without having all this information at my fingertips!!

Data Analysts Captured By R’s Power

the first thing i thought when i saw it was, way to go R! i had never heard of R when i first started my graduate program — i was still using SPSS. but as a poor graduate student, i didn’t want to pay yearly renewal fees for expensive licenses, and i started learning R because it is free. i’ve been hooked ever since.

as for the article in the nytimes, i think this statement isn’t quite accurate:

But R has also quickly found a following because statisticians, engineers and scientists without computer programming skills find it easy to use.

i think you definitely need some programming experience to get the command-line interface of R, which is where the power is. i never had much luck with the GUI point-and-click add-on. but i do agree with Daryl Pregibon, who is apparently a research scientist at Google and a fan of R:

R is really important to the point that it’s hard to overvalue it.

and, i love the part in the article about the reaction of SAS Institute to R:

“R has really become the second language for people coming out of grad school now, and there’s an amazing amount of code being written for it,” said Max Kuhn, associate director of nonclinical statistics at Pfizer. “You can look on the SAS message boards and see there is a proportional downturn in traffic.”

SAS says it has noticed R’s rising popularity at universities, despite educational discounts on its own software, but it dismisses the technology as being of interest to a limited set of people working on very hard tasks.

“I think it addresses a niche market for high-end data analysts that want free, readily available code,” said Anne H. Milley, director of technology product marketing at SAS. She adds, “We have customers who build engines for aircraft. I am happy they are not using freeware when I get on a jet.”

sounds a little defensive, eh?

for me, one of the biggest benefits of using R is the really nice looking, professional quality, complex graphs and charts R produces — they kick the ass of anything Excel or SPSS can generate. and, there are times when R reminds me of Lisp, which makes me happy. using R confidently also requires that one be more knowledgeable about the statistics one is performing (and why) than the point-and-click stats packages do — and i believe this is a good thing! i feel like i’ve become a much better researcher and user of statistics because of R, and i’ve been able to do data munging and analysis that i think would be difficult-to-impossible in SPSS.

however, the R documentation can be a bit maddening at times. either it is too terse, or doesn’t quite cover your exact situation… but R is open source after all, so my expectations are lower than they might be if i had actually paid for the software. a little persistence usually does the trick.

if you’re interested in learning more about R, you can find the software at r-project.org, and my favorite introduction to doing basic stuff with R is hosted by the UCLA Statistical Computing service.

UPDATE: the NY Times article hit the “most emailed” feed finally, only after it was posted to Slashdot yesterday evening.

So, this is my final post on NHST, or Null Hypothesis Significance Testing. Not, as Cohen (1994) — see last post for citation — pseudo-jokingly wrote: Statistical Hypothesis Inference Testing. I am working my way through a really comprehensive paper on the subject:

Nickerson R. (2000) Null Hypothesis Significance Testing: A Review of an Old and Continuing Controversy. Psychological Methods, 5(2), 241-301. [ pdf ]

I am finding that the authors of papers damning NHST and all those who practice it seem to take an extreme stance in making arguments about the damage being done to science by researchers who have not yet seen the light about Bayesian statistics. Nickerson (2000) seems to agree with me. He wrote:

Although many of the participants have stated their positions objectively and gracefully, I have been struck with the stridency of the attacks by some on views that oppose their own. NHST has been described as “thoroughly discredited,” a “perversion of the scientific method,” a “bone-headedly misguided procedure,” “grotesquely fallacious,” a “disaster,” and “mindless,” among other things. Positions, pro or con, have been labeled “absurd,” “senseless,” “nonsensical,” “ridiculous,” and “silly.” The surety of the pronouncements of some participants on both sides of the debate is remarkable. (p289)

Nickerson (2000) is a fairly balanced treatment of the issue. If you’re only going to read one thing on the subject, his is the paper I recommend. For example, he makes this point about an implicit assumption behind NHST experiments, and it is one I haven’t seen other authors make:

If one is willing to make the assumption that p(D | H_A) [the probability of the data, assuming the alternative hypothesis is true] is large relative to p(D | H₀) [the probability of the data, assuming the null hypothesis is true], then one has a legitimate basis for interpreting a small p as evidence for increasing the likelihood of H_A relative to that of H₀. Perhaps this assumption underlines many applications of NHST, but seldom does one see an explicit acknowledgment of it. (p263)

What I think he is saying is that researchers conducting an experiment have implicit assumptions and expectations, based on previous work and their own intuition, guiding both the design of the experiment and their interpretation of patterns in the data they’ve collected. If they’ve done a thorough job and designed a good experiment, it seems reasonable to interpret small p values as support for the alternate hypothesis (rather than just a sample that is unlikely to occur under the null hypothesis. However, you rarely see an explicit discussion of this type of assumption in the results section of a paper. This also seems like an inherently Bayesian assumption, interpreting p in light of expectations at the beginning of the experiment (or prior probability?).

There are two things I have certainly learned from my foray into NHST “controversy”. The first is that the history of science is really interesting and I should take advantage of opportunities to learn more about it. The second is that I really don’t know enough about probability — this is something I absolutely should work on.

So, what am I going to do with all of this new knowledge? I am definitely going to be thinking more about prior probabilities and implicit assumptions when I design experiments. And, I will be sure to include a discussion of practical significance whenever I write up research results, including placing more emphasis on power analysis and effect sizes. It seems like the Bayesian approach on a conceptual level (i.e. ignoring the math) involves being more explicit about assumptions and predictions in an experiment, and assigning specific, numeric probabilities to potential outcomes. This level of specificity then allows one to be more precise about where the experiment data fit in the bigger picture. In general, this seems like good advice for becoming a better scientist; however, I am not ready to make the leap and become a Bayesian statistician just yet. Nickerson (2000) has convinced me that it is possible to do real, valid science with NHST as long as one is aware of its weaknesses and the mistaken assumptions that researchers sometimes make:

What is the great harm if many people who use NHST believe that p is the probability that the null hypothesis is true, or that a small p is evidence of replicability, or that α is the probability that if one has rejected the null hypothesis one has made a Type I error? Claims to the contrary notwithstanding, there is room for doubt as to whether acquisition of psychological knowledge through experimentation has been greatly impeded by the prevalence of such beliefs or by any of the many other shortcomings of NHST that have been ably identified by its critics. (p289)