More on the language-as-fixed-effect fallacy: Monte Carlo estimates of error rates for F₁,F₂,F′, and min F′

The properties of four different tests of the treatment effect in experiments using linguistic materials are examined using Monte Carlo procedures for estimating Type I error rates. It is shown that: (a) in extreme cases, the Type I error rates for F₁ and F₂ can exceed the desired rate by a factor of at least 10; (b) minF′ tends to be a very close estimate of F′; (c) both minF′ and F′ are very conservative tests when between item variance or subject-by-treatment variance is low; (d) requiring both F₁ and F₂ to be significant before H₀ is rejected does not prevent the nominal Type I error rate from being exceeded; (e) most of these problems can be minimized by using multistage decision rules which select the most appropriate test on the basis of preliminary tests of item variance and subject-by-treatment variance.