Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

How does LocationTest select its "AutomaticTest" when testing two samples?

Background: A previous post poses a similar question but in that case the choice concerned a single sample. The question involving two samples is harder since the idiom apparently in use - using the significant level set for the overarching test to set significance levels for "Normality", "EqualVariances" and "Symmetry" diagnostic tests - has more complex interactions and raises issues about the entire idiom (including for the family of hypothesis-testing functions - VarianceTest, DistributionFitTest, IndependentTest etc) while also suggesting certain usage guidelines.

Note that the following applies two user-defined functions (both defined in the related post); ShowSignificanceLevelThresholds - for showing those significant levels at which LocationTest changes its choice of "AutomaticTest" and the "HighlightAutomaticTest" option - for highlighting (in a the gray background) the selected "AutomaticTest".

First define two samples:

SampleA = 
  {17.375, 20.375, 20, 20, 18.375, 18.625, 18.625, 
   15.25, 16.5, 18, 16.25, 18, 12.75, 15.5, 18};

SampleB = 
  {23.5, 12, 21, 22, 19.125, 21.5, 22.125, 20.375, 
   18.25,21.625, 23.25, 21, 22.125, 23, 12};

Now find the significance level thresholds.

LocationTest[{SampleA, SampleB}, Automatic, {"TestDataTable", All}, 
  "HighlightAutomaticTest" -> True] // ShowSignificanceLevelThresholds

enter image description here

Similar to the single sample case, Normality of now both samples affects the test chosen and hints at the generation of the first threshold

Row[{
  DistributionFitTest[SampleA, Automatic, 
    {"PValueTable", All},"HighlightAutomaticTest" -> True],
  Spacer @ 50,
  DistributionFitTest[SampleB, Automatic,
    {"PValueTable", All},"HighlightAutomaticTest" -> True]
}]

enter image description here

Perhaps, the normality of the second sample is being assumed for significance levels less than 0.016 as when the overarching significance level is passed down to the PearsonChiSquare normality test it is halved to correspond to the extra evidence required (the number of tests being effectively doubled due to the two samples)?

As for the other transition point, 0.826, "EqualVariance" and "Symmetry" are likely to be involved. As far as "EqualVariance" is concerned, the same threshold analysis can be performed.

VarianceTest[{SampleA, SampleB}, Automatic, {"TestDataTable", All}, 
  "HighlightAutomaticTest" -> True] // ShowSignificanceLevelThresholds

enter image description here

Now disentangling becomes trickier - bear in mind that the "EqualVariance" test presumably used by LocationTest presumably uses the passed-down significance level to presumably both select the test assessing an assumption and to determine its outcome - by comparing this (adjusted) significance level with one of the tabled p-values above. Furthermore, VarianceTest presumably applies its own tests of normality so how this interacts with LocationTest's own normality test and selected significant values rapidly renders rabbit-warren-like - and this is even before factoring in (undocumented?) tests of "Symmetry".

Furthermore, in the last transition significance value, 0.016, doesn't change Conover as the chosen "AutomaticTest" but rather those "appropriate" or "applicable" (plausible?) tests - but under exactly what criteria does a test become "appropriate" or "applicable"? According to the documentation "LocationTest" chooses all tests that apply ..." suggesting "AutomaticTest" as the most powerful, applicable test and implying the nub of the post's question a more general, "How does LocationTest select its applicable tests?" In such a scenario the significance level further affects "AutomaticTest"'s selection since it impacts on relative powers between applicable tests (particularly for tests with large degrees of freedom and low significance levels such as, for example, with genome-wide association studies) further adding to the tangle.

Apart from a lack of transparancy (surely impacting LocationTest's prospects for use in published research?) there is something epistemologically suspect about this significance level inheritance that goes against the grain of NHST's logic. The point of a significance level is that it is sets a standard of evidence to be met in order to make a decision/accept an effect. The point of a p-value is that it constitutes this evidence - a measure (probability) of the "extremeness" of a given experimental outcome. They are separate. This is the basis of their subsequent comparison: If the outcome is "sufficiently extreme" this apriori-decided standard of evidence is reached. The notion that a different significance level can change an outcome's p-value subverts this relationship and completely alters the methodology's logic while setting up convoluted dependencies inimical to repeatability, intuitive understanding and natural invocation - all fundamental tenets of scientific practice.

Ultimately however, this is not a function of Wolfram Language's design per se nor does it superficially follow from the specific decision to link the significance level of diagnostic tests` significance levels; it is instead, an inevitable consequence of the decision to adopt formal, diagnostic tests for assessing such assumptions (or, as some have argued, the inherent incoherence of the NHST paradigm itself). This changes the definition (and perception) of Type 1 and Type 2 errors including for example when a t-test is selected:

enter image description here

This approach has long been regarded as highly problematic as far back as 19441 and with plenty of empirical evidence emerging since2-5 and yet ... for example,

Preliminary tests of equality of variances used before a test of location are no longer widely recommended by statisticians, although they persist in some textbooks and software packages.3

In particular, it is used in SPSS, SAS and Statistica although seemingly, studiously eschewed in R. The Wolfram Langugage apparently adopts this strategy of the former group and despite the above inherent complications and conventional criticism just outlined, the concerns can perhaps be ameliorated (albeit IMO not in "natural usage" of the current implementation) while retaining certain advantages. More specifically, balancing Type I/II errors in many ways represents a theoretician's mirage resting as it does on distributional assumptions requiring judgement in the real world. The problems flowing from replacing this judgement with automatic computation in diagnostic tests are well made by numerous researchers but what they also consistently fail to do is demonstrate the superiority of the alternative (or any hybrid thereof):

In practice we conclude that graphical diagnostics are better than a formal pretest.2

How so? That the shortcomings of a formal pretest can be analyzed, quantified and demonstrated fails to imply bettter outcomes from a vaguely defined and never-measured alternative.

Hence, LocationTest has scope for improvement and attempting to tap into its test-choosing machinations is, I'd argue, far from idle spelunking; it's directly related to reproducibility concerns plagueing NHST as a scientific methodology virtually since its inception. In its current incarnatation, LocationTest is already useful in terms of suggesting relevant and powerful tests and for providing overviews but IMO caveats are needed for its routine usage:

1) Caution is needed in using its defaults as currently implemented - instead it may currently be more prudent to manually set assumptions via "VerifyTestAssumptions" option (surely a more natural, and in the light of the above, emphasised distinction would include a dedicated "SetTestAssumptions" option?)

2) If manually setting assumptions, confirm the logic dovetails with the automatic test chosen as described in the literature.

Improvements would therefore see the significant values of diagnostic tests demonstrably preserving both meaningful diagnostics and overarching Type 1 and Type 2 errors (at least to a point where feasible and in particular in a more refined way than " ... using \[Alpha]/k where k is the number of datasets being tested." or alternatively when not feasible having an option to set these levels separately) while having the logic used to select automatic and applicable tests made much more transparent. These changes would not only satisfy Frequentists in this NHST implementation but also lay the groundwork for a casting/translation more amenable to Baysians (in a way beyond this post's scope).

This would require an extension of the language - in particular of the input types and generated "HypothesisTestData" object as part of resolving multiple design tensions: human judgment vs machine determinations; smallest possible functional unit vs increasing functional scope; command-line vs object-like input & output etc all of which, albeit in a different guise, similarly (and perhaps even more pervasively) arise in more general data science invocations ... but this post digresses.

enter image description here

share|improve this question

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.