14
$\begingroup$

Bug introduced in 11.3 or earlier and persisting through 11.3.0 or later


Just before upgrading to 11.3.0 on last Friday, i was having p-value results from 11.2.0.

Not worrying about upgrade may affect built-in tools output, I re-compiled same untouched file (my codes and data) in a new version. But I received shocking results, as given in screenshots below.

First screen shot, sourced from 11.2 version. enter image description here Second screen shot, sourced from 11.3 version. enter image description here Below are the main source information in file, i have applied in both v11.2 and v11.3.

Data generated as:

n=500; 
Dist = MixtureDistribution[{0.631111,0.368889},{BetaDistribution[2.21477,2.44873],GammaDistribution[11.1681,0.0990735]}];
(SeedRandom[243]; MixData = RandomVariate[Dist,n]);

Tested distributions:

Dist1 = GammaDistribution[2.63261, 0.265247];
Dist2 = GammaDistribution[1.01794, 0.773567, 1.77486, 0.00184678]; 

There are other customized distributions in screenshots as APDF and sPDF are kept hold.

Worrying about received results from version 11.3,I tried following,

  1. First I'd compared randomly generated data, generated in both versions using the same information as given above,

    SeedRandom[243]; MixData112 = RandomVariate[Dist,n] (*generated in v11.2; same as MixData*);
    SeedRandom[243];MixData113 = RandomVariate[Dist,n] (*generated in v11.3*);
    

    Note: Data set "MixData112" created on another machine already installed v11.2 and are not in plain text form.

    MapThread[Equal,{MixData112, MixData113}]
     (*Returned TRUE for all elements*) 
    
  2. Then manual usage for KS test-statics,

    tstat1 = Max[Abs[Range[0, n - 1]/n - CDF[Dist1, Sort@MixData]]]
     (*0.0406447*)
    
    tstat2 = Max[Abs[Range[0, n - 1]/n - CDF[Dist2, Sort@MixData]]]
     (*0.0305432*)
    

Note: Test statistics are exactly same in both v11.2 and v11.3.

From v11.3,

 KolmogorovSmirnovTest[MixData113, Dist1, "TestStatistic"]
  (*0.0406447*)
 KolmogorovSmirnovTest[MixData113, Dist2, "TestStatistic"]
  (*0.0305432*)
  1. Then I executed built-in command,

     KolmogorovSmirnovTest[MixData113, Dist1]
      (*1.*)
     KolmogorovSmirnovTest[MixData113, Dist2]
      (*1.*)
    

But in v11.2, for particular seed I was always getting p-value 0.37073 for Dist1 and 0.727397 for Dist2).

Additional information:

For customized distribution, e.g. in v11.2

 APDF[5.6476, 2.32299, 8]
 DistributionFitTest[MixData112, APDF[5.6476, 2.32299, 8]], {"TestDataTable", {"KolmogorovSmirnov", "AndersonDarling", "CramerVonMises"}}] 
 (*p-values returned, {0.815692, 0.882876, 0.886041}*)

But in v11.3,

 APDF[5.6476, 2.32299, 8]
 DistributionFitTest[MixData113, APDF[5.6476, 2.32299, 8]], {"TestDataTable", {"KolmogorovSmirnov", "AndersonDarling", "CramerVonMises"}}]  
 (*p-values now returned as, {0., 0., 2.9976*10^-15}*)

In KS-test, when p-value returned 0. for customized distribution, together following message received,

General::munfl: Exp[-1009.17] is too small to represent as a normalized machine number; precision may be lost.(*New in 11.3*)

Beyond my limitation on such, i am wondering about, is this a bug that p-value behaves like that or some new setting require in v11.3 that is not apparent?

Kindly help on above to resolve facing problem.

$\endgroup$
  • $\begingroup$ That looks pretty much like bug. You basically say that KolmogorovSmirnovTest leads to completely different results when applied to the same data in different version? Since the documentation does not state that there was a change in (the API of) KolmogorovSmirnovTest in 11.3, it must be a bug in the backend. General::munfl suggests some method in the backend was exchanged for a much more instable one. I strongly suggest that you contact the support about it. Give them also the link to your post. $\endgroup$ – Henrik Schumacher Mar 25 '18 at 17:38
  • $\begingroup$ Btw.: You did a great job to track that down. $\endgroup$ – Henrik Schumacher Mar 25 '18 at 17:38
  • 2
    $\begingroup$ Hm. In this case, KolmogorovSmirnovTest returns 1 for large data sets telling me that the data sets are likely distributed with GammaDistribution[3.13505, 0.235861] (which they are). That's good isn't it? But for even longer list of random numbers, the return value of KolmogorovSmirnovTest is again way below 1. That's not good. $\endgroup$ – Henrik Schumacher Mar 25 '18 at 18:38
  • 1
    $\begingroup$ @SjoerdC.deVries. I have reported together referring "this post" to both Wolfram Customer Support/Technical Support on 26th March. They have forwarded to developers. And assured to notified when a fix is available. I'll keep updating "here" on any improvement. Thanks. $\endgroup$ – step-by-step Mar 28 '18 at 1:42
  • 1
    $\begingroup$ Wolfram technical support replied as, "Just to update on this old issue: I can't guarantee that a fix will go out with any specific version of Mathematica (changes sometimes happen at the last second). However, I can say that our developers took your problem seriously, and found that it was part of a larger set of issues that came up in 11.3. The problems can't be fixed in a paclet update, but will be fixed in some future version of Mathematica. I think that it will be the next version of Mathematica, but I can't guarantee that for the reasons above." $\endgroup$ – step-by-step Jun 16 '18 at 14:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.