Large increase in integrate result leaf size after version 9. Looking for common cause

Question

reported to WRI. [CASE:3960295]

---

I run the CAS integration tests which contains 15000 integrals over number of Mathematica versions.

I found that the average leaf size of the anti-derivatives blows up after version 9. Here is diagram

To find the integrals which cause this increase in leaf size, I picked those in version 11.2 which had leaf size of at least 5 times as large as in version 9.0. I found 260 such integrals. Many have much larger ratio than 5. some have ratio of 50 times as large and 100 times as large.

The question is, why did this happen after version 9? Is there a common cause among these 260 integrals? I will give one such example, and link to the report which contain list of the 260 integrals and also the zip file of the raw test files obtained from Rubi web site.

First integral

command: Integrate[Tan[x]/Sqrt[1 + Sec[x]^3],x] 
Version 11.2 leaf size: 37353  
Version 9 leaf size: 296  
Ratio of version 11.2 leaf size to verion 9:      126

Version 11.2

Integrate[Tan[x]/Sqrt[1+Sec[x]^3],x]
(*output too large to post*)

In version 9

The report in PDF and HTML can be found here

Appendix

Here is partial part of the list. Too long to post here in full.

Command used: Integrate[Tan[x]/Sqrt[1 + Sec[x]^3],x]
Version 11.2 leaf size: 37353.
Version 9 leaf size: 296.
Ratio of version 11.2 leaf size to verion 9 leaf size: 126

Command used: Integrate[ArcTan[Sqrt[-1 + Sec[x]]]*Sin[x],x]
Version 11.2 leaf size: 6968.
Version 9 leaf size: 1043.
Ratio of version 11.2 leaf size to verion 9 leaf size: 7

Command used: Integrate[(1 + x^2)/((1 - x^2)*Sqrt[1 + x^4]),x]
Version 11.2 leaf size: 269.
Version 9 leaf size: 36.
Ratio of version 11.2 leaf size to verion 9 leaf size: 7

Command used: Integrate[(1 - x^2)/((1 + x^2)*Sqrt[1 + x^4]),x]
Version 11.2 leaf size: 266.
Version 9 leaf size: 40.
Ratio of version 11.2 leaf size to verion 9 leaf size: 7

Command used: Integrate[(1 - x^2)/((1 + x^2)*Sqrt[1 + x^4]),x]
Version 11.2 leaf size: 266.
Version 9 leaf size: 40.
Ratio of version 11.2 leaf size to verion 9 leaf size: 7


Command used: Integrate[(1 + x^2)/((1 - x^2)*Sqrt[1 + x^4]),x]
Version 11.2 leaf size: 269.
Version 9 leaf size: 36.
Ratio of version 11.2 leaf size to verion 9 leaf size: 7


Command used: Integrate[(1 - x^2)/((1 + x^2)*Sqrt[1 + x^2 + x^4]),x]
Version 11.2 leaf size: 567.
Version 9 leaf size: 94.
Ratio of version 11.2 leaf size to verion 9 leaf size: 6

Command used: Integrate[(Sqrt[Cos[x]*Sin[x]^3] - 2*Sin[2*x])/(-Sqrt[Cos[x]^3*Sin[x]] + Sqrt[Tan[x]]),x]
Version 11.2 leaf size: 68457.
Version 9 leaf size: 2051.
Ratio of version 11.2 leaf size to verion 9 leaf size: 33

Command used: Integrate[(Cosh[x]*(-Cosh[2*x] + Tanh[x]))/(Sqrt[Sinh[2*x]]*(Sinh[x]^2 + Sinh[2*x])),x]
Version 11.2 leaf size: 5173.
Version 9 leaf size: 490.
Ratio of version 11.2 leaf size to verion 9 leaf size: 11

Command used: Integrate[Sqrt[1 + p*x^2 - x^4]/(1 + x^4),x]
Version 11.2 leaf size: 10200.
Version 9 leaf size: 322.
Ratio of version 11.2 leaf size to verion 9 leaf size: 32

Command used: Integrate[Sqrt[e + f*x^2]/((a + b*x^2)*(c + d*x^2)^(7/2)),x]
Version 11.2 leaf size: 12924.
Version 9 leaf size: 577.
Ratio of version 11.2 leaf size to verion 9 leaf size: 22

Command used: Integrate[(Sqrt[c + d*x^2]*(e + f*x^2)^(3/2))/(a + b*x^2),x]
Version 11.2 leaf size: 11662.
Version 9 leaf size: 739.
Ratio of version 11.2 leaf size to verion 9 leaf size: 16

Command used: Integrate[(Sqrt[2 + d*x^2]*Sqrt[3 + f*x^2])/(a + b*x^2),x]
Version 11.2 leaf size: 7470.
Version 9 leaf size: 134.
Ratio of version 11.2 leaf size to verion 9 leaf size: 56

Command used: Integrate[(Sqrt[c + d*x^2]*Sqrt[e + f*x^2])/(a + b*x^2)^2,x]
Version 11.2 leaf size: 3879.
Version 9 leaf size: 401.
Ratio of version 11.2 leaf size to verion 9 leaf size: 10

Command used: Integrate[1/((a + b*x^2)^2*Sqrt[c - d*x^2]*Sqrt[e + f*x^2]),x]
Version 11.2 leaf size: 7216.
Version 9 leaf size: 773.
Ratio of version 11.2 leaf size to verion 9 leaf size: 9

Command used: Integrate[((e*x)^m*(a + b*x^n)^2*(A + B*x^n))/(c + d*x^n)^3,x]
Version 11.2 leaf size: 1924.
Version 9 leaf size: 345.
Ratio of version 11.2 leaf size to verion 9 leaf size: 6

Command used: Integrate[((b + 2*c*x)*Sqrt[a + b*x + c*x^2])/(d + e*x)^(3/2),x]
Version 11.2 leaf size: 5706.
Version 9 leaf size: 1045.
Ratio of version 11.2 leaf size to verion 9 leaf size: 5

Command used: Integrate[1/(x*(1 + x)^(5/2)*(1 - x + x^2)^(5/2)),x]
Version 11.2 leaf size: 2539.
Version 9 leaf size: 98.
Ratio of version 11.2 leaf size to verion 9 leaf size: 26

Command used: Integrate[Sqrt[2 + 3*x^2 + x^4]/(7 + 5*x^2),x]
Version 11.2 leaf size: 520.
Version 9 leaf size: 90.
Ratio of version 11.2 leaf size to verion 9 leaf size: 6

Command used: Integrate[1/((d + e*x^2)^2*Sqrt[a + b*x^2 - c*x^4]),x]
Version 11.2 leaf size: 10996.
Version 9 leaf size: 1341.
Ratio of version 11.2 leaf size to verion 9 leaf size: 8

Command used: Integrate[Sqrt[a + b*x^2 + c*x^4]/(x^3*(d + e*x^2)),x]
Version 11.2 leaf size: 10848.
Version 9 leaf size: 244.
Ratio of version 11.2 leaf size to verion 9 leaf size: 44

Command used: Integrate[Sqrt[1 + 2*x^2 + 2*x^4]/(x^4*(3 + 2*x^2)),x]
Version 11.2 leaf size: 1546.
Version 9 leaf size: 154.
Ratio of version 11.2 leaf size to verion 9 leaf size: 10

Command used: Integrate[Sqrt[1 + 2*x^2 + 2*x^4]/(x^6*(3 + 2*x^2)),x]
Version 11.2 leaf size: 1650.
Version 9 leaf size: 224.
Ratio of version 11.2 leaf size to verion 9 leaf size: 7

Command used: Integrate[(1 + 2*x^2 + 2*x^4)^(3/2)/(x^2*(3 - 2*x^2)),x]
Version 11.2 leaf size: 1609.
Version 9 leaf size: 213.
Ratio of version 11.2 leaf size to verion 9 leaf size: 8

Command used: Integrate[1/(x*(d + e*x^2)*Sqrt[a + b*x^2 + c*x^4]),x]
Version 11.2 leaf size: 3622.
Version 9 leaf size: 174.
Ratio of version 11.2 leaf size to verion 9 leaf size: 21

Command used: Integrate[x^2/((3 + 2*x^2)*Sqrt[1 + 2*x^2 + 2*x^4]),x]
Version 11.2 leaf size: 1518.
Version 9 leaf size: 99.
Ratio of version 11.2 leaf size to verion 9 leaf size: 15

Command used: Integrate[x^2/((3 + 2*x^2)*(1 + 2*x^2 + 2*x^4)^(3/2)),x]
Version 11.2 leaf size: 1636.
Version 9 leaf size: 199.
Ratio of version 11.2 leaf size to verion 9 leaf size: 8

Command used: Integrate[1/((3 + 2*x^2)*(1 + 2*x^2 + 2*x^4)^(3/2)),x]
Version 11.2 leaf size: 1636.
Version 9 leaf size: 199.
Ratio of version 11.2 leaf size to verion 9 leaf size: 8

Command used: Integrate[Sqrt[1 - x^2]/(x^3*(a + b*x^2 + c*x^4)),x]
Version 11.2 leaf size: 6953.
Version 9 leaf size: 298.
Ratio of version 11.2 leaf size to verion 9 leaf size: 23

Command used: Integrate[(d + e*x^n)*(a + b*x^n + c*x^(2*n))^(3/2),x]
Version 11.2 leaf size: 8312.
Version 9 leaf size: 690.
Ratio of version 11.2 leaf size to verion 9 leaf size: 12

Command used: Integrate[(a - c*x^4)/(Sqrt[a + b*x^2 + c*x^4]*(a*d + a*e*x^2 + c*d*x^4)),x]
Version 11.2 leaf size: 26953.
Version 9 leaf size: 419.
Ratio of version 11.2 leaf size to verion 9 leaf size: 64

Command used: Integrate[(a - c*x^4)/((a*e + c*d*x^2)*(d + e*x^2)*Sqrt[a + b*x^2 + c*x^4]),x]
Version 11.2 leaf size: 15438.
Version 9 leaf size: 383.
Ratio of version 11.2 leaf size to verion 9 leaf size: 40

Command used: Integrate[Cos[c + d*x]^6*Sqrt[a + a*Sin[c + d*x]],x]
Version 11.2 leaf size: 995.
Version 9 leaf size: 99.
Ratio of version 11.2 leaf size to verion 9 leaf size: 10

Command used: Integrate[1/((a + b*Sin[e + f*x])^2*(c + d*Sin[e + f*x])^(3/2)),x]
Version 11.2 leaf size: 6518.
Version 9 leaf size: 1057.
Ratio of version 11.2 leaf size to verion 9 leaf size: 6

Command used: Integrate[1/((a + b*Sin[e + f*x])^2*(c + d*Sin[e + f*x])^(5/2)),x]
Version 11.2 leaf size: 6780.
Version 9 leaf size: 1319.
Ratio of version 11.2 leaf size to verion 9 leaf size: 5

Command used: Integrate[(c + d*Sin[e + f*x])^(5/2)/(a + b*Sin[e + f*x])^3,x]
Version 11.2 leaf size: 6610.
Version 9 leaf size: 1149.
Ratio of version 11.2 leaf size to verion 9 leaf size: 6

Command used: Integrate[Sqrt[c + d*Sin[e + f*x]]/(a + b*Sin[e + f*x])^3,x]
Version 11.2 leaf size: 6499.
Version 9 leaf size: 1038.
Ratio of version 11.2 leaf size to verion 9 leaf size: 6

Command used: Integrate[1/((a + b*Sin[e + f*x])^3*(c + d*Sin[e + f*x])^(3/2)),x]
Version 11.2 leaf size: 6779.
Version 9 leaf size: 1318.
Ratio of version 11.2 leaf size to verion 9 leaf size: 5

Command used: Integrate[Cos[c + d*x]*Sin[c + d*x]^n*(a + a*Sin[c + d*x])^4,x]
Version 11.2 leaf size: 457.
Version 9 leaf size: 80.
Ratio of version 11.2 leaf size to verion 9 leaf size: 6

Command used: Integrate[Sqrt[g*Sin[e + f*x]]/(Sqrt[a + a*Sin[e + f*x]]*(c - c*Sin[e + f*x])),x]
Version 11.2 leaf size: 58053.
Version 9 leaf size: 232.
Ratio of version 11.2 leaf size to verion 9 leaf size: 250

Command used: Integrate[1/(Sqrt[g*Sin[e + f*x]]*Sqrt[a + a*Sin[e + f*x]]*(c - c*Sin[e + f*x])),x]
Version 11.2 leaf size: 58056.
Version 9 leaf size: 234.
Ratio of version 11.2 leaf size to verion 9 leaf size: 248

Command used: Integrate[1/(Sqrt[g*Sin[e + f*x]]*Sqrt[a + a*Sin[e + f*x]]*(c + d*Sin[e + f*x])),x]
Version 11.2 leaf size: 663005.
Version 9 leaf size: 99997.
Ratio of version 11.2 leaf size to verion 9 leaf size: 7

Command used: Integrate[Sqrt[3 + 4*Cos[c + d*x]]*Sec[c + d*x]^2,x]
Version 11.2 leaf size: 987.
Version 9 leaf size: 157.
Ratio of version 11.2 leaf size to verion 9 leaf size: 6

Command used: Integrate[Cos[c + d*x]^(3/2)/(a + b*Cos[c + d*x]),x]
Version 11.2 leaf size: 642.
Version 9 leaf size: 84.
Ratio of version 11.2 leaf size to verion 9 leaf size: 8

Command used: Integrate[Cos[c + d*x]^(3/2)/(a + b*Cos[c + d*x])^3,x]
Version 11.2 leaf size: 1390.
Version 9 leaf size: 276.
Ratio of version 11.2 leaf size to verion 9 leaf size: 5

....

Answer 1

In general, it can be quite hard to determine why Integrate does what it does. There are too many competing rules, and it's tricky to make the best rule (assuming even everyone agrees on which one is best) fire at the right time.

The good news is that in our internal development build, this issue appears to be fixed.

In[19]:= Integrate[Tan[x]/Sqrt[1 + Sec[x]^3], x] // LeafCount
Out[19]= 296

In[20]:= Integrate[ArcTan[Sqrt[-1 + Sec[x]]]*Sin[x], x] // LeafCount
Out[20]= 285

In[21]:= Integrate[(1 + x^2)/((1 - x^2)*Sqrt[1 + x^4]), x] // LeafCount
Out[21]= 36

In[22]:= Integrate[(1 - x^2)/((1 + x^2)*Sqrt[1 + x^4]), x] // LeafCount
Out[22]= 40

Answer 2

I've just run the Integration regression tests and added 11.3. This result below confirms the answer above.

The average leaf size of 11.3 Integrate result is much smaller than in 11.2. But it is still over 3 times as large as in version 9.0. But it is much better than in 11.2 which was over 6 times as large.

Found 146 Integrals in the test suite in which version 11.3 generate result at least 5 times as large as in version 9.0. This is down from 260 integrals in version 11.2

This below is average (normalized) leaf size. Normalized against optimal size. This means 11.3 generates, on average, result which is 6.3 as large as the optimal result.

This is average of the actual leaf size. (not normalized).

Also Found 27 integrals which now fails in 11.3 but did not fail in 11.2.

11.3 Integrate was a little faster than 11.2 on same PC

Full report can be found here