The following is for Mathematica 8.0.4 running on Windows 7 laptop (6GB memory).
I have eight expressions that are unexpanded multivariate polynomials in 6 variables. They are huge expressions, so big that I cannot run Exponent to get the degrees in the variables. (After waiting for several minutes, I have to abort.)
Similarly, I abort any call to Expand after waiting a few minutes (or maybe an hour).
Then, I inspected the first "layer" of each expression and wrote a function that expands each part of the first layer and then combines the results. The function does Prints so that I know there is progress. For example, the first expression is the sum of three sub-expressions. The function calculates and expands the individual sub-expressions and then combines them.
Starting with the simplest expression, I ran the function. Each sub-expression had (order of magnitude) 500K terms. There was a lot of cancellation and the sum of the three sub-expressions ended up being about 500K terms. The polynomials have rational coefficients (actually integer) with the exception of a Sqrt. I replaced Sqrt with a 7th variable to make it rational. Then, I ran Factor on the results and got it down to 80K terms. Almost manageable, also note the Exponent call now worked.
I repeated this for four more expressions, each expression being more complicated than the previous. Each expression had lots of cancellation and then factored. So far, I have 5 out of the 8 expressions simplified. I had some sub-expressions with nearly 2M terms.
For the 6th expression, I am stuck. It is the sum of four sub-expressions. My function expanded the first two successfully (2.3M terms and 1.9M terms, respectively). It has been working on the third sub-expression for 2 days.
Trying to assess whether it was making progress, I have been looking at the "Windows Task Manager" to see how the "mathkernel.exe" process is doing. It barely using any CPU% but it is slowly accumulating CPU Time (maybe 30s in the last hour). It is doing something, because I see the memory changing. It is maxing out memory (goes up to approx 3.7GB, drops and rises again) and having some hard faults.
I also tried to use the "Evaluation->Kernel Status" menu command. This puts me into a cycle of getting the prompt "The kernel is not responding to a dynamic evaluation. You may either choose abort and restart the kernel or continue waiting". I finally chose Abort and this killed the kernel. It had accumulated about 8Hrs of CPU time over a few days.
Has anyone had success or knowledge about expanding huge expressions that simplify and factor? Maybe I can go another layer deeper and expand those sub-sub-expressions.
Update. I have made some tweaks to this evaluation. I finally got Mathematica to explicitly tell me that it ran out of memory. I think it had to have enough memory to finish the calculation. I got the message:
No more memory available. Mathematica kernel has shut down. Try quitting other applications and then retry.