I really can't make a "smaller" example, but the files are here, for the data and for the notebook. It will cost a LOT of memory, up to 30G of RAM (maybe more).

Here is the problem, (from a Linux Server, MMA10.0.2 X64)

enter image description here

It does look like it finished the job. But res did not save as expected. I think this rise from a memory problem? Because at some point, the server(system) hang for a while.

Instead, if I use

(res = ParallelTable[
     myclosepop[dat[indat], kdat[indat], m], {indat, data}, {m, models}
 ];) // AbsoluteTiming

res will finish and save without any problem. The memory usage during the calculation will also be "smaller".

My main question is:

1 - can the code be improved to avoid this? 2 - What does the error message tell me?? Join::heads:...... does really not tell me much. I did not use it explicitly. And I dont even know whether this is a memory related problem.


Update: Individual ones do run OK, say with a smaller subsets of the models

enter image description here

I am doing in this way, hoping to get a "nice" table, with all results in it.

  • $\begingroup$ What happens if you run a subset of the data/models, or use Table instead of ParallelTable? $\endgroup$ – 2012rcampion Jan 29 '15 at 23:30
  • $\begingroup$ Single ones do run fine, even with different methods and search points ect. But I want to make a nice table like this, say run models = {"Bin", "BetaBin"}; then the res = .... then {lmax, aics, mles} = Transpose[res, {2, 3, 1}]; TableForm[lmax, TableHeadings -> {data, models}]. These gives an informative table, see updated post. $\endgroup$ – Chen Stats Yu Jan 30 '15 at 1:04

My guess is you are running into the same issue as was reported here. A possible workaround is to give NMinimize an explicit PostProcess option value, e.g."PostProcess" -> {"KKT"} (this goes inside the Method option setting).

  • $\begingroup$ I will try and see if that works. I have also updated the post, explaining why I am doing this. I will report the result back tomorrow. $\endgroup$ – Chen Stats Yu Jan 30 '15 at 1:07
  • $\begingroup$ I think the same problem still occur. On the server, the memory usage quickly goes up to about 40G and the process become a status "D" (it hangs basically). $\endgroup$ – Chen Stats Yu Jan 30 '15 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.