3
$\begingroup$

I have a function that does a large number of calculations (by calling out to other smaller functions), and then builds a large Dataset object. Then I do two things with the object:

(a) I write it to disk as an MX file using Import; (b) I write the rows of the MX file to a SQL Server database (by using DatabaseLink).

So, the sequence of calls is something like this:

mainFunction[param1_, param2_] := Module[{dbConn, result},
    result = Composition[
        innerFunctionN,
        .
        .
        .
        innerFunction2,
        innerFunction1
    ][];
    (* At this point, result is a 250 MB Dataset object *)
    (* We now write this object to disk *)
    Export["result.mx", result]; (* This step happens correctly *)
    (* Set dbConn to a valid SQLConnection object *)
    DatabaseLink`SQLExecute[
        dbConn,
        queryBuilder[result]
    ]; (* This step fails; JDBC parse error from string to float *)
];

But curiously, the following succeeds:

mainFunction[param1_, param2_] := Module[{dbConn, result, result1},
    result = Composition[
        innerFunctionN,
        .
        .
        .
        innerFunction2,
        innerFunction1
    ][];
    (* At this point, result is a 250 MB Dataset object *)
    (* We now write this object to disk *)
    Export["result.mx", result]; (* This step happens correctly *)
    (* Set dbConn to a valid SQLConnection object *)
    (* NEW STEP: NOW READ RESULT BACK FROM DISK *)
    result1 = Import["result.mx"];
    DatabaseLink`SQLExecute[
        dbConn,
        queryBuilder[result1]
    ]; (* This succeeds! *)
];

This has been driving me crazy. The only conclusion I can draw is that something about the MX write is corrupting result in memory.

I am currently using the second method as a workaround. But do you have a better suggestion? Can you please help me understand the problem?

Thanks in advance for your help.

$\endgroup$
  • $\begingroup$ In the second case, is result == result1? $\endgroup$ – MarcoB May 22 '15 at 21:12
  • $\begingroup$ And also result === result1? $\endgroup$ – Mr.Wizard May 22 '15 at 21:13
  • $\begingroup$ Unfortunately, I don't have a ready answer to your questions. The sequence of calls innerFunction1, innerFunction2, ..., take about four hours to complete. I will have to set off the computations once again. But is there any reason why they should not be equal? $\endgroup$ – Shredderroy May 22 '15 at 21:23
  • $\begingroup$ Not that I can think of; it sounds like a strange problem indeed, but it wouldn't be the first time there was an explanation for something I thought was surely a bug. If a smaller working example is not possible I think you should contact Wolfram support about this. $\endgroup$ – Mr.Wizard May 22 '15 at 22:26
  • $\begingroup$ I have no solution, but I have seen similar behavior when I was writing (relatively) large Datasets in an earlier version 10 (I think it was 10.0.1). I have never investigated what it was and just reloaded as you do in the second example (which was almost immediate in my case). I just checked and found it does not happen for my case with 10.0.2 on Windows anymore. Which version are you using? $\endgroup$ – Albert Retey May 24 '15 at 8:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.