4
$\begingroup$

The following code is supposed to sequentially read data from a large CSV file and put it row by row into an SQL server. Although I thought RAM consumption should be constant the system keeps asking for more and more RAM.

$HistoryLength=3;
fileDestination="C:\\Users\\atfai\\Desktop\\destination.txt"
fileSource="C:\\Users\\atfai\\Desktop\\source.txt"

data=ReadString[fileSource,"\n"];
While[data=!=EndOfFile,
result=SQLInsert[conn,SQLTABLE["table_name"],availableColumns,StringSplit[data,","]];
If[result==$Failed,WriteString[fileDestination,data<>"\n"]];
data=ReadString[fileSource,"\n"];
];

Close[fileSource];
Close[fileDestination];

This code runs out of memory despite there being reading taking place row by row. What am I doing wrong here?

Also this code is slow since it it reading each row one by one. Is there a way to turn this into bulk insert but also be able to place rows which can't be inserted into the fileDestination?

I need help urgently if anyone can help.

$\endgroup$
  • 2
    $\begingroup$ A couple of observations: the $HistoryLength=3 is largely irrelevant here, as it only limits the length of completely evaluated expressions. The whole While loop is one complete expression, and retrhns Null in any event, so you're not saving memory. You could try printing the ByteCount of result, conn, and data, (and possibly also MemoryInUse[]) so you can track where the memoroy blow up is happening. I don't see anyting obviously wrong. $\endgroup$ – Itai Seggev Jul 11 '17 at 5:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.