The following code is supposed to sequentially read data from a large CSV file and put it row by row into an SQL server. Although I thought RAM consumption should be constant the system keeps asking for more and more RAM.
$HistoryLength=3; fileDestination="C:\\Users\\atfai\\Desktop\\destination.txt" fileSource="C:\\Users\\atfai\\Desktop\\source.txt" data=ReadString[fileSource,"\n"]; While[data=!=EndOfFile, result=SQLInsert[conn,SQLTABLE["table_name"],availableColumns,StringSplit[data,","]]; If[result==$Failed,WriteString[fileDestination,data<>"\n"]]; data=ReadString[fileSource,"\n"]; ]; Close[fileSource]; Close[fileDestination];
This code runs out of memory despite there being reading taking place row by row. What am I doing wrong here?
Also this code is slow since it it reading each row one by one. Is there a way to turn this into bulk insert but also be able to place rows which can't be inserted into the
I need help urgently if anyone can help.