1
$\begingroup$

I have a very large CSV data file, which I imported it to my active Notebook:

data = Import["filepath/myData.csv"];  (* data is a square matrix*)

Some calculations:

output = Inverse[data];

Then:

Export["output.mx", newFile];

Then:

Import["newFile.mx"]

To Extract the first 3 columns:

newFile[[All,3]]

I only get the 1st column because Import does not take the entire file. Why?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ I think you meant newfile[[All, ;;3]] to get the first three columns. Your newFile[[All, 3]] means "take only the third column of all rows". Also confused by your code here: What is in the variable newFile? Did you mean that to be output? Also, you export to "output.mx", but import from "newFile.mx": is that right? $\endgroup$ – MarcoB Feb 1 at 16:01
  • $\begingroup$ @MarcoB: You are right in all of your comments. I confused myself with newFile, with which I meant output. All these data operations with import/export aim to circumvent my computer's RAM problem. I do calculations to produce the output but at the time the output is produced, RAM is full. Therefore, I save the output to my hard disk & recall it for further calculations. Can you suggest the shortest way of doing what I described here? Thanks for your comment and would accept it as an answer if you post it. $\endgroup$ – Tugrul Temel Feb 1 at 16:29
  • 3
    $\begingroup$ My first recommendation would be to avoid inverting the matrix altogether, if you can (and you generally can). What do you want to do with that inverse? It might perhaps be best to abandon this question (delete it / close it) and ask a new one focusing on your underlying problem of avoiding matrix inversion / saving memory. An example of a matrix that will cause the problem (you can put it in a pastebin) and code to show what you want to achieve with it would be a good idea. $\endgroup$ – MarcoB Feb 1 at 16:41
2
$\begingroup$

Credit goes to @MarcoB as he suggested the following answer to my question that arose from a programming mistake. I am providing his answer to help others who may face the same issue.

Create a matrix mmB (can be a very very large matrix) and Save it as an MX file because it is the format that allows the fastest data transfer:

Export["mmB.mx", mmB];  (*exported to the hard disk*)

Then, retrieve it for further computations:

Import["mmB.mx"];

MX format is really the fastest data packing, and this Export/Import method is useful especially for those who have RAM limitations.

Improve this answer
$\endgroup$
2
  • $\begingroup$ I don’t think this answers your question nor do I really know what your question/goal was. Can you, please, further clarify this in your original question? What is the issue that you run into? This, to my understanding, would not circumvent “RAM limitations” as you have said, purely due to the fact that if you are not Quit-ing between these runs, the data continues to live in the kernel even after the Export is performed. You might also benefit from looking into the DumpSave function that is the intended means to save an expression/data in theMX format. $\endgroup$ – CA Trevillian Feb 1 at 23:42
  • $\begingroup$ @CA Trevillian: You might want to read my reply to @MarcoB above. I face the RAM problem as I have a large CSV matrix plus performed various operations with it. I convert the CSV data to MX and keep it on my hard disk and retrieve it when needed. I avoided the RAM limitation by doing this. I will take a look at DumpSave function. Thanks for your comments. $\endgroup$ – Tugrul Temel Feb 1 at 23:55

Not the answer you're looking for? Browse other questions tagged or ask your own question.