How might I work with the following functions on a list, if the list is a disk based file?


The file is too large to ReadList[] or Import[] the data in the first place.

  • $\begingroup$ You might find information that will help you by reading this question and its answers. $\endgroup$ – m_goldberg Mar 23 '13 at 21:47

If your data set is too large to read in one big chunk, it seems unlikely that you'd be able to use those functions on it since they need to hold the entire set in memory.

However, if there are a lot of duplicates, you may be able to work with it if you read the data line-by-line instead. Use Read[] on a stream you Open[] on the file. Prune the duplicates inside a function that reads the data that way.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.