Take the 2-minute tour ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.

How might I work with the following functions on a list, if the list is a disk based file?

Partition[]
DeleteDuplicates[]
Split[]  
SortBy[]

The file is too large to ReadList[] or Import[] the data in the first place.

share|improve this question
    
You might find information that will help you by reading this question and its answers. –  m_goldberg Mar 23 '13 at 21:47
add comment

1 Answer 1

If your data set is too large to read in one big chunk, it seems unlikely that you'd be able to use those functions on it since they need to hold the entire set in memory.

However, if there are a lot of duplicates, you may be able to work with it if you read the data line-by-line instead. Use Read[] on a stream you Open[] on the file. Prune the duplicates inside a function that reads the data that way.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.