# How can I use list processing functions on data in a file?

How might I work with the following functions on a list, if the list is a disk based file?

Partition[]
DeleteDuplicates[]
Split[]
SortBy[]


The file is too large to ReadList[] or Import[] the data in the first place.

-
However, if there are a lot of duplicates, you may be able to work with it if you read the data line-by-line instead. Use Read[] on a stream you Open[] on the file. Prune the duplicates inside a function that reads the data that way.