I have a gigantic data file that needs to be processed. I am reading it in chunks using
ReadList with an
InputStream. In my configuration
ReadList has a maximum reading speed of 30 MB/s. I was thinking of opening multiple streams from the same file and reading from them simultaneously, so that it can be done a little bit faster:
stream1=OpenRead["data.txt"] stream2=stream1 Skip[stream2,String,5000000]
tp = Parallelize[ Join[ReadList[stream1, Real, 5000000], ReadList[stream2, Real, 5000000]]];
However, I get a message saying that this task cannot be parallelized.
Is there a way to let it work?
Many thanks in advance.