I have a gigantic data file that needs to be processed. I am reading it in chunks using ReadList with an InputStream. In my configuration ReadList has a maximum reading speed of 30 MB/s. I was thinking of opening multiple streams from the same file and reading from them simultaneously, so that it can be done a little bit faster:


and then

tp = Parallelize[
Join[ReadList[stream1, Real, 5000000], 
ReadList[stream2, Real, 5000000]]];

However, I get a message saying that this task cannot be parallelized.

Is there a way to let it work?

Many thanks in advance.

  • $\begingroup$ I think it's a hardware (hard drive) restriction. $\endgroup$
    – user202729
    Commented Feb 3, 2018 at 13:36
  • $\begingroup$ Have you tried Join @@ ParallelMap[ReadList[#, Real, 5000000] &, {stream1, stream2}]? $\endgroup$
    – Michael E2
    Commented Feb 3, 2018 at 13:37
  • $\begingroup$ A stream can not be simultaneously accessed by multiple reads. If you had a workflow where you read a chunk and then did some processing on that chunk then you could initiate additional reads on the stream while the chunk is being processed. However, this does not sound like what you are trying to do. $\endgroup$
    – Edmund
    Commented Feb 3, 2018 at 13:39
  • $\begingroup$ what if he use a 0 raid configuration on multiple hdd ? $\endgroup$
    – Alucard
    Commented Feb 3, 2018 at 14:06


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.