0
$\begingroup$

I have a gigantic data file that needs to be processed. I am reading it in chunks using ReadList with an InputStream. In my configuration ReadList has a maximum reading speed of 30 MB/s. I was thinking of opening multiple streams from the same file and reading from them simultaneously, so that it can be done a little bit faster:

stream1=OpenRead["data.txt"]
stream2=stream1
Skip[stream2,String,5000000] 

and then

tp = Parallelize[
Join[ReadList[stream1, Real, 5000000], 
ReadList[stream2, Real, 5000000]]];

However, I get a message saying that this task cannot be parallelized.

Is there a way to let it work?

Many thanks in advance.

$\endgroup$
  • $\begingroup$ I think it's a hardware (hard drive) restriction. $\endgroup$ – user202729 Feb 3 '18 at 13:36
  • $\begingroup$ Have you tried Join @@ ParallelMap[ReadList[#, Real, 5000000] &, {stream1, stream2}]? $\endgroup$ – Michael E2 Feb 3 '18 at 13:37
  • $\begingroup$ A stream can not be simultaneously accessed by multiple reads. If you had a workflow where you read a chunk and then did some processing on that chunk then you could initiate additional reads on the stream while the chunk is being processed. However, this does not sound like what you are trying to do. $\endgroup$ – Edmund Feb 3 '18 at 13:39
  • $\begingroup$ what if he use a 0 raid configuration on multiple hdd ? $\endgroup$ – Alucard Feb 3 '18 at 14:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.