3 hour calculation? ParallelMap & ParallelTable

Question

I'm having some problems with slow Mathematica code... I'm basically doing a computation that's is expected to take 3 hours (!) in my Core i7 with 4 processors... I'm basically computing an index (of stock market options implied volatilities), with my data coming from a (359,835 x 17) matrix, which I call CompleteMatrix. However, I know that my code is too basic, and I'm trying to find some ways to improve it with ParallelMap / ParallelTable or even Compile (don't know how though). And, more interesting, for some calculations Map is faster than ParallelMap and Table is faster than ParallelTable... Here are some samples of my code and how long it takes to calculate:

PregaoMC[v_] := CompleteMatrix[[v, 1.]];
n = 359835;

(Is there any way to improve this function with Compile?!)

In[17]:= DeleteDuplicates[Table[PregaoMC[x], {x, 2., n}]];//AbsoluteTiming
Out[17]= {0.824105, Null}

In[18]:= DeleteDuplicates[ParallelTable[PregaoMC[x], {x, 2., n}]];//AbsoluteTiming
Out[18]= {1.080137, Null}

In[19]:= DeleteDuplicates[Map[PregaoMC, Range[2., n]]];//AbsoluteTiming
Out[19]= {0.707090, Null}

In[20]:= DeleteDuplicates[ParallelMap[PregaoMC, Range[2., n]]];//AbsoluteTiming
Out[20]= {1.022129, Null}

Another calculation which is taking a lot of time is:

AtivoMC[v_] := CompleteMatrix[[v, 2.]];

In[32]:= Select[Flatten[Position[{Table[AtivoMC[x], {x, 1, n}]}, "StockName"]], # > 1 &]; // AbsoluteTiming
Out[32]= {0.887112, Null}

In[33]:= Select[Flatten[Position[{Map[AtivoMC, Range[ 1., n]]}, "StockName"]], # > 1 &]; // AbsoluteTiming
Out[33]= {0.793101, Null}

In[34]:= Select[Flatten[Position[{ParallelTable[AtivoMC[x], {x, 1, n}]}, "StockName"]], # > 1 &]; // AbsoluteTiming
Out[34]= {1.150145, Null}

In[35]:= Select[Flatten[Position[{ParallelMap[AtivoMC, Range[ 1., n]]}, "StockName"]], # > 1 &]; // AbsoluteTiming
Out[35]= {1.040132, Null}

Apparently these calculations don't take too long, however I have to do them several times, for several different stocks and several different days... Is there any way to improve whis code with Compile?

Thank you! Rodrigo

Answer 1

First, I notice that you are using Real numbers such as 1. and 2. for Part indexes. While this works it would be better to use Integer indexes, 1 and 2.

Your use of PregaoMC and then Table, etc., is highly inefficient. Part and Span will be better. Observe:

Table[PregaoMC[x], {x, 2, n}] === CompleteMatrix[[2 ;; n, 1]]

True

n = 359835;
CompleteMatrix = RandomReal[9, {n, 17}];

DeleteDuplicates @ CompleteMatrix[[2 ;; n, 1]]; // AbsoluteTiming

 {0.0250000, Null}

I believe your second code section is equivalent to this:

Join @@ Position[CompleteMatrix[[All, 2]], "StockName"]

This will likewise be much faster.