How can I reduce the memory cost of this parallel table?

Question

The main question

I was trying to obtain the piecewise cubic-spline interpolating polynomials of a series of very long lists of (x, f(x)) pairs (O(100) of them with length ~350,000), i.e. to get InterpolatingPolynomial on tab[[1;;4]], tab[[2;;5]], ..., with tab[[i]]={xi,yi}. The xi for all those lists are the same, on the good side.

Given that I'm working on a cluster, I tried to use ParallelTable to accelerate the calculation, with the code

(* tabvr is a list of 100*O(350,000) of (x, f(x)) pairs. For toy data of tabvr, see below *)

totpoint=Length[tabvr[[1]]];
wp=100;

LaunchKernels[100];
SetSharedVariable[tabvr];
vrpp$poly=ParallelTable[SetPrecision[InterpolatingPolynomial[Take[tabvr[[kk]],{i-3,i}],x]//Expand,wp]
          ,{kk,1,100},{i,4,totpoint}];

I requested 120 cores (5 nodes of 24 cores) and 400GB of memory. However, it turned out to be not enough and my job was killed as it goes beyond the memory limit. Moreover, the ParallelTable didn't seem to run as fast as I expected, as it takes ~50s for a single-core machine to process a single list, while it took about 2 hours before my job was terminated.

So, did I do anything wrong in this parallel calculation, or is such situation something I should expect for a parallel calculation? If it's the first case, what should I do to make the code work?

Edit:

If anyone want to actually play with the data, you may find a slice of the data here (the vrpp86.dat therein). You may just copy it 100 times to make a "complete" tabvr.

A side question

I also realized that, though I requested 5*24 cores, when I check the processor number

Print[$ProcessorCount];

I got the answer as 24 instead of 120. Does this mean that MMA's parallelization cannot work across different nodes?

---

Update

As inspired by @Henrik Schumacher 's answer, I managed to shorten the running time of the code by a lot after using Compile

xgrid=tabvr[[1]];
ygrid=tabvr[[2]];
totpoint=Length[xgrid];
wp=100;

result0=SetPrecision[Table[CoefficientList[InterpolatingPolynomial[tabvr[[i-3;;i]],x],x],{i,4,totpoints}].{1,x,x^2,x^3}, wp]//AbsoluteTiming;
Print[result0[[1]]];

(*
to get the code working, mat in the following line should be replaced by the result of
D[CoefficientList[InterpolatingPolynomial[Array[{x[[#]], a[#]} &, 4], x], x], {Array[a, 4], 1}]//Simplify
*) 
calc$coeff2 = Compile[{{x, _Real, 1}, {y, _Real, 1}}, mat.y, 
CompilationTarget -> "C", RuntimeAttributes -> {Listable}, Parallelization -> True, RuntimeOptions -> "Speed"];

result2 = (Table[SetPrecision[calc$coeff2[tabxgrid[[i - 3 ;; i]], ygrid[[i - 3 ;; i]]], wp], {i, 4, totpoints}].{1, x, x^2, x^3}) // AbsoluteTiming;
Print[result2[[1]]];

And I get the result

(* result0 *)
19.95942
(* result1 *)
2.976688

Answer 1

Just don't use a cluster. You can do the full job on a costumer computer within a couple of seconds, at least if 15 digits of precision are sufficient. (I really wonder why you ask for 100 digits of precision.)

This is what I did: I downloaded the file vrpp86.dat into my home directory:

tabvr = ToPack[Join[##, 2] & @@ ConstantArray[ToPack@Import["~/vrpp86.dat"], 50]];

n = 10000;
x =.;
result0 = Table[
     CoefficientList[
      InterpolatingPolynomial[tabvr[[kk, i - 3 ;; i]] , x], x],
     {kk, 1, n}, {i, 4, Dimensions[tabvr][[2]]}]; // AbsoluteTiming // First

38.6408

The combination of CoefficientList and InterpolatingPolynomial is actually a linear operator acting on 4-vectors. We can generate the according matrix as follows:

A = Developer`ToPackedArray@N@
    D[
     CoefficientList[InterpolatingPolynomial[Array[a, 4], x], x],
     {Array[a, 4], 1}
     ];

This allows us to avoid all symbolic calculations and leads to a first speed-up:

result1 = Table[
     A . tabvr[[kk, i - 3 ;; i]],
     {kk, 1, n}, {i, 4, Dimensions[tabvr][[2]]}]; // AbsoluteTiming // First

2.59673

Compiling this leads to a greater speed-up:

cf = Compile[{{tabvr, _Real, 2}, {A, _Real, 2}, {k, _Integer}},
   Block[{n, m},
    n = Dimensions[A][[2]] - 1;
    m = Dimensions[tabvr][[2]];
    Table[A . tabvr[[k, i - n ;; i]], {i, n + 1, m}]
    ],
   CompilationTarget -> "C",
   RuntimeAttributes -> {Listable},
   Parallelization -> True,
   RuntimeOptions -> "Speed"
   ];

result2 = cf[tabvr, A, Range[n]]; // AbsoluteTiming // First

0.153785

And with a bit more effort, we can squeeze even more optimizations out of the compiler:

cf2 = Block[{k, i, X, XX},
   XX = Table[ Compile`GetElement[X, k, l], {l, {i - 3, i - 2, i - 1, i}}];
   With[{code = A . XX},
    Compile[{{X, _Real, 2}, {k, _Integer}},
     Table[code, {i, 4, Dimensions[X][[2]]}],
     CompilationTarget -> "C",
     RuntimeAttributes -> {Listable},
     Parallelization -> True,
     RuntimeOptions -> "Speed"
     ]
    ]
   ];
result3 = cf2[tabvr, Range[n]]; // AbsoluteTiming // First

0.028772

All variants have 15 digits of precision:

Max[Abs[1 - result0/result1]]
Max[Abs[1 - result0/result2]]
Max[Abs[1 - result0/result3]]

4.44089*10^-16

4.44089*10^-16

4.44089*10^-16

The compiled code is automatically parallelized. (This won't work on distributed systems.)

Timings were taken on my Quad Core Haswell CPU.

The full job is completed in less than a second:

fullresult = cf2[tabvr, Range[Length[tabvr]]]; // AbsoluteTiming // First

0.861428