The main question
I was trying to obtain the piecewise cubic-spline interpolating polynomials of a series of very long lists of (x, f(x)) pairs (O(100) of them with length ~350,000), i.e. to get InterpolatingPolynomial on tab[[1;;4]], tab[[2;;5]], ..., with tab[[i]]={xi,yi}. The xi for all those lists are the same, on the good side.
Given that I'm working on a cluster, I tried to use ParallelTable to accelerate the calculation, with the code
(* tabvr is a list of 100*O(350,000) of (x, f(x)) pairs. For toy data of tabvr, see below *)
totpoint=Length[tabvr[[1]]];
wp=100;
LaunchKernels[100];
SetSharedVariable[tabvr];
vrpp$poly=ParallelTable[SetPrecision[InterpolatingPolynomial[Take[tabvr[[kk]],{i-3,i}],x]//Expand,wp]
,{kk,1,100},{i,4,totpoint}];
I requested 120 cores (5 nodes of 24 cores) and 400GB of memory. However, it turned out to be not enough and my job was killed as it goes beyond the memory limit. Moreover, the ParallelTable didn't seem to run as fast as I expected, as it takes ~50s for a single-core machine to process a single list, while it took about 2 hours before my job was terminated.
So, did I do anything wrong in this parallel calculation, or is such situation something I should expect for a parallel calculation? If it's the first case, what should I do to make the code work?
Edit:
If anyone want to actually play with the data, you may find a slice of the data here (the vrpp86.dat therein). You may just copy it 100 times to make a "complete" tabvr.
A side question
I also realized that, though I requested 5*24 cores, when I check the processor number
Print[$ProcessorCount];
I got the answer as 24 instead of 120. Does this mean that MMA's parallelization cannot work across different nodes?
Update
As inspired by @Henrik Schumacher 's answer, I managed to shorten the running time of the code by a lot after using Compile
xgrid=tabvr[[1]];
ygrid=tabvr[[2]];
totpoint=Length[xgrid];
wp=100;
result0=SetPrecision[Table[CoefficientList[InterpolatingPolynomial[tabvr[[i-3;;i]],x],x],{i,4,totpoints}].{1,x,x^2,x^3}, wp]//AbsoluteTiming;
Print[result0[[1]]];
(*
to get the code working, mat in the following line should be replaced by the result of
D[CoefficientList[InterpolatingPolynomial[Array[{x[[#]], a[#]} &, 4], x], x], {Array[a, 4], 1}]//Simplify
*)
calc$coeff2 = Compile[{{x, _Real, 1}, {y, _Real, 1}}, mat.y,
CompilationTarget -> "C", RuntimeAttributes -> {Listable}, Parallelization -> True, RuntimeOptions -> "Speed"];
result2 = (Table[SetPrecision[calc$coeff2[tabxgrid[[i - 3 ;; i]], ygrid[[i - 3 ;; i]]], wp], {i, 4, totpoints}].{1, x, x^2, x^3}) // AbsoluteTiming;
Print[result2[[1]]];
And I get the result
(* result0 *)
19.95942
(* result1 *)
2.976688