Modify s and let Dot do the computation.
s2 = Normal@SparseArray[Thread[Abs@s -> Sign@s], Length@First@l];
l.s2 // RepeatedTiming
(* {4.7051429748535154` *^-7, {6,14,19,19}}*)
#1 - #3 + #5 & @@@ l // RepeatedTiming
(* {1.889423370361328` *^-6, {6,14,19,19}}*)
l[[;; , Abs@s]] . Sign@s // RepeatedTiming
(* {2.0671768188476562` *^-6, {6,14,19,19}}*)
It's slightly faster than the other answers. Aesthetically, I like the simple dot product.
lMax = RandomInteger[{1, Times @@ #}, #] &@{31, 37};
sMax = RandomChoice[{-1, 0, 0, 0, 1}, Length@First@lMax];
lMax.sMax // RepeatedTiming
(* {7.610435485839844` *^-7 , {-3202,-4288,-2123,-6777,-4301,-1779,-1781,-\
2826,-2023,-1972,-2624,-5320,-2773,-3360,-3533,-2929,-3197,-700,-2891,\
-5077,-3149,-2768,-2477,-508,-2841,-4150,-3876,-4284,-3164,-2091,-\
2712}}*)
ss = Flatten[Position[sMax,1|-1]*Sign@Cases[sMax,1|-1]];
lMax[[;; , Abs@ss]] . Sign@ss // RepeatedTiming
(* {1.4854793548583985` *^-6 , {-3202,-4288,-2123,-6777,-4301,-1779,-1781,\
-2826,-2023,-1972,-2624,-5320,-2773,-3360,-3533,-2929,-3197,-700,-\
2891,-5077,-3149,-2768,-2477,-508,-2841,-4150,-3876,-4284,-3164,-2091,\
-2712}}*)
edit:
I forgot that RandomInteger and RandomChoice return a packed array (Developer`PackedArrayQ), which speeds up the computation. Here are the times if it's not packed (name is Min rather than Max if it's not packed):
lMax.sMin -> 1.01245*10^-6
lMin.sMax -> 3.54877*10^-6
lMin.sMin -> 0.0000138674
lMax[[;;,Abs@ss]].Sign@ss -> 1.48547 *^-6