3
$\begingroup$

Consider some test data:

DataSamples = 
  Join[Partition[RandomReal[{0, 1}, 10^7], 1], 
   Partition[Sqrt[RandomReal[{0, 1}, 10^7]], 1], 2];
DataSamples2 = Exp[-DataSamples];

I would like to perform a test selection:

index1 = 1;
index2 = 2;
SelData[data_, index1_, index2_, value1_, value2_] := 
 Select[data, #[[index1]] > value1 && #[[index2]] < value2 &]

This realization is very slow:

SelData[DataSamples2, index1, index2, 0.3, 0.5]; // AbsoluteTiming

{17.1745,Null}

Alternatively, I may make a compiled selection:

SelDataCompiled = 
 Hold@Compile[{{data, _Real, 
      2}, {index1, _Integer}, {index2, _Integer}, {value1, _Real}, \
{value2, _Real}}, 
    Select[data, #[[index1]] > value1 && #[[index2]] < value2 &], 
    CompilationTarget -> "C", RuntimeOptions -> "Speed"] // 
  ReleaseHold
SelDataCompiled[DataSamples2, index1, index2, 0.3, 
   0.5]; // AbsoluteTiming

{0.671522,Null}

Is there any way to speedup this selection and similar ones further?

$\endgroup$

1 Answer 1

3
$\begingroup$

This is quite fast. Does it compile too? (I haven't installed a compiler.)

PickData[data_, index1_, index2_, value1_, value2_] :=
 Pick[data,
  UnitStep[-data[[All, index1]] + value1] +
   UnitStep[data[[All, index2]] - value2], 0]

PickData[DataSamples2, index1, index2, 0.3, 0.5]; // AbsoluteTiming

{0.747338, Null}

Compared with

SelData[DataSamples2, index1, index2, 0.3, 0.5]; // AbsoluteTiming

{19.7634, Null}

For compiling

Pick may not compile, i.e.

Map[MemberQ[Map[ToString, Compile`CompilerFunctions[]], #] &,
 {"Pick", "UnitStep"}]

{False, True}

so perhaps use this version

PartData[data_, index1_, index2_, value1_, value2_] :=
 Part[data, Flatten@Position[
    UnitStep[-data[[All, index1]] + value1] +
     UnitStep[data[[All, index2]] - value2], 0]]

All the above functions are compilable

Map[MemberQ[Map[ToString, Compile`CompilerFunctions[]], #] &,
 {"Part", "Flatten", "Position", "UnitStep"}]

{True, True, True, True}

PartDataCompiled = Hold@Compile[{
     {data, _Real, 2}, {index1, _Integer},
     {index2, _Integer}, {value1, _Real}, {value2, _Real}},
    Part[data, Flatten@Position[
       UnitStep[-data[[All, index1]] + value1] +
        UnitStep[data[[All, index2]] - value2], 0]],
    CompilationTarget -> "C", RuntimeOptions -> "Speed"] // ReleaseHold
PartDataCompiled[DataSamples2, index1, index2, 0.3, 0.5]; // AbsoluteTiming
$\endgroup$
3
  • 1
    $\begingroup$ On my machine PickData is 0.64s and PartDataCompiled is 0.75s $\endgroup$ Nov 23, 2022 at 12:08
  • 1
    $\begingroup$ @IntroductionToProbability Thanks for the info. $\endgroup$ Nov 23, 2022 at 12:10
  • 1
    $\begingroup$ I love the way you used CompilerFunctions[] to check if a function could be compiled. I have not thought to do this algorithmically and been referencing stackexchange pages to check manually "-_- $\endgroup$ Nov 23, 2022 at 12:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.