3
$\begingroup$

I have a dataset (satellite data) - Here is the link:

https://drive.google.com/drive/folders/1qBOROSDHSRMykZjWgP7iRMtpQ3gBMceL?usp=sharing

To read a specific part of the dataset one uses:

LISTA = FileNames[]; (* Here you assume you have your files in one specific Directory and list them*)

data1 = Import[
   LISTA[[1]], {"Datasets", "/PRODUCT/methane_mixing_ratio"}];

Checking the data structure one gets

Dimensions[data1]

{1, 4173, 215}

I want to get rid of some of the background (or NaN data) and put 0 (zeroes)

Newdata1 = Flatten[ReplaceAll[data1, s_ /; s > 5000 -> 0]]

When this is done however the data structure is "destroyed" and becomes only large one element list

Dimensions[Newdata1]

{897195}

Is there a way to replace those number and keep its structure (1 x 4173 x 214) ?

I have tried doing it without using the Flatten option but then I does not work I have seen some of the questions here and this might be a case of nested list issue, but I am not sure how to solve that .

$\endgroup$
6
  • 4
    $\begingroup$ You can use Replace to specify the level where you want to do the replacements. E.g., Replace[RandomReal[1, {3, 4, 5}], s_ /; s > 0.5 :> -1, {3}] $\endgroup$ Sep 30, 2020 at 15:50
  • $\begingroup$ Alternatively, also have a look at Select and Cases. $\endgroup$
    – Natas
    Sep 30, 2020 at 15:51
  • 1
    $\begingroup$ @Natas Select and Cases will most certainly destroy the structure of the data. $\endgroup$ Sep 30, 2020 at 15:53
  • $\begingroup$ @SjoerdSmit I guess it would be a bit cumbersome to keep the structure with those. Thanks for pointing this out. $\endgroup$
    – Natas
    Sep 30, 2020 at 15:58
  • 1
    $\begingroup$ It looks like the reason you lost the structure is because of the Flatten command in Newdata1, not because of the substitution. $\endgroup$
    – bill s
    Sep 30, 2020 at 19:13

2 Answers 2

1
$\begingroup$

A case for Clip?

Clip[RandomReal[10000, {3, 4, 5}],{5000,Infinity},{0,Infinity}]
$\endgroup$
1
$\begingroup$

You can also use Threshold, which, to my surprise, is faster than Clip.

SeedRandom[1]
data1 = RandomReal[10000, {1, 4173, 215}];

Replacing numbers greater than 5000 with 0:

t1a = First @ RepeatedTiming[
    res1a = Subtract[data1, Threshold[data1, 5000 + $MachineEpsilon]]];
t2a = First @ RepeatedTiming[res2a = Clip[data1, {0, 5000}, {0, 0}]];
t3a = First @ RepeatedTiming[res3a = data1 UnitStep[Subtract[5000, data1]]];
t4a = First @ RepeatedTiming[res4a = Replace[data1, s_ /; s > 5000 -> 0., All]];
t5a = First @ RepeatedTiming[res5a = ReplaceAll[data1, s_ /; s > 5000 -> 0.]];

res1a == res2a == res3a == res4a == res5a
True
Dimensions @ res1a
 {1, 4173, 215}

Replacing numbers less than 5000 with 0:

t1b = First @ RepeatedTiming[res1b = Threshold[data1, 5000 + $MachineEpsilon]];
t2b = First @ RepeatedTiming[res2b = Clip[data1, {5000, ∞}, {0, ∞}]];
t3b = First @ RepeatedTiming[res3b = data1 UnitStep[Subtract[data1, 5000]]];
t4b = First @ RepeatedTiming[res4b = Replace[data1, s_ /; s < 5000 -> 0., All]];
t5b = First @ RepeatedTiming[res5b = ReplaceAll[data1, s_ /; s < 5000 -> 0.]];

res1b == res2b == res3b == res4b == res5b
True
Dimensions @ res1b
 {1, 4173, 215}

Timing comparisons

{timingsa, timingsb} = {{t1a, t2a, t3a, t4a, t5a}, {t1b, t2b, t3b, t4b, t5b}};

functionsa = {"Subtract[data1,Threshold[data1,5000+$MachineEpsilon]]",
    "Clip[data1,{0,5000},{0,0}]", 
   "data1 UnitStep[Subtract[5000,data1]]", 
   "Replace[data1,s_/;s>5000\[Rule]0., All]", 
   "ReplaceAll[data1,s_/;s>5000\[Rule]0.]"};

functionsb = {"Threshold[data1,5000+$MachineEpsilon]", 
   "Clip[data1,{5000, Infinity},{0,Infinity}]", 
   "data1 UnitStep[Subtract[data1,5000]]", 
   "Replace[data1,s_/;s<5000\[Rule]0., All]", 
   "ReplaceAll[data1,s_/;s<5000\[Rule]0.]"};

Grid[Join[{{Item["Replace numbers greater than 5000 with 0", 
     Background -> LightBlue], SpanFromLeft}}, 
  Transpose[{functionsa, timingsa}], 
 {{Item["Replace numbers less than 5000 with 0", 
     Background -> LightBlue], SpanFromLeft}}, 
  Transpose[{functionsb, timingsb}]], 
 Alignment -> {{Left, "."}, Center}, Dividers -> All] 

enter image description here

$\endgroup$
1
  • $\begingroup$ Wow. Good to know Threshold! $\endgroup$
    – Lou
    Oct 31, 2020 at 9:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.