3
$\begingroup$

I have sorted some data into a dataset, but when exporting the dataset to be an external file I would only need certain columns within the dataset. I would like to drop the everything within the "Repeats" column (nested columns within etc) and the "Fitting Parameter" column. I know keyDrop can be used for outer keys for example, it would work fine at removing "Experiment Identifier", however I don't think it works for nested keys. An example dataset (which is empty) is coded underneath. The reason for wanting to do this is to save on storage space, as these columns would contain raw data that is no longer needed.

NumberOfBlocks = 100;
NumberOfRepeats = 2;
NumberOfWaveforms = 2;
NumberOfExperiments = 10;

(*Create dataset for data*)
Blocks = Table[<|"Block Number" -> i, 
    "Parameters" -> <|"Freq" -> 5, "Drifted Freq" -> {}|>, 
    "Waveforms" -> 
     Table[<|"Waveform Type" -> j, 
       "Repeats" -> 
        Table[<|"Repeat number" -> k, "Spectrum" -> {}, 
          "Fitting Parameters" -> {}|>, {k, NumberOfRepeats}], 
       "Fitting Parameter" -> {}, "Spectral Splitting" -> {}|>, {j, 
       NumberOfWaveforms}]|>, {i, NumberOfBlocks}];

DatasetRawData = 
  Table[<|"Experiment" -> i, 
    "Experiment Identifier" -> ToString["Example " <> ToString[i]], 
    "Data" -> Blocks|>, {i, NumberOfExperiments}];
(*End of Dataset creation*)
DatasetRawData // Dataset
```
$\endgroup$

1 Answer 1

4
$\begingroup$

We can use the Query (or Dataset) "part application" operator syntax {part->operator} to apply KeyDrop at the appropriate level:

DatasetRawData //
  Query[
    All
  , {"Data"->{All->{"Waveforms"->KeyDrop[{"Repeats","Fitting Parameter"}]}}}
  ] //
  Dataset

or we could use MapAt to do the same:

MapAt[
  KeyDrop[{"Repeats", "Fitting Parameter"}]
, DatasetRawData
, {All, "Data", All, "Waveforms"}
] // Dataset

Both return this result: dataset snapshot

If desired, these strategies can also be applied directly to a dataset object (named DatasetRaw in these examples):

DatasetRaw[
  All
, {"Data"->{All->{"Waveforms"->KeyDrop[{"Repeats","Fitting Parameter"}]}}}
]
MapAt[
  KeyDrop[{"Repeats", "Fitting Parameter"}]
, DatasetRaw
, {All, "Data", All, "Waveforms"}
]
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.