3
$\begingroup$

I have experimental data where every run is stored in subsaves, e.g. Run 1 has subsaves run1.1, run1.2, run1.3, run1.4. and the data are cumulative. I import the data of a run using

files = FileNames[
  "run1*.dat", \
{"PathToData"}]
data = Import /@ files;

resulting in a list of lists.

To eliminate experimental problems (which are present) I would like to subtract the subsaves from each other that way, that I have each of the datasets isolated in order to reject the data containing the problems.

I demonstrate the task on a small part of my data:

run1 = {{{1., 38}, {2., 33}, {3., 31}, {4., 37}, {5., 30}, {6., 34}, {7., 
   37}, {8., 33}, {9., 34}}, {{1., 40}, {2., 35}, {3., 34}, {4., 
   40}, {5., 31}, {6., 36}, {7., 42}, {8., 36}, {9., 35}, {10., 
   42}}, {{1., 186}, {2., 165}, {3., 150}, {4., 150}, {5., 145}, {6., 
   146}, {7., 161}, {8., 146}, {9., 156}, {10., 167}}, {{1., 
   187}, {2., 166}, {3., 152}, {4., 153}, {5., 146}, {6., 148}, {7., 
   162}, {8., 148}, {9., 158}, {10., 169}}};
subsave11 = run1[[1]];
subsave12 = run1[[2]];
subsave13 = run1[[3]];
subsave14 = run1[[4]];

The resulting lists would be

pure1 = run1[[1]]
pure2 = Transpose[{subsave11[[All, 1]], 
    subsave12[[All, 2]] - subsave11[[All, 2]]}]
pure3 = Transpose[{subsave11[[All, 1]], 
    subsave13[[All, 2]] - subsave12[[All, 2]]}]
pure4 = Transpose[{subsave11[[All, 1]], 
    subsave14[[All, 2]] - subsave13[[All, 2]]}]


{{1., 38}, {2., 33}, {3., 31}, {4., 37}, {5., 30}, {6., 
  34}, {7., 37}, {8., 33}, {9., 34}, {10., 39}}

{{1., 2}, {2., 2}, {3., 3}, {4., 3}, {5., 1}, {6., 2}, {7., 
  5}, {8., 3}, {9., 1}, {10., 3}}

 {{1., 146}, {2., 130}, {3., 116}, {4., 110}, {5., 114}, {6.,
   110}, {7., 119}, {8., 110}, {9., 121}, {10., 125}}

 {{1., 1}, {2., 1}, {3., 2}, {4., 3}, {5., 1}, {6., 2}, {7., 
  1}, {8., 2}, {9., 2}, {10., 2}}

I know, that I can subtract a particular list from a list of lists with Map, but is there a way to tell Mathematica to run over all files in a folder and subtract n+1-n? Meaning to somehow put the last shown code block in a function to automate the process.

As I have a lot of runs and in total ~600 subsaves, it would be very annoying and time consuming to do it on foot...

Any advice or hint would be highly appreciated!

$\endgroup$
6
  • $\begingroup$ Welcome to Mathematica Stack Exchange. What is a "varying" list? All your lists containing pairs {_ , _} have the same length. Also if your question is about reading files from a directory, a sample of what's in the file would be helpful as this data may have to be imported. Would you like to append to those files, or create separate files etc. Perhaps you should break up this question in at least two questions. With which aspect of your solution are you having trouble? $\endgroup$
    – Syed
    Nov 4, 2021 at 14:33
  • $\begingroup$ There is such a way in Mma to import data from a file in a certain folder to the actual notebook as a list, do something with that list, store the result in that notebook and save the notebook. All these operations can be done programmatically. Therefore, this can be done in a loop going through all files in a given folder. Have a look at such functions as Import and NotebookDirectory and there around you will find some other useful ones that you will fill you need. Without a working example it is difficult to help you more. $\endgroup$ Nov 4, 2021 at 14:38
  • $\begingroup$ Hello and thank you for the replies. Importing the data is not the problem, though. I import the data with filenames and therefore have a list of lists containing all the subsaves of a specific run. I think Ill edit the question, so that the point gets more clear. $\endgroup$
    – sammyGreek
    Nov 4, 2021 at 14:44
  • $\begingroup$ @Syed I meant that it is not the same list every time, which can easily be made by Map. I want to subtract list4 from list5, list3 from list4, list2 from list3 ... $\endgroup$
    – sammyGreek
    Nov 4, 2021 at 15:29
  • 1
    $\begingroup$ @Syed the first sublist was miscopied, but you fixed it, as I saw in your answer $\endgroup$
    – sammyGreek
    Nov 4, 2021 at 19:33

2 Answers 2

3
$\begingroup$

(* I have added a tenth element to run1[[1]] *) This answer presents the subtraction automation part once you read a data file with the equivalent of run1 in it. The output will be called pure.

-------------------------------------

Utility function: subtracts last item

f[k_List, p_List] := Module[{},
  {k[[1]], Last@p - Last@k}
  ]

As an example (for comparison):

t1 = Transpose[{run1[[1]], run1[[2]]}]

This makes pairs of pairs that will each be an input to the utility function when it is applied.

{{{1., 38}, {1., 40}}, {{2., 33}, {2., 35}}, {{3., 31}, {3., 
   34}}, {{4., 37}, {4., 40}}, {{5., 30}, {5., 31}}, {{6., 34}, {6., 
   36}}, {{7., 37}, {7., 42}}, {{8., 33}, {8., 36}}, {{9., 34}, {9., 
   35}}, {{10, 36}, {10., 42}}}
f @@@ t1
{{1., 2}, {2., 2}, {3., 3}, {4., 3}, {5., 1}, {6., 2}, {7., 5}, {8., 
  3}, {9., 1}, {10, 6}}

Putting this into a Table; notice the table range.

t2 = Table[
  f @@@ Transpose[{run1[[i]], run1[[i + 1]]}], {i, 1, 
   Length@run1 - 1}]
{{{1., 2}, {2., 2}, {3., 3}, {4., 3}, {5., 1}, {6., 2}, {7., 5}, {8., 
   3}, {9., 1}, {10, 6}}, {{1., 146}, {2., 130}, {3., 116}, {4., 
   110}, {5., 114}, {6., 110}, {7., 119}, {8., 110}, {9., 121}, {10., 
   125}}, {{1., 1}, {2., 1}, {3., 2}, {4., 3}, {5., 1}, {6., 2}, {7., 
   1}, {8., 2}, {9., 2}, {10., 2}}}

Finally (if you want):

pure = {run1[[1]]}~Join~t2

This is ready to export to a file of your choice.

$\endgroup$
2
  • $\begingroup$ Great!Thank you so much @Syed, I used it on a larger dataset and iit works perfect. Especially the Module with the subsequent tabbing was exactly what I was looking for, as the number of subsaves varies, which I forgot to mention! $\endgroup$
    – sammyGreek
    Nov 4, 2021 at 19:29
  • $\begingroup$ @sammyGreek Thanks for the accept; I am glad I was able to help you. $\endgroup$
    – Syed
    Nov 4, 2021 at 19:33
2
$\begingroup$

Here is an example. For this I write the original data in a file using Put and read it with Get. You will have to adopt this according to your needs. I write the resulting data in a file with the prefix "New". Further, you may use "FileNames" to get all the file names in a directory.

Here I write the data file:

Put[{{{1, 10}, {2, 13}, {3, 15}, {4, 12}, {5, 14}}, {{1, 8}, {2, 
     7}, {3, 9}, {4, 6}, {5, 10}}, {{1, 11}, {2, 13}, {3, 14}, {4, 
     19}, {5, 11}}, {{1, 5}, {2, 16}, {3, 8}, {4, 6}, {5, 7}}}, 
  "d:/tmp/test.dat"];

And this reads the data and processes it:

Function[{file},
   dat = Get[file];
   dat[[2 ;;, All, 2]] = # - dat[[1, All, 2]] & /@ 
     dat[[2 ;;, All, 2]] ;
   dat[[All, All, 1]] = dat[[1, All, 1]];
   Put[dat, DirectoryName[file] <> "New" <> FileNameTake[file]];
   ] /@ FileNames["test.dat", "d:/tmp/"];
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.