1
$\begingroup$

I'm trying to write a short program that does the following over and over again.

  1. Import all files in a folder
  2. Automatically generate names for the imported files based on source file name.
  3. Perform a simple operation on the file.
  4. Re-Export the modified files with a new name.

For concreteness, here's an example of how I would do this manually. It is important that I be able to have a descriptive name for the files (e.g. file1 etc).

root = NotebookDirectory[];
file1 = Import[root <> "file1.csv"];
file2 = Import[root <> "file2.csv"];

These files are just .csv files.
file1 is {{1, 1}, {2, 9}, {3, 3}, {4, 2}, {5, 9}, {6, 1}, {7, 6}}

file2 is {{1, 6}, {2, 0}, {3, 4}, {4, 7}, {5, 10}, {6, 9}, {7, 3}}

Then I would modify the files. I've defined a simple function that adds a shift to each column of the table with name 'FILEname':

MODdata[FILEname_] := 
 Table[{FILEname[[i]][[1]] + 2, FILEname[[i]][[2]] + 10}, {i, 1, 
   Length[FILEname], 1}]
file1mod = MODdata[file1];
file2mod = MODdata[file2];

Then finally I want to export the files with a new name that indicates that they've been modified.

Export[root <> "file1_mod.csv", file1mod]
Export[root <> "file2_mod.csv", file2mod]

Any help with this would be greatly appreciated!

......... EDIT ............

I tried to adapt @TumbiSapichu suggestion to my case but am still having issues. In the code below I replaced myAnalysedData=readCurrentCSV+1 with my "MODdata" function above:

root = NotebookDirectory[];
myCurrentCSVs = FileNames[root <> "*.csv"];
i = 1;
While[i <= Length[myCurrentCSVs], 
 readCurrentCSV = Import[myCurrentCSVs[[i]]];
 (*Make some operation on the data,say,add 1*)
 myAnalysedData = 
  Table[{readCurrentCSV[[k]][[1]] + 2, 
    readCurrentCSV[[k]][[2]] + 10}, {k, 1, Length[readCurrentCSV], 1}];
 (*Now,export the analysed data to new file*)
 Export[StringJoin["Analyzed", myCurrentCSVs[[i]]], myAnalysedData];
 i++]

I'm getting both an 'Export: Directory xxxx does not exist' and an 'OpenWrite' Cannot open xxxx/file1.csv' error for both files. Maybe I'm not specifying the file paths correctly with my root = NotebookDirectory[]?

$\endgroup$
2
  • $\begingroup$ Are the CSV files in the same directory as NotebookDirectory[]? $\endgroup$ Jan 30 at 1:47
  • 1
    $\begingroup$ A better way to implement MODdata: modify[data_] := {First@# + 2, Last@# + 10} & /@ data. $\endgroup$ Jan 30 at 1:56
3
$\begingroup$

This is a very common task for me and most probably many other Mathematica users as well. It typically looks somewhat like the following when I do this:

Scan[
  Function[filename,
    Module[{data,newdata,newfilename},
      Print["working on "<>FileBaseName[filename]<>"..."];
      data=Import[filename];
      newdata=manipulateData[data];
      newfilename=FileNameJoin[{
        DirectoryName[filename],
        StringJoin[FileBaseName[filename],"_mod.csv"]
      }];
      Export[newfilename,newdata]
    ]
  ],
  FileNames["*.csv",NotebookDirectory[]]
]

manipulateData would be a function you can write, test and debug before running it on the files with ad hoc data which makes it much easier to get it right.

The Print is of course optional, but for most realistic cases data manipulation will take a while and it is a simple way to see the progress and relate potential errors to the correct file. For more details about the rest of the code, I think the documentation of the functions used should be sufficient...

An implementation of manipulateData (based on Rohit's suggestion) for your specific case could look like:

manipulateData[data_]:=({#1+2,#2+10}&@@@data);
$\endgroup$
1
$\begingroup$

Suppose you have many .csv files in your current directory, named "File1.csv", "File2.csv", etc.

You can first get the names of all .csv files:

myCurrentCSVs=FileNames["*.csv"];

Then, you can make some operation, opening each at a time, for instance, and exporting the new file with some "prefix" that indicates they have been modified/analysed:

i=1;
While[i<=Length[myCurrentCSVs],
readCurrentCSV=Import[myCurrentCSVs[[i]]];
(*Make some operation on the data, say, add 1*)
myAnalysedData=readCurrentCSV+1;
(*Now, export the analysed data to new file*)
Export[StringJoin["Analyzed", myCurrentCSVs[[i]]], myAnalysedData];
i++]

I'm sure you can adapt the process above for your needs. For some complicated processes, where each analysis involves some sort of indexing, change of parameters, or any other number that you would like to add to the exported file name I recommend checking IntegerString[]. That is pretty useful to take those numbers and making them strings with leading zeroes. For instance, if you know you'll have from 1 to 100 indexes for the exported names, you can have strings like {001,002,003,...,100} with IntegerString[#,10,3].

$\endgroup$
3
  • $\begingroup$ Rather than While, use Map. The symbol i is only used for indexing. Rather than StringJoin, you might want to use FileNameJoin if e.g. you want the exported files to be in a different directory. $\endgroup$ Jan 29 at 23:58
  • 1
    $\begingroup$ @TumbiSapichu It may be simple to adapt that to what I want but I couldn't get it to work. This failed: root = NotebookDirectory[]; myCurrentCSVs = FileNames[root <> "*.csv"]; i = 1; While[i <= Length[myCurrentCSVs], readCurrentCSV = Import[myCurrentCSVs[[i]]]; (*Make some operation on the data,say,add 1*) myAnalysedData = Table[{readCurrentCSV[[k]][[1]] + 2, readCurrentCSV[[k]][[1]] + 10}, {k, 1, Length[readCurrentCSV], 1}];; (*Now,export the analysed data to new file*) Export[root <> StringJoin["Analyzed", myCurrentCSVs[[i]]], myAnalysedData]; i++] $\endgroup$
    – Drotar
    Jan 29 at 23:59
  • $\begingroup$ I am not sure I understand your comment. The code I gave should work, as @RohitNamjoshi mentioned, you can also export to a different folder. What I'm doing is reading one file at a time, do some calculation and exporting the result to another file, why would you want to load ALL the files? You could also do that, but depending on how large they are you might run out of memory. Maybe you can edit your original question to add more details. $\endgroup$ Jan 30 at 0:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.