# Loading and processing five files at a time from a directory containing many, many files

I have 3000 files (for example: aa101.csv, aa108.csv, aa120.csv, ...) and I need to process them. I need to load five files from a folder, process them, and export them to new list in Excel. Then I need to take another five files from the same folder and process them, and so on.

I tried

printer[n_] :=
Module[{file = n},

calculation...
...

Export[name <> ".xlsx", {data}, "Data"];
]

printer[Import["E:\\mathematica\\a\\*", "text"]];


But printer loads all files from the specified folder.

• Function printer take name of the file or data from file? – molekyla777 Jul 8 '14 at 12:30
• This function takes data from files – Lefl Jul 9 '14 at 7:00

This somewhat depends on your specific file names, desired target filename and location, the details of procedure etc. There would be a number of approaches. I post this for motivation.

In the following I assume:

• all the files for processing are in the same directory (and number 0 Mod 5)
• they have a sequential nomenclature that will be preserved
• they are .csv
• the target directory is the same folder

and will use "file001.xlsx","file002.xlsx",...etc and the procedure is a function fun:

SetDirectory["your directory"]
(* get file names and check in desired order*)
fn=FileNames["*.csv"]
(* partition into desired set of 5 *)
pfn=Partition[fn,5];
(* procedure on 5 files *)
proc=Map[Function[x,fun[Import[#,"Text"]&/@x]],pfn]
(*this produces the processed files in a list*)
(*exporting list*)
MapIndexed[Export[
"file" <> IntegerString[First@#2, 10, 3] <> ".xlsx",#1,"Data"] &, proc]


Whatever approach you ultimately use I suggest testing on small test set to confirm desired outcome.

• There is problem with function fun, it doesnt load data from files. Output: Import::chtype: First argument #1 is not a valid file, directory, or URL specification. >> {fun[{$Failed["file001.csv"],$Failed["file002.csv"], $Failed["file003.csv"],$Failed["file004.csv"], $Failed["file005.csv"]}], fun[{$Failed["file006.csv"], $Failed["file007.csv"],$Failed["TS2404_T1-8042.csv"], $Failed["TS2406_T1-806new2.csv"],$Failed[ "TS5620_T1-1484new.csv"]}]} – Lefl Jul 9 '14 at 7:11
• @Lefl as I commented fun was just the name I used for the user defined function/procedure hence in this code it is "dummy code" for insert whatever your procedure. I explicitly listed the point of the post was motivation. There was insufficient information in the question for a "solution". – ubpdqn Jul 9 '14 at 9:30
• I can´t give my calculations in "fun" because I can´t load data from files. Do you know please what I have to change to load data from files? – Lefl Jul 9 '14 at 9:55
• @Lefl if you just wish to get import the data then run your procedure: imported=Map[Function[x,Import[#,"Text"]/@x],pfn] should produce a list, each element will 5 imported csv files. Then you can run your procedure. – ubpdqn Jul 9 '14 at 10:09
• It still doesnt work :( – Lefl Jul 9 '14 at 11:20