# Doing same operation on multiple files

Is there a way in mathematica to do same operation on multiple files at once? And preferably have the output in a table or list form?

I want to do the following operation preferably in few non-labor intensive steps as possible. (As before this, I did it allll by hand, pluggging in each number for each file. AND IT WAS NO FUN;;-( )

I have files, a001, ... a100, which contains two field(column) of numerical data that look like the following

2 4

5 6

3 7

...
..
.

For each file, I need to import/use the data in a variable in an equation and locate the global maximum which boundaries.

f:= Beta[2+a,4+b]Beta[5+a,6+b]Beta[3+a,7+b]......../(Beta[a,b]^n)]


To make the problem more complicated n is the number of rows in each file.

Next I would use NMaximize function with ristriction on 100>a>0, 100>b>0 like the following

 NMaximize[{Log[f], 100 > a > .1^10, 100 > b > .1^10}, {a, b},
Method -> "NelderMead", MaxIterations -> 10000]


And if possible I would like to get the results of all NMaximize methods.

 NMaximize[{Log10[f], 100 > a > .1^10, 100 > b > .1^10}, {a, b},
Method -> "NelderMead", MaxIterations -> 10000]

NMaximize[{Log10[f], 100 > a > .1^10, 100 > b > .1^10}, {a, b},
Method -> "DifferentialEvolution", MaxIterations -> 10000]

NMaximize[{Log10[f], 100 > a > .1^10, 100 > b > .1^10}, {a, b},
Method -> "SimulatedAnnealing", MaxIterations -> 10000]

NMaximize[{Log10[f], 100 > a > .1^10, 100 > b > .1^10}, {a, b},
Method -> "RandomSearch", MaxIterations -> 10000]


The wanted result would look something like this : ( header not necessary ) ( Method name not necessary, but I want 4 results for each file. )

filename Method a b

f1 NM 1 4

f1 DE .2 .5

f1 SA .1 .5

f2 DE .2 .3

f2 NM 1 4

f2 DE 3 4

f2 SA .2 .1

f2 DE .5 .6

...

..

.

EDIT: SampleData

• Please put copies of three typical data files somewhere that others can download them without needing to register, log in, etc. and edit your original post to tell where you put them. Then someone can try a method and verify that it works. – Bill Aug 12 '15 at 16:35
• Can you do this operation in an automated way for one file, if so put it in a table with all the filenames ...Table[f[n],{n,listOfFileNames}] – image_doctor Aug 12 '15 at 16:50
• @Bill thanks hadn't thought of that. I've added link to 5 sample data files.@image_doctor If need be I can do that. Now I am working with around 100 files, but ultimately, I would have about 100,000 files. So I was wondering if it could be done automatically. If not, I guess I'll have to come up with something else later. – agathusia Aug 12 '15 at 16:58
• @agathusia Hi there, it would be automated, you would build the list of filenames using FileNames. – image_doctor Aug 12 '15 at 17:32
• @agathusia You may want to read some of the introductory programming material for Mathematica. It's basic data structure is the List. Oliver's solution builds you a list of functions, equivalent to the fs in your code, capable of computing the values you want. These functions are then applied to the values you want to compute using Map. This is typical of the functional programming style mathematica supports. Does Oliver's solution not provide the values you want, if so, in which way ? As an aside TableForm is just a formatting command to display a list nicely. – image_doctor Aug 13 '15 at 6:44

Import the data from the files:

files = FileNames["/sample_data/1_5ormore_methy_00*"];
data = Import[#, "Table"] & /@ files;


Define a function to compute your values:

f[s1_, s2_, vals_] :=
Times @@ (Beta[First@# + s1, Last@# + s2] & /@vals)/Beta[s1, s2]^Length@vals


Apply the function to the data for each of the desired methods:

res = Table[
NMaximize[{Log[f[a, b, #]], 100 > a > .1^10, 100 > b > .1^10}, {a,
b}, Method -> m, MaxIterations -> 10],
/@ data


Extract just the values for a and b and reshape to a list of a,b pairs:

Flatten[{a, b} /. #[[All, 2]] & /@ res , 1]
Export["/sample_data/1_5ormore_methy_all.csv", %]

• Whilst you are experimenting with this you might want to drop MaxIterations to 10 say and maybe truncate the data to a small size by something like: smallData = #[[1 ;; 4]] & /@ data – image_doctor Aug 13 '15 at 9:04
• Thanks! I am currently running it. I've replaced s1_ s2_ with a_ b_. The immedieate problem I am seeing is that nnum error is showing and I think this will create some problem when Flattening. No? Is there a way to suppress nnum error? And I have worked on about 25 files now but haven't run into nnum error, so it concerns me a little. – agathusia Aug 13 '15 at 9:28
• you don't need to replace s1 and s2, they are parameters allowing you to use a,b or any other symbol you like , what was your reasoning for doing that ? :) – image_doctor Aug 13 '15 at 12:06
• I changed it because I didn't know that it will allow me to use a,b in the next step. ;p – agathusia Aug 13 '15 at 14:20
• It is a variable, it will contain whatever symbol name is passed into the function :) – image_doctor Aug 13 '15 at 17:25

Of course this is possible. But I suggest that you break the problem down in smaller steps. Let's start with the problem of reading many files. If you have them in one directory, it can be as simple as

   files = FileNames["a*"];
data = Import[#,"Table"] &/@ files;


This fills the list data with sub-lists, that contain the data n each file. Now you can create the expressions (no need to use functions, you can, if you want to, though) that you want to maximize

f = Function[u,Product[Beta[ #[[k,1]]+a, #[[k,2]]+b],{k,Length[ u] ]}]/Beta[a,b]^Length[u]] &/@ data


In simple words, this creates the 'Beta...Beta/Beta' function for one file. We assume that the data is stored in an array called u. We run through each row of the array (numbered k) and multiply everything (Product). Then we divide by the approriate power of the Beta-function (n is here simply the number of rows in the array u). We run this over all the top-level components of data (i.e. over all files), which is why this is a Function that is then mapped (/@) onto the data array. This gives an array of expressions f, one for each file.

And now you can maximize:

max=Table[ NMaximize[{Log[#], 100 > a > .1^10, 100 > b > .1^10}, {a, b}, Method -> m, MaxIterations -> 10000], {m,{"NelderMead", "DifferentialEvolution", "SimulatedAnnealing","RandomSearch"}}] &/@ f;


Run the maximizer of each component of f once for each method (this is what the Table does, it replaces m by the methods one by one), yielding an array with each of entry containing 4 solutions, one for each of the methods.

Now you just need to print them. The filename is still in files, so you want a table a bit like this

files[[1]], {a,b}/. max[[1,1]], {a,b}/. max[[1,2]], {a,b}/. max[[1,3]], {a,b}/. max[[1,4]]
.
.
.
files[[n]], {a,b}/. max[[n,1]], {a,b}/. max[[n,2]], {a,b}/. max[[n,3]], {a,b}/. max[[n,4]]


Use Map (or the /@ shorthand) to replace all four solutions and then a MapThread to simultaneously run through the files

 tt = MapThread[ {#1 , ({a,b}/. #2 ) } & ,{files, max}];


Now print it:

TableForm[ tt];


Or export it into a file for future use

Export[ tt, "myfile.dat", "Table"];


Done.

• Could you explain the part after the maximizing step? I run max which results in an a table of value? What do you mean to 'use Map(/@) ot replace all four solutions..." – agathusia Aug 12 '15 at 22:22
• Also it seems like for some of the files NMax is running into unidentified value problems. Is there anyway to handle this too? maybe just output an x? – agathusia Aug 12 '15 at 22:31
• And it seems like the 'f=Function..' command gives a array of {Function[u,BetaBeta/Beta]},{Function[u,BetaBeta/Beta},.. Instead of {BetaBeta/Beta,BetaBeta/Beta,BetaBeta/Beta..} – agathusia Aug 13 '15 at 2:49
• 'tt = MapThread[ {#1 ({a,b}/. #2 &) ,{files, max}];' seems like it is missing a bracket. Since i don't quite understand the function of MapThread I don't know where to place or remove one of the brackets. – agathusia Aug 13 '15 at 9:16