1
$\begingroup$

Is there a way in mathematica to do same operation on multiple files at once? And preferably have the output in a table or list form?

I want to do the following operation preferably in few non-labor intensive steps as possible. (As before this, I did it allll by hand, pluggging in each number for each file. AND IT WAS NO FUN;;-( )

I have files, a001, ... a100, which contains two field(column) of numerical data that look like the following

2 4

5 6

3 7

...
..
.

For each file, I need to import/use the data in a variable in an equation and locate the global maximum which boundaries.

f:= Beta[2+a,4+b]Beta[5+a,6+b]Beta[3+a,7+b]......../(Beta[a,b]^n)]

To make the problem more complicated n is the number of rows in each file.

Next I would use NMaximize function with ristriction on 100>a>0, 100>b>0 like the following

 NMaximize[{Log[f], 100 > a > .1^10, 100 > b > .1^10}, {a, b}, 
 Method -> "NelderMead", MaxIterations -> 10000]

And if possible I would like to get the results of all NMaximize methods.

 NMaximize[{Log10[f], 100 > a > .1^10, 100 > b > .1^10}, {a, b}, 
 Method -> "NelderMead", MaxIterations -> 10000]

 NMaximize[{Log10[f], 100 > a > .1^10, 100 > b > .1^10}, {a, b}, 
 Method -> "DifferentialEvolution", MaxIterations -> 10000]

 NMaximize[{Log10[f], 100 > a > .1^10, 100 > b > .1^10}, {a, b}, 
 Method -> "SimulatedAnnealing", MaxIterations -> 10000]

 NMaximize[{Log10[f], 100 > a > .1^10, 100 > b > .1^10}, {a, b}, 
 Method -> "RandomSearch", MaxIterations -> 10000]

The wanted result would look something like this : ( header not necessary ) ( Method name not necessary, but I want 4 results for each file. )

filename Method a b

f1 NM 1 4

f1 DE .2 .5

f1 SA .1 .5

f2 DE .2 .3

f2 NM 1 4

f2 DE 3 4

f2 SA .2 .1

f2 DE .5 .6

...

..

.

PLEASE HELP. IS THIS IN ANYWAY POSSIBLE?

EDIT: SampleData

$\endgroup$
9
  • 3
    $\begingroup$ Please put copies of three typical data files somewhere that others can download them without needing to register, log in, etc. and edit your original post to tell where you put them. Then someone can try a method and verify that it works. $\endgroup$
    – Bill
    Aug 12, 2015 at 16:35
  • 1
    $\begingroup$ Can you do this operation in an automated way for one file, if so put it in a table with all the filenames ...Table[f[n],{n,listOfFileNames}] $\endgroup$ Aug 12, 2015 at 16:50
  • $\begingroup$ @Bill thanks hadn't thought of that. I've added link to 5 sample data files.@image_doctor If need be I can do that. Now I am working with around 100 files, but ultimately, I would have about 100,000 files. So I was wondering if it could be done automatically. If not, I guess I'll have to come up with something else later. $\endgroup$
    – agathusia
    Aug 12, 2015 at 16:58
  • $\begingroup$ @agathusia Hi there, it would be automated, you would build the list of filenames using FileNames. $\endgroup$ Aug 12, 2015 at 17:32
  • 1
    $\begingroup$ @agathusia You may want to read some of the introductory programming material for Mathematica. It's basic data structure is the List. Oliver's solution builds you a list of functions, equivalent to the fs in your code, capable of computing the values you want. These functions are then applied to the values you want to compute using Map. This is typical of the functional programming style mathematica supports. Does Oliver's solution not provide the values you want, if so, in which way ? As an aside TableForm is just a formatting command to display a list nicely. $\endgroup$ Aug 13, 2015 at 6:44

2 Answers 2

1
$\begingroup$

Here is my quick answer.

Import the data from the files:

files = FileNames["/sample_data/1_5ormore_methy_00*"];
data = Import[#, "Table"] & /@ files;

Define a function to compute your values:

f[s1_, s2_, vals_] := 
   Times @@ (Beta[First@# + s1, Last@# + s2] & /@vals)/Beta[s1, s2]^Length@vals

Apply the function to the data for each of the desired methods:

res = Table[
    NMaximize[{Log[f[a, b, #]], 100 > a > .1^10, 100 > b > .1^10}, {a,
       b}, Method -> m, MaxIterations -> 10], 
     {m, {"NelderMead", "DifferentialEvolution","SimulatedAnnealing", "RandomSearch"}}] & 
    /@ data

Extract just the values for a and b and reshape to a list of a,b pairs:

Flatten[{a, b} /. #[[All, 2]] & /@ res , 1]
Export["/sample_data/1_5ormore_methy_all.csv", %]
$\endgroup$
8
  • $\begingroup$ Whilst you are experimenting with this you might want to drop MaxIterations to 10 say and maybe truncate the data to a small size by something like: smallData = #[[1 ;; 4]] & /@ data $\endgroup$ Aug 13, 2015 at 9:04
  • $\begingroup$ Thanks! I am currently running it. I've replaced s1_ s2_ with a_ b_. The immedieate problem I am seeing is that nnum error is showing and I think this will create some problem when Flattening. No? Is there a way to suppress nnum error? And I have worked on about 25 files now but haven't run into nnum error, so it concerns me a little. $\endgroup$
    – agathusia
    Aug 13, 2015 at 9:28
  • $\begingroup$ you don't need to replace s1 and s2, they are parameters allowing you to use a,b or any other symbol you like , what was your reasoning for doing that ? :) $\endgroup$ Aug 13, 2015 at 12:06
  • $\begingroup$ I changed it because I didn't know that it will allow me to use a,b in the next step. ;p $\endgroup$
    – agathusia
    Aug 13, 2015 at 14:20
  • $\begingroup$ It is a variable, it will contain whatever symbol name is passed into the function :) $\endgroup$ Aug 13, 2015 at 17:25
1
$\begingroup$

Of course this is possible. But I suggest that you break the problem down in smaller steps. Let's start with the problem of reading many files. If you have them in one directory, it can be as simple as

   files = FileNames["a*"];
   data = Import[#,"Table"] &/@ files;

This fills the list data with sub-lists, that contain the data n each file. Now you can create the expressions (no need to use functions, you can, if you want to, though) that you want to maximize

f = Function[u,Product[Beta[ #[[k,1]]+a, #[[k,2]]+b],{k,Length[ u] ]}]/Beta[a,b]^Length[u]] &/@ data

In simple words, this creates the 'Beta...Beta/Beta' function for one file. We assume that the data is stored in an array called u. We run through each row of the array (numbered k) and multiply everything (Product). Then we divide by the approriate power of the Beta-function (n is here simply the number of rows in the array u). We run this over all the top-level components of data (i.e. over all files), which is why this is a Function that is then mapped (/@) onto the data array. This gives an array of expressions f, one for each file.

And now you can maximize:

max=Table[ NMaximize[{Log[#], 100 > a > .1^10, 100 > b > .1^10}, {a, b}, Method -> m, MaxIterations -> 10000], {m,{"NelderMead", "DifferentialEvolution", "SimulatedAnnealing","RandomSearch"}}] &/@ f;

Run the maximizer of each component of f once for each method (this is what the Table does, it replaces m by the methods one by one), yielding an array with each of entry containing 4 solutions, one for each of the methods.

Now you just need to print them. The filename is still in files, so you want a table a bit like this

files[[1]], {a,b}/. max[[1,1]], {a,b}/. max[[1,2]], {a,b}/. max[[1,3]], {a,b}/. max[[1,4]]
.
.
.
files[[n]], {a,b}/. max[[n,1]], {a,b}/. max[[n,2]], {a,b}/. max[[n,3]], {a,b}/. max[[n,4]]

Use Map (or the /@ shorthand) to replace all four solutions and then a MapThread to simultaneously run through the files

 tt = MapThread[ {#1 , ({a,b}/. #2 ) } & ,{files, max}];

Now print it:

TableForm[ tt];

Or export it into a file for future use

Export[ tt, "myfile.dat", "Table"];

Done.

$\endgroup$
4
  • $\begingroup$ Could you explain the part after the maximizing step? I run max which results in an a table of value? What do you mean to 'use Map(/@) ot replace all four solutions..." $\endgroup$
    – agathusia
    Aug 12, 2015 at 22:22
  • $\begingroup$ Also it seems like for some of the files NMax is running into unidentified value problems. Is there anyway to handle this too? maybe just output an x? $\endgroup$
    – agathusia
    Aug 12, 2015 at 22:31
  • $\begingroup$ And it seems like the 'f=Function..' command gives a array of {Function[u,BetaBeta/Beta]},{Function[u,BetaBeta/Beta},.. Instead of {BetaBeta/Beta,BetaBeta/Beta,BetaBeta/Beta..} $\endgroup$
    – agathusia
    Aug 13, 2015 at 2:49
  • $\begingroup$ 'tt = MapThread[ {#1 ({a,b}/. #2 &) ,{files, max}];' seems like it is missing a bracket. Since i don't quite understand the function of MapThread I don't know where to place or remove one of the brackets. $\endgroup$
    – agathusia
    Aug 13, 2015 at 9:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.