# Dealing with a large number of optimizations performed with parameters read in from a CSV file

I have a large CSV file (1.5M rows), with each row containing a set of parameters that I need for an optimization routine in Mathematica. I'm hoping to import the data, run the computation, and then append the output to each respective row in the CSV. What is the most efficient way to do this? I've been going in circles with modules, blocks, and functions. I would use the apply family in R but I'm confused as to what to do in Mathematica.

Specifically, here is what I'm hoping to do. I have two expressions:

Gq = b γ-(E^((b Sqrt[μ^2+2 λ σ^2])/σ^2) (E^((b Sqrt[μ^2+2 λ σ^2])/σ^2) mc+E^((b μ)/σ^2) me) ((E^((b Sqrt[μ^2+2 λ σ^2])/(2 σ^2)) Sqrt[E^((b Sqrt[μ^2+2 λ σ^2])/σ^2) mc+E^((b μ)/σ^2) me])/Sqrt[mc+E^((b (μ+Sqrt[μ^2+2 λ σ^2]))/σ^2) me])^(-1-μ/Sqrt[μ^2+2 λ σ^2]) (-μ^2-2 λ σ^2+μ Sqrt[μ^2+2 λ σ^2]))/((-1+E^((2 b Sqrt[μ^2+2 λ σ^2])/σ^2)) λ (-μ+Sqrt[μ^2+2 λ σ^2]))


and

sq = (σ^2 Log[(E^((b Sqrt[μ^2+2 λ σ^2])/(2 σ^2)) Sqrt[E^((b Sqrt[μ^2+2 λ σ^2])/σ^2) mc+E^((b μ)/σ^2) me])/Sqrt[mc+E^((b μ)/σ^2+(b Sqrt[μ^2+2 λ σ^2])/σ^2) me]])/Sqrt[μ^2+2 λ σ^2]


Then I have a separate set of parameters σ, μ,λ,mc, me, γ in each row of the file. Then, I'd like to run the following routine, with the corresponding parameters contained in each row of the CSV file. I've substituted parameter values here as an example.

Optb =
FindMinimum[Gq /. {σ -> 10000, μ -> 1000, λ -> .1, mc -> .0085, me -> .0055, γ -> .0055}, {b, 1, 10000000}]
optcost = Optb[]
optbud = b /. Optb[]
optq =
sq /. {σ -> 10000, μ ->  1000, λ -> .1, mc -> .0085, me -> .0055, γ -> .0055, Optb[[2, 1]]}


I'd like to append the computed values of optcost, optbud, and optq to each corresponding row in the CSV file. I'm a Mathematica beginner, so explicit steps would be VERY appreciated.

• Check Apply and Map. Also, defining sq or so as functions will help. – Wjx Jun 20 '16 at 23:18

I'm worried that your problem is intractable because of the large number of rows in your parameter file. Processing a few rows would be no problem, where few is defined as a thousand or so.

Also if you were to accumulate lists of all the target values optcost, optbud, and optq, what would you do with such long lists? Each would have 1.5M elements according to your question.

I recommend doing the optimization with managable chucks of the parameters. You can learn about strategies for reading in chunks by searching on "huge csv files" on this site.

Now let's suppose you have a adopted a strategy for breaking up your processing into chunks, and let's suppose the you have imported a chunk of parameters as a list of rows. I will make a toy chunk as follows.

 params = ConstantArray[{10000, 1000, .1, .0085, .0055, .0055}, 5];


This produces a array where each row is repeats the single parameter set you give in your question. It is good enough to demonstrate the techniques I want to present here.

First I would rewrite your two expressions as functions.

Gq[b_, {σ_, μ_, λ_, mc_, me_, γ_}] :=
b γ -
(E^((b Sqrt[μ^2 + 2 λ σ^2])/σ^2)
(E^((b Sqrt[μ^2 + 2 λ σ^2])/σ^2) mc + E^((b μ)/σ^2) me)
((E^((b Sqrt[μ^2 + 2 λ σ^2])/(2 σ^2))
Sqrt[E^((b Sqrt[μ^2 + 2 λ σ^2])/σ^2) mc + E^((b μ)/σ^2) me])/
Sqrt[mc + E^((b (μ + Sqrt[μ^2 + 2 λ σ^2]))/σ^2) me]) ^
(-1 - μ/Sqrt[μ^2 + 2 λ σ^2])
(-μ^2 - 2 λ σ^2 + μ Sqrt[μ^2 + 2 λ σ^2]))/
((-1 + E^((2 b Sqrt[μ^2 + 2 λ σ^2])/σ^2)) λ (-μ + Sqrt[μ^2 + 2 λ σ^2]))

sq[b_, {σ_, μ_, λ_, mc_, me_, γ_}] :=
(σ^2
Log[
(E^((b Sqrt[μ^2 + 2 λ σ^2])/(2 σ^2))
Sqrt[E^((b Sqrt[μ^2 + 2 λ σ^2])/σ^2) mc +  E^((b μ)/σ^2) me])/
Sqrt[mc + E^((b μ)/σ^2 + (b Sqrt[μ^2 + 2 λ σ^2])/σ^2) me]])/
Sqrt[μ^2 + 2 λ σ^2]


Now I compute over the row of params.

{optcost, bud} =
Transpose[(FindMinimum[Gq[b, #], {b, 1, 10000000}] & /@ params)];
optcost


{{367.078, 367.078, 367.078, 367.078, 367.078}

This gives the optcost vector. To get the optbud vector

optbud = bud[[All, 1, 2]]


{36437., 36437., 36437., 36437., 36437.}

And, finally, to get optq.

optq = MapThread[sq[#1, #2] &, {optbud, params}]


{18747.2, 18747.2, 18747.2, 18747.2, 18747.2}

All values in each vector are the same, of course, because each row of parameters was the same, but the code illustrates a technique that will handle, I would think, a thousand rows of parameters at a time.

• This is great, thank you! I'm struggling a bit with the import and export from CSV -- it looks like I need to import a list of lists with each row being a list, but params = Import["path/to/data.csv", "List"] just imports a single list. I'm also struggling on the other side -- appending the list to the CSV file with Export[]. Any advice? Thank you! – Karthik Jun 22 '16 at 2:01
• Actually just figured it out -- use the "Table" option in the Import[] command. Thanks again for the help! – Karthik Jun 22 '16 at 18:13