4
$\begingroup$

I have a two-part question about the automatic generation of file names and Plot Legends, Labels.

Part 1: I typically work with very large matrices of data with columns of the form:

MASTERdataset: {a-parameter, b-parameter, c-parameter, x-var, y-var}

A sample dataset is below:

masterDATA = {{1200, 700, 150, 285.29323135045837`, 
    124.81439541987501`}, {1200, 700, 150, 286.60945594708426`, 
    126.30947680625`}, {1200, 700, 150, 287.92561104172626`, 
    127.73505620875001`}, {1200, 700, 150, 289.24169417515805`, 
    129.08867440106252`}, {1200, 700, 150, 290.5577029045866`, 
    130.367888590125`}, {1200, 700, 150, 291.8736347915017`, 
    131.5702602656875`}, {1200, 700, 150, 293.1894874091171`, 
    132.69336264156252`}, {1200, 700, 150, 294.5052583400343`, 
    133.7347783193125`}, {1200, 700, 150, 295.8209451736646`, 
    134.692096710125`}, {1200, 700, 150, 297.1365455140142`, 
    135.5629218201875`}, {1200, 700, 150, 298.4520569668892`, 
    136.3448594556875`}, {1200, 700, 150, 299.7674771578958`, 
    137.0355352228125`}, {1200, 700, 150, 301.0828037094413`, 
    137.63257152881252`}, {1200, 700, 150, 302.39803426883464`, 
    138.13361568262502`}, {1200, 700, 150, 303.7131664753014`, 
    138.5363069099375`}, {1200, 700, 150, 305.02819799763193`, 
    138.83831400093752`}, {1200, 700, 150, 306.34312649281645`, 
    139.0372939459375`}, {1200, 700, 150, 307.65794964948805`, 
    139.1309353780625`}, {1200, 700, 150, 308.9726651454576`, 
    139.116916108125`}, {1100, 700, 150, 285.20258900103653`, 
    136.52080780656252`}, {1100, 700, 150, 286.51872642831677`, 
    138.33050686200002`}, {1100, 700, 150, 287.8347978496311`, 
    140.07419995162502`}, {1100, 700, 150, 289.15080077907106`, 
    141.7494011668125`}, {1100, 700, 150, 290.4667327507974`, 
    143.3536446684375`}, {1100, 700, 150, 291.7825912998558`, 
    144.8844655020625`}, {1100, 700, 150, 293.0983739766935`, 
    146.339414115`}, {1100, 700, 150, 294.41407833772337`, 
    147.716046920125`}, {1100, 700, 150, 295.729701949836`, 
    149.01193080800002`}, {1100, 700, 150, 297.0452423911218`, 
    150.2246438690625`}, {1100, 700, 150, 298.36069724507706`, 
    151.35176959956252`}, {1100, 700, 150, 299.6760641116824`, 
    152.3909079801875`}, {1100, 700, 150, 300.99134059121155`, 
    153.33965928456252`}, {1100, 700, 150, 302.30652430565584`, 
    154.195645504125`}, {1100, 700, 150, 303.62161287225007`, 
    154.95648387356252`}, {1100, 700, 150, 304.93660393503`, 
    155.61981842875002`}, {1100, 700, 150, 306.2514951283887`, 
    156.1832835626875`}, {1100, 700, 150, 307.5662841177902`, 
    156.64454473943752`}, {1100, 700, 150, 308.8809685567008`, 
    157.0012554253125`}, {1200, 650, 140, 294.568511670944`, 
    152.44652887431252`}, {1200, 650, 140, 295.5572385555695`, 
    153.244130479`}, {1200, 650, 140, 296.5459073608416`, 
    153.983652730375`}, {1200, 650, 140, 297.5345169136301`, 
    154.663922498125`}, {1200, 650, 140, 298.52306602239275`, 
    155.28374823993752`}, {1200, 650, 140, 299.5115535361726`, 
    155.84197899893752`}, {1200, 650, 140, 300.49997828288434`, 
    156.33744269000002`}, {1200, 650, 140, 301.48833910612274`, 
    156.768982907625`}, {1200, 650, 140, 302.4766348644878`, 
    157.135458251875`}, {1200, 650, 140, 303.46486439545527`, 
    157.43570619850001`}, {1200, 650, 140, 304.4530265778131`, 
    157.668605535625`}, {1200, 650, 140, 305.44112027103967`, 
    157.83301574143752`}, {1200, 650, 140, 306.4291443464566`, 
    157.92780813756252`}, {1200, 650, 140, 307.41709769382004`, 
    157.95187248018752`}, {1200, 650, 140, 308.4049791815327`, 
    157.904077172125`}, {1200, 650, 140, 309.39278771614994`, 
    157.7833287685625`}, {1200, 650, 140, 310.3805221893237`, 
    157.5885189215`}, {1200, 650, 140, 311.3681814999415`, 
    157.3185465185`}, {1200, 650, 140, 312.3557645686668`, 
    156.97233222306252`}, {1200, 650, 140, 313.34327029479664`, 
    156.54877533212502`}, {1200, 650, 140, 314.33069761222396`, 
    156.046809738625`}, {1200, 650, 140, 315.3180454445702`, 
    155.4653590640625`}, {1200, 650, 140, 316.3053127179819`, 
    154.80334945506252`}, {1200, 650, 140, 317.29249838358635`, 
    154.05973203875`}, {1150, 600, 140, 298.23873411795546`, 
    164.05272120981252`}, {1150, 600, 140, 299.2273312653466`, 
    164.975212389875`}, {1150, 600, 140, 300.2158717606931`, 
    165.8410515253125`}, {1150, 600, 140, 301.20435441922945`, 
    166.64905385056252`}, {1150, 600, 140, 302.1927780834343`, 
    167.3980618443125`}, {1150, 600, 140, 303.18114157670266`, 
    168.0868989015625`}, {1150, 600, 140, 304.169443745497`, 
    168.7144114848125`}, {1150, 600, 140, 305.1576834377865`, 
    169.27944756325002`}, {1150, 600, 140, 306.14585949313545`, 
    169.7808467010625`}, {1150, 600, 140, 307.13397078169055`, 
    170.2174790450625`}, {1150, 600, 140, 308.1220161542818`, 
    170.58819542525`}, {1150, 600, 140, 309.1099944821404`, 
    170.89186707275002`}, {1150, 600, 140, 310.0979046420354`, 
    171.1273707566875`}, {1150, 600, 140, 311.08574549827347`, 
    171.293570783625`}, {1150, 600, 140, 312.0735159488727`, 
    171.38936517175`}, {1150, 600, 140, 313.06121487255257`, 
    171.4136326405625`}, {1150, 600, 140, 314.0488411653967`, 
    171.36526927356252`}, {1150, 600, 140, 315.03639373319686`, 
    171.24318086262502`}, {1150, 600, 140, 316.0238714652131`, 
    171.04625666781251`}, {1150, 600, 140, 317.011273287306`, 
    170.77342254962502`}, {1150, 600, 140, 317.99859810628294`, 
    170.42358531543752`}, {1150, 600, 140, 318.98584484300574`, 
    169.995665827125`}, {1150, 600, 140, 319.9730124318714`, 
    169.48859848168752`}, {1150, 600, 140, 320.96009978832717`, 
    168.90129872643752`}};

I partition the matrix into sub-matrices corresponding to the values of the (a,b,c) parameters with:

SELECTfxn[data_,a_,b_,c_]:=Select[data,#[[1";;" 3]]=={a,b,c}&]"

To plot the data or perform other manipulations I need to delete the first three columns and assign a file name that uses the (a,b,c) parameters so that I can keep track of which is which later. Up to now I’ve been doing this “by hand” with lots of strategic copy/paste and find/replace operations. I’ve been naming the files with this scheme:

a1200b700c150 = SELECTfxn[masterDATA, 1200, 700, 150][[All , {4, 5}]];
a1100b700c150 = SELECTfxn[masterDATA, 1100, 700, 150][[All , {4, 5}]];
a1200b650c140 = SELECTfxn[masterDATA, 1200, 650, 140][[All , {4, 5}]];
a1150b600c140 = SELECTfxn[masterDATA, 1150, 600, 140][[All , {4, 5}]];

This takes an enormous amount of time and is prone to mistakes. I would like an operation that would achieve the following:

AUTOfileNAMES[data_]:= “Perform operation SELECTfxn[data_,a_,b_,c_] for all unique combinations of (a,b,c) and assign filenames according to the values (a,b,c) similar to above `by hand' example“

I'm sure this is possible but I can't figure it out. Can someone help?

Part 2: The second part of my question is how to incorporate this information into PlotLegends.

I like the data with different groupings and look for patterns. Doing this by hand takes forever and I often make mistakes in the labels for PlotLegends. Is there a way to assign Legend labels automatically that have the form:

a=1200, b = 700, c =150

where the values (1200,700,150) are taken from file names of the form: file=a1200b700c150?

Here's an example of how I'd like the end-product to look:

fontsize = 16;
SAMPLEdataset = {a1200b700c150, a1100b700c150, a1200b650c140, 
   a1150b600c140};
ListPlot[SAMPLEdataset, PlotStyle -> PointSize[Large], Frame -> True, 
 Axes -> False, FrameLabel -> {"x", "y"}, 
 PlotLegends -> 
  Placed[LineLegend[{"a=1200, b=700, c=150", "a=1100, b=700, c=150", 
     "a=1200,b=650, c=140", 
     "a=1150, b=600, c = 140"},(*LegendFunction\[Rule]"Frame",*)
    LegendMarkerSize -> 20, LabelStyle -> Directive[Bold, fontsize], 
    LegendLayout -> "Column"], Right]]

Has someone done something like this before?

Thank you!

$\endgroup$

2 Answers 2

5
$\begingroup$

Here's an approach using GroupBy:

groupData = GroupBy[#[[;; 3]] & -> (#[[-2 ;;]] &)]@masterDATA
(* <|{1200, 700, 150} -> {
     {285.293, 124.814}, {286.609, 126.309}, {287.926, 127.735}, {289.242, 129.089}, 
     {290.558, 130.368}, {291.874, 131.57}, {293.189, 132.693}, {294.505, 133.735}, 
     {295.821, 134.692}, {297.137, 135.563}, {298.452, 136.345}, {299.767, 137.036},
     {301.083, 137.633}, {302.398, 138.134}, {303.713, 138.536}, {305.028, 138.838}, 
     {306.343, 139.037}, {307.658, 139.131}, {308.973, 139.117}
    }, 
    {1100, 700, 150} -> {
     {285.203, 136.521}, {286.519, 138.331}, {287.835, 140.074}, {289.151, 141.749}, 
     {290.467, 143.354}, {291.783, 144.884}, {293.098, 146.339}, {294.414, 147.716}, 
     {295.73, 149.012}, {297.045, 150.225}, {298.361, 151.352}, {299.676, 152.391}, 
     {300.991, 153.34}, {302.307, 154.196}, {303.622, 154.956}, {304.937, 155.62}, 
     {306.251, 156.183}, {307.566, 156.645}, {308.881, 157.001}
    }, 
    {1200, 650, 140} -> {
     {294.569, 152.447}, {295.557, 153.244}, {296.546, 153.984}, {297.535, 154.664}, 
     {298.523, 155.284}, {299.512, 155.842}, {300.5, 156.337}, {301.488, 156.769}, 
     {302.477, 157.135}, {303.465, 157.436}, {304.453, 157.669}, {305.441, 157.833}, 
     {306.429, 157.928}, {307.417, 157.952}, {308.405, 157.904}, {309.393, 157.783}, 
     {310.381, 157.589}, {311.368, 157.319}, {312.356, 156.972}, {313.343, 156.549}, 
     {314.331, 156.047}, {315.318, 155.465}, {316.305, 154.803}, {317.292, 154.06}
    },
    {1150, 600, 140} -> {
     {298.239, 164.053}, {299.227, 164.975}, {300.216, 165.841}, {301.204, 166.649}, 
     {302.193, 167.398}, {303.181, 168.087}, {304.169, 168.714}, {305.158, 169.279}, 
     {306.146, 169.781}, {307.134, 170.217}, {308.122, 170.588}, {309.11, 170.892}, 
     {310.098, 171.127}, {311.086, 171.294}, {312.074, 171.389}, {313.061, 171.414}, 
     {314.049, 171.365}, {315.036, 171.243}, {316.024, 171.046}, {317.011, 170.773}, 
     {317.999, 170.424}, {318.986, 169.996}, {319.973, 169.489}, {320.96, 168.901}
    }|> *)

It groups all data points by the first three columns, and keeps the last two as entries. The returned Association can easily be used with ListPlot, we simply Apply a StringTemplate to the Keys to get the needed legend entries:

ListPlot[
 KeyMap[Apply@StringTemplate["a=``, b=``, c=``"]]@
  groupData[[Key /@ {
     {1200, 700, 150},
     {1100, 700, 150},
     {1200, 650, 140},
     {1150, 600, 140}
     }]],
 PlotStyle -> PointSize@Large,
 Frame -> True,
 FrameLabel -> {"x", "y"},
 PlotLegends -> 
  Placed[LineLegend[Automatic, LegendMarkerSize -> 20, 
    LabelStyle -> Directive[Bold, 16], LegendLayout -> "Column"], 
   Right]
 ]

enter image description here

Note how we use Part ([[…]]) with Key to extract some of the entries of groupData. We also specify Automatic instead of the entries of the legend to specify that they should be extracted from the data if possible (in this case, the string keys work for this purpose)

To access individual dat asets, you can use e.g.

groupData[{1200, 700, 150}]
(* {{285.293, 124.814}, {286.609, 126.309}, {287.926, 
  127.735}, {289.242, 129.089}, {290.558, 130.368}, {291.874, 
  131.57}, {293.189, 132.693}, {294.505, 133.735}, {295.821, 
  134.692}, {297.137, 135.563}, {298.452, 136.345}, {299.767, 
  137.036}, {301.083, 137.633}, {302.398, 138.134}, {303.713, 
  138.536}, {305.028, 138.838}, {306.343, 139.037}, {307.658, 
  139.131}, {308.973, 139.117}} *)

groupData[[Key@{1200, 700, 150}]]
(* same output *)

To export the data sets to individual files, you can use KeyValueMap, StringTemplate and Export:

KeyValueMap[Export[StringTemplate["a``b``c``.csv"] @@ #, #2] &]@groupData

You can also use StringTemplate and KeyMap to convert the keys to strings of the form a***b***c*** (but I'm not sure what the advantage would be over keys that can easily be generated and manipulated):

KeyMap[Apply@StringTemplate["a``b``c``"]]@groupData
(* <|"a1200b700c150" -> {{285.293, 124.814}, …}, 
 "a1100b700c150" -> {{285.203, 136.521}, …}, 
 "a1200b650c140" -> {{294.569, 152.447}, …}, 
 "a1150b600c140" -> {{298.239, 164.053}, …}|> *)
$\endgroup$
4
  • 1
    $\begingroup$ Thank you for rapid reply! $\endgroup$
    – Drotar
    Commented Feb 12, 2020 at 23:16
  • $\begingroup$ @Drotar See the updates for examples on how to access the individual data sets and how to export them. However, I'm still not completely sure I understand what you mean exactly with "storing the filenames for later" - what do you need them for, other than to export to that file? For what purpose do you need tons of variables with long names, instead of a single list that can be easily manipulated? $\endgroup$
    – Lukas Lang
    Commented Feb 13, 2020 at 20:25
  • $\begingroup$ I have to calculate an ENORMOUS number of parameters. Imagine tasks like I outlined but for a 8x50,000 matrix. $\endgroup$
    – Drotar
    Commented Feb 14, 2020 at 18:19
  • $\begingroup$ (sorry the full message got cutoff) For each of the 8x50,000 matrices I need to perform about 10 calculations. The first few of these are consecutive but the rest are independent and I don't always need all of them. Many of these calculations involve 12th order fits, 3D interpolations etc which are very time-consuming. It's much faster to just generate the results of the first several calculations once without having to calculate them repeatedly for each of the remaining 7 calculations. $\endgroup$
    – Drotar
    Commented Feb 14, 2020 at 23:50
3
$\begingroup$

Update: Save each key to a CSV file in the same directory as the notebook.

namedGroups = 
  groupData // KeyMap[Map[ToString] /* (Riffle[{"a", "b", "c"}, #] &) /* StringJoin]

namedGroups // KeyValueMap[Export[NotebookDirectory[] <> # <> ".csv", #2] &]

To generate "file names" using groupData from @Lukas Lang's answer.

groupData // KeyMap[Map[ToString] /* (Riffle[{"a", "b", "c"}, #] &) /* StringJoin]

(*
<|"a1200b700c150" -> {{285.293, 124.814}, {286.609, 
    126.309}, {287.926, 127.735}, {289.242, 129.089}, {290.558, 
    130.368}, {291.874, 131.57}, {293.189, 132.693}, {294.505, 
    133.735}, {295.821, 134.692}, {297.137, 135.563}, {298.452, 
    136.345}, {299.767, 137.036}, {301.083, 137.633}, {302.398, 
    138.134}, {303.713, 138.536}, {305.028, 138.838}, {306.343, 
    139.037}, {307.658, 139.131}, {308.973, 139.117}}, 
 "a1100b700c150" -> {{285.203, 136.521}, {286.519, 138.331}, {287.835,
     140.074}, {289.151, 141.749}, {290.467, 143.354}, {291.783, 
    144.884}, {293.098, 146.339}, {294.414, 147.716}, {295.73, 
    149.012}, {297.045, 150.225}, {298.361, 151.352}, {299.676, 
    152.391}, {300.991, 153.34}, {302.307, 154.196}, {303.622, 
    154.956}, {304.937, 155.62}, {306.251, 156.183}, {307.566, 
    156.645}, {308.881, 157.001}}, 
 "a1200b650c140" -> {{294.569, 152.447}, {295.557, 153.244}, {296.546,
     153.984}, {297.535, 154.664}, {298.523, 155.284}, {299.512, 
    155.842}, {300.5, 156.337}, {301.488, 156.769}, {302.477, 
    157.135}, {303.465, 157.436}, {304.453, 157.669}, {305.441, 
    157.833}, {306.429, 157.928}, {307.417, 157.952}, {308.405, 
    157.904}, {309.393, 157.783}, {310.381, 157.589}, {311.368, 
    157.319}, {312.356, 156.972}, {313.343, 156.549}, {314.331, 
    156.047}, {315.318, 155.465}, {316.305, 154.803}, {317.292, 
    154.06}}, 
 "a1150b600c140" -> {{298.239, 164.053}, {299.227, 164.975}, {300.216,
     165.841}, {301.204, 166.649}, {302.193, 167.398}, {303.181, 
    168.087}, {304.169, 168.714}, {305.158, 169.279}, {306.146, 
    169.781}, {307.134, 170.217}, {308.122, 170.588}, {309.11, 
    170.892}, {310.098, 171.127}, {311.086, 171.294}, {312.074, 
    171.389}, {313.061, 171.414}, {314.049, 171.365}, {315.036, 
    171.243}, {316.024, 171.046}, {317.011, 170.773}, {317.999, 
    170.424}, {318.986, 169.996}, {319.973, 169.489}, {320.96, 
    168.901}}|>
*)
$\endgroup$
6
  • $\begingroup$ This is great. Thanks! $\endgroup$
    – Drotar
    Commented Feb 12, 2020 at 23:16
  • $\begingroup$ Actually, this doesn't seem to generate a variable with that name. It just generates a string. Is there a way to save the datasets (a1200b700c150, for example) as a table of values? $\endgroup$
    – Drotar
    Commented Feb 12, 2020 at 23:31
  • $\begingroup$ You want to save the values for each key to a file with the corresponding name? e.g. File a1200b700c150.dat has the values {{285.293, 124.814}, ... $\endgroup$ Commented Feb 13, 2020 at 0:11
  • $\begingroup$ @Drotar To expand on RohitNamjoshi's comment: Could you please explain what exactly you intend to use variables for? Do you just want to access the individual data sets, do you want to save them to files, or something else? $\endgroup$
    – Lukas Lang
    Commented Feb 13, 2020 at 10:14
  • $\begingroup$ @RohitNamjoshi I'd like to have files saved in a format like masterDATA. I'd likely to export them at some point as a .csv but the critical piece is just storing the file names so I can use them in later calculations. $\endgroup$
    – Drotar
    Commented Feb 13, 2020 at 17:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.