# Efficient Data Entry / Manipulation with Many Datasets

I currently have a simple task of graphing many different datasets (26 here, with many different instances where I'll need to do, say, 26 different ones from the same template). I'm trying to minimize the amount of "name changing" I'll have to do manually, as the treatment of each dataset is virtually identical aside from the names and (obviously) individual data points being graphed.

For example, I have a large table (specifically imported using ExcelRead with ExcelLink) with:

data = {{DataSetName1, m1, m2, m3,m4},{DataSetName2, m1, m2, m3, m4},...}

etc. for all 26 datasets and their measurements 1 - 4 as a function of time.

Next I have to extract each datasets information individually using

time=[5, 10, 20, 1440] for the time intervals my data is taken at,

DataSet1raw = Rest[Part[data, 2, 1;;5] and then DataSet1 = Transpose[{time, DataSet1raw}] DataSetList = {"DataSet1Name","DataSet2Name","...", "DataSet26Name"}

to finally yield the time-series {{t1, m1},{t2, m2}, ...} form and a list of the names of each set.

Combining the list of time-series (& names respectively) to graph in thirds for ease of viewing,

AllSets = {DataSet1, DataSet2, ..., DataSet26} FirstCollection = Take[AllSets, 1;;9] SecondCollection = Take[AllSets, 10;;18] FinalCollection = Take[AllSets, 19;;26]

After doing this individually for 26 numbered data sets (such as NB1,NB2,NB3, etc., all of which contain name-specific components previously used), it is finally in the form where I can graph the datasets easily using:

ListLogLinearPlot[FirstCollection, Legending, etc.]

So, my question is a general one of how to optimize this process. Currently I'm leaning towards generalizing the terms like I've done above, and trying to use a simple process of "Specific name = generic name," but suspect my lack of general organizational Mathematica familiarity (and coding in general) has me complicating much of this process.

Thank you for taking the time to respond!

Update:

After investigating further as to how to possibly simplify the process, I've at least found in the documentation a means of possibly doing the renaming of each drug iteration (26 times here) by a series of ToString and ToExpression functions, in case anyone else is facing a similar dilemma.

• I posted an answer that I hope sets you in the right direction. If there are parts you don't understand, or issues with applying these methods to your actual data please ask and I will try to assist. – Mr.Wizard Apr 29 '13 at 9:19

I am not certain where your trouble lies, but since you mention ToString and ToExpression you're surely making this harder than it needs to be. Let me show you a couple of methods to format your data in the way I think you intend. Sample data:

dat = {{"cat", 8, 18, 9, 1}, {"dog", 19, 15, 15, 7}, {"fox", 7, 10, 9, 20},
{"bird", 17, 10, 1, 5}, {"lynx", 19, 9, 7, 1}, {"snail", 3, 12, 17, 2}};

time = {5, 10, 20, 1440};


We may write a function such as:

f[time_][{name_, val___}] := Prepend[{time, {val}}\[Transpose], name]


This uses a SubValues pattern to simplify application while preventing implicit reliance on global values, by which I mean using time directly in the function rather than as a parameter. Also, this function does no argument testing, only Destructuring. To be robust you may wish to check that time and {val} have the same Length, etc.

Now:

f[time] /@ dat

{{"cat", {5, 8}, {10, 18}, {20, 9}, {1440, 1}},
{"dog", {5, 19}, {10, 15}, {20, 15}, {1440, 7}},
{"fox", {5, 7}, {10, 10}, {20, 9}, {1440, 20}},
{"bird", {5, 17}, {10, 10}, {20, 1}, {1440, 5}},
{"lynx", {5, 19}, {10, 9}, {20, 7}, {1440, 1}},
{"snail", {5, 3}, {10, 12}, {20, 17}, {1440, 2}}}


Alternatively you could use an assignment on Part, either on a copy of dat or on dat itself for in-place modification.

dat2 = dat;
dat2[[All, 2 ;; 5]] = {time, #}\[Transpose] & /@ dat[[All, 2 ;; 5]];

(* dat2 will be the same as the output above *)


For splitting your data into thirds I recommend using Partition, e.g.:

thirds[lst_List] := Partition[lst, #, #, 1, {}] & @ Ceiling[Length@lst/3]

Range@26 // thirds

{{1, 2, 3, 4, 5, 6, 7, 8, 9},
{10, 11, 12, 13, 14, 15, 16, 17, 18},
{19, 20, 21, 22, 23, 24, 25, 26}}


If you need to assign these chunks to symbols you can use e.g.:

{one, two, three} = Range@26 // thirds;

two

{10, 11, 12, 13, 14, 15, 16, 17, 18}


In general however I recommend not using a separate symbol for each part, but instead using Part, or indexed objects, or replacement rules.

• Yes, thank you sir for acknowledging my noob struggle. Apologies for the tedium. Currently I have data = {"5min avg", "10min avg", "20min avg", "24h avg", "5m stdev", "10m stdev", "20m stdev", "24h stdev", {"DataSet1", y1, y2, y3, y4},{"DataSet2", y1, y2, y3, y4},...} I have used Rest & Part to "rest apart" each individual DataSet and name it invidually, I suppose I need to reapproach the whole table I've imported from Excel. I'll certainly try the things you've recommended here and in your linked posts. Thank you. – Ghersic Apr 30 '13 at 3:16
• The actual manipulation of such a table is what baffles me, particularly the name being list first in each dataset and the stage at which the time-series Transpose[] phase happens before graphing. Thanks again. Do you think such functions as Prepend/Append, Rest, Part and others like them are sufficient to manipulate the data to where it is graphable, or should I go for indices manipulation as you've suggested in your conclusion / linked posts? – Ghersic Apr 30 '13 at 3:21
• Above in comment one, that is, I have so far imported only data without the names within the lists. Sorry – Ghersic Apr 30 '13 at 3:26
• @Ghersic please comment here when you get on-line; if I am still around we can discuss this in Chat. It will be helpful to me if you have a copy of the data that I can download. – Mr.Wizard Apr 30 '13 at 12:39
• Certainly, I appreciate your willingness to help me. I am at work currently (without Mathematica unfortunately) but will be certain to message you this evening. If this evening doesn't work, my schedule is fairly open the rest of the week for whenever is convenient for you. Thank you again. @Mr.Wizard – Ghersic Apr 30 '13 at 16:46

I'd recommend rethinking your data structure a bit. How about defining lists like:

data[[i]] = {{DataSetName1, m1, m2, m3,m4},{DataSetName2, m1, m2, m3, m4},...}


for i=1 to however many pieces of data you have. Then the corresponding

time[[i]] =[5, 10, 20, 1440]


and etc for all the different items. Now you can just index through the i, instead of renaming everything at each step.

• Thanks, that seems simple enough. I'm aiming at smoothly graphing (on a few different, otherwise identical graphs) many datasets' time-series (y(t)) lists. How does indexing incorporate Transpose[] in the creation of a graphable set of lists? Thank you again for bringing light into a thus far dark place! @bill s – Ghersic Apr 30 '13 at 3:30
• I'm not completely sure I get the question... but if you mean something like your original DataSet1 = Transpose[{time, data}], anything you previously did to time and data you can now do to time[[i]] and data[[i]]. If you want to keep (almost) the same code you can use Table[Transpose[{time[[i]],data[[i]]}],{i,1,Length[time]}]. – bill s Apr 30 '13 at 6:42