# Statistical analysis of a data file and its Plot [closed]

As the title says, I have an external data file which contains thousands lines of data. The data file contains eight columns and here is what I would like to plot:

Let's assume that the length of the file is $N_0$.

N0=Length[data];


Now, some statistical analysis: if the seventh column is non zero (in other words, either 1 or 2), plot $\rm log_{10}(N/N_0)$ as a function of $n$, where $n$ is the integer of the eight column and $N$ is the number of lines inside the date file that have non zero seventh column and the respective values $n$ of the eight column. This is a rather complicated aspect, so I hope I explained correctly and clear what I want to plot. Nevertheless, if there is any confusing issue, please ask for additional details.

Here is a small sample of the data file:

data1={{-0.63, -0.05, 0.607293, -1.01899, 244.7, 8.9221*10^-10, 2, 33},
{-0.63, -0.04, 0.750888, -0.914848, 99.06, 0.00192017, 2, 13},
{-0.63, -0.03, 0.866068, -0.807778, 240.9, 8.3753*10^-11, 2, 30},
{-0.63, -0.02, 0.5362, -1.05766, 105.95, 0.0000915176, 2, 14},
{-0.63, -0.01, 0.818972, -0.85418, 109.99, 7.34825*10^-8, 2, 14},
{-0.63, 0., 0.644631, 0.994565, 75.04, 0.0000214851, 1, 10},
{-0.63, 0.01, 0.678989, 0.972796, 131.81, 0.000396984, 1, 19},
{-0.63, 0.02, 0.572168, 1.03831, 212.27, 2.26474*10^-9, 1, 31},
{-0.63, 0.03, 0.593172, 1.02434, 125.65, 0.000119734, 1, 18},
{-0.63, 0.04, 0.620368, 1.00784, 80.25, 0.302882, 1, 11},
{-0.07, 0.47, 0.687758, 0.963341, 2.71, 1.07464, 1, 0},
{-0.07,0.48, 0.680021, 0.973429, 2.76, 1.10912, 1, 0},
{-0.07, 0.49, 0.668844, 0.980709, 2.81, 1.1496, 1, 0},
{-0.07, 0.5, 0.654742, 0.989591, 2.87, 1.19496, 1, 251},
{-0.07, 0.51, 0.636733, 0.999036, 2.94, 1.24555, 0, 72},
{-0.07, 0.52, 0.614364, 1.01248, 3.03, 1.29982, 1, 0},
{-0.07, 0.53, 0.586779, 1.03184, 3.15, 1.35616, 1, 0},
{-0.07, 0.54, 0.552593, 1.05116, 3.3, 1.41155, 0, 30},
{-0.07, 0.55, 0.512858, 1.06758, 3.5, 1.33486, 1, 0},
{-0.07, 0.56, 0.48694, 1.08078, 3.84, 1.17199, 1, 27}}


The complete data file can be obtained from here: data.

Many thanks in advance. I have not tried anything, simply because I have no idea where and how to start. This however, should be quiet easy for a list-manipulation guru!

EDIT

Let me explain more what I want to plot. The seventh column has three options 0,1,2 and the last column contains only integers whose values go from zero to an unknown maximum value. I want to plot $\rm log_{10}(N/N_0)$ as function of $n$, in other words how $\rm log_{10}(N/N_0)$ evolves for every value of $n$ $(n = 1, n = 2, n = 3, ....., n = n_{max})$ when the corresponding seventh column is non zero (1 or 2).

EDIT 2

The final plot should be look like the following plot. The data file correspond to the darker green line (0.20).

-

## closed as off-topic by Ajasja, Sjoerd C. de Vries, Pinguin Dirk, rm -rf♦Oct 11 '13 at 17:07

This question appears to be off-topic. The users who voted to close gave this specific reason:

• "The question is out of scope for this site. The answer to this question requires either advice from Wolfram support or the services of a professional consultant." – Ajasja, Sjoerd C. de Vries, Pinguin Dirk, rm -rf
If this question can be reworded to fit the rules in the help center, please edit the question.

At the beginning you state "The data file contains eight rows", but in the rest of your post you only mention columns. Is "rows" a typo? –  m_goldberg Oct 11 '13 at 8:37
Also, it would help you get answers if you were to extract a few typical lines, say 10, from your huge data set and include them in your question. It shows consideration to those who would help if you do what work you can and save them some effort. –  m_goldberg Oct 11 '13 at 8:42
Now that I've seen some of your data, I'm having trouble with the statement "N is the number of lines inside the date file that have non zero seventh column and the respective values n of the eight column". Perhaps you mean, "N is the number of lines with a non-zero seventh column and having the same value of n in the eight column"? Also, to make the sample data more typical, I think you should add some lines with zero in the 7th column and a value on n that duplicate one or more that are already in the sample. –  m_goldberg Oct 11 '13 at 9:29
You are no newcomer, so why do you not post your list data as such, i.e. in a format that can be pasted properly? Please consider the time other people have to spend to make your posts marginally useful... –  Yves Klett Oct 11 '13 at 9:48
After 34 or so questions you really should take care to post good questions. Again, consider how much time people already courteously spent to help you. –  Yves Klett Oct 11 '13 at 10:50

I may not have understood this. It appears the desired plot depends only on columns 7 and 8. As such:

d = GatherBy[Select[data1[[All, {7, 8}]], #[[1]] != 0 &], #[[2]] &];
N0 = Length[data1];
ListPlot[Sort[{#[[1, -1]], Log[10, Length[#]/N0]} & /@ d],
Joined -> True, PlotRange -> {-2, 0}]


-
This is exactly what I wanted! Simple and elegant solution. –  Vaggelis_Z Oct 11 '13 at 13:29

Here is a step-by-step approach to what I think you are looking for.

n0 = Length @ data1


20

Strip data down to columns 7 and 8.

data2 = data1[[All, -2 ;;]];


Get rid of data points with zero in column 7.

data3 = DeleteCases[data2, {0, _}];


A function to separate and aggregate the data

selectN[val_Integer] := SortBy[Tally@Cases[data3, {val, x_} :> x], First]


Perform the separation and aggregation.

n1 = selectN[1]
n2 = selectN[2]

{{0, 6}, {10, 1}, {11, 1}, {18, 1}, {19, 1}, {27, 1}, {31, 1}, {251, 1}}
{{13, 1}, {14, 2}, {30, 1}, {33, 1}}


A function to convert the dependent variable in normalized Log10 values.

logData[{x_, y_}] := {x, N[Log10[y/n0]]}


Make the plot data sets.

plotData = logData /@ # & /@ {n1, n2}

{{{0, -0.522879}, {10, -1.30103}, {11, -1.30103}, {18, -1.30103},
{19, -1.30103}, {27, -1.30103}, {31, -1.30103}, {251, -1.30103}},
{{13, -1.30103}, {14, -1.}, {30, -1.30103}, {33, -1.30103}}}


Note: the above can also be expressed as

plotData = Map[logData, {n1, n2}, {2}]


Plot the data.

ListPlot[plotData, PlotStyle -> PointSize[Large]]


The resulting plot isn't very exciting because the data set is so small.

-
Why there are two different colors in the plot? Does this imply two different sublists? Note, that the points should be joined together. –  Vaggelis_Z Oct 11 '13 at 13:31
@Vaggelis_Z. There are two colors because I thought you wanted the data associated with 1 in column 7 separated from the data associated with 2. As for joining or any other plot decorations that may you want, I leave them to you to do on your own. Surely, you can do something toward solving your own problem. –  m_goldberg Oct 11 '13 at 15:38

I might regret answering this but here goes.

data1 = Import["~/Downloads/data_stat.out", "Table"];(*that's your data file*)
pnonz = Select[data1, #[[-2]] != 0 &];(*subset of data with nonzero 7th entry*)
n0 = N@Length@data1;(*lists in data1*)


Now if you observe that you only have few integers as an option for the last entry you can define downvalues for a function

Block[{subl, range, dv},

subl = pnonz[[ All, -1]];
(*last column of pnonz*)

range = Sort@DeleteDuplicates@subl;
(*separate entries of subl and order*)

ClearAll@g;
SetAttributes[g, Listable];

DownValues[g] =
Thread[g[range] -> ((Log[10, Count[subl, #]/n0]) & /@ range)];
(*assign downvalues to g. Count[subl,#] is the number of columns \
with nonzero 7th entry that have the number #. We map that to all \
possible last entries*)

]


Now g has been assigned all the values it can take so we map it onto the last element of every sublist of data1.

With[{column = data1[[All, -1]]},
lst = Sort@Transpose@{column, g[column]};
ListLinePlot[lst, PlotRange -> {All, {-6, 0}}]]


-
Why regretting it? It seems that your approach is more or less quite similar to that of @ubpdqn. I upvoted your answer but I accepted ubpdqn's mainly because it is more compact. –  Vaggelis_Z Oct 11 '13 at 13:33