1
$\begingroup$

I have got different files with the result of an experiment varying 2 variables. The files are in this link. Each file has 3 columns of data: the first column is the same for all the rows of the file but changes from file to file and represents the first variable; a second column is a float number and represents the second variable. The third column in each file represents the success with 1 and, the failure of the experiment with a 0.

I would like to get the frequency of successes in XY bins. For example, if {X,0.1,0.7} and {Y,0,30}, divide this region like it does the function FindDivisions[] and count the successes found in this division and divide this number by the total number of points in this division.

With these generated data points, I would like to plot the result with stems or with a surface. Also, I would like to compute an interpolation function with them to get the success probability passing any XY values to the function.

$\endgroup$

closed as unclear what you're asking by m_goldberg, JimB, Alex Trounev, José Antonio Díaz Navas, bbgodfrey Mar 28 at 22:40

Please clarify your specific problem or add additional details to highlight exactly what you need. As it's currently written, it’s hard to tell exactly what you're asking. See the How to Ask page for help clarifying this question. If this question can be reworded to fit the rules in the help center, please edit the question.

  • $\begingroup$ There's no indication that these questions are related to the software package Mathematica. Was there something unclear about the Mathematica documentation on Import and Plot3D? $\endgroup$ – JimB Mar 25 at 21:49
  • $\begingroup$ I find the sentence "I would like to ask how to make bins the points and substitute them by a center XY datapoint and with a Z value equivalent to the probability of success (number_of_1's/ number_of_(0's+1's))." totally unclear. I believe it to be the critical sentence of your post. Should the others reading your post find it unclear as well, you will not get an answer and your question will be closed.. $\endgroup$ – m_goldberg Mar 25 at 22:10
  • $\begingroup$ @m_goldberg You are right. The sentence is unclear. I have changed it. I hope now will be clearer. I am sorry. $\endgroup$ – user1993416 Mar 25 at 22:33
  • $\begingroup$ @JimB I just wanted some indication of how to proceed and solve the problem. $\endgroup$ – user1993416 Mar 25 at 22:35
  • 1
    $\begingroup$ It sounds like an appropriate model might be a logistic regression which can be found using GeneralizedLinearModelFit or LogitModelFit. A related type of model would be ProbitModelFit. That would give you a predictive model rather than just a data display (although a display is essential, too). $\endgroup$ – JimB Mar 26 at 0:44
2
$\begingroup$

This is just an extended comment. It's many times a good idea to get a plot of the raw data.

data = Flatten[Import["*.txt", "Data"], 1];

(* Jitter the first variable by a little bit *)
data2 = data;
data2[[All, 1]] = data2[[All, 1]] + RandomVariate[NormalDistribution[0, 0.0015], Length[data]];

(* Separate the data in the 0's and 1's *)
data0 = Select[data2, #[[3]] == 0 &][[All, {1, 2}]];
data1 = Select[data2, #[[3]] == 1 &][[All, {1, 2}]];

(* Plot the data clouds *)
ListPlot[data0, PlotRange -> {{0, 0.8}, {-5, 35}}, ImageSize -> Large,
 PlotLabel -> Style["0", 18, Bold], PlotStyle -> Red]
ListPlot[data1, PlotRange -> {{0, 0.8}, {-5, 35}}, ImageSize -> Large,
 PlotLabel -> Style["1", 18, Bold], PlotStyle -> Green]
ListPlot[{data0, data1}, PlotRange -> {{0, 0.8}, {-5, 35}}, 
 ImageSize -> Large, PlotLabel -> Style["Both 0 and 1", 18, Bold], 
 PlotStyle -> {Red, Green}]

data with zeros

data with ones

data with zeros and ones

Addition: Maybe it's easier than I thought to get the proportion estimates:

(* Set some bins *)
bins = {{9/100, 71/100, 2/100}, {-5, 33, 2}};

(* Total number of observations in each bin *)
n = HistogramList[data[[All, {1, 2}]], bins];
(* Number of ones in each bin *);
ones = HistogramList[Select[data, #[[3]] == 1 &][[All, {1, 2}]], bins];

(* Generate a table of triplets with bin centers and proportion of ones *)
{d1, d2} = Dimensions[n[[2]]]
proportions = 
  Flatten[Table[{(n[[1, 1, i - 1]] + n[[1, 1, i]])/2, (n[[1, 2, j - 1]] + n[[1, 2, j]])/2,
     If[n[[2, i, j]] == 0, -1, ones[[2, i, j]]/n[[2, i, j]]]}, {i, 2, d1}, {j, 2, d2}], 1];
proportions = Select[proportions, #[[3]] >= 0 &];

(* Show results *)
ListPointPlot3D[proportions]

3D plot of proportions

$\endgroup$
  • $\begingroup$ I wanted to thank you for the solution. I have tested each step of the solution, and it is what I needed. It requires practice to know how to visualize the data introducing the jitter :-). I would also ask you how to select the correct bins, and if it is a way to select the bins if the values of the data change. Thank you. $\endgroup$ – user1993416 Mar 26 at 10:10
  • $\begingroup$ The jittering was just to give folks an idea as to where all of the data was (since it was very systematically collected and it would be harder to assess with just looking at the raw data values). There is no "correct" bin size or orientation. Because what you almost certainly need is to create a predictive equation the bin size is irrelevant for that but definitely good for looking at the goodness-of-fit for the model being fitted to the data. $\endgroup$ – JimB Mar 26 at 15:15
  • $\begingroup$ You are right. I need to find a function to interpolate XY points. I have tried Interpolation[] of proportions, but since the function is not defined with data points in the part without bins, the interpolated function gives Z values that are incorrect. I have made some tests and posted a new question in mathematica.stackexchange.com/questions/193972/… $\endgroup$ – user1993416 Mar 26 at 15:27

Not the answer you're looking for? Browse other questions tagged or ask your own question.