0
$\begingroup$

I have a data set with 500,000 elements. I am trying to use SmoothHistogram to plot the distribution, but it doesn't work and the kernel quits. I used R on the same problem and it was fine. Am I doing something wrong? I'm really new to Wolfram Language. The file can be downloaded from here or here.

data = Import["https://pastebin.com/raw/0gGvcMxQ"];
SmoothHistogram[data, Automatic, "PDF"]

Update

Thank you for your help, but I have also a similar problem but probably with a different cause.

dist := ProbabilityDistribution[(2/Sqrt[Pi]*Exp[-1/x])/x^(5/2), {x, 
    0, \[Infinity]}];
g[x_] = CDF[dist, x];
h = Table[{g[x], x}, {x, 0, 5500, 0.001}] // N // 
   Interpolation[#, InterpolationOrder -> 3] &;
data2 = Table[Total[h /@ (RandomReal[{0, 1}, {500}])], {i, 100000}]
p1 = SmoothHistogram[data2, Automatic, 
  PlotRange -> {{900, 1100}, {0, 1000}}

The plot doesn't appear with no error. Thank you again for your patience.

$\endgroup$
  • 2
    $\begingroup$ Could you put your file on a site that doesn't have so many "Download" buttons? Or at least has an obvious button associated with your file? $\endgroup$ – JimB Nov 15 '19 at 18:14
  • $\begingroup$ Note that the structure of your data may not be what you expect. If you execute Dimensions[data], you might see that your data is not a flat list, but a list of lists. That's just how CSV files are imported, i.e. each line ends up as a list. You would want to Flatten it before further work. Even then, perhaps start with Histogram first to get an idea of what bandwidth and other parameters might be appropriate for your data. $\endgroup$ – MarcoB Nov 15 '19 at 18:14
  • $\begingroup$ Thank you, I tried with Flatten but the situation is the same. Is it possible to use SmoothHistogram in Log-Log scale? The distribution in question is just x^(-5/2). $\endgroup$ – davideor Nov 15 '19 at 18:27
  • $\begingroup$ @davideor Sure, add ScalingFunctions -> {"Log", "Log"} as an option to the SmoothHistogram. $\endgroup$ – MarcoB Nov 15 '19 at 18:39
2
$\begingroup$

The problem is that the data is imported as a list of lists.

data = Import["~/Downloads/polydistr.csv"];
data[[1;;5]]
(* {{1.37525}, {1.09473}, {2.20973}, {1.28192}, {1.15815}} *)

Import as list

data = Import["~/Downloads/polydistr.csv", "List"];
data[[1;;5]]
(* {1.37525, 1.09473, 2.20973, 1.28192, 1.15815} *)

Histogram[data]

enter image description here

SmoothHistogram[data, Automatic, "PDF", PlotRange -> {{0, 20}, {0, 1}}]

enter image description here

Update

Use log-log scale

SmoothHistogram[data, Automatic, ScalingFunctions -> {"Log", "Log"}, PlotRange -> {{0, 100}, {10^-5, 1}}]

enter image description here

| improve this answer | |
$\endgroup$
  • $\begingroup$ Thank you very much for your help, is it possible to display it in log-log scale? $\endgroup$ – davideor Nov 15 '19 at 18:29
  • $\begingroup$ @davideor You are welcome. See update for log-log scaling. $\endgroup$ – Rohit Namjoshi Nov 15 '19 at 18:46
  • $\begingroup$ Could you give a look to the update? Thank you again $\endgroup$ – davideor Nov 16 '19 at 11:34
  • $\begingroup$ @davideor First experiment with small numbers, e.g. change 5500 to 100 and step size of .01, change 500 to 50 and 100000 to 100. Then you can take a look at the structure of h and data2. The problem will then be obvious. $\endgroup$ – Rohit Namjoshi Nov 16 '19 at 14:56
  • $\begingroup$ I'm sorry, I still don't get it.. $\endgroup$ – davideor Nov 26 '19 at 15:31
2
$\begingroup$

Because your data is bounded by 1 (i.e., values less than 1 are likely impossible) and because the density isn't zero at 1, you need to use the "Bounded" option in SmoothHistogram (which unfortunately is only documented in SmoothKernelDistribution).

SmoothHistogram[data, {Automatic, {"Bounded", {1, ∞}, "Gaussian"}}, "PDF",
  PlotRange -> {{0, 20}, {0, 1.2}}, Frame -> True, 
  FrameLabel -> (Style[#, Bold, 18] &) /@ {"X", "Probability density"}]

Estimate of pdf of x

If one doesn't use the "Bounded" option in such a case, then part of the estimated density is positive where values can't happen. That's not good.

Some in this forum suggest a log transform of the vertical axis is a good thing. I think doing so is nonsense. If one log transforms the vertical axis, then the area under the curve is no longer 1 and that makes it impossible (or at least inappropriate) to compare density estimates from other datasets. (But if there is a good reason for doing so, I'm partially open-minded to hear about that.)

What can make sense is log transforming the data and then find the resulting estimate of the probability density of the log of the original variable.

SmoothHistogram[Log[data], {Automatic, {"Bounded", {0, ∞}, "Gaussian"}}, "PDF",
  PlotRange -> {{0, Log[20]}, {0, 1.5}}, Frame -> True, 
  FrameLabel -> (Style[#, Bold, 18] &) /@ {"Log[X]", "Probability density"}]

Estimated pdf for Log[x]

| improve this answer | |
$\endgroup$
  • $\begingroup$ Could you give a look to the update? Thank you $\endgroup$ – davideor Nov 16 '19 at 12:17
  • $\begingroup$ I did but I think you have a totally different question there. You should open a new question. But...your added coded works if you change the PlotRange to {{900,1100},Automatic}. $\endgroup$ – JimB Nov 16 '19 at 15:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.