Hot answers tagged data
20
You are looking for ideas, so I will venture a partial solution in the hope it might inspire something useful. The idea presented here is to exploit a statistical model of the heartbeat intervals as a way to test the goodness of any attempted clustering of the data.
The approach is general, because it is based on maximum likelihood methods, but I will ...
12
Seeing as how someone has been nice enough to write the C-code for you, you could just use that. Assuming you have a C compiler on your machine, here's how you use it within Mathematica. Note that code is defined below.
(* Be sure to define code first! *)
Needs["CCompilerDriver`"];
url = "http://www.stackexchange.com/";
checksum = CreateExecutable[code, ...
11
I am not sure what exactly that rank mean. But here's direct rough porting of code:
ConvertStrToInt[url_String, init_, factor_] :=
Fold[FromDigits[IntegerDigits[#1*factor + #2, 16, 8], 16] &, init,
ToCharacterCode[url]];
HashURL[url_String]:=
Block[{c1, c2,t1, t2},
c1=ConvertStrToInt[url, 5381,33];
c2=ConvertStrToInt[url, 0,65599];
...
10
You want to remove high-frequency noise while retaining the low-frequency signal. This is a job for a bandpass filter. A simple one is the MovingAverage, which you can apply like so:
xsi = Interpolation[MovingAverage[data, 20]]
Plot[{Derivative[1][xsi][t], Cos[t]}, {t, 0, 6.25}, PlotRange -> {-1.1, 1.1}]
10
This filters out your points by their EuclideanDistance. I think it makes a pretty good job preserving the curve features with very few points:
rx[n_] := Accumulate[Prepend[RandomVariate[ExponentialDistribution[1000], n], 0]]
ry[n_] := Accumulate[Prepend[RandomVariate[NormalDistribution[0, .001], n], 0]]
c = Transpose[{rx@#, ry@#}] &@100000;
t = ...
10
Here is a modified version of belisarius' answer I ended up using. Instead of using the euclidean distance to find out if two points are close to each other I set a fixed rectangle in an area and collapse it into a line. For instance, consider the same example.
rx[n_] := Accumulate[Prepend[RandomVariate[ExponentialDistribution[1000], n], 0]]
ry[n_] := ...
9
You can read lines from an InputStream strm (opened with OpenRead) in batches:
lines=ReadList[strm, "String", 4000]
You can vary the chunk size based on your application, 4000 is a number I found to work well for reading web server logs with lines that aren't crazy-long.
You can also reposition for random access on startup. Version 9 improves the use of ...
9
LinearModelFit does too much--it computes residuals, etc., etc. When working with large problems, just compute what you need when you need it. It all begins with the fit itself, which should be done with LeastSquares:
First[Timing[
LeastSquares[{ConstantArray[1, #], RandomReal[NormalDistribution[], #]}\[Transpose],
RandomReal[NormalDistribution[], #]]]] ...
9
Using your values for data, this seems to work:
error = StandardDeviation[data];
data //. {a___, b_, c__, d___} /; Abs[b + c - Median[data]] < error :> {a, b + c, d}
I used StandardDeviation[data] because that's what you used, but you can put in whatever error bound you think is best there. Also note that I replaced Mean[data] with Median[data], ...
8
A simple Mathematica-only solution is:
CountLines[file_String /; FileExistsQ[file]] :=
Module[{counter = 0, str = OpenRead@file},
While[ Read[str, Record, NullRecords -> True] =!= EndOfFile,
counter++
];
Close[str];
counter];
which is quite slow of course, so 123 MB (1978142 ...
7
NIntegrate has many advanced options that let you control which algorithms and strategies it will use. I'm quite sure that you can find a set of options that will make NIntegrate work well enough for the desired task, but of course these numerical algorithms will never be quite as fast and precise as an exact solution, which your sums and Integrates results ...
7
The simplest and most efficient way would probably be using the common wc external utility.
For example,
In[33]:= Import["!wc ~/test.m", "Table"]
Out[33]= {{6, 5, 56, "/Users/szhorvat/test.m"}}
You'll get wc by default on Linux/OSX, but you can install it on Windows too.
7
Lots of answers, but none of them leveraging this, so here is another.
null[_String] := Null
Length @ ReadList["data.txt", null @ String, NullRecords -> True]
On my system this is more than three times as fast as Rolf Mertig's CountLines, and a lot more concise as well.
If even one Null for every record is too much memory usage then read in blocks of ...
6
The simplest thing would probably be to put all heart beats in a list and perform a Fourier transform:
beats = ConstantArray[0., Total[data]];
beats[[Accumulate[data]]] = 1;
beats = GaussianFilter[beats, 100, Padding -> "Cyclic"];
Which gives you an array like this (I've picked a section of your data with a few extra pulses):
ft = ...
6
Tally is good help in this case, it counts the number of occurances.
Using this it's easy to construct a replacement rule that replaces each element with the number of occurances, which you can then plot.
bd = BinomialDistribution[10, 0.4];
m = RandomVariate[bd, {100, 100}];
counts = Tally[Flatten[m]]
rule = Rule @@@ counts
ArrayPlot[m /. rule, ...
6
As explained in this question, you can do a non-parametric fit to your data using B-Splines, and differentiate this fit:
pts = Table[{x, Sin[2 Pi x] + RandomReal[{-.15, .15}]}, {x, 0,
1, .0125}];
kfun[n_, d_] :=
Join[ConstantArray[0, d], Range[0, 1, 1/(n - d)],
ConstantArray[1, d]];
uparam[pts_] := N[Range[0, 1, 1/(Length[pts] - 1)]];
...
6
f = Interpolation[l3= (Last /@ Sort /@ GatherBy[l[[All, 1 ;; 2]], #[[1]] &])]
ListContourPlot[l, RegionFunction -> (#2 < f[#1] &)]
Edit
If you want a smoother curve, you could use for example whuber's method here for getting something similar to an "envolvent" curve:
l4 = {(#[[1]] - 14) 2 + 1, #[[2]]} & /@ l3;
nrow = 32;
i = ...
6
MATLAB's surf plots a parametric surface with the point $(x_i,y_j)$ colored according to the value in $\mathrm{dat}_{i,j}$. This is easily achieved with ListDensityPlot as follows:
ListDensityPlot[Thread[Flatten /@ {x, y, dat}], PlotRange -> {{-50, 50}, {-50, 50}}]
5
You're looking for Skip. This does not check for an EndOfFile condition, and should use something like BlockStream to handle aborts, but the following should work:
strm = OpenRead["filename"];
(* Repeat the following until done *)
Skip[strm, String, A];
AppendTo[ results, Read[strm, String]]
(*
String just gets the entire line. If the number of ...
4
Here is some sample data, hopefully of the sort you are looking for
a=Table[{i, i^2 - 3 i + 2 + 0.3 Random[]}, {i, -1, 3, 0.1}];
ListPlot[a]
will display the data.
This will create a derivable interpolation.
b = Interpolation[a, Method -> "Spline"];
This is its derivative
c = b';
Plot them together.
Plot[Evaluate[{b[x], c[x]}], {x, -1, 3}]
...
4
A bit crude, but it works. Idea: Find lowliers, then merge those that are contiguous.
lowposns = Flatten[Position[data, aa_ /; aa < 4/5*Max[data]]];
groups = Split[lowposns, #2 - #1 == 1 &];
regroup =
SortBy[Join[
Transpose[{Complement[Range[Length[data]], Flatten[groups]]}],
groups], #[[1]] &]
(* {{1}, {2}, {3}, {4}, {5, 6}, {7}, {8}, ...
4
You can read in the first 500 elements like this:
data = ToExpression@ReadList["myfile.txt", Record, 500, RecordSeparators -> " "];
On Linux/OS X, items 500 to 1000 can be read in this way:
n = 500; m = 1000;
data =ReadList["!cat myfile.txt | tr ' ' '\n' | head -" <>
ToString@m <>" | tail -" <> ToString[m - (n - 1) ]];
...
4
You can try to force NIntegrate into doing a similar approach than you did by calculating the sum directly. This reduces your error quite a bit, although I don't understand the reasons behind this exercise:
ni = NIntegrate[int[ww], {ww, dataF[[1, 1]], dataF[[-1, 1]]},
Method -> {"TrapezoidalRule",
"Points" -> Length[dataF]}, MaxRecursion -> ...
4
The documentation is misleading here. On one hand, the only export option is "Append" which can be found under the Options tab. On the other hand, the general documentation reads
I really wonder, why it is necessary to put Import only behind an option value when "DataEncoding" isn't an export option at all.
Anyway, I have the same behaviour in MacOSX ...
4
For not so big data, what I do often is the following
data = RandomReal[{-1, 1}, 100000];
ByteCount@data
(* 800144 *)
Now, you write data = Interpretation["hidden data", Evaluate@data], then select ONLY THE INTERPRETATION EXPRESSION, and evaluate IN PLACE.
As a result, you get a cell which can be used to load the data, that is fully stored in the ...
4
Add Method -> "NMinimize" to the NonlinearModelFit call.
data = {{248, 0.032}, {280, 0.0498}, {327, 0.0971}, {360, 0.162}};
NonlinearModelFit[data, a Exp[b t], {a, b}, t,
Method -> "NMinimize"]["BestFitParameters"]
{a -> 0.000783053, b -> 0.0147985}
The fit looks good:
Show[ListPlot@data, Plot[a Exp[b t] /. fit, {t, 248, ...
4
This is merely a skeleton answer. I will leave you to implement these ideas as without knowing your data format I cannot give usable code.
Open an input stream with OpenRead, and an output stream with OpenWrite or OpenAppend.
Read a block of a manageable size from the input stream using ReadList
Process this data and apply your function
Format the data, ...
4
wkstrngs = StringJoin /@ Map[ToString, PadLeft[IntegerDigits /@ Range[52]], {-1}];
wkurls = Quiet["http://www.boxofficemojo.com/weekend/chart/?yr=2012&wknd=" ~~
# ~~ "&\p=.htm" & /@ wkstrngs[[;; 5]]] (* remove [[;;5]] for all 52 weeks *)
(* {"http://www.boxofficemojo.com/weekend/chart/?yr=2012&wknd=01&\\p=.htm",
...
4
Here is a Java-based solution, based on this answer. You will need the Java reloader, which was described here. Load it first, then evaluate the following:
JCompileLoad@
"
import java.io.*;
public class LineCounter{
public static int count(String filename) throws IOException {
InputStream is = new BufferedInputStream(new ...
4
Surely not the fastest or ideal solution, but this should work (it assumes that you have two numbers in each row):
randomline[str_InputStream, num_] := (
SetStreamPosition[ str, 0];
Skip[str, Record, num-1];
Read[str, {Number, Number}])
str = OpenRead["data.txt"];
Now, if you have 2000 lines and want 100 random samples:
Map[randomline[str, #] ...
Only top voted, non community-wiki answers of a minimum length are eligible
