Questions on systematic data handling and analysis, including data correction, manipulation, or selection of user-data or built-in data available with Mathematica.

learn more… | top users | synonyms

5
votes
1answer
309 views

LinearModelFit with millions of rows

I'd like to do a simple regression with millions of rows of (x,y) data. It seems to be too long for Mathematica. Is there a way to speed it up? It shouldn't be that hard to do, I am not sure what ...
4
votes
3answers
592 views

How do I make a frequency heatmap?

I have an array of data that has repeats in it. I want to generate a heatmap depending on how much repetition each point has, as in how many times it is repeated.
0
votes
1answer
469 views

importing and manipulating data file

I have a large text file containing numeric data of 1000 groups each of 200 lines. The lines are unsorted. There is a 9 line header at the beginning of each group, i.e., 209 ⨉ 1000 lines altogether. ...
16
votes
2answers
2k views

How do you deal with very large datasets in Mathematica?

This question is in two related parts. The first is about dataset size, the second data wrangling. Part 1: I've built a naive Bayesian classifier on Mathematica. It uses ...
3
votes
0answers
109 views

External file Sub Matrix Import

Is there a Built-In Mathematica Function to extract a Sub matrix from a huge external file? Functions getting pieces of internal nested Lists, (arrays) are in place. Example: ...
7
votes
2answers
5k views

How do I plot the derivative of a set of experimental points?

I have a set of points like this: P = {{$x_1$,$y_1$},{$x_2$,$y_2$},...,{$x_n$,$y_n$}}. I want to plot them and also plot the derivative of that graph. How can I do this with Mathematica? Is there a ...
8
votes
3answers
1k views

Analytically finding critical values in a hysteresis curve made with ListPlot

Let me start with a super basic explanation of what I'm trying to do, and I will follow with a more detailed discussion afterwards. From data in the form shown below (a ...
16
votes
2answers
1k views

Dealing with large files

I have a *.dat file in which the contents are basically one giant list: {1,2,3,4} {5,6,7,8} {9,10,11,12} and so on. I'd like to read in the first 500 members { ...
28
votes
4answers
1k views

Filtering beat-to-beat heart rate data

From an experiment, I have a dataset of beat-to-beat heart rate data: a list of the time between each heart beat in [ms]. The data is measured using an infrared optic sensor at the finger tip. The ...
4
votes
0answers
303 views

How to save (large) data generated from simulations

The following is a simple example of a much larger simulation that I want to run Define some processes ...
0
votes
1answer
482 views

How to create function that classifies sample data

I want to create expression, that classifies samples ( 4 weather features -> decision). Example training data: For the values that i have data for, I can use rules: ...
10
votes
2answers
4k views

How to get Matlab data imported with the same dimensions?

I have some Matlab image data with the following dimensions (output is from Matlab): >> size(im) ans = 86 86 3 45 The data imports just fine ...
1
vote
4answers
6k views

Creating a data1 versus data2 plot?

I have a table of data with multiple rows and columns, and I have to make a versus data table using two of these columns. Essentially, I have to set one column of data (column 3) as the y-axis and ...