Questions on systematic data handling and analysis, including data correction, manipulation, or selection of user-data or built-in data available with Mathematica.

learn more… | top users | synonyms

28
votes
4answers
1k views

Filtering beat-to-beat heart rate data

From an experiment, I have a dataset of beat-to-beat heart rate data: a list of the time between each heart beat in [ms]. The data is measured using an infrared optic sensor at the finger tip. The ...
27
votes
5answers
6k views

How to perform a multi-peak fitting?

I am wondering how to implement the multi-peak detecting and fitting in Mathematica. Following is an example of fitting the data using three peaks (such that the data ~ peak1 + peak2 + peak3). The ...
22
votes
1answer
407 views

Visual representation of a Ranking over time (image provided)

[Disclaimer: I initially posted this question on stackOverflow 2 months ago and think it might be better suited for this forum (link to original question)] The graph below shows a ranking of ...
20
votes
4answers
1k views

Graph databases / Nosql with Mathematica

I am very new to mathematica but it looks like an interesting language and I would like to explore it for a future project to create a web search engine. It seems like it would be fairly well suited ...
19
votes
5answers
1k views

Moving maximum function?

How can I construct a moving maximum function? For example, if I have a list of 12 values: { 5, 6, 9, 3, 2, 6, 7, 8, 1, 1, 4, 7 } and I want to maximize over 3 ...
19
votes
3answers
992 views

How to extract the edge from a set of points

I have an accumulation of points representing a body as you can see in the image below. From this data I want to generate a list of points or a curve, which is the envelope. The first idea was a ...
19
votes
4answers
2k views

How to integrate functions of linearly interpolated data?

At first, consider integration of pure InterpolatingFunction. Importing some data (works in v.9, for earlier versions one can use this link to download zipped ...
17
votes
4answers
656 views

How: 2D scatterplots with quantitative density-dependent coloring

Consider the following scatterplot of a 50K-point dataset: ...
17
votes
2answers
429 views

Can A Plot Be Overlayed With A Map?

Some brief, only slightly important background. I am doing a research project using the data from NASA's GRACE mission. I wrote a short Perl script to take two data files and find the change in ...
16
votes
2answers
1k views

Dealing with large files

I have a *.dat file in which the contents are basically one giant list: {1,2,3,4} {5,6,7,8} {9,10,11,12} and so on. I'd like to read in the first 500 members { ...
15
votes
4answers
1k views

Plotting data points: Optimizing size and visuals

The following problem is one that may have showed up to you while plotting a large data collection. Suppose you have a data set of say 1 million points. ...
15
votes
3answers
972 views

Parametric Interpolation of 2D data

I want to create an interpolation through the following 2dim data points: ...
15
votes
2answers
2k views

How do you deal with very large datasets in Mathematica?

This question is in two related parts. The first is about dataset size, the second data wrangling. Part 1: I've built a naive Bayesian classifier on Mathematica. It uses ...
15
votes
1answer
513 views

Export dataset to JSON

I'm experiencing a weird issue. Consider a dataset defined like this: dataset = Dataset[{<|"a" -> 1, "b" -> 2|>, <|"a" -> 3, "b" -> 4|>}]; ...
14
votes
3answers
561 views

How to Import random elements of huge data files

I am calculating huge data files with an external program. I would like then to import the data into Mathematica for analysis. The files are 2 columns and up to many millions of rows. So for ...
12
votes
6answers
1k views

Determine the Number of Lines in a Text File

I have looked and looked but I do not see a simple efficient way to get MMA to return the number of lines in a text file? I thought about reading the file until it returned the EOF marker, but that ...
12
votes
3answers
545 views

WaveletScalogram in polar coordinates

For given dataset data = Re[Zeta[1/2 + I Range[0, 100, 0.01]]]; It is nice that Mathematica can plot data in both Cartesian and Polar coordinates. ...
12
votes
2answers
826 views

Filtering and Replacing outliers

I have a set of data that measurements of temperature vs time (date). It is easy to realize when the sensor went wrong as one might see a blip on the data. How to teach to Mathematica to: 1) Detect ...
11
votes
3answers
1k views

Smoothing ListContourPlot contours

I have some data of type {{x1, y1, f1}, {x2, y2, f2}, ...} and want to do ListContourPlot. However, there is a problem that the ...
11
votes
2answers
2k views

Finding fourier transform of data and hence frequencies

I looked at all the other questions related to mine before posting this, and they didn't solve my problem. I have a large data set which can be downloaded from here. I'm using the following code to ...
11
votes
2answers
473 views

Map of United States as a 3D histogram

By combining the state data from here and the extrusion code from here I have managed to make prisms of the various US states such as However, the extrusion method looses information about the ...
10
votes
2answers
3k views

How to get Matlab data imported with the same dimensions?

I have some Matlab image data with the following dimensions (output is from Matlab): >> size(im) ans = 86 86 3 45 The data imports just fine ...
10
votes
5answers
671 views

Relational joining of tables

A common thing to want to do with data is to combine it (at least I thought this was a common thing). In SQL there is the idea of table joins "select id,v1,v2 from A,B where A.id=B.id" kind of thing. ...
9
votes
3answers
993 views

Approximately Fit Data Without FindFit

I have some noisy data, and I'd like to analyze the derivative of the function the data approximately matches. Interpolation is normally great, but doesn't work ...
9
votes
3answers
238 views

Improving selection of elements from large multidimensional array

Frequently in my profession I deal with large arrays of data. Typically these are mixed types including integers, strings, dates, and reals. To recreate a typical data array: ...
9
votes
1answer
207 views

Median of Dataset in Mathematica 10

I'm having trouble calculating the median of a Dataset[] in Mathematica 10. The situation is as follows. Consider a dataset that was defined as follows: ...
9
votes
0answers
75 views

How does Dataset compare to DataFrame in pandas?

Pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data. Just like ...
8
votes
3answers
300 views

“NotAvailable” and “NotApplicable” data points returned by WeatherData[]

I am trying to use ListVectorPlot in combination with WeatherData and CountryData to show ...
8
votes
3answers
407 views

How to parse information contained in a file name

I have a set of files containing my data, and the file-names contain relevant information that I would like to extract. What I have is: ...
8
votes
3answers
398 views

Turning a set of random walk data 45 degrees

I have a set of data that is just a "random" (generated by me, not by computer) sequence of length 2000 of 1's and (-1)'s. I used it to plot a 1-D random walk where +1 is step up, (-1) is step down, ...
8
votes
4answers
778 views

Replacing a for-loop with something functional

I'm a physics grad and have been using Mathematica mostly for data visualization. The evaluation part I normally do with C/Python, but I've been wanting to try out Mathematica for these tasks, too. ...
8
votes
1answer
240 views

How to reconstruct a missing data point

In the output of a large computer simulation(thousands of cpus for 3 days), I found that one data point is missing in the result. Is it possible to reconstruct the missing value using the remaining ...
8
votes
3answers
993 views

Analytically finding critical values in a hysteresis curve made with ListPlot

Let me start with a super basic explanation of what I'm trying to do, and I will follow with a more detailed discussion afterwards. From data in the form shown below (a ...
7
votes
1answer
106 views

Why does Rescale performance vary significantly based on the real number values in a list?

I have a list of the form: testList = Table[{Flatten[RandomReal[{-200, 200}, {1, 2}], 1], RandomReal[{-200, 200}, {69, 2}]}, {i, 1, 100}]; And I would like to ...
7
votes
1answer
241 views

Assign the contents of a “Copy” operation programmatically to a variable?

While an earlier question Cut and paste data from a spreadsheet nominally addresses the issue, a year has past since its OP asked it and I hope that others have come across simpler or more direct ...
7
votes
2answers
115 views

How can I recover a graph from a graphical output in a notebook after resetting the kernel?

I have an old Mathematica notebook with an output for a graph structure $G$ I would like to recover. However, the routine for building up the graph is stochastic, so I can't simply rerun the code. ...
7
votes
1answer
140 views

Is Mathematica's (v10.0.2) NetCDF exporter broken?

I'm trying something very simple, namely: Export["file.nc", {"Datasets" -> {"seqs" -> {"AAA", "BBB"}}}, "Rules"] And getting Java errors about "illegal ...
6
votes
2answers
502 views

Correct fourier scaling and high-resolution frequency identification

I have a dataset of amplitude versus time $(t,A(t))$ and I need to extract the dominant frequency and amplitude, and also get the amplitude at one other specific frequency. My data looks like this: ...
6
votes
3answers
217 views

Looking for ideas for handling a complicated fitting problem

I have the following data: ...
6
votes
2answers
222 views

Implement planar correction for skewed 3D profilometric data

I have some data that represents small-scale surface topography. The data is planar but not flat because it's impractical to level this microscope to a tolerance of microns. It looks like this: I'd ...
6
votes
3answers
2k views

Find a best fitting curve for some data with no regular pattern

Suppose some data with no regular pattern is given, so common patterns of functions all fail to fit it. Then how can one find the best fitting curve? I believe there will be some data for which we can ...
6
votes
2answers
4k views

How do I plot the derivative of a set of experimental points?

I have a set of points like this: P = {{$x_1$,$y_1$},{$x_2$,$y_2$},...,{$x_n$,$y_n$}}. I want to plot them and also plot the derivative of that graph. How can I do this with Mathematica? Is there a ...
6
votes
2answers
97 views

Removing a non linear noise trend from data

I have a dataset that looks as follows Now, this is from an actual measurement (of the reflection coefficient of a microchip with four LC resonators connected in a periodic fashion) so there is ...
6
votes
2answers
291 views

Find fit of an experimental PDF

After quite some struggle, I managed to put up a code that is syntaxically correct. The goal is to find the parameters for a gamma distribution that best match experimental data. The data is a set of ...
6
votes
3answers
704 views

Calculate mean of values in bins

Problem description: I have a list of $\{x,y\}$ pairs. I'd like to divide $x$ into equal[*] bins, say $bx_1, bx_2, \ldots$, calculate $\left<y\right>$ for every bin and then plot the bin values ...
6
votes
1answer
542 views

Having trouble fitting data using NonlinearModelFit

I'm trying to fit data = {{2002, 45000}, {2004, 90000}, {2005, 120000}, {2013, 211000}}; to an exponential function ...
6
votes
2answers
189 views

Building bounded polygon around heatmap (or points)

I have a set of data for world marine piracy. I'd like to build polygons encircling areas of active piracy. So to start with I get piracy data and make a heatmap from it. ...
6
votes
3answers
271 views

Storing data in the Notebook in readable form

The common problem in my theretical work is a need to import some experimental data to work with. This means that my notebook depends on some external files which makes it difficult to share it with ...
6
votes
1answer
91 views

How to provide data to users in a way similar to Mathematica's curated data

How can one go about implementing a custom data backend usable by Mathematica? For example something like CountryData or ...
5
votes
4answers
428 views

Breaking a sorted list into bins of a specified size

Can somebody please explain me the following: If I try to split data every increase in 0.5: ...