Questions on systematic data handling and analysis, including data correction, manipulation, or selection of user-data or built-in data available with Mathematica.

learn more… | top users | synonyms

33
votes
5answers
9k views

How to perform a multi-peak fitting?

I am wondering how to implement the multi-peak detecting and fitting in Mathematica. Following is an example of fitting the data using three peaks (such that the data ~ peak1 + peak2 + peak3). The ...
28
votes
4answers
1k views

Filtering beat-to-beat heart rate data

From an experiment, I have a dataset of beat-to-beat heart rate data: a list of the time between each heart beat in [ms]. The data is measured using an infrared optic sensor at the finger tip. The ...
28
votes
1answer
589 views

Visual representation of a Ranking over time (image provided)

[Disclaimer: I initially posted this question on stackOverflow 2 months ago and think it might be better suited for this forum (link to original question)] The graph below shows a ranking of ...
23
votes
5answers
2k views

Graph databases / Nosql with Mathematica

I am very new to mathematica but it looks like an interesting language and I would like to explore it for a future project to create a web search engine. It seems like it would be fairly well suited ...
22
votes
4answers
2k views

How to integrate functions of linearly interpolated data?

At first, consider integration of pure InterpolatingFunction. Importing some data (works in v.9, for earlier versions one can use this link to download zipped ...
20
votes
3answers
578 views

Detecting components in timeseries

I'm looking at this sequence of values: I'd like to detect the points where the center of the time-series shifts (around x=1000 and x=2000). Many of the transforms and smoothing methods I have ...
19
votes
5answers
2k views

Moving maximum function?

How can I construct a moving maximum function? For example, if I have a list of 12 values: { 5, 6, 9, 3, 2, 6, 7, 8, 1, 1, 4, 7 } and I want to maximize over 3 ...
19
votes
3answers
1k views

How to extract the edge from a set of points

I have an accumulation of points representing a body as you can see in the image below. From this data I want to generate a list of points or a curve, which is the envelope. The first idea was a ...
19
votes
4answers
1k views

How: 2D scatterplots with quantitative density-dependent coloring

Consider the following scatterplot of a 50K-point dataset: ...
18
votes
0answers
342 views

How does Dataset compare to DataFrame in pandas?

Pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data. Just like ...
17
votes
4answers
366 views

upper envelope of data

Here is a ListPlot[] of some data. Clearly, there is a fairly smooth upper envelope - the question is whether there is an nice way of extracting it...
17
votes
2answers
587 views

Can A Plot Be Overlayed With A Map?

Some brief, only slightly important background. I am doing a research project using the data from NASA's GRACE mission. I wrote a short Perl script to take two data files and find the change in ...
16
votes
6answers
1k views

Determine the Number of Lines in a Text File

I have looked and looked but I do not see a simple efficient way to get MMA to return the number of lines in a text file? I thought about reading the file until it returned the EOF marker, but that ...
16
votes
2answers
3k views

How do you deal with very large datasets in Mathematica?

This question is in two related parts. The first is about dataset size, the second data wrangling. Part 1: I've built a naive Bayesian classifier on Mathematica. It uses ...
16
votes
2answers
2k views

Dealing with large files

I have a *.dat file in which the contents are basically one giant list: {1,2,3,4} {5,6,7,8} {9,10,11,12} and so on. I'd like to read in the first 500 members { ...
16
votes
3answers
777 views

Export dataset to JSON including Unicode Characters

I'm experiencing a weird issue. Consider a dataset defined like this: dataset = Dataset[{<|"a" -> 1, "b" -> 2|>, <|"a" -> 3, "b" -> 4|>}]; ...
15
votes
4answers
1k views

Plotting data points: Optimizing size and visuals

The following problem is one that may have showed up to you while plotting a large data collection. Suppose you have a data set of say 1 million points. ...
15
votes
3answers
1k views

Parametric Interpolation of 2D data

I want to create an interpolation through the following 2dim data points: ...
15
votes
3answers
657 views

How to Import random elements of huge data files

I am calculating huge data files with an external program. I would like then to import the data into Mathematica for analysis. The files are 2 columns and up to many millions of rows. So for ...
13
votes
2answers
246 views

How to do Independent Component Analysis?

The new experimental function in 10.3.1 DimensionReduce[], has the following three options for Method: ...
12
votes
3answers
2k views

Smoothing ListContourPlot contours

I have some data of type {{x1, y1, f1}, {x2, y2, f2}, ...} and want to do ListContourPlot. However, there is a problem that the ...
12
votes
2answers
549 views

Construct a master curve from data

I will try to be as informative as possible. Clear["Global`*"] I have the following data ...
12
votes
2answers
4k views

Finding fourier transform of data and hence frequencies

I looked at all the other questions related to mine before posting this, and they didn't solve my problem. I have a large data set which can be downloaded from here. I'm using the following code to ...
12
votes
2answers
603 views

Map of United States as a 3D histogram

By combining the state data from here and the extrusion code from here I have managed to make prisms of the various US states such as However, the extrusion method looses information about the ...
12
votes
3answers
607 views

WaveletScalogram in polar coordinates

For given dataset data = Re[Zeta[1/2 + I Range[0, 100, 0.01]]]; It is nice that Mathematica can plot data in both Cartesian and Polar coordinates. ...
12
votes
1answer
325 views

Integration of incomplete rings (ellipses) pattern

I am facing a problem in trying to do a series of averages in incomplete images. I have a series of X-ray data images, like this one: As you can see my image is a series of partial rings/ellipses ...
12
votes
2answers
1k views

Filtering and Replacing outliers

I have a set of data that measurements of temperature vs time (date). It is easy to realize when the sensor went wrong as one might see a blip on the data. How to teach to Mathematica to: 1) Detect ...
11
votes
3answers
1k views

Approximately Fit Data Without FindFit

I have some noisy data, and I'd like to analyze the derivative of the function the data approximately matches. Interpolation is normally great, but doesn't work ...
10
votes
2answers
4k views

How to get Matlab data imported with the same dimensions?

I have some Matlab image data with the following dimensions (output is from Matlab): >> size(im) ans = 86 86 3 45 The data imports just fine ...
10
votes
5answers
870 views

Relational joining of tables

A common thing to want to do with data is to combine it (at least I thought this was a common thing). In SQL there is the idea of table joins "select id,v1,v2 from A,B where A.id=B.id" kind of thing. ...
9
votes
3answers
270 views

Improving selection of elements from large multidimensional array

Frequently in my profession I deal with large arrays of data. Typically these are mixed types including integers, strings, dates, and reals. To recreate a typical data array: ...
8
votes
3answers
379 views

“NotAvailable” and “NotApplicable” data points returned by WeatherData[]

I am trying to use ListVectorPlot in combination with WeatherData and CountryData to show ...
8
votes
2answers
785 views

Correct fourier scaling and high-resolution frequency identification

I have a dataset of amplitude versus time $(t,A(t))$ and I need to extract the dominant frequency and amplitude, and also get the amplitude at one other specific frequency. My data looks like this: ...
8
votes
3answers
850 views

How to parse information contained in a file name

I have a set of files containing my data, and the file-names contain relevant information that I would like to extract. What I have is: ...
8
votes
3answers
468 views

Turning a set of random walk data 45 degrees

I have a set of data that is just a "random" (generated by me, not by computer) sequence of length 2000 of 1's and (-1)'s. I used it to plot a 1-D random walk where +1 is step up, (-1) is step down, ...
8
votes
4answers
835 views

Replacing a for-loop with something functional

I'm a physics grad and have been using Mathematica mostly for data visualization. The evaluation part I normally do with C/Python, but I've been wanting to try out Mathematica for these tasks, too. ...
8
votes
1answer
263 views

How to reconstruct a missing data point

In the output of a large computer simulation(thousands of cpus for 3 days), I found that one data point is missing in the result. Is it possible to reconstruct the missing value using the remaining ...
8
votes
3answers
1k views

Analytically finding critical values in a hysteresis curve made with ListPlot

Let me start with a super basic explanation of what I'm trying to do, and I will follow with a more detailed discussion afterwards. From data in the form shown below (a ...
8
votes
1answer
197 views

Is Mathematica's (v10.0.2) NetCDF exporter broken? [duplicate]

I'm trying something very simple, namely: Export["file.nc", {"Datasets" -> {"seqs" -> {"AAA", "BBB"}}}, "Rules"] And getting Java errors about "illegal ...
7
votes
5answers
534 views

How to extract points from a closed curve

OK, let's create a simple closed curve C0 = ContourPlot[x^2/4 + y^2/9 == 1, {x, -5, 5}, {y, -5, 5}]; and let's plot the corresponding points ...
7
votes
3answers
199 views

Fitting data with interdependence

I have an issue, of which I'm not sure it can be solved, with or without Mathematica. I feel as if it should be possible, but I'm quite clueless as to how. Some introduction is required. I'm working ...
7
votes
2answers
6k views

How do I plot the derivative of a set of experimental points?

I have a set of points like this: P = {{$x_1$,$y_1$},{$x_2$,$y_2$},...,{$x_n$,$y_n$}}. I want to plot them and also plot the derivative of that graph. How can I do this with Mathematica? Is there a ...
7
votes
1answer
112 views

Why does Rescale performance vary significantly based on the real number values in a list?

I have a list of the form: testList = Table[{Flatten[RandomReal[{-200, 200}, {1, 2}], 1], RandomReal[{-200, 200}, {69, 2}]}, {i, 1, 100}]; And I would like to ...
7
votes
2answers
408 views

Find fit of an experimental PDF

After quite some struggle, I managed to put up a code that is syntaxically correct. The goal is to find the parameters for a gamma distribution that best match experimental data. The data is a set of ...
7
votes
5answers
376 views

Storing data in the Notebook in readable form

The common problem in my work is a need to import some experimental data to work with. This means that my notebook depends on some external files which makes it difficult to share it with others and ...
7
votes
1answer
275 views

Assign the contents of a “Copy” operation programmatically to a variable?

While an earlier question Cut and paste data from a spreadsheet nominally addresses the issue, a year has past since its OP asked it and I hope that others have come across simpler or more direct ...
7
votes
2answers
121 views

How can I recover a graph from a graphical output in a notebook after resetting the kernel?

I have an old Mathematica notebook with an output for a graph structure $G$ I would like to recover. However, the routine for building up the graph is stochastic, so I can't simply rerun the code. ...
7
votes
1answer
49 views

KaryTree of Text .csv and Correlation

How can I make a KaryTree of this raw-data (raw-data at the end of the question): This is is data from a political survey in which I ask 5 questions with the following possible answers: Q1: 'Y'= ...
7
votes
1answer
452 views

How to create a real-time dashboard in Mathematica?

I am trying to figure out how to use Mathematica as a real-time interface. I've used Excel/ODBC connection to a SQL database to use the graphing functions within to get my real-time dashboard. I ...
6
votes
4answers
260 views

Average dataset with many repetitions?

I have a dataset where each item is of the form: <|"p1" -> x1, "p2" -> x2, "r1" -> y1, "r2" -> y2|> The interpretation is this: ...