Questions on systematic data handling and analysis, including data correction, manipulation, or selection of user-data or built-in data available with Mathematica.

learn more… | top users | synonyms

33
votes
5answers
8k views

How to perform a multi-peak fitting?

I am wondering how to implement the multi-peak detecting and fitting in Mathematica. Following is an example of fitting the data using three peaks (such that the data ~ peak1 + peak2 + peak3). The ...
28
votes
4answers
1k views

Filtering beat-to-beat heart rate data

From an experiment, I have a dataset of beat-to-beat heart rate data: a list of the time between each heart beat in [ms]. The data is measured using an infrared optic sensor at the finger tip. The ...
28
votes
1answer
556 views

Visual representation of a Ranking over time (image provided)

[Disclaimer: I initially posted this question on stackOverflow 2 months ago and think it might be better suited for this forum (link to original question)] The graph below shows a ranking of ...
23
votes
5answers
2k views

Graph databases / Nosql with Mathematica

I am very new to mathematica but it looks like an interesting language and I would like to explore it for a future project to create a web search engine. It seems like it would be fairly well suited ...
21
votes
4answers
2k views

How to integrate functions of linearly interpolated data?

At first, consider integration of pure InterpolatingFunction. Importing some data (works in v.9, for earlier versions one can use this link to download zipped ...
19
votes
5answers
2k views

Moving maximum function?

How can I construct a moving maximum function? For example, if I have a list of 12 values: { 5, 6, 9, 3, 2, 6, 7, 8, 1, 1, 4, 7 } and I want to maximize over 3 ...
19
votes
3answers
1k views

How to extract the edge from a set of points

I have an accumulation of points representing a body as you can see in the image below. From this data I want to generate a list of points or a curve, which is the envelope. The first idea was a ...
19
votes
4answers
1k views

How: 2D scatterplots with quantitative density-dependent coloring

Consider the following scatterplot of a 50K-point dataset: ...
18
votes
3answers
471 views

Detecting components in timeseries

I'm looking at this sequence of values: I'd like to detect the points where the center of the time-series shifts (around x=1000 and x=2000). Many of the transforms and smoothing methods I have ...
17
votes
2answers
533 views

Can A Plot Be Overlayed With A Map?

Some brief, only slightly important background. I am doing a research project using the data from NASA's GRACE mission. I wrote a short Perl script to take two data files and find the change in ...
16
votes
2answers
2k views

How do you deal with very large datasets in Mathematica?

This question is in two related parts. The first is about dataset size, the second data wrangling. Part 1: I've built a naive Bayesian classifier on Mathematica. It uses ...
16
votes
2answers
1k views

Dealing with large files

I have a *.dat file in which the contents are basically one giant list: {1,2,3,4} {5,6,7,8} {9,10,11,12} and so on. I'd like to read in the first 500 members { ...
16
votes
3answers
710 views

Export dataset to JSON including Unicode Characters

I'm experiencing a weird issue. Consider a dataset defined like this: dataset = Dataset[{<|"a" -> 1, "b" -> 2|>, <|"a" -> 3, "b" -> 4|>}]; ...
15
votes
4answers
1k views

Plotting data points: Optimizing size and visuals

The following problem is one that may have showed up to you while plotting a large data collection. Suppose you have a data set of say 1 million points. ...
15
votes
3answers
1k views

Parametric Interpolation of 2D data

I want to create an interpolation through the following 2dim data points: ...
15
votes
3answers
628 views

How to Import random elements of huge data files

I am calculating huge data files with an external program. I would like then to import the data into Mathematica for analysis. The files are 2 columns and up to many millions of rows. So for ...
13
votes
6answers
1k views

Determine the Number of Lines in a Text File

I have looked and looked but I do not see a simple efficient way to get MMA to return the number of lines in a text file? I thought about reading the file until it returned the EOF marker, but that ...
12
votes
2answers
525 views

Construct a master curve from data

I will try to be as informative as possible. Clear["Global`*"] I have the following data ...
12
votes
2answers
3k views

Finding fourier transform of data and hence frequencies

I looked at all the other questions related to mine before posting this, and they didn't solve my problem. I have a large data set which can be downloaded from here. I'm using the following code to ...
12
votes
3answers
584 views

WaveletScalogram in polar coordinates

For given dataset data = Re[Zeta[1/2 + I Range[0, 100, 0.01]]]; It is nice that Mathematica can plot data in both Cartesian and Polar coordinates. ...
12
votes
1answer
310 views

Integration of incomplete rings (ellipses) pattern

I am facing a problem in trying to do a series of averages in incomplete images. I have a series of X-ray data images, like this one: As you can see my image is a series of partial rings/ellipses ...
12
votes
2answers
1k views

Filtering and Replacing outliers

I have a set of data that measurements of temperature vs time (date). It is easy to realize when the sensor went wrong as one might see a blip on the data. How to teach to Mathematica to: 1) Detect ...
12
votes
1answer
206 views

How does Dataset compare to DataFrame in pandas?

Pandas is a Python package providing fast, flexible, and expressive data structures designed to make working with “relational” or “labeled” data. Just like ...
11
votes
3answers
2k views

Smoothing ListContourPlot contours

I have some data of type {{x1, y1, f1}, {x2, y2, f2}, ...} and want to do ListContourPlot. However, there is a problem that the ...
11
votes
3answers
1k views

Approximately Fit Data Without FindFit

I have some noisy data, and I'd like to analyze the derivative of the function the data approximately matches. Interpolation is normally great, but doesn't work ...
11
votes
2answers
561 views

Map of United States as a 3D histogram

By combining the state data from here and the extrusion code from here I have managed to make prisms of the various US states such as However, the extrusion method looses information about the ...
10
votes
2answers
4k views

How to get Matlab data imported with the same dimensions?

I have some Matlab image data with the following dimensions (output is from Matlab): >> size(im) ans = 86 86 3 45 The data imports just fine ...
10
votes
5answers
802 views

Relational joining of tables

A common thing to want to do with data is to combine it (at least I thought this was a common thing). In SQL there is the idea of table joins "select id,v1,v2 from A,B where A.id=B.id" kind of thing. ...
9
votes
3answers
257 views

Improving selection of elements from large multidimensional array

Frequently in my profession I deal with large arrays of data. Typically these are mixed types including integers, strings, dates, and reals. To recreate a typical data array: ...
9
votes
1answer
239 views

Median of Dataset in Mathematica 10

I'm having trouble calculating the median of a Dataset[] in Mathematica 10. The situation is as follows. Consider a dataset that was defined as follows: ...
8
votes
3answers
353 views

“NotAvailable” and “NotApplicable” data points returned by WeatherData[]

I am trying to use ListVectorPlot in combination with WeatherData and CountryData to show ...
8
votes
2answers
706 views

Correct fourier scaling and high-resolution frequency identification

I have a dataset of amplitude versus time $(t,A(t))$ and I need to extract the dominant frequency and amplitude, and also get the amplitude at one other specific frequency. My data looks like this: ...
8
votes
3answers
650 views

How to parse information contained in a file name

I have a set of files containing my data, and the file-names contain relevant information that I would like to extract. What I have is: ...
8
votes
1answer
161 views

upper envelope of data

Here is a ListPlot[] of some data. Clearly, there is a fairly smooth upper envelope - the question is whether there is an nice way of extracting it...
8
votes
3answers
444 views

Turning a set of random walk data 45 degrees

I have a set of data that is just a "random" (generated by me, not by computer) sequence of length 2000 of 1's and (-1)'s. I used it to plot a 1-D random walk where +1 is step up, (-1) is step down, ...
8
votes
4answers
816 views

Replacing a for-loop with something functional

I'm a physics grad and have been using Mathematica mostly for data visualization. The evaluation part I normally do with C/Python, but I've been wanting to try out Mathematica for these tasks, too. ...
8
votes
1answer
255 views

How to reconstruct a missing data point

In the output of a large computer simulation(thousands of cpus for 3 days), I found that one data point is missing in the result. Is it possible to reconstruct the missing value using the remaining ...
8
votes
3answers
1k views

Analytically finding critical values in a hysteresis curve made with ListPlot

Let me start with a super basic explanation of what I'm trying to do, and I will follow with a more detailed discussion afterwards. From data in the form shown below (a ...
8
votes
1answer
167 views

Is Mathematica's (v10.0.2) NetCDF exporter broken?

I'm trying something very simple, namely: Export["file.nc", {"Datasets" -> {"seqs" -> {"AAA", "BBB"}}}, "Rules"] And getting Java errors about "illegal ...
7
votes
5answers
513 views

How to extract points from a closed curve

OK, let's create a simple closed curve C0 = ContourPlot[x^2/4 + y^2/9 == 1, {x, -5, 5}, {y, -5, 5}]; and let's plot the corresponding points ...
7
votes
3answers
184 views

Fitting data with interdependence

I have an issue, of which I'm not sure it can be solved, with or without Mathematica. I feel as if it should be possible, but I'm quite clueless as to how. Some introduction is required. I'm working ...
7
votes
2answers
5k views

How do I plot the derivative of a set of experimental points?

I have a set of points like this: P = {{$x_1$,$y_1$},{$x_2$,$y_2$},...,{$x_n$,$y_n$}}. I want to plot them and also plot the derivative of that graph. How can I do this with Mathematica? Is there a ...
7
votes
1answer
112 views

Why does Rescale performance vary significantly based on the real number values in a list?

I have a list of the form: testList = Table[{Flatten[RandomReal[{-200, 200}, {1, 2}], 1], RandomReal[{-200, 200}, {69, 2}]}, {i, 1, 100}]; And I would like to ...
7
votes
5answers
346 views

Storing data in the Notebook in readable form

The common problem in my work is a need to import some experimental data to work with. This means that my notebook depends on some external files which makes it difficult to share it with others and ...
7
votes
1answer
258 views

Assign the contents of a “Copy” operation programmatically to a variable?

While an earlier question Cut and paste data from a spreadsheet nominally addresses the issue, a year has past since its OP asked it and I hope that others have come across simpler or more direct ...
7
votes
2answers
121 views

How can I recover a graph from a graphical output in a notebook after resetting the kernel?

I have an old Mathematica notebook with an output for a graph structure $G$ I would like to recover. However, the routine for building up the graph is stochastic, so I can't simply rerun the code. ...
7
votes
1answer
381 views

How to create a real-time dashboard in Mathematica?

I am trying to figure out how to use Mathematica as a real-time interface. I've used Excel/ODBC connection to a SQL database to use the graphing functions within to get my real-time dashboard. I ...
7
votes
1answer
94 views

Why some GeoProjections show just half of Chukotka region (Russian map)

I've moved to V.10.3 as suggested and proceed with Russian map (the aim is to geoplot regional distributions of some values). Notice most eastern Russian region - Chukotka. Chukotka, unlike other ...
6
votes
4answers
241 views

Average dataset with many repetitions?

I have a dataset where each item is of the form: <|"p1" -> x1, "p2" -> x2, "r1" -> y1, "r2" -> y2|> The interpretation is this: ...
6
votes
3answers
228 views

Looking for ideas for handling a complicated fitting problem

I have the following data: ...