Questions on systematic data handling and analysis, including data correction, manipulation, or selection of user-data or built-in data available with Mathematica.

learn more… | top users | synonyms

33
votes
5answers
9k views

How to perform a multi-peak fitting?

I am wondering how to implement the multi-peak detecting and fitting in Mathematica. Following is an example of fitting the data using three peaks (such that the data ~ peak1 + peak2 + peak3). The ...
22
votes
4answers
2k views

How to integrate functions of linearly interpolated data?

At first, consider integration of pure InterpolatingFunction. Importing some data (works in v.9, for earlier versions one can use this link to download zipped <...
12
votes
3answers
2k views

Smoothing ListContourPlot contours

I have some data of type {{x1, y1, f1}, {x2, y2, f2}, ...} and want to do ListContourPlot. However, there is a problem that the ...
5
votes
2answers
2k views

How to extract data from contour plot as a text file?

I have the following expression from which I want to extract data for a contour plot. ...
4
votes
3answers
186 views

Making large calculation results persistent

Very often, I need to do same calculations under different parameters for comparing the results. Every calculation may generate a large data set; for example, the calculation result of ...
16
votes
2answers
3k views

How do you deal with very large datasets in Mathematica?

This question is in two related parts. The first is about dataset size, the second data wrangling. Part 1: I've built a naive Bayesian classifier on Mathematica. It uses ...
23
votes
5answers
2k views

Graph databases / Nosql with Mathematica

I am very new to mathematica but it looks like an interesting language and I would like to explore it for a future project to create a web search engine. It seems like it would be fairly well suited ...
17
votes
2answers
2k views

Dealing with large files

I have a *.dat file in which the contents are basically one giant list: {1,2,3,4} {5,6,7,8} {9,10,11,12} and so on. I'd like to read in the first 500 members { }...
10
votes
2answers
4k views

How to get Matlab data imported with the same dimensions?

I have some Matlab image data with the following dimensions (output is from Matlab): >> size(im) ans = 86 86 3 45 The data imports just fine ...
2
votes
1answer
859 views

Data filtering using Butterworth filter

I want to filter data using Butterworth filter. I am currently using Matlab and I want to know how to do it in Mathematica. I have the following data: ...
5
votes
3answers
252 views

Removing background from peak function

Let's say I have a a very large list of data in the format {{n,f[n]}} so something like {{1,600},{2,700},{3,1100}...} and if this data is plotted, then it will look like a sum of Gaussian functions+a ...
16
votes
3answers
1k views

Parametric Interpolation of 2D data

I want to create an interpolation through the following 2dim data points: ...
8
votes
2answers
828 views

Correct fourier scaling and high-resolution frequency identification

I have a dataset of amplitude versus time $(t,A(t))$ and I need to extract the dominant frequency and amplitude, and also get the amplitude at one other specific frequency. My data looks like this: <...
7
votes
5answers
385 views

Storing data in the Notebook in readable form

The common problem in my work is a need to import some experimental data to work with. This means that my notebook depends on some external files which makes it difficult to share it with others and ...
1
vote
3answers
507 views

How to import files with headers?

R has excellent header supports such as read.table with header options. I don't want to mess up with text-processing or stream-processing. I am looking for a similar tool as in R because it saves a ...
8
votes
3answers
473 views

Turning a set of random walk data 45 degrees

I have a set of data that is just a "random" (generated by me, not by computer) sequence of length 2000 of 1's and (-1)'s. I used it to plot a 1-D random walk where +1 is step up, (-1) is step down, ...
0
votes
3answers
306 views

Importing multiple files with a loop

I have 101 files with data that I want to import into Mathematica. All the file names are numbered as follows datafile.20.0000.out, datafile.20.1000.out, datafile.20.2000.out and so on. I used the ...
17
votes
4answers
410 views

upper envelope of data

Here is a ListPlot[] of some data. Clearly, there is a fairly smooth upper envelope - the question is whether there is an nice way of extracting it...
15
votes
4answers
1k views

Plotting data points: Optimizing size and visuals

The following problem is one that may have showed up to you while plotting a large data collection. Suppose you have a data set of say 1 million points. ListLinePlot...
12
votes
2answers
4k views

Finding fourier transform of data and hence frequencies

I looked at all the other questions related to mine before posting this, and they didn't solve my problem. I have a large data set which can be downloaded from here. I'm using the following code to ...
6
votes
3answers
1k views

Calculate mean of values in bins

Problem description: I have a list of $\{x,y\}$ pairs. I'd like to divide $x$ into equal[*] bins, say $bx_1, bx_2, \ldots$, calculate $\left<y\right>$ for every bin and then plot the bin values ...
12
votes
2answers
1k views

Filtering and Replacing outliers

I have a set of data that measurements of temperature vs time (date). It is easy to realize when the sensor went wrong as one might see a blip on the data. How to teach to Mathematica to: 1) Detect ...
7
votes
1answer
281 views

Assign the contents of a “Copy” operation programmatically to a variable?

While an earlier question Cut and paste data from a spreadsheet nominally addresses the issue, a year has past since its OP asked it and I hope that others have come across simpler or more direct ...
6
votes
2answers
934 views

ListContourPlot interpolation screws up on concave set of data

I want to create ListContourPlots for sets of Data (from a physics simulation) given as Triplets {x, y, f(x, y)}. Certain parameter regions in the x-y-plane are ...
5
votes
3answers
383 views

Histogram of a very large dataset?

I have a very large dataset stored in a file (over 2GB). The file contains a tab-separated table of floating-point numbers. I want to do an Histogram of all the numbers in the table. But if I try to ...
4
votes
2answers
230 views

Deseasonalize the data of traffic growth for the past data

In another question (Using TimeSeriesForecast for forecasting the traffic growth) I asked to use the TimeSeriesForecast on this set of data the answer provided the plot with and without the ...
4
votes
1answer
992 views

ReadList with header and foot lines

I want to do some large data analysis, and i want to be able to import files in a fast way. I tried Import, but it is quite slow; ReadList takes half the time. This is the code I was using: ...
4
votes
4answers
616 views

More elegant way

I read ReplaceOrRemoveInvalidOrMissingData Mathematica documentation, and I tried to apply the same approach to my dataset. I've got two problems, but one is only a style problem! There is one more ...
3
votes
2answers
347 views

Selecting for 2D points that are within a threshold distance of an upper- and lower-bound number of points

I have a very large set of 2D points: numberOf2DPoints = 10^6; pointList = RandomReal[{0, 1000}, {numberOf2DPoints, 2}]; I'd like to find a way to quickly ...
0
votes
1answer
2k views

How to plot data with error bars on a log scale in Mathematica 9?

I have data in the form d={{x1,y1,dy1},{x2,y2,dy2},..{xn,yn,dyn}}; so the error bars are different for each coordinate pair. I want to plot on a logarithmic x-...
5
votes
2answers
1k views

How can I control the number format of exported data?

I'm having issues while exporting a tables of numerical values to "*.dat" data files. Here's the thing, sometimes when I export a table, I get more decimals on some random values. The problem is that ...
2
votes
5answers
614 views

How do I find the maximum value from the given set of data?

I have a set of data like {{x1, y1},{x2, y2}, ...,{xn, yn}}. From this data can I find the value of x for which ...
2
votes
2answers
192 views

Animation of moving rectangles in 2D from data file

I am not very familiar with mathematica but I know the basics so I thought I give it a try. For a project I have to animate rectangles moving around in 2D (later also 3D). The data come from a C ...
19
votes
3answers
1k views

How to extract the edge from a set of points

I have an accumulation of points representing a body as you can see in the image below. From this data I want to generate a list of points or a curve, which is the envelope. The first idea was a ...
20
votes
3answers
608 views

Detecting components in timeseries

I'm looking at this sequence of values: I'd like to detect the points where the center of the time-series shifts (around x=1000 and x=2000). Many of the transforms and smoothing methods I have ...
29
votes
4answers
1k views

Filtering beat-to-beat heart rate data

From an experiment, I have a dataset of beat-to-beat heart rate data: a list of the time between each heart beat in [ms]. The data is measured using an infrared optic sensor at the finger tip. The ...
18
votes
3answers
825 views

Export dataset to JSON including Unicode Characters

I'm experiencing a weird issue. Consider a dataset defined like this: dataset = Dataset[{<|"a" -> 1, "b" -> 2|>, <|"a" -> 3, "b" -> 4|>}]; ...
10
votes
5answers
893 views

Relational joining of tables

A common thing to want to do with data is to combine it (at least I thought this was a common thing). In SQL there is the idea of table joins "select id,v1,v2 from A,B where A.id=B.id" kind of thing. ...
18
votes
2answers
625 views

Can A Plot Be Overlayed With A Map?

Some brief, only slightly important background. I am doing a research project using the data from NASA's GRACE mission. I wrote a short Perl script to take two data files and find the change in ...
16
votes
6answers
1k views

Determine the Number of Lines in a Text File

I have looked and looked but I do not see a simple efficient way to get MMA to return the number of lines in a text file? I thought about reading the file until it returned the EOF marker, but that ...
19
votes
5answers
2k views

Moving maximum function?

How can I construct a moving maximum function? For example, if I have a list of 12 values: { 5, 6, 9, 3, 2, 6, 7, 8, 1, 1, 4, 7 } and I want to maximize over 3 ...
15
votes
3answers
669 views

How to Import random elements of huge data files

I am calculating huge data files with an external program. I would like then to import the data into Mathematica for analysis. The files are 2 columns and up to many millions of rows. So for ...
8
votes
3answers
401 views

“NotAvailable” and “NotApplicable” data points returned by WeatherData[]

I am trying to use ListVectorPlot in combination with WeatherData and CountryData to show ...
6
votes
2answers
623 views

Map of Swedish regions?

There have been some questions dealing with how to draw individual US states since such data doesn't come with Mathematica through its CountryData which offers ...
3
votes
1answer
152 views

How can I export definitions of several symbols to a single file?

the problem related to this question What I want to do is to Export definition of several symbol into a single file. The reason I choose ...
2
votes
2answers
372 views

Using Periodogram to investigate the seasonality of the data

After asking how to get rid of the seasonality of this set of data, I'd like to ask how to investigate the seasonality in data using Mathematica. Which tool would you use to study seasonality? I.e. ...
5
votes
0answers
136 views

Dropbox datastore

Dropbox seems to have a feature where it can be used as a database. This would be really cool to use for memoized functions, that can be restored on several computers and sessions. I have a function ...
4
votes
1answer
378 views

BarChart that lines up with dates in DateListPlot

I am trying to create a chart similar to the view offered by TradingChart, but with custom data not relating to finance. ...
1
vote
4answers
7k views

Creating a data1 versus data2 plot?

I have a table of data with multiple rows and columns, and I have to make a versus data table using two of these columns. Essentially, I have to set one column of data (column 3) as the y-axis and ...
8
votes
1answer
200 views

Is Mathematica's (v10.0.2) NetCDF exporter broken? [duplicate]

I'm trying something very simple, namely: Export["file.nc", {"Datasets" -> {"seqs" -> {"AAA", "BBB"}}}, "Rules"] And getting Java errors about "illegal ...