Questions on systematic data handling and analysis, including data correction, manipulation, or selection of user-data or built-in data available with Mathematica.

learn more… | top users | synonyms

33
votes
5answers
9k views

How to perform a multi-peak fitting?

I am wondering how to implement the multi-peak detecting and fitting in Mathematica. Following is an example of fitting the data using three peaks (such that the data ~ peak1 + peak2 + peak3). The ...
22
votes
4answers
2k views

How to integrate functions of linearly interpolated data?

At first, consider integration of pure InterpolatingFunction. Importing some data (works in v.9, for earlier versions one can use this link to download zipped ...
12
votes
3answers
2k views

Smoothing ListContourPlot contours

I have some data of type {{x1, y1, f1}, {x2, y2, f2}, ...} and want to do ListContourPlot. However, there is a problem that the ...
5
votes
2answers
2k views

How to extract data from contour plot as a text file?

I have the following expression from which I want to extract data for a contour plot. ...
16
votes
2answers
3k views

How do you deal with very large datasets in Mathematica?

This question is in two related parts. The first is about dataset size, the second data wrangling. Part 1: I've built a naive Bayesian classifier on Mathematica. It uses ...
23
votes
5answers
2k views

Graph databases / Nosql with Mathematica

I am very new to mathematica but it looks like an interesting language and I would like to explore it for a future project to create a web search engine. It seems like it would be fairly well suited ...
12
votes
2answers
4k views

Finding fourier transform of data and hence frequencies

I looked at all the other questions related to mine before posting this, and they didn't solve my problem. I have a large data set which can be downloaded from here. I'm using the following code to ...
16
votes
2answers
2k views

Dealing with large files

I have a *.dat file in which the contents are basically one giant list: {1,2,3,4} {5,6,7,8} {9,10,11,12} and so on. I'd like to read in the first 500 members { ...
10
votes
2answers
4k views

How to get Matlab data imported with the same dimensions?

I have some Matlab image data with the following dimensions (output is from Matlab): >> size(im) ans = 86 86 3 45 The data imports just fine ...
12
votes
2answers
1k views

Filtering and Replacing outliers

I have a set of data that measurements of temperature vs time (date). It is easy to realize when the sensor went wrong as one might see a blip on the data. How to teach to Mathematica to: 1) Detect ...
5
votes
2answers
900 views

ListContourPlot interpolation screws up on concave set of data

I want to create ListContourPlots for sets of Data (from a physics simulation) given as Triplets {x, y, f(x, y)}. Certain parameter regions in the x-y-plane are ...
3
votes
3answers
173 views

Making large calculation results persistent

Very often, I need to do same calculations under different parameters for comparing the results. Every calculation may generate a large data set; for example, the calculation result of ...
5
votes
3answers
239 views

Removing background from peak function

Let's say I have a a very large list of data in the format {{n,f[n]}} so something like {{1,600},{2,700},{3,1100}...} and if this data is plotted, then it will look like a sum of Gaussian functions+a ...
2
votes
1answer
818 views

Data filtering using Butterworth filter

I want to filter data using Butterworth filter. I am currently using Matlab and I want to know how to do it in Mathematica. I have the following data: ...
15
votes
3answers
1k views

Parametric Interpolation of 2D data

I want to create an interpolation through the following 2dim data points: ...
8
votes
2answers
788 views

Correct fourier scaling and high-resolution frequency identification

I have a dataset of amplitude versus time $(t,A(t))$ and I need to extract the dominant frequency and amplitude, and also get the amplitude at one other specific frequency. My data looks like this: ...
7
votes
5answers
376 views

Storing data in the Notebook in readable form

The common problem in my work is a need to import some experimental data to work with. This means that my notebook depends on some external files which makes it difficult to share it with others and ...
1
vote
3answers
474 views

How to import files with headers?

R has excellent header supports such as read.table with header options. I don't want to mess up with text-processing or stream-processing. I am looking for a similar tool as in R because it saves a ...
8
votes
3answers
468 views

Turning a set of random walk data 45 degrees

I have a set of data that is just a "random" (generated by me, not by computer) sequence of length 2000 of 1's and (-1)'s. I used it to plot a 1-D random walk where +1 is step up, (-1) is step down, ...
3
votes
3answers
986 views

Import data from first/last 100 rows of a file

If one wants import data from the first three rows of a file "myfile.data", one can use the command Import["myfile.data",{"Data",{1,2,3},All}]; However, if I ...
2
votes
5answers
571 views

How do I find the maximum value from the given set of data?

I have a set of data like {{x1, y1},{x2, y2}, ...,{xn, yn}}. From this data can I find the value of x for which ...
0
votes
3answers
263 views

Importing multiple files with a loop

I have 101 files with data that I want to import into Mathematica. All the file names are numbered as follows datafile.20.0000.out, datafile.20.1000.out, datafile.20.2000.out and so on. I used the ...
17
votes
4answers
368 views

upper envelope of data

Here is a ListPlot[] of some data. Clearly, there is a fairly smooth upper envelope - the question is whether there is an nice way of extracting it...
15
votes
4answers
1k views

Plotting data points: Optimizing size and visuals

The following problem is one that may have showed up to you while plotting a large data collection. Suppose you have a data set of say 1 million points. ...
6
votes
3answers
992 views

Calculate mean of values in bins

Problem description: I have a list of $\{x,y\}$ pairs. I'd like to divide $x$ into equal[*] bins, say $bx_1, bx_2, \ldots$, calculate $\left<y\right>$ for every bin and then plot the bin values ...
7
votes
1answer
275 views

Assign the contents of a “Copy” operation programmatically to a variable?

While an earlier question Cut and paste data from a spreadsheet nominally addresses the issue, a year has past since its OP asked it and I hope that others have come across simpler or more direct ...
5
votes
3answers
328 views

Histogram of a very large dataset?

I have a very large dataset stored in a file (over 2GB). The file contains a tab-separated table of floating-point numbers. I want to do an Histogram of all the numbers in the table. But if I try to ...
4
votes
2answers
218 views

Deseasonalize the data of traffic growth for the past data

In another question (Using TimeSeriesForecast for forecasting the traffic growth) I asked to use the TimeSeriesForecast on this set of data the answer provided the plot with and without the ...
4
votes
4answers
608 views

More elegant way

I read ReplaceOrRemoveInvalidOrMissingData Mathematica documentation, and I tried to apply the same approach to my dataset. I've got two problems, but one is only a style problem! There is one more ...
4
votes
1answer
924 views

ReadList with header and foot lines

I want to do some large data analysis, and i want to be able to import files in a fast way. I tried Import, but it is quite slow; ReadList takes half the time. This is the code I was using: ...
3
votes
2answers
339 views

Selecting for 2D points that are within a threshold distance of an upper- and lower-bound number of points

I have a very large set of 2D points: numberOf2DPoints = 10^6; pointList = RandomReal[{0, 1000}, {numberOf2DPoints, 2}]; I'd like to find a way to quickly ...
0
votes
1answer
2k views

How to plot data with error bars on a log scale in Mathematica 9?

I have data in the form d={{x1,y1,dy1},{x2,y2,dy2},..{xn,yn,dyn}}; so the error bars are different for each coordinate pair. I want to plot on a logarithmic ...
5
votes
2answers
483 views

Exporting row vector as a column in excel

I want to export a vector in Mathematica to appear as a column in Excel. I am trying A={1, 2, 3}; Export["mfile.xls", {"A" -> A}] But it is showing 1,2,3 in ...
5
votes
2answers
1k views

How can I control the number format of exported data?

I'm having issues while exporting a tables of numerical values to "*.dat" data files. Here's the thing, sometimes when I export a table, I get more decimals on some random values. The problem is that ...
2
votes
2answers
186 views

Animation of moving rectangles in 2D from data file

I am not very familiar with mathematica but I know the basics so I thought I give it a try. For a project I have to animate rectangles moving around in 2D (later also 3D). The data come from a C ...
0
votes
1answer
40 views

save all inputs as list [duplicate]

I received three results of calculations: {k->0.190729} {k->0.197575} {k->0.249319} Could you help me write the results in the following form: ...
19
votes
3answers
1k views

How to extract the edge from a set of points

I have an accumulation of points representing a body as you can see in the image below. From this data I want to generate a list of points or a curve, which is the envelope. The first idea was a ...
28
votes
4answers
1k views

Filtering beat-to-beat heart rate data

From an experiment, I have a dataset of beat-to-beat heart rate data: a list of the time between each heart beat in [ms]. The data is measured using an infrared optic sensor at the finger tip. The ...
16
votes
3answers
783 views

Export dataset to JSON including Unicode Characters

I'm experiencing a weird issue. Consider a dataset defined like this: dataset = Dataset[{<|"a" -> 1, "b" -> 2|>, <|"a" -> 3, "b" -> 4|>}]; ...
10
votes
5answers
870 views

Relational joining of tables

A common thing to want to do with data is to combine it (at least I thought this was a common thing). In SQL there is the idea of table joins "select id,v1,v2 from A,B where A.id=B.id" kind of thing. ...
16
votes
6answers
1k views

Determine the Number of Lines in a Text File

I have looked and looked but I do not see a simple efficient way to get MMA to return the number of lines in a text file? I thought about reading the file until it returned the EOF marker, but that ...
19
votes
5answers
2k views

Moving maximum function?

How can I construct a moving maximum function? For example, if I have a list of 12 values: { 5, 6, 9, 3, 2, 6, 7, 8, 1, 1, 4, 7 } and I want to maximize over 3 ...
15
votes
3answers
657 views

How to Import random elements of huge data files

I am calculating huge data files with an external program. I would like then to import the data into Mathematica for analysis. The files are 2 columns and up to many millions of rows. So for ...
12
votes
2answers
605 views

Map of United States as a 3D histogram

By combining the state data from here and the extrusion code from here I have managed to make prisms of the various US states such as However, the extrusion method looses information about the ...
12
votes
3answers
607 views

WaveletScalogram in polar coordinates

For given dataset data = Re[Zeta[1/2 + I Range[0, 100, 0.01]]]; It is nice that Mathematica can plot data in both Cartesian and Polar coordinates. ...
8
votes
3answers
381 views

“NotAvailable” and “NotApplicable” data points returned by WeatherData[]

I am trying to use ListVectorPlot in combination with WeatherData and CountryData to show ...
6
votes
2answers
601 views

Map of Swedish regions?

There have been some questions dealing with how to draw individual US states since such data doesn't come with Mathematica through its CountryData which offers ...
3
votes
1answer
149 views

How can I export definitions of several symbols to a single file?

the problem related to this question What I want to do is to Export definition of several symbol into a single file. The reason I choose ...
2
votes
2answers
361 views

Using Periodogram to investigate the seasonality of the data

After asking how to get rid of the seasonality of this set of data, I'd like to ask how to investigate the seasonality in data using Mathematica. Which tool would you use to study seasonality? I.e. ...
5
votes
0answers
121 views

Dropbox datastore

Dropbox seems to have a feature where it can be used as a database. This would be really cool to use for memoized functions, that can be restored on several computers and sessions. I have a function ...