Questions about getting data, files, images etc. from various formats into a form that can be read, tranformed and manipulated by Mathematica
40
votes
3answers
2k views
Recovering data points from an image
Unfortunately, some data can only be obtained in picture form (Japanese publications anyone?). Since this cannot be worked with, it has to be converted to a dataset that can; I was wondering whether ...
25
votes
5answers
2k views
How can I extract data points from a black and white image?
Consider the following graph (source):
Is there any way to extract the data points from this image in a semi-automatic way? I have seen, and tried the methods suggested in this question, but they ...
21
votes
3answers
639 views
How to do http POST in Mathematica?
The Import command lets you do a HTTP GET from a URL, like this
Import @ "http://merkosoncampus.com/wp-content/uploads/2011/05/higgs-boson.jpg"
This sends an ...
20
votes
3answers
840 views
How to read data file quickly?
I have a tab separated value file with 10 million rows each of which has three tab separated values. The first value is a string, the second an integer, and the third another string. How to read ...
20
votes
1answer
796 views
Programmatically retrieving e-mails from a remote server
Mathematica has a function for sending mail (SendMail) and can import the MBOX format. But what I'd like to do is ...
19
votes
2answers
691 views
Extracting values from nested rules in JSON data
I have been using Mathematica to analyse some data from the StackExchange API. It is conveniently available in JSON form, which Mathematica interprets as replacement rules. However, some of the rules ...
19
votes
3answers
737 views
Importing videos in Mathematica
Mathematica's Import command is purportedly able to import .AVI files. However, I find that many files that I want to import do only result in MMA showing a blank ...
18
votes
5answers
2k views
How do you convert a string containing a number in C scientific notation to a Mathematica number?
Suppose I have a string containing the C-representation of a floating point number, for example
s = "1.23e-5"
and I want to convert this to a Mathematica number. ...
16
votes
3answers
613 views
Reading periodic elements from a large file
I have a large binary data file (big endian) with 100+ million "rows" of 11 elements, combination of floats and integers.
This is the format:
...
16
votes
2answers
347 views
Why does ImageData need four times more memory?
Background
We are dealing with very large TIFF image files that are imported, processed and exported using Mathematica 9. Many of our algorithms only work with an array representation of the images, ...
15
votes
1answer
282 views
Importing Zip files
Mathematica can import zipped files automatically, but when the content of the zip file it's another zipped file, I just get the list of files in the second zip:
...
15
votes
1answer
310 views
When importing GIF animation, how to find the correct list of “DisplayDurations”?
When importing GIF animations with variable frame durations, the settings for "DisplayDurations" are not recovered correctly. As an example, I'm using the movie from this earlier question:
Here I'm ...
15
votes
1answer
414 views
How to export and import Graphs with additional data?
I'm trying to export and then later import graphs with additional data added such as VertexLabels and EdgeLabels. When I export ...
14
votes
2answers
257 views
How to load thumbnail, not image?
Is there any way to import only the thumbnail of an image instead of the full image? I frequently work with lots of images and it would be nice if the contents of a directory could be quickly ...
14
votes
1answer
195 views
Getting Mathematica.SE reputation data
I'm asking the following on behalf of Mr.Wizard.
On StackOverflow I used the old v1.0 SO API and Mathematica to obtain user reputation data. Now we are no longer a tag on StackOverflow and the v2.1 ...
14
votes
1answer
667 views
How to manipulate web pages on Mathematica?
I wanted to input some data on this page through Mathematica:
http://www.fundamentus.com.br/buscaavancada.php
Then I would press the "buscar" button also through mathematica coding and get the ...
14
votes
2answers
300 views
Is Compress[] compatible between different Mathematica versions?
Compress[] was introduced in version 6 of Mathematica. It gives a compressed version of expressions represented as a string of ASCII characters. It preserves more ...
13
votes
3answers
421 views
How to scrape the headlines from New York Times and Wall Street Journal?
How to scrape headlines from New York Times, Wall Street Journal main pages to create datasets similar to this service?
Importing HTML from nyt.com (HTML4) results in a ...
13
votes
3answers
622 views
Is it possible to use Mathematica to record sound and/or vision?
I think the title says it all. I know about ImageCapture which seems to capture stills.
Can ImageCapture be made to capture a ...
13
votes
1answer
291 views
Gravatars/identicons with Mathematica
The default avatar/identicon is based on an MD5 hash of your e-mail address. With some e-mail providers (e.g., GMail) the address gmailuser+foo@gmail.com is still ...
12
votes
2answers
646 views
How to save all data in all variables so that loading it is fast?
I've got this CSV file I've imported that has tens of millions of lines in it. It takes around 20 minutes to import. I've been working with it for a while and have the processed data spread out in a ...
12
votes
2answers
347 views
How to develop an Import/Export converter for Compress[]ed data?
Mathematica has a cross platform data exchange format, WDX. Unfortunately importing or exporting large data from/to WDX is very slow. Using MX files is very fast, but they are not compatible across ...
12
votes
4answers
571 views
Cut and paste data from a spreadsheet
I have a set of data points in two columns in a spreadsheet (OpenOffice Calc):
I would like to get these into Mathematica in this format:
...
12
votes
2answers
651 views
How to import all files of a folder at once?
I know how to import one textfile by calling its name
filestring = Import["myfile.tex", "Text"];
Then "filestring" is a string with the myfile content.
How do ...
12
votes
2answers
302 views
How to monitor the progress of Read?
I use
f = OpenRead["largefile.m"]
x = Read[f];
Close[f];
to read in a Mathematica expression from a large file largefile.m ...
12
votes
2answers
1k views
Importing a large Excel file
I'm trying to import a simple Excel file containing 94,000 rows and 52 columns of various types, mostly Reals. It's a 38-MB file. I get this super helpful error ...
12
votes
3answers
169 views
How to Import random elements of huge data files
I am calculating huge data files with an external program. I would like then to import the data into Mathematica for analysis. The files are 2 columns and up to many millions of rows.
So for ...
12
votes
1answer
243 views
How to efficiently import videos
I need to do frame-by-frame analysis of videos. I can import the video files I need, but each file is too large to import at once. I can use the ...
11
votes
3answers
311 views
Data of badly formatted file
Similar as in this question, I have to read in data from a file (exampleFile.txt) which is not so well formatted.
Unfortunately, I cannot find a solution by myself. Since it is a bit complicated to ...
11
votes
1answer
306 views
Release the web camera after using CurrentImage[]
CurrentImage[] will capture an image from the webcam. It is even possible to create a live video in the notebook using
...
10
votes
3answers
1k views
Plot data from txt file?
I have data in file.txt which can be expressed by 20000x11 matrix. I do not want to import all of these data. How can I plot graph of the column 1 & column 4?
10
votes
4answers
449 views
Import[ ] drops leading zeros from data when importing a CSV file
I need to load data from the CSV file shown below.
Import[ ] drops the leading zeros in column 3 when it imports the CSV file, but they are security identifiers, and I need them. ReadList[ ] ...
10
votes
2answers
373 views
Maximum size of TIF files in Mathematica
I have a question concerning the maximum size of TIFF files Mathematica can handle. According to the TIFF 6.0 specification -revision 6.0 (see: ...
10
votes
1answer
151 views
Exporting and reimporting a list of lists changes the dimensions
Could anyone explain to me what in the world is going on here and please help me in reading in dummy.dat properly?
...
10
votes
1answer
665 views
Web Browsing Automation with Mathematica
I want to build a bot to automate web browsing, this mean something like:
filling forms
press "submit" buttons
find certain text inside pages
and so on...
How can I do this with Mathematica?
The ...
9
votes
5answers
367 views
convert long string to integer list
at first, i would like to thank all the people here. i'm working with mathematica for half a year now and always found great advice and tricks.
now, i came up with a problem i couldn't find a ...
9
votes
6answers
806 views
How do you skip missing data values in a data file?
In scanning through a simple imported file of weather station data, some dates have missing data.
What is a test to use to check whether a datum is valid? I tried ...
9
votes
2answers
159 views
How to extract a specific element from EXIF photographic metadata?
It's now possible to import photographs into Mathematica and import the EXIF data at the same time:
i = Import["ExampleData/coneflower.jpg", "ImageWithExif"]
...
9
votes
2answers
901 views
How to get Matlab data imported with the same dimensions?
I have some Matlab image data with the following dimensions (output is from Matlab):
>> size(im)
ans =
86 86 3 45
The data imports just fine ...
9
votes
1answer
341 views
How to import date (YYYY-MM-DD) data from a file?
I have a file with data in form {date, value}:
2010-05-19 17
2010-05-20 20
...
9
votes
1answer
200 views
Possible to import Excel formatted text?
Is it possible to import Excel text markup such as underlines? For example:
Yet
...
9
votes
3answers
489 views
Is it possible to import dates and times directly as AbsoluteTime and by pass DateLists?
Date and time calculations are quite slow so I am exploring ways to minimize the number of calculations performed. As an example:
...
8
votes
2answers
320 views
How can I animate a set of points imported from file?
I have an external program which generates a set of data - X points at the time t1, X points at the time t2 and so on. These points are stored in CSV file and I can change the format if necessary.
...
8
votes
1answer
164 views
Import the “thumbnail” of an internet page
I have a list of internet sites links from which I would like to automatically recover the page image/thumbnail and page title.
Let's suppose that one of the links is https://www.google.com/, I would ...
8
votes
3answers
862 views
Excel Headers --> Mathematica
I have a large Excel .xls file. Miniature version shown below:
(formatting should appear as two columns...)
...
8
votes
2answers
162 views
8
votes
3answers
176 views
8
votes
3answers
542 views
Importing HDF5 with compound data
I am looking to Import HDF5 data with compound data. It appears, at least from what I read, that Mathematica 8.0.4 (and previous versions) does not support reading ...
8
votes
1answer
265 views
How to properly DumpSave & Get Global`s symbols inside packages while not touching Global context?
For efficiency reasons I prefer to use DumpSave instead of Save.
For ease of access I prefer to save symbols in ...
8
votes
1answer
262 views
Dealing with large files
I have a *.dat file in which the contents are basically one giant list:
{1,2,3,4} {5,6,7,8} {9,10,11,12}
and so on. I'd like to read in the first 500 members { ...
