Questions about getting data, files, images etc. from various formats into a form that can be read, tranformed and manipulated by Mathematica

learn more… | top users | synonyms

58
votes
3answers
5k views

Recovering data points from an image

Unfortunately, some data can only be obtained in picture form (Japanese publications anyone?). Since this cannot be worked with, it has to be converted to a dataset that can; I was wondering whether ...
35
votes
8answers
4k views

How do you convert a string containing a number in C scientific notation to a Mathematica number?

Suppose I have a string containing the C-representation of a floating point number, for example s = "1.23e-5" and I want to convert this to a Mathematica number. ...
35
votes
5answers
4k views

How can I extract data points from a black and white image?

Consider the following graph (source): Is there any way to extract the data points from this image in a semi-automatic way? I have seen, and tried the methods suggested in this question, but they ...
30
votes
3answers
2k views

How to do http POST in Mathematica?

The Import command lets you do a HTTP GET from a URL, like this Import @ "http://merkosoncampus.com/wp-content/uploads/2011/05/higgs-boson.jpg" This sends an ...
29
votes
4answers
2k views

Importing videos in Mathematica

Mathematica's Import command is purportedly able to import .AVI files. However, I find that many files that I want to import do only result in MMA showing a blank ...
25
votes
3answers
2k views

How to read data file quickly?

I have a tab separated value file with 10 million rows each of which has three tab separated values. The first value is a string, the second an integer, and the third another string. How to read ...
22
votes
3answers
1k views

Reading periodic elements from a large file

I have a large binary data file (big endian) with 100+ million "rows" of 11 elements, combination of floats and integers. This is the format: ...
22
votes
1answer
2k views

Programmatically retrieving e-mails from a remote server

Mathematica has a function for sending mail (SendMail) and can import the MBOX format. But what I'd like to do is ...
21
votes
2answers
2k views

How to save all data in all variables so that loading it is fast?

I've got this CSV file I've imported that has tens of millions of lines in it. It takes around 20 minutes to import. I've been working with it for a while and have the processed data spread out in a ...
21
votes
2answers
1k views

How to manipulate web pages on Mathematica?

I wanted to input some data on this page through Mathematica: http://www.fundamentus.com.br/buscaavancada.php Then I would press the "buscar" button also through mathematica coding and get the ...
21
votes
2answers
2k views

Extracting values from nested rules in JSON data

I have been using Mathematica to analyse some data from the StackExchange API. It is conveniently available in JSON form, which Mathematica interprets as replacement rules. However, some of the rules ...
20
votes
2answers
405 views

Is Compress[] compatible between different Mathematica versions?

Compress[] was introduced in version 6 of Mathematica. It gives a compressed version of expressions represented as a string of ASCII characters. It preserves more ...
19
votes
2answers
2k views

How to import all files of a folder at once?

I know how to import one textfile by calling its name filestring = Import["myfile.tex", "Text"]; Then "filestring" is a string with the myfile content. How do ...
19
votes
1answer
258 views

Getting Mathematica.SE reputation data

I'm asking the following on behalf of Mr.Wizard. On StackOverflow I used the old v1.0 SO API and Mathematica to obtain user reputation data. Now we are no longer a tag on StackOverflow and the v2.1 ...
19
votes
2answers
702 views

Why does ImageData need four times more memory?

Background We are dealing with very large TIFF image files that are imported, processed and exported using Mathematica 9.0.1. Many of our algorithms only work with an array representation of the ...
18
votes
2answers
576 views

How to develop an Import/Export converter for Compress[]ed data?

Mathematica has a cross platform data exchange format, WDX. Unfortunately importing or exporting large data from/to WDX is very slow. Using MX files is very fast, but they are not compatible across ...
18
votes
1answer
558 views

When importing GIF animation, how to find the correct list of “DisplayDurations”?

When importing GIF animations with variable frame durations, the settings for "DisplayDurations" are not recovered correctly. As an example, I'm using the movie from this earlier question: Here I'm ...
18
votes
4answers
793 views

How to efficiently import videos

I need to do frame-by-frame analysis of videos. I can import the video files I need, but each file is too large to import at once. I can use the ...
18
votes
1answer
1k views

Web Browsing Automation with Mathematica

I want to build a bot to automate web browsing, this mean something like: filling forms press "submit" buttons find certain text inside pages and so on... How can I do this with Mathematica? The ...
16
votes
3answers
1k views

Is it possible to use Mathematica to record sound and/or vision?

I think the title says it all. I know about ImageCapture which seems to capture stills. Can ImageCapture be made to capture a ...
16
votes
1answer
518 views

How to programmatically extract mathematical formulas from Word with Mathematica 9?

I have a bunch of(1000+) Microsoft Word document in .Docx format. How can I programmatically extract the mathematical formulas from MS Word using Mathematica 9? This is what example looks like (or ...
16
votes
1answer
827 views

Slow import of multigigabyte TIF image stacks

Update 6: After contacting Wolfram Technical Support, I was advised that the frame-to-frame import timing variability can be eliminated by reading in the frame stack from frame to frame with the ...
15
votes
4answers
3k views

Cut and paste data from a spreadsheet

I have a set of data points in two columns in a spreadsheet (OpenOffice Calc): I would like to get these into Mathematica in this format: ...
15
votes
1answer
305 views

Why does ImportString[“1c”, “Table”] eat the letter c?

I have some whitespace-separated matrix data that I read with Import[..., "Table"]. The data contained mixed strings and numbers (the strings are for row and ...
15
votes
1answer
396 views

Importing Zip files

Mathematica can import zipped files automatically, but when the content of the zip file it's another zipped file, I just get the list of files in the second zip: ...
15
votes
1answer
447 views

Gravatars/identicons with Mathematica

The default avatar/identicon is based on an MD5 hash of your e-mail address. With some e-mail providers (e.g., GMail) the address gmailuser+foo@gmail.com is still ...
14
votes
2answers
356 views

How to load thumbnail, not image?

Is there any way to import only the thumbnail of an image instead of the full image? I frequently work with lots of images and it would be nice if the contents of a directory could be quickly ...
14
votes
1answer
793 views

Release the web camera after using CurrentImage[]

CurrentImage[] will capture an image from the webcam. It is even possible to create a live video in the notebook using ...
14
votes
2answers
2k views

Importing a large Excel file

I'm trying to import a simple Excel file containing 94,000 rows and 52 columns of various types, mostly Reals. It's a 38-MB file. I get this super helpful error ...
14
votes
3answers
464 views

How to Import random elements of huge data files

I am calculating huge data files with an external program. I would like then to import the data into Mathematica for analysis. The files are 2 columns and up to many millions of rows. So for ...
14
votes
2answers
803 views

Dealing with large files

I have a *.dat file in which the contents are basically one giant list: {1,2,3,4} {5,6,7,8} {9,10,11,12} and so on. I'd like to read in the first 500 members { ...
13
votes
3answers
893 views

How to scrape the headlines from New York Times and Wall Street Journal?

How to scrape headlines from New York Times, Wall Street Journal main pages to create datasets similar to this service? Importing HTML from nyt.com (HTML4) results in a ...
13
votes
3answers
5k views

Plot data from txt file?

I have data in file.txt which can be expressed by 20000x11 matrix. I do not want to import all of these data. How can I plot graph of the column 1 & column 4?
13
votes
2answers
509 views

How to monitor the progress of Read?

I use f = OpenRead["largefile.m"] x = Read[f]; Close[f]; to read in a Mathematica expression from a large file largefile.m ...
13
votes
2answers
424 views

Problem importing URL with Greek characters

I'm considering buying a car. So I thought why not make a web-crawler in Mathematica to pile-up car data? Brilliant idea. Then I found this Greek website, gocar.gr, which just so happens to have all ...
12
votes
6answers
2k views

How do you skip missing data values in a data file?

In scanning through a simple imported file of weather station data, some dates have missing data. What is a test to use to check whether a datum is valid? I tried ...
12
votes
3answers
3k views

Excel Headers --> Mathematica

I have a large Excel .xls file. Miniature version shown below: (formatting should appear as two columns...) ...
12
votes
2answers
492 views

Parsing a CSV file using Interpreter

The new Interpreter functionality in version 10 looks like it has the potential to make parsing custom data formats very easy. I'm trying to create a CSV parser. ...
12
votes
1answer
257 views

Exporting and reimporting a list of lists changes the dimensions

Could anyone explain to me what in the world is going on here and please help me in reading in dummy.dat properly? ...
11
votes
2answers
664 views

How to build a bvh player in Mathematica?

How to build a bvh player that can be used to visualize human motions in Mathematica? Any idea or sample code would be appreciated. ...
11
votes
3answers
929 views

How can I transfer mathematical expressions between Mathematica and Maple?

Is there a simple way to copy mathematical expressions between Mathematica and Maple (or at least in one direction)? I mean only expressions built from numbers and predefined mathematical functions, ...
11
votes
3answers
415 views

Data of badly formatted file

Similar as in this question, I have to read in data from a file (exampleFile.txt) which is not so well formatted. Unfortunately, I cannot find a solution by myself. Since it is a bit complicated to ...
11
votes
4answers
1k views

Import[ ] drops leading zeros from data when importing a CSV file

I need to load data from the CSV file shown below. Import[ ] drops the leading zeros in column 3 when it imports the CSV file, but they are security identifiers, and I need them. ReadList[ ] ...
11
votes
2answers
744 views

Maximum size of TIF files in Mathematica

I have a question concerning the maximum size of TIFF files Mathematica can handle. According to the TIFF 6.0 specification -revision 6.0 (see: ...
11
votes
3answers
235 views

ToExpression fails with long nested string expressions

Consider the following conversion from String to Mathematica Expression: ...
11
votes
1answer
501 views

Get both output and return code of external program

Run["program"] will run an external program and return the return code. Import["!program", "String"] will run the program and ...
10
votes
2answers
230 views

How to extract a specific element from EXIF photographic metadata?

It's now possible to import photographs into Mathematica and import the EXIF data at the same time: i = Import["ExampleData/coneflower.jpg", "ImageWithExif"] ...
10
votes
2answers
3k views

How to get Matlab data imported with the same dimensions?

I have some Matlab image data with the following dimensions (output is from Matlab): >> size(im) ans = 86 86 3 45 The data imports just fine ...
10
votes
1answer
327 views

Possible to import Excel formatted text?

Is it possible to import Excel text markup such as underlines? For example: Yet ...
10
votes
1answer
492 views

How to import date (YYYY-MM-DD) data from a file?

I have a file with data in form {date, value}: 2010-05-19 17 2010-05-20 20 ...