Questions about getting data, files, images etc. from various formats into a form that can be read, tranformed and manipulated by Mathematica

learn more… | top users | synonyms

8
votes
1answer
762 views

How to properly DumpSave & Get Global`s symbols inside packages while not touching Global context?

For efficiency reasons I prefer to use DumpSave instead of Save. For ease of access I prefer to save symbols in ...
6
votes
2answers
322 views

Importing multiple images for machine learning classification

I want to create a ML classifier using images just like this example. I have two classes of images in two different folders. Is there an easier way to import all images from both folders rather than ...
5
votes
2answers
376 views

Sorting a large file

I have some huge logs, which are text files, containing up to a million rows each, and I need them sorted. Because trying to import one obviously makes Mathematica run out of memory, we must rule out ...
15
votes
3answers
652 views

How to Import random elements of huge data files

I am calculating huge data files with an external program. I would like then to import the data into Mathematica for analysis. The files are 2 columns and up to many millions of rows. So for ...
9
votes
2answers
462 views

Convert quoted printable aka PrintableASCII characters to UTF-8

I have several .eml email files to import. Some of them were saved as quoted-printable 7-bit encoded. I skipped through the Mathematica documentation and found ...
6
votes
1answer
559 views

Accessing data on local machine from remote kernel

I have a remote kernel on a server that I would like to use to process some data. The connection works, i.e, I'm able to use the local front end and the remote kernel. However, I cannot figure out how ...
6
votes
1answer
1k views

When do I use Mathematica, when do I use Gephi, and how do I use them together?

Mathematica has great operations for analyzing graphs but when the graphs are a bit bigger, Gephi seems to be more efficient (at least on my computer) at drawing and analyzing, i.e. visualizing the ...
6
votes
1answer
438 views

Operations on online files via public URL access

I need Mathematica to access an online public directory via some URL (fake in this case) http://example.com/images and perform operations similar to those Mathematica can do on OS directories and ...
5
votes
1answer
279 views

How to transform a DumpSaved .mx file to .m file conveniently?

I have many DumpSave-d .mx files under a directory. They are generated under Linux. The .mx ...
5
votes
1answer
957 views

Efficiently exporting many images as a video

I have many .gif images and I would like to export them as a single video file (I prefer .mov, but any video format is fine). But what if I have several tens of megabytes of .gif images? Is it ...
3
votes
2answers
101 views

Most efficient way of splitting a file into length-prefixed blocks

Let's say I have a really big file (>100MB) that consists of length-prefixed blocks, i.e. the first byte of each byte block indicates the length of the rest of the block. This is a simplified example ...
13
votes
4answers
406 views

Unencoding encoded URLs in a function

Is there a built-in function for decoding encoded URLs in Mathematica? I would like to convert an encoded URL, for example ...
12
votes
2answers
726 views

Parsing a CSV file using Interpreter

The new Interpreter functionality in version 10 looks like it has the potential to make parsing custom data formats very easy. I'm trying to create a CSV parser. ...
10
votes
1answer
673 views

How to import date (YYYY-MM-DD) data from a file?

I have a file with data in form {date, value}: 2010-05-19 17 2010-05-20 20 ...
9
votes
2answers
1k views

How can I animate a set of points imported from file?

I have an external program which generates a set of data - X points at the time t1, X points at the time t2 and so on. These points are stored in CSV file and I can change the format if necessary. ...
8
votes
1answer
122 views

Is it possible to move .mx file from one server to other?

I have mathematica file in one server along with .mx file. Is there any way to move the .mx file from one server to other such ...
8
votes
1answer
158 views

Organizing similar datasets using Contexts or otherwise

I often import several long time series to analyze something about a city. When I have multiple cities, I use contexts. As a simple example, ...
7
votes
1answer
955 views

Import file from FTP site

I would like to Import files from an FTP site. The code below works on a test site: file = Import["ftp://test:test@ftp.secureftp-test.com/pigs.xml"]; But when I ...
6
votes
1answer
244 views

How to use Get on a folder

In the documentation for Get it says: If the file found by <<name is a directory, the Wolfram Language will try to ...
6
votes
1answer
3k views

Reading csv/xlsx data into Mathematica with column headings as list names

It's old question. However, I could not find satisfactory answer. Decided to post this question. I have a .csv file with column heading. How can I import this ...
4
votes
2answers
214 views

How can I encrypt/decrypt a directory?

I'd like to use Mathematica 10's new Encrypt and Decrypt functions to secure files and folders on my mac. Here's what I have so far but something goes wrong along the way, since by the end of this ...
4
votes
1answer
260 views

mathematica use too much memory to Import large dat file

Err... When I try to Import a .dat file by dat = Import["D:\\TIEGCM\\solar_input\\omni_m_all_years.dat", "Table"]; or ...
4
votes
1answer
201 views

Importing malformed XML: Import[… “XMLObject”] vs ImportString[…, “XML”]

It appears that Import[ url , "XMLObject" ] is more forgiving of improperly formatted XML than ImportString[ string, "XML"]. For ...
4
votes
3answers
3k views

Import specific data from Excel to make a plot

I need some help with this: I have imported data from an Excel file and I need to make a contour plot of it. The data are in table-like form. Here is a sample of the first few 'rows': ...
3
votes
2answers
142 views

Importing CSV with multiline text fields

Here is a valid CSV: a1,b1,c1,d1,"e1 e1 e1" a2,b2,"c2 c2 c2",d2,e2 It has two rows and five columns, and contains two fields with embedded CRLF. (See the CSV ...
3
votes
1answer
420 views

how to delete specific data from text file and then update it

I have a large text file. I read data from text file one at a time and check whether data agree with my condition. If not I want to delete that data in text file and update it. For example say my ...
3
votes
1answer
514 views

Mathematica - set end-of-line characters when importing

I have a plain text file of tab-separated values that was created on a Windows machine. I want to import it into Mathematica on a Mac. ...
1
vote
2answers
1k views

Can I improve the importing speed of Excel data?

I have data in an Excel spreadsheet that has 4 rows and 1,000 columns. I am using Import as follows: ...
9
votes
2answers
289 views

Working with Java hashmaps from Mathematica

I'm using J/Link to access data from an external Java program. The data are represented as a ConcurrentHashMap (see the official specification here). ...
8
votes
1answer
176 views

Reading a HDF file im Mathematica

Bug introduced in 8.0 or earlier and persisting though 10.2 On OS X, the problem is present in 9.0 but not in 10.0, 10.1 or 10.2. I am trying to read a HDF file in Mathematica 9 on OS X and get ...
7
votes
1answer
252 views

Incorrect result while using SemanticImport for CSV file import

I might use this incorrectly. I want to use a .csv file that (as list) looks like: ...
7
votes
2answers
995 views

Why does importing a URL fail the second time and produce a FetchURL::conopen error?

I have Mathematica code that uses Import[] to download URLs. I noticed that first time I use the Import[] command it works. ...
7
votes
3answers
243 views

How can I read in specific information from a TeX-file?

I have several TeX-files and they contain information of the form ...
6
votes
1answer
395 views

How to correctly import data zipped with the “deflate” algorithm?

I'm writing an importer for the GIFTI file format. The details of the format are not particularly important, but the basic idea is that it is a relatively simple XML file which includes binary arrays ...
6
votes
1answer
174 views

Can I read Chinese characters with ReadList correctly?

Just try this example: Export["a.txt", "这乱码问题该怎么解决呢\n***\n怎么解决嘛"] ReadList["a.txt", Record] ...
6
votes
3answers
201 views

Is it possible to Import urls not type “”

These days I often copy urls from Internet Explorer's address bar. When I paste the url into a Mathematica cell it is neither a string nor a expression. ...
6
votes
1answer
297 views

Importing images from the web using ImportString & URLFetch

I can bring images from the web using the Import command like this: Import["http://blog.wolfram.com/wp-content/uploads/2008/06/se-30.jpg"] Sometimes the web ...
4
votes
1answer
306 views

Importing FITS file from HEALPix

I am trying to read FITS file created by HEALPix software. The file contains a spherical map of a scalar field. There is the online support on how to use the file in IDL, but not for Mathematica. I ...
4
votes
2answers
175 views

Workaround for Import not supporting the \leqslant TeX macro

$\{A\leqslant x\leqslant B\}$ Copy the code into Notebook, and you'll lose the less sign, this can be rendered well in ...
3
votes
2answers
220 views

Why are dictionary keys reordered upon JSON import?

After importing a JSON file, Mathematica changed the order of the fields. For example, I downloaded a sample file from: http://api.openweathermap.org/data/2.5/find?lat=55.5&lon=37.5&cnt=10 I ...
3
votes
2answers
294 views

Speeding up TIF import

After trying to import a $\approx 2$ GB TIF file with the Import command, I noticed that the process takes $\approx 15$ minutes or so, which seems excessive, especially when compared to other software ...
3
votes
2answers
1k views

3D scatter plot from a txt file

I have a txt file with space-delimited x, y, z coordinates of several points. The points are organized into groups of unequal size; the groups are delimited by skipping a line, like this: ...
3
votes
2answers
463 views

Extracting a single layer from a SHP file

I have an ESRI ShapeFile (SHP) that I've loaded. It has a number of layers which I can see the names of by using: Import["file.shp", "LayerNames"] I get ...
3
votes
1answer
727 views

Importing data sets from plain-text files and performing operations on them

I am working with a lot of daily weather files in the txt format. I need to calculate a daily quantity from each of them, e.g., daily light intensity, which ...
2
votes
1answer
59 views

Import tabular text data

I have a text file created with a bash script I wanted to read the data avoiding the first line. I have written few dirty codes, but I wondered if anybody can do it in fewest possible nice one ...
2
votes
1answer
570 views

How to work with 3D images of STL format

After importing a 3D image of STL format, my possibilities in regard to "3D Volumetric Image Processing" seems rather limited. All examples are given with TIFF files. Is there a way to convert from ...
2
votes
1answer
269 views

importing large data files with a constraint

As I have learned in this community, one way to import a hefty data files (e.g. millions of rows of {x1, y1} data) if you don't care about the individual elements (maybe just want to obtain some ...
0
votes
1answer
82 views

Package to read in data from excel files

I want make a package to add a function to Mathematica that reads all excel files is the working directory that is set after the package is loaded. ...
0
votes
1answer
73 views

SocialMediaData[“Twitter”] - strange output

I am getting error messages when I use SocialMediaData["Twitter"]: ...
0
votes
1answer
69 views

Elegant ways of solving this file batching task

There are a batch of data files in a folder (dir = "D:\\Documents\\datadir"), each of which consists of a list of data items, one per line. Each data item is a pair ...