Linked Questions

4
votes
0answers
265 views

How to speed up import of large csv table? [duplicate]

Loading less than 2 gB of csv data took 5.29 hours: ...
35
votes
4answers
5k views

Looking for “Longest Common Substring” solution

I'm looking for robust code to solve the "Longest Common Substring" problem: Find the longest string (or strings) that is a substring (or are substrings) of two or more strings. I can just code it ...
36
votes
5answers
8k views

How can I import a huge CSV file quickly?

I have two CSV files, each one is around 1 GB of data. When I use Import["file.csv"], it takes a very, very, long time to import the data. So how can I accelerate ...
27
votes
3answers
3k views

Reading periodic elements from a large file

I have a large binary data file (big endian) with 100+ million "rows" of 11 elements, combination of floats and integers. This is the format: ...
12
votes
2answers
11k views

When importing is it possible to set the delimiter?

When importing a csv file is there a way to set the delimiter. I have a csv that uses ";" as its delimiter. I tried ...
9
votes
3answers
4k views

Histogram of a very large dataset?

I have a very large dataset stored in a file (over 2GB). The file contains a tab-separated table of floating-point numbers. I want to do an Histogram of all the numbers in the table. But if I try to ...
11
votes
2answers
2k views

Stream CSV or TSV files

RAM limitations make import impractical for large CSV files. Is it possible to Stream CSV correctly? str = OpenRead["EU.csv"]; Read[str, "CSV"] Read::readn: ...
11
votes
2answers
3k views

Export to CSV is very slow - can performance be improved?

I have to provide data exports from Mathematica to CSV format and it is very slow compared to other formats as can be seen in the following table: JSON, 15 sec, 8.4 MB TXT, 23 sec, 5.6 MB XLSX, 37 ...
5
votes
2answers
1k views

ReadList and Import cannot read data separated by “,”

I have a data file contains several lines of numbers that separated by "," and I cannot load it correctly with ReadList and Import. Here is a example: with data contains "," and "\n", simple ...
7
votes
1answer
1k views

Reading specific entries from a CSV file

This question is related to the following: Speeding up Import and Export in CSV format Stream CSV or TSV files Reading periodic elements from a large file I have a set of very large CSV files (a ...
6
votes
1answer
1k views

Efficiently reading CSV with ReadList

I have a large CSV file (~1gb) with mixed data. Basically with strings and integers. It takes very long to import this whole dataset with Import[] so I would like ...
2
votes
2answers
2k views

Export complex number to excel or txt file

I'm doing some calculations on complex numbers in Mathematica. I can use Export to generate a file (*.CSV or *.dat, or anything else) of complex numbers. If I want to use excel, origin or Matlab to ...
7
votes
1answer
533 views

Working with a CSV data file with 100 millions rows (and 30 columns)

I am new to working with large datasets.. Currently I am struggling to upload a "big data" file into Mathematica. The file is in the CSV format and has 14GB. There are 100 millions of rows and 30 ...
5
votes
0answers
3k views

Import Large Excel File

Currently, I am working on a large time series dataset, which contains 928,586 rows and 35 time intervals in columns, in excel format (.xlsx) and using Import to ...
5
votes
1answer
312 views

Huge CSV file with numeric and text data

I try to load a huge CSV file (~ 4 Gbyte), the data is both numeric and textual. So, I can't load the data as a string. ...

15 30 50 per page