Linked Questions
16 questions linked to/from Speeding up Import and Export in CSV format
4
votes
0
answers
273
views
How to speed up import of large csv table? [duplicate]
Loading less than 2 gB of csv data took 5.29 hours:
...
- 31.8k
36
votes
4
answers
5k
views
Looking for "Longest Common Substring" solution
I'm looking for robust code to solve the "Longest Common Substring" problem:
Find the longest string (or strings) that is a substring (or are substrings) of two or more strings.
I can just code it ...
- 8,763
39
votes
5
answers
11k
views
How can I import a huge CSV file quickly?
I have two CSV files, each one is around 1 GB of data. When I use Import["file.csv"], it takes a very, very, long time to import the data. So how can I accelerate ...
- 1,582
18
votes
2
answers
18k
views
When importing is it possible to set the delimiter?
When importing a csv file is there a way to set the delimiter. I have a csv that uses ";" as its delimiter.
I tried
...
- 181
28
votes
3
answers
3k
views
Reading periodic elements from a large file
I have a large binary data file (big endian) with 100+ million "rows" of 11 elements, combination of floats and integers.
This is the format:
...
- 9,692
13
votes
3
answers
8k
views
Histogram of a very large dataset?
I have a very large dataset stored in a file (over 2GB). The file contains a tab-separated table of floating-point numbers.
I want to do an Histogram of all the numbers in the table. But if I try to
...
- 11.5k
12
votes
2
answers
2k
views
Stream CSV or TSV files
RAM limitations make import impractical for large CSV files. Is it possible to Stream CSV correctly?
str = OpenRead["EU.csv"];
Read[str, "CSV"]
Read::readn: ...
- 1,211
11
votes
2
answers
4k
views
Export to CSV is very slow - can performance be improved?
I have to provide data exports from Mathematica to CSV format and it is very slow compared to other formats as can be seen in the following table:
JSON, 15 sec, 8.4 MB
TXT, 23 sec, 5.6 MB
XLSX, 37 ...
- 717
7
votes
1
answer
7k
views
Working with a CSV data file with 100 millions rows (and 30 columns)
I am new to working with large datasets.. Currently I am struggling to upload a "big data" file into Mathematica. The file is in the CSV format and has 14GB. There are 100 millions of rows and 30 ...
- 671
5
votes
2
answers
2k
views
ReadList and Import cannot read data separated by ","
I have a data file contains several lines of numbers that separated by "," and I cannot load it correctly with ReadList and Import.
Here is a example:
with data contains "," and "\n", simple ...
- 27.8k
9
votes
1
answer
2k
views
Reading specific entries from a CSV file
This question is related to the following:
Speeding up Import and Export in CSV format
Stream CSV or TSV files
Reading periodic elements from a large file
I have a set of very large CSV files (a ...
- 1,867
6
votes
1
answer
1k
views
Efficiently reading CSV with ReadList
I have a large CSV file (~1gb) with mixed data. Basically with strings and integers. It takes very long to import this whole dataset with Import[] so I would like ...
- 680
2
votes
2
answers
3k
views
Export complex number to excel or txt file
I'm doing some calculations on complex numbers in Mathematica. I can use Export to generate a file (*.CSV or *.dat, or anything else) of complex numbers. If I want to use excel, origin or Matlab to ...
- 4,855
5
votes
1
answer
504
views
Huge CSV file with numeric and text data
I try to load a huge CSV file (~ 4 Gbyte), the data is both numeric and textual.
So, I can't load the data as a string.
...
- 2,019
5
votes
0
answers
3k
views
Import Large Excel File
Currently, I am working on a large time series dataset, which contains 928,586 rows and 35 time intervals in columns, in excel format (.xlsx) and using Import to ...
- 467