Linked Questions
21 questions linked to/from Reading periodic elements from a large file
36
votes
4
answers
5k
views
Looking for "Longest Common Substring" solution
I'm looking for robust code to solve the "Longest Common Substring" problem:
Find the longest string (or strings) that is a substring (or are substrings) of two or more strings.
I can just code it ...
- 8,763
34
votes
3
answers
6k
views
How to read data file quickly?
I have a tab separated value file with 10 million rows each of which has three tab separated values. The first value is a string, the second an integer, and the third another string. How to read ...
- 8,674
20
votes
2
answers
8k
views
Speeding up Import and Export in CSV format
I am handling large numerical data in Mathematica. In smaller problems everything worked fine using Export and Import with the ...
- 323
11
votes
3
answers
17k
views
Import selected columns from a tab delimited text file
I have a very huge file (1GB) from which I desire to import only the 5th column. It is a tab delimited text file with a .txt extension. Can you please help me with ...
- 833
20
votes
2
answers
6k
views
How do you deal with very large datasets in Mathematica?
This question is in two related parts. The first is about dataset size, the second data wrangling.
Part 1:
I've built a naive Bayesian classifier on Mathematica. It uses ...
user4860
14
votes
5
answers
2k
views
Using ReadList and Skip to read a file with periodic alternation of useful and useless data
I am trying a read a file that's arranged such that there's alternating set of rows containing data that I don't need and data that I need. So for my case the first n lines in the files are useless to ...
- 33.3k
15
votes
2
answers
2k
views
Stream reading 1000 times faster in Python
I have a 26GB CSV. I want to extract all the rows that have data for a specific sensor. In Mathematica I write:
...
15
votes
3
answers
990
views
How to Import random elements of huge data files
I am calculating huge data files with an external program. I would like then to import the data into Mathematica for analysis. The files are 2 columns and up to many millions of rows.
So for ...
- 2,887
9
votes
1
answer
2k
views
Reading specific entries from a CSV file
This question is related to the following:
Speeding up Import and Export in CSV format
Stream CSV or TSV files
Reading periodic elements from a large file
I have a set of very large CSV files (a ...
- 1,867
3
votes
1
answer
1k
views
Read and filter a text file without importing
I have a large text file of a million sections. Each section contains A lines of header and B lines of numeric data.
I want to read the first and last line of B from each section (to have a list of 2 ...
- 169
1
vote
2
answers
706
views
Import large amount of data with time stamps
I have a large data file (150 000 lines) which I want to import in Mathematica. The first few lines look like this
...
- 399
2
votes
2
answers
2k
views
Importing several huge files with headers & footers
having here several files in ASCII format
http://ge.tt/91rv0Jf/v/0
I want to import, while removing headers and footers and then join the several lists.
...
- 247
6
votes
1
answer
1k
views
Reading in Huge Text Files
I would like to read in really large text files (about 2GB) into Mathematica. The structure of the file is such that the first row and the first column are text i.e strings. Rest is all numbers. Is it ...
- 833
3
votes
2
answers
485
views
Import columns from a long .txt file
I have to import columns 1,2,3, and 6 from a long (2 * 10^5 rows) file with this structure:
...
- 6,837
2
votes
1
answer
360
views
importing large data files with a constraint
As I have learned in this community, one way to import a hefty data files (e.g. millions of rows of {x1, y1} data) if you don't care about the individual elements (maybe just want to obtain some ...
- 2,887