Linked Questions
19 questions linked to/from Reading periodic elements from a large file
35
votes
4answers
5k views
Looking for “Longest Common Substring” solution
I'm looking for robust code to solve the "Longest Common Substring" problem:
Find the longest string (or strings) that is a substring (or are substrings) of two or more strings.
I can just code it ...
33
votes
3answers
6k views
How to read data file quickly?
I have a tab separated value file with 10 million rows each of which has three tab separated values. The first value is a string, the second an integer, and the third another string. How to read ...
20
votes
2answers
7k views
Speeding up Import and Export in CSV format
I am handling large numerical data in Mathematica. In smaller problems everything worked fine using Export and Import with the ...
11
votes
3answers
15k views
Import selected columns from a tab delimited text file
I have a very huge file (1GB) from which I desire to import only the 5th column. It is a tab delimited text file with a .txt extension. Can you please help me with ...
19
votes
2answers
5k views
How do you deal with very large datasets in Mathematica?
This question is in two related parts. The first is about dataset size, the second data wrangling.
Part 1:
I've built a naive Bayesian classifier on Mathematica. It uses ...
14
votes
5answers
2k views
Using ReadList and Skip to read a file with periodic alternation of useful and useless data
I am trying a read a file that's arranged such that there's alternating set of rows containing data that I don't need and data that I need. So for my case the first n lines in the files are useless to ...
15
votes
3answers
868 views
How to Import random elements of huge data files
I am calculating huge data files with an external program. I would like then to import the data into Mathematica for analysis. The files are 2 columns and up to many millions of rows.
So for ...
3
votes
1answer
1k views
Read and filter a text file without importing
I have a large text file of a million sections. Each section contains A lines of header and B lines of numeric data.
I want to read the first and last line of B from each section (to have a list of 2 ...
1
vote
2answers
688 views
Import large amount of data with time stamps
I have a large data file (150 000 lines) which I want to import in Mathematica. The first few lines look like this
...
8
votes
1answer
1k views
Reading specific entries from a CSV file
This question is related to the following:
Speeding up Import and Export in CSV format
Stream CSV or TSV files
Reading periodic elements from a large file
I have a set of very large CSV files (a ...
2
votes
2answers
1k views
Importing several huge files with headers & footers
having here several files in ASCII format
http://ge.tt/91rv0Jf/v/0
I want to import, while removing headers and footers and then join the several lists.
...
6
votes
1answer
1k views
Reading in Huge Text Files
I would like to read in really large text files (about 2GB) into Mathematica. The structure of the file is such that the first row and the first column are text i.e strings. Rest is all numbers. Is it ...
2
votes
1answer
342 views
importing large data files with a constraint
As I have learned in this community, one way to import a hefty data files (e.g. millions of rows of {x1, y1} data) if you don't care about the individual elements (maybe just want to obtain some ...
4
votes
1answer
360 views
Importing large datasets using streams
I work with large data sets with 22 columns and up to a million rows. Each row contains one observation with several variables. All variables are numbers.
Simplyfied example:
1, 1, 2, 3, 4, 5, 6,...
3
votes
2answers
170 views
Import columns from a long .txt file
I have to import columns 1,2,3, and 6 from a long (2 * 10^5 rows) file with this structure:
...