Linked Questions

35
votes
4answers
5k views

Looking for “Longest Common Substring” solution

I'm looking for robust code to solve the "Longest Common Substring" problem: Find the longest string (or strings) that is a substring (or are substrings) of two or more strings. I can just code it ...
33
votes
3answers
5k views

How to read data file quickly?

I have a tab separated value file with 10 million rows each of which has three tab separated values. The first value is a string, the second an integer, and the third another string. How to read ...
19
votes
2answers
6k views

Speeding up Import and Export in CSV format

I am handling large numerical data in Mathematica. In smaller problems everything worked fine using Export and Import with the ...
11
votes
3answers
14k views

Import selected columns from a tab delimited text file

I have a very huge file (1GB) from which I desire to import only the 5th column. It is a tab delimited text file with a .txt extension. Can you please help me with ...
19
votes
2answers
5k views

How do you deal with very large datasets in Mathematica?

This question is in two related parts. The first is about dataset size, the second data wrangling. Part 1: I've built a naive Bayesian classifier on Mathematica. It uses ...
14
votes
5answers
2k views

Using ReadList and Skip to read a file with periodic alternation of useful and useless data

I am trying a read a file that's arranged such that there's alternating set of rows containing data that I don't need and data that I need. So for my case the first n lines in the files are useless to ...
15
votes
3answers
845 views

How to Import random elements of huge data files

I am calculating huge data files with an external program. I would like then to import the data into Mathematica for analysis. The files are 2 columns and up to many millions of rows. So for ...
3
votes
1answer
1k views

Read and filter a text file without importing

I have a large text file of a million sections. Each section contains A lines of header and B lines of numeric data. I want to read the first and last line of B from each section (to have a list of 2 ...
1
vote
2answers
677 views

Import large amount of data with time stamps

I have a large data file (150 000 lines) which I want to import in Mathematica. The first few lines look like this ...
2
votes
2answers
1k views

Importing several huge files with headers & footers

having here several files in ASCII format http://ge.tt/91rv0Jf/v/0 I want to import, while removing headers and footers and then join the several lists. ...
7
votes
1answer
1k views

Reading specific entries from a CSV file

This question is related to the following: Speeding up Import and Export in CSV format Stream CSV or TSV files Reading periodic elements from a large file I have a set of very large CSV files (a ...
6
votes
1answer
1k views

Reading in Huge Text Files

I would like to read in really large text files (about 2GB) into Mathematica. The structure of the file is such that the first row and the first column are text i.e strings. Rest is all numbers. Is it ...
2
votes
1answer
335 views

importing large data files with a constraint

As I have learned in this community, one way to import a hefty data files (e.g. millions of rows of {x1, y1} data) if you don't care about the individual elements (maybe just want to obtain some ...
4
votes
1answer
286 views

Importing large datasets using streams

I work with large data sets with 22 columns and up to a million rows. Each row contains one observation with several variables. All variables are numbers. Simplyfied example: 1, 1, 2, 3, 4, 5, 6,...
0
votes
1answer
202 views

Importing large data without using Import [closed]

I have around 240 files with lots of data, I have been using Import[] to import these data for further processing. But as my simulations are growing bigger, I am ...

15 30 50 per page