144 reputation
9
bio website
location
age
visits member for 1 year, 8 months
seen Jul 25 at 23:27

Jul
18
revised Using cursors with result sets to reduce memory usage
added 84 characters in body
Jul
18
awarded  Promoter
Jul
18
revised Using cursors with result sets to reduce memory usage
Light editing, clarified reasons for disliking SQLExecute approach.
Jul
16
asked Using cursors with result sets to reduce memory usage
May
27
awarded  Popular Question
Dec
14
awarded  Popular Question
Sep
14
awarded  Nice Question
Feb
4
accepted How do you force a decimal output?
Dec
12
revised How do you force a decimal output?
added 151 characters in body
Dec
12
comment How do you force a decimal output?
NumberForm[2.601519253458693*10^8, 30] Still results in scientific notation. Annoyingly. AccountingForm seems to be the answer to my question. AccountingForm[2.601519253458693*10^-8, 30] gives a nice decimal format I can use.
Dec
12
asked How do you force a decimal output?
Dec
12
accepted How do you deal with very large datasets in Mathematica?
Dec
11
revised How do you deal with very large datasets in Mathematica?
added 1221 characters in body
Dec
11
revised How do you deal with very large datasets in Mathematica?
added 467 characters in body
Dec
11
comment How do you deal with very large datasets in Mathematica?
Well, I know which columns are needed or not needed ahead of time. I get those when I'm training my classifier. But you still have to load a file or a row before discarding it in mathematica. So I could write some code to go through, read a column I want, skip the next few values, and read the next column I want. But that seems like the wrong way to do it. Particularly next to just taking the pain of some wasted memory, and doing it in chunks of 5K rows.
Dec
10
comment How do you deal with very large datasets in Mathematica?
This stream approach looks very promising. Is there any way to read a line of a CSV file in as a list? If so, I can hack up something that'll work quite easily. I see RecordSeparators, but apparently those are three fixed CR/LF Types. It also gets confused by CSV field quoting.
Dec
10
comment How do you deal with very large datasets in Mathematica?
mark: Sorry, finance style. ->2mm=2 million. Joel. You're probably right, I'll think about the phrasing and post a separate question.
Dec
10
comment How do you deal with very large datasets in Mathematica?
I can throw out unused columns, but have to load up the entire row first. Which is where my trouble lies. I think my best approach is just to do predictions in chunks of say 5000 rows at a time. Which will easily fit in memory.
Dec
10
comment How do you deal with very large datasets in Mathematica?
After looking at some of the work involved, there seems to be two options that don't radically increase complexity: 1. Buy more memory. 2. Spit my prediction file into chunks to make do for now, then rewrite in Clojure later. (Clojure is a similar language to mma for my purposes.). I've still no clue how to deal with the varying/shifted fields in otherwise similar datasets.
Dec
10
asked How do you deal with very large datasets in Mathematica?