Efficiently Visualising Very Large Data Sets (without running out of memory)

I have put a few really hard problems in combinatorics up against Mathematica 8. I'd have to say that it works really well, until you want to view the data. If you look at my question Advanced Tupling there is an elegant solution. One line function. Data of about 12 million sets given the sample data presented.

Let's use that question to illustrate. Here are the data:

a = {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 17, 18, 21, 22, 23, 25, 26, 28};
b = {2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 25, 26, 27, 28, 30, 31, 33, 37, 41};
c = {6, 10, 13, 14, 15, 16, 18, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 32, 35, 37, 39};
d = {17, 19, 25, 30, 31, 33, 34, 35, 36, 38, 44};
e = {31, 41, 45, 47};
f = {23, 26, 31, 32, 33, 34, 35, 36, 38, 39, 40, 41, 42, 43, 45, 46, 47, 48, 49, 50, 51, 52, 53};


And here is one of the solutions posted in that question:

newTuples[t_, x_] :=
Flatten[ ParallelTable[Append[s, #] & /@ Complement[x, s], {s, t}], 1];


One would however like to verify the results. From everything seen, it looks good. But one wants to see it all. Click 'Show All' output and the program grabs all available memory resources, including consuming all available disk space for page file and still takes literally days to complete. Is there an efficient way to render data?

I have had the same problem with graphical plots, it just chokes up after processing. It is as if the entire problem has to be reprocessed for it to render. Surely this cannot be how Mathematica was intended to work. I must be missing something.

What are some memory-efficient ways to visualize a data set as large as the output of the above function?

-
@Sinistar It seems that you are asking how to visualize the data from a separate question on this site. If so, could you update the question here to include the original data set so that users could more easily play around with it? –  tkott Mar 6 '12 at 15:10
It "gags" because you have about 2GB of results and are trying to render them in the front end. I suppose I don't see how you hope to gain anything by seeing the full output. Try a small problem and verify that it works. Look at a subset of individual results. Check that the dimensions are what you expect etc.. –  Andy Ross Mar 6 '12 at 15:33
But your output isn't packed and ByteCount[ts] verifies the size. If you apply DeveloperToPackedArray@ts it will consume far less memory. –  Andy Ross Mar 6 '12 at 15:38
I would suggest editing your question so that is specifically addresses what you are hoping to accomplish with this data. Maybe responses will give you a sense of how to handle similar issues in the future. –  Andy Ross Mar 6 '12 at 15:41
Sinistar, could you please come up with a more descriptive title for your question? This will help future users searching for an answer to similar questions. –  Sjoerd C. de Vries Mar 6 '12 at 20:57

Showing humongous data by screen-fulls can be done using Manipulateas follows:

(* generate some data to show *)
res = Tuples[{1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}, 7];

(* size of the data*)
ByteCount[res]

(* ==> 1003290792 *)

screenNumbers = 100;
Manipulate[
Take[res, {i screenNumbers + 1, (i + 1) screenNumbers}], {i, 0,
Length[res]/screenNumbers - 1, 1}
]


Since the slider slides over 100,000's of values in a short distance you can slow down its movement using the modifier keys when you drag the slider.

The resulting slider can be finely manipulated by holding down the Alt key (or Option on Macintosh) while dragging the mouse. This causes the slider to move at 1/20 the rate of the mouse. The slider can be even more finely manipulated by also holding the Shift and/or Ctrl keys. [Last bullet of the More Information part of the Slider doc page]

Scrolling through the 1GB of data is almost instantaneous.

-
Cool! I have been wanting to work more with the Manipulate object for a while now for other reasons. This is very nice solution to the rendering problem! –  Sinistar Mar 7 '12 at 17:44

The difficulty lies in the rendering process used by the Front-end. If you have a large data-set, then it is entirely possible to lock up your machine when you try to show it all. While a visual inspection of such data sets would be nice, it is likely that it would be useless as you could unintentionally skip over data that is incorrect. I would suggest an automated approach instead.

The simplest test is to determine if each element of the tuple is a member of the correct distribution. To do that simply,

Select[data,
Function[{f},
! And @@ MapIndexed[
With[{dist = #, idx = #2}, MemberQ[dist, f[[ idx[[1]] ]]] ] &,
{a, b, c, d, e, f}]]
]


which will return any data that does not conform. If you just wish to strip out any such data, instead, just remove the ! in the above function.

-

I deal with lots of text as well, and this is going to be a serious problem. In short, the comments are correct that you probably won't be able to just throw 2GB of information up there on your screen. One function that a person mentoring me in Mathematica provided was this:

viewData[x_] :=
Framed[Pane[x, {Automatic, 200}, Scrollbars -> True]]


You could then ViewData[x] where x is the information you're trying to visualize. You still won't be able to view 2GB of data all at once, but this might give you an ability to view a bit more of it for whatever reason.

There have been some discussions over at Stackoverflow on the related problem of viewing large files, if you were to export it textually, including some text editor suggestions (here and here).

-
Yes I use Ultraedit. It has been able to handle about anything I throw at it. But, I have to get it there first. If I try to write the contents of the variable to disk will it still hang and take three days? I suspect not because I was able to translate an algorithm from PDP11 to M! and write out a 500+ MB text file, but it did one result at a time. The alg was so ingenious it didnt need to use any memory other than a couple registers in the orginal PDP11. –  Sinistar Mar 6 '12 at 15:52
@Sinistar as I point out in my answer, it is the display process itself that takes the time. Dumping your data to disk won't take anywhere near as long. –  rcollyer Mar 6 '12 at 15:53
This doesn't really work. Try viewData[Tuples[{1, 2, 3, 4, 5, 6, 7, 8}, 5]] for instance. –  Sjoerd C. de Vries Mar 6 '12 at 20:44

The solution to this is after processing, write a text stream to a text file, open it with a text editor capable of handling large files. I used UltraEdit. This example created a ~319MB text file.

fname = ["C:/YourDir/Yourfile.txt"];
stream = Openwrite[fname];
Write[stream, ts];
Close [stream];
`

This took about the same time to write out as the original tupling solution took to process.

I think the problem is kind of like when Windows 95 was released. Sure you could play DOOM 2 in windowed mode, but the lag was atrocious and the frame rate was unacceptable. Pop back into full screen mode and voila you were good!

So, it appears to be a similar problem with mma that it performs some kind of a formatted rendering that could be dolled up with proper adjectives on the commands. Would be nice just to have raw unformatted, good old fashioned text output as an option, but I'm not aware of any such facility in mma.

-