# Importing data multiple times from the same file

I have a large data file (.txt) with bivariate data in the following form

x1,y1
x2,y2
x3,y3
.
.
.
xn,yn
---------F 0
x1, y1
x2, y2
.
.
.
xn, yn
--------F 1
.
.
.



and so on.

What I want to do is import the X, Y data into Mathematica up to the first "------- F " and plot a Histogram3D. This, I would repeat until the end of the file, each time plotting a Histogram3D. In the end, I would like to make an Animation of the Histogram3Ds that I plotted.

How should I go about this?

I do

data = Import[
"C:\\Users\\Debajyoti \
Sengupta\\Documents\\Processing\\collide_pde\\spatial_density.txt", \
{"Data", Range[10]}]
Histogram3D[data,Automatic]



to import and plot a Histogram3D for the first 10 data points. I understand that I need to do a loop somehow and store the Histogram3Ds somehow, but I am not sure how to do that.

The files are really large (~2 GB). Any ideas about speeding up import are welcome.

Thanks

• Hi @Debajyoti and welcome. A minimal data file for doing a working example would be nice for helping you... – José Antonio Díaz Navas Feb 16 '19 at 11:35
• I can't find the option to upload the data file here. Here's a pastebin.com/Nrk7SeTi pastebin link containing the data. – Debajyoti Sengupta Feb 16 '19 at 11:55
• It seems to me the easiest way is to use ReadLine[] and loop through all the lines of the input file. – Somos Feb 16 '19 at 15:27
• I have to save n entries to the data variable and keep doing till EOF. I will also need to skip the line where it goes ------ F. – Debajyoti Sengupta Feb 16 '19 at 16:00

## 1 Answer

Import the raw data from your pastebin as comma-separated values:

list = Import["https://pastebin.com/raw/Nrk7SeTi", "CSV"];


This will give a list of lists with the following structure:

{{290.73, 62.61}, {71.91, 122.72}, {548.1, 326.38}, {331.88, 358.37}, {109.96, 74.38},
{179.66, 364.96}, {221.43, 166.61}, {177.08, 188.76}, {331.75, 207.65}, {247.19, 277.06},
{"------------ F 0"}, {291.21, 62.95}, {71.42, 122.6}, {548.02, 325.96}, {332.3, 358.63},
{109.88, 73.98}, {180.1, 365.03}, {221.55, 166.49}, {177.15, 188.68}, {331.88, 207.85},
{247.21, 277.53}, {"------------ F 1"},
...


You can exploit the fact that these are lists of pairs (i.e. elements with Length = 2), separated by a list with a single element in it, the separator string, i.e. an element with Length = 1. You can use SplitBy[list, Length] to get separated lists. Then you have to remove the separators, which are still lists with one element, so you can use Select with a length criterion:

split = Select[SplitBy[list, Length], Length[#] != 1 &];


Finally, generate BarCharts from the pairs of values, and animate the list of plots using ListAnimate:

ListAnimate[BarChart /@ split]


• Hello @MacroB, Thank you for your reply. The following code works perfectly. But what I really wanted to plot was a Histogram3D (or a SmoothDensityHistogram) so I do - ListAnimate[SmoothDensityHistogram /@ split]. It produces the desired output, except that it keeps readjusting the axes scale every now and then. Is there a way to fix the axes scale? Say Y range - 0 : 700, X range : 0 - 600? – Debajyoti Sengupta Feb 17 '19 at 10:08
• Also, Is there a way I can export the ListAnimate to a gif or a mov? I tried Export["evo.mov",%] but it seems to take forever – Debajyoti Sengupta Feb 17 '19 at 11:11