Fast way to export large amount of data in "Table" format

Question

I have about a million 3D points, say

data = Table[{1. x, 1. y, 0.}, {x, 1000}, {y, 10^3}]~Flatten~1;

I want to visualize them, but Graphics3D@Point@data or ListPointPlot3D@data are too slow.

MeshLab or CloudCompare can do it, but I need to export it in some way to get it there. It can read ".txt" files with the "Table" format that Mathematica produces, but that takes a very long time to write:

AbsoluteTiming@Export["test 1mio points.txt", data, "Table"]

{23.9979, "test 1mio points.txt"}

CSV file writing is not much faster. It doesn't look like there is a general bottleneck in converting numbers to text format, as '.m' is written quite quickly:

AbsoluteTiming@Export["test 1mio points.m", data]

{1.92666, "test 1mio points.m"}

but MeshLab cannot handle it.

Any alternatives? Is there a way I can convert a list of points to a GraphicsComplex or Mesh or so and then export that in some format (that might be faster).

Answer 1

I started working on a package and LinkedLibrary achieving significant speedups with this job:

data = Table[{1. x, 1. y, 0.}, {x, 1000}, {y, 10^3}]~Flatten~1;
<< ExportTable`
AbsoluteTiming@ExportTable["test 1mio points.txt", data]

{0.0381513, Null}

You can find it here: https://github.com/Masterxilo/ExportTable

Answer 2

Here's an alternative: The package https://github.com/Masterxilo/PLYExport implements exporting of Points in PLY format, and extends Export with {"PLY", "BinaryFormat" -> True} to use it, allowing to also export lots of points very quickly:

data = RandomReal[1., {10^6, 3}];
(*Graphics3D@Point[data]*)(*don't do this for large n: Use MeshLab or CloudCompare*)
AbsoluteTiming@
 Export["test 1 mio points.ply", 
  Graphics3D@Point[data], {"PLY", "BinaryFormat" -> True}]

{0.308942, Null}

Answer 3

Not very fast, but acceptable.

data = RandomReal[1, {10^5, 3}];
Export["test1.txt", data, "Table"] // AbsoluteTiming

AbsoluteTiming[
 str = Row[#, "\t"] & /@ data // Column // NumberForm[AccountingForm@#, 16] & // ToString;
 Export["test2.txt", str, "String"]
 ]

Equal @@ (ReadList[#, {Number,Number,Number}] & /@ {"test1.txt","test2.txt"}) // AbsoluteTiming

I also noticed that the same code version 11.2 more slowly than 11.1