5
$\begingroup$

I have a graph data file infile.txt.

3 2 0 1 1 2
3 2 0 1 2 1
3 2 1 0 1 2
3 3 0 1 1 0 1 2
3 3 0 1 1 0 2 1
3 4 0 1 1 0 1 2 2 1

In this file, each row corresponds to a digraph in the following way:

  • The first number is the number of vertices in this digraph;
  • The second number is the number of arcs in this digraph;
  • Remaining numbers represent arc in pairs.

For example, the last row in the given file corresponds to a digraph with 3 vertices and 4 arcs, and arcs are $(0,1)$,$(1,0)$,$(1,2)$,$(2,1)$.

I want to import it into Mathematica and construct a list of graphs accordingly. And my solution is:

mylist=Import["/path/to/infile.txt","Table"];
mylist=Drop[#,2]&/@mylist;
glist=Graph[DirectedEdge@@@Partition[#,2]]&/@mylist

My method is OK with small files. However, it is extremely slow when the input file is large. But I need to deal with large files a lot. So my questions are

  1. How to input such large files efficiently? ReadList seems to be good option. But how to read the file row by row?

  2. What is best importing format for constructing a list of graphs, number, string or something else?

  3. Are there other ways to construct the list of graph efficiently from a list given above?

Update: Following are the running times for a file with 531441 rows.

AbsoluteTiming[mylist=Import["infile.txt","Table"];]
{20.3156, Null}

And the method suggested by @RunnyKine gives

AbsoluteTiming[mylist=ReadList["infile.txt", Number, RecordLists -> True];]
{4.45186, Null}

That's really a big improvement.
The input file (around 35MB) is available from this link.

$\endgroup$
7
$\begingroup$

The following should be fast enough:

myList = ReadList["infile.txt", Number, RecordLists -> True]; // AbsoluteTiming

{4.64517, Null}

$\endgroup$
  • $\begingroup$ I tested your solution on a 35MB file. It turns out it is not as fast as expected. Is it a problem of Mathematica itself? I'm using Mathematica 10.3. $\endgroup$ – Han Xiao Apr 22 '16 at 5:44
  • $\begingroup$ @HanXiao. See my edit. $\endgroup$ – RunnyKine Apr 22 '16 at 5:45
  • $\begingroup$ Thanks for the edit. Now it is almost 5 times faster than mine. $\endgroup$ – Han Xiao Apr 22 '16 at 6:06
  • 2
    $\begingroup$ Just a side note: ReadList also uses less memory during the import. Import causes a memory usage peak of almost 2GB, while ReadList isn't really noticeable on the memory side $\endgroup$ – jens_bo Apr 22 '16 at 6:46
  • 4
    $\begingroup$ @jens_bo. Yes it does use less memory. It's a low level function that's why I prefer using it. It's almost always (if used right) much faster than Import $\endgroup$ – RunnyKine Apr 22 '16 at 6:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.