I have a c++ program that generates large numbers of small graphs (millions). I would like to remove graphs that are isomorphic to each other so I am thinking about trying to use the callable interface to mathematica to do this. I see there is something called WSTP with essentially string interfaces. I see mentions here of MathLink but that isn't in the list of topics when I look in help so maybe that's an older interface. I am wondering if it's realistic to call mathematica a large number of times via the WSTP interface to essentially just check if two small graphs are isomorphic. My graphs are acyclic multi-digraphs with a single source and sink. My notion of isomorphic is a path-edge incidence matrix that is a row and column permutation of another path-edge incidence matrix. So I will generate a kind of line graph that has a vertex for an edge and a vertex for a path and a directed edge if the edge is in the path and test these for isomorphism. Maybe ~100 paths and ~10 edges. A naive isomorphism test like in boost or my own code is too slow.
You should not test pairs of graphs for isomorphism, as that is quadratic in the number of graphs, and it's hopeless. You should compute a canonical labelling of each graph with a suitable library (bliss, nauty, traces, etc.—not Boost which only has VF2 for testing pairs of graphs), then use a set data structure to remove duplicates.
I do not recommend trying to call Mathematica to do this. It will be very slow. I know this as I created IGraph/M and suffered plenty of frustration due to Mathematica's lack of performant facilities to convert a
Graph to/from some format that is friendly to C++. They didn't add this even after years of requests.
Just use a C or a C++ library from your C++ program. It'll be faster than what Mathematica has anyway. I believe Mathematica may be using an old version of nauty.