Graph databases / Nosql with Mathematica

Question

I am very new to mathematica but it looks like an interesting language and I would like to explore it for a future project to create a web search engine. It seems like it would be fairly well suited for this as it seems to already be powering wolfram alpha.

I was wondering if the only databases you can hook into are SQL like databases. I can't see anywhere that talks about hooking into Mongo or a graph database like Neo4j. Why is this? Does this not make sense for some reason?

Any thoughts would be great. Thanks.

Edit:

So ya. If you have examples of ways to connect Mathematica to MongoDB that would be awesome. If possible I would also like to see an example of connecting it to Neo4j. (Or reasons why that would not make sense). I guess I am wondering why everything for Mathematica is being used to connect to SQL databases thus far? Is there a reason for that?

Answer 1

Here are the two ways I have successfully used MongoDB with Mathematica. AFAIK much of this would also apply to similar DBs such as CouchDB.

A. Easy way

One working example used a third party Mongo supplier such as 28msec.io and use URLFetch e.g.

With this supplier you can both read from and write to the database using HTTP GET and simple query strings. While I have not tested it, it seems to me that you should be able to make calls to a set up such as this from a free CDF (it is my understanding that you can import from a URL from free CDF).

B. Stand alone implementation

I have installed MongoDB on my Mac and got it working with Mathematica using this method:

I do not imagine that the set up would be too different on other systems. To interface with Mathematica I have used the REST API. For me as a non-Java programmer this was far more straight forward than trying to make a JLink connector. I have no idea of performance issues of the REST API vs purpose built JLink connector. For the intended application there were no noticeable problems, slowdowns etc.

1 Download and install MongoDB.

2 Start the webserver

$ sudo apachectl start

3 Start Mongo DB

$ mongod --rest

4 Check that everything is working on port 28017

127.0.0.1:28017

The problem with the install is that the bundled REST interface does not allow inserting etc. Full REST capabilities can be achieved with an external tool such as sleepy mongoose.

5 Install pymongo

$ easy_install pymongo

6 Download and install sleepy.mongoose.

You are now in business and can read from and write to your local MongoDB from Mathematica using URLFetch.

Edit

Forgot to mention that you use ImportString[ string, "JSON"] to convert the JSON to rules. Also forgot to mention that for easy option A, or similar database providers, if you can enter the login and password as part of the query string then you can use Import to read from, and write to, the database.

Edit 2

I've been using RESTHeart for the past few months and find that a better API. Add your instructions using the "Body" option for URLFetch.

Answer 2

Mathematica can also connect to Neo4J via the REST API. Instructions for use on OS X:

1. Install Neo4j

2. start the webserver

$ sudo apachectl start

3. start Neo4j (installation in usr/local/bin)

$ neo4j start

4. Make sure all is working

127.0.0.1:7474

Examples:

URLFetch["http://localhost:7474/db/data/node",
 {"StatusCode", "Content"},
 "Method" -> "POST"]

Neo4J has a cypher query language that can be used from Mathematica in an analogous way to how we use SQL queries from SQLExecute:

tmp = URLFetch["http://localhost:7474/db/data/cypher",
  {"StatusCode", "Content"},
  "Method" -> "POST",
  "Parameters" -> {"query" -> 
     "MATCH (a)-[r]->(b)RETURN DISTINCT head(labels(a)) AS This, type(r) AS To,   
head(labels(b)) AS That LIMIT 100"}
]

(*

{200, "{
   \"columns\" : [ \"This\", \"To\", \"That\" ],
   \"data\" : [ [ \"BusinessLine\", \"LOCATED_IN\", \"Building\" ], [ \
\"BusinessProcess\", \"CONTAINS\", \"Process\" ], [ \"BusinessProcess\
\", \"BUSINESSPROCESS_HAS_RTO\", \"RTO\" ], [ \"Process\", \
\"USED_BY\", \"BusinessLine\" ], [ \"Process\", \"PROCESS_HAS_RTO\", \
\"RTO\" ], [ \"Process\", \"USES\", \"Application\" ], [ \"RTO\", \
\"PRECEDES\", \"RTO\" ] ]
 }"}

*)

The output is JSON so

tmp2 = ImportString[tmp[[2]], "JSON"];
TableForm["data" /. tmp2]

Things get a bit tricky for more complex queries that return graphs but above gives the structure for connecting and querying.

Answer 3

For MongoDB instead of using the Rest interface using urls as suggested by Mike, you can use the Java driver.

Here's the proof of concept that should allow a lot of fruitful things ...
I'm still amazed by the possibilities offered by JLink.

Prerequisites

Install MongoDB, for example on Windows http://docs.mongodb.org/manual/tutorial/install-mongodb-on-windows/
Follow the getting started http://docs.mongodb.org/manual/tutorial/getting-started/
Download the Java driver http://docs.mongodb.org/ecosystem/drivers/java/

Some useful links as a start

http://www.mkyong.com/mongodb/java-mongodb-convert-json-data-to-dbobject
http://howtodoinjava.com/2014/06/03/java-mongodb-insert-documents-in-collection-examples/
http://docs.mongodb.org/manual/reference/sql-aggregation-comparison/
http://www.mathematica-journal.com/issue/v9i1/contents/UI_With_JLink/UI_With_JLink_2.html

Proof of concept

Installing the Java driver

Needs["JLink`"]
ReinstallJava[ClassPath->"yourPath\\mongo-2.10.1.jar"];

Connecting to the database

mongo=JavaNew["com.mongodb.Mongo","127.0.0.1",27017];

Here's an example if you need to authenticate, for example on the cloud with MongoLab

mongo=JavaNew["com.mongodb.MongoClient",JavaNew["com.mongodb.MongoClientURI","mongodb://user:[email protected]:63140/mydb"]]

Defining a collection

db=mongo@getDB["mydb"];
collection=db@getCollection["testData"];

Converting an association to an insertable document. You could just use rules instead of Associations.

assocToJson[expr_]:=expr//.a_Association:>Normal[a]//ExportString[#,"JSON"]&;
json=<|"x"->3.`,"y"-><|"z"->48|>|>//assocToJson;
LoadJavaClass@"com.mongodb.util.JSON";
dbObject=JSON`parse[json];
dbObject@toString[] (*not necessary, just for display*)

Other way to create a document

document=JavaNew["com.mongodb.BasicDBObject"];
document@put["keyTest",MakeJavaObject@"valueTest2"];
document@toString[]

Inserting two documents

collection@insert[{dbObject,document}];

Query and converting the results back to Associations

cursorDoc=collection@find[];
Needs["GeneralUtilities`"]; (*for ToAssociations*)
While[cursorDoc@hasNext[],
    Print[(cursorDoc@next[])@toString[]//ImportString[#,"JSON"]&//ToAssociations]
]

Answer 4

Here is one example of how to create a neo4j connection using JDBC driver from this git.

The steeps are:

1- Put the JAR file (neo4j-jdbc-2.0.1-SNAPSHOT-jar-with-dependencies.jar) together with other jar drivers in: FileNameJoin@{ $InstallationDirectory, "Contents/SystemFiles/Links/DatabaseLink/Java"}

2- Restart you Mathematica

3- Now, using Movie Graph as example (that comes with neo4j).

Needs["DatabaseLink`"]
conn=OpenSQLConnection[ JDBC["org.neo4j.jdbc.Driver", "jdbc:neo4j://localhost:7474"]];
sql="
    MATCH (tom:Person {name: \"Tom Hanks\"})-[:ACTED_IN]->(movies)
    RETURN tom.name,movies.title
    LIMIT 10
";
SQLExecute[conn,sql,"ShowColumnHeadings"-> True]//TableForm
CloseSQLConnection[conn];

UPDATE

The maven repository sounds to be the best place to download the latest JDBC jar version. https://maven-repository.com/artifact/org.neo4j/neo4j-jdbc

I retested the 2.1.4 file (with dependency), and worked just nice.

Answer 5

I have an open-source MongoDB Mathematica driver here: https://github.com/zbjornson/MongoDBLink. There are quick setup instructions and an overview in the readme.

It's built on the official Java driver, like @faysou's answer, but it's optimized to reduce the JLink overhead when serializing/deserializing many documents. It also transparently maps some of the Wolfram Dataset/Query operations to MongoDB aggregation operations, so you can do things like myCollection[Total, "myKey"].

If you have any issues or feature requests, please open a GitHub issue, as it's much easier to discuss there than on StackExchange :).

Answer 6

Wolfram Language version 11.3 now has support for interfacing with MongoDB.

You can connect to a MongoDB database via MongoConnect. For example, connecting to a database running locally:

In[1]:= Needs["MongoLink`"]
client = MongoConnect[];
MongoGetDatabaseNames[client]

Out[2]= {"Test", "admin"}