30
$\begingroup$

I am very new to mathematica but it looks like an interesting language and I would like to explore it for a future project to create a web search engine. It seems like it would be fairly well suited for this as it seems to already be powering wolfram alpha.

I was wondering if the only databases you can hook into are SQL like databases. I can't see anywhere that talks about hooking into Mongo or a graph database like Neo4j. Why is this? Does this not make sense for some reason?

Any thoughts would be great. Thanks.

Edit:

So ya. If you have examples of ways to connect Mathematica to MongoDB that would be awesome. If possible I would also like to see an example of connecting it to Neo4j. (Or reasons why that would not make sense). I guess I am wondering why everything for Mathematica is being used to connect to SQL databases thus far? Is there a reason for that?

$\endgroup$
3
  • $\begingroup$ You can connect MongoDB to Mathematica but it is not as straight forward as using the built in SQL connectors. A possible advantage of Mongo, and possibly Couch and similar, is that in theory you should be able to access these from a free CDF. $\endgroup$ Jan 19, 2014 at 22:17
  • 5
    $\begingroup$ If you re-word your question I'll dig up some MongoDB set up instructions and code and post later today. $\endgroup$ Jan 19, 2014 at 22:28
  • $\begingroup$ Cool. Thanks. I updated it. Why are NoSQL databases less popular with Mathematica? And what about using a graph database like neo4j. The one way I was thinking of going about this could be using Clojuratica (call mathematica from clojure), and use clojure to access the database. $\endgroup$
    – Jonovono
    Jan 21, 2014 at 2:44

6 Answers 6

26
$\begingroup$

Here are the two ways I have successfully used MongoDB with Mathematica. AFAIK much of this would also apply to similar DBs such as CouchDB.

A. Easy way

One working example used a third party Mongo supplier such as 28msec.io and use URLFetch e.g.

enter image description here

With this supplier you can both read from and write to the database using HTTP GET and simple query strings. While I have not tested it, it seems to me that you should be able to make calls to a set up such as this from a free CDF (it is my understanding that you can import from a URL from free CDF).

B. Stand alone implementation

I have installed MongoDB on my Mac and got it working with Mathematica using this method:

enter image description here

I do not imagine that the set up would be too different on other systems. To interface with Mathematica I have used the REST API. For me as a non-Java programmer this was far more straight forward than trying to make a JLink connector. I have no idea of performance issues of the REST API vs purpose built JLink connector. For the intended application there were no noticeable problems, slowdowns etc.

1 Download and install MongoDB.

2 Start the webserver

$ sudo apachectl start

3 Start Mongo DB

$ mongod --rest

4 Check that everything is working on port 28017

127.0.0.1:28017

enter image description here

The problem with the install is that the bundled REST interface does not allow inserting etc. Full REST capabilities can be achieved with an external tool such as sleepy mongoose.

5 Install pymongo

$ easy_install pymongo

6 Download and install sleepy.mongoose.

You are now in business and can read from and write to your local MongoDB from Mathematica using URLFetch.

enter image description here

Edit

Forgot to mention that you use ImportString[ string, "JSON"] to convert the JSON to rules. Also forgot to mention that for easy option A, or similar database providers, if you can enter the login and password as part of the query string then you can use Import to read from, and write to, the database.

Edit 2

I've been using RESTHeart for the past few months and find that a better API. Add your instructions using the "Body" option for URLFetch.

$\endgroup$
2
  • $\begingroup$ Wow this is awesome. Thanks! $\endgroup$
    – Jonovono
    Jan 21, 2014 at 21:32
  • 1
    $\begingroup$ Since 10.2, ImportString[jsonText, "RawJSON"] is perfect to this job. Transforming JSON into an Association. $\endgroup$
    – Murta
    Oct 28, 2015 at 17:53
17
$\begingroup$

Mathematica can also connect to Neo4J via the REST API. Instructions for use on OS X:

1. Install Neo4j

2. start the webserver

$ sudo apachectl start

3. start Neo4j (installation in usr/local/bin)

$ neo4j start

4. Make sure all is working

127.0.0.1:7474

enter image description here

Examples:

URLFetch["http://localhost:7474/db/data/node",
 {"StatusCode", "Content"},
 "Method" -> "POST"]

Neo4J has a cypher query language that can be used from Mathematica in an analogous way to how we use SQL queries from SQLExecute:

enter image description here

tmp = URLFetch["http://localhost:7474/db/data/cypher",
  {"StatusCode", "Content"},
  "Method" -> "POST",
  "Parameters" -> {"query" -> 
     "MATCH (a)-[r]->(b)RETURN DISTINCT head(labels(a)) AS This, type(r) AS To,   
head(labels(b)) AS That LIMIT 100"}
]

(*

{200, "{
   \"columns\" : [ \"This\", \"To\", \"That\" ],
   \"data\" : [ [ \"BusinessLine\", \"LOCATED_IN\", \"Building\" ], [ \
\"BusinessProcess\", \"CONTAINS\", \"Process\" ], [ \"BusinessProcess\
\", \"BUSINESSPROCESS_HAS_RTO\", \"RTO\" ], [ \"Process\", \
\"USED_BY\", \"BusinessLine\" ], [ \"Process\", \"PROCESS_HAS_RTO\", \
\"RTO\" ], [ \"Process\", \"USES\", \"Application\" ], [ \"RTO\", \
\"PRECEDES\", \"RTO\" ] ]
 }"}

*)

The output is JSON so

tmp2 = ImportString[tmp[[2]], "JSON"];
TableForm["data" /. tmp2]

enter image description here

Things get a bit tricky for more complex queries that return graphs but above gives the structure for connecting and querying.

$\endgroup$
14
$\begingroup$

For MongoDB instead of using the Rest interface using urls as suggested by Mike, you can use the Java driver.

Here's the proof of concept that should allow a lot of fruitful things ...
I'm still amazed by the possibilities offered by JLink.

Prerequisites

Install MongoDB, for example on Windows http://docs.mongodb.org/manual/tutorial/install-mongodb-on-windows/
Follow the getting started http://docs.mongodb.org/manual/tutorial/getting-started/
Download the Java driver http://docs.mongodb.org/ecosystem/drivers/java/

Some useful links as a start

http://www.mkyong.com/mongodb/java-mongodb-convert-json-data-to-dbobject
http://howtodoinjava.com/2014/06/03/java-mongodb-insert-documents-in-collection-examples/
http://docs.mongodb.org/manual/reference/sql-aggregation-comparison/
http://www.mathematica-journal.com/issue/v9i1/contents/UI_With_JLink/UI_With_JLink_2.html

Proof of concept

Installing the Java driver

Needs["JLink`"]
ReinstallJava[ClassPath->"yourPath\\mongo-2.10.1.jar"];

Connecting to the database

mongo=JavaNew["com.mongodb.Mongo","127.0.0.1",27017];

Here's an example if you need to authenticate, for example on the cloud with MongoLab

mongo=JavaNew["com.mongodb.MongoClient",JavaNew["com.mongodb.MongoClientURI","mongodb://user:[email protected]:63140/mydb"]]

Defining a collection

db=mongo@getDB["mydb"];
collection=db@getCollection["testData"];

Converting an association to an insertable document. You could just use rules instead of Associations.

assocToJson[expr_]:=expr//.a_Association:>Normal[a]//ExportString[#,"JSON"]&;
json=<|"x"->3.`,"y"-><|"z"->48|>|>//assocToJson;
LoadJavaClass@"com.mongodb.util.JSON";
dbObject=JSON`parse[json];
dbObject@toString[] (*not necessary, just for display*)

Other way to create a document

document=JavaNew["com.mongodb.BasicDBObject"];
document@put["keyTest",MakeJavaObject@"valueTest2"];
document@toString[]

Inserting two documents

collection@insert[{dbObject,document}];

Query and converting the results back to Associations

cursorDoc=collection@find[];
Needs["GeneralUtilities`"]; (*for ToAssociations*)
While[cursorDoc@hasNext[],
    Print[(cursorDoc@next[])@toString[]//ImportString[#,"JSON"]&//ToAssociations]
]
$\endgroup$
8
  • $\begingroup$ 2 votes ! Finally ! Lol $\endgroup$
    – faysou
    Jan 7, 2015 at 7:11
  • $\begingroup$ I only noticed this yesterday! +1 of course $\endgroup$ Jan 8, 2015 at 2:53
  • $\begingroup$ Could something similar to @Murta method be employed for this as well? $\endgroup$ Jan 8, 2015 at 2:54
  • $\begingroup$ I've switched to Couchbase recently, it's similar to MongoDB but better (for me). It's much faster on a Windows machine and there's a query language called N1QL that is an SQL adapted to documents. The same ideas as in my post above can be used for this db. $\endgroup$
    – faysou
    Dec 12, 2015 at 13:22
  • $\begingroup$ I've been using a different REST API for Mongo since my post. It works fine but I'd like to try other approaches but cannot get the java driver to work $\endgroup$ Dec 12, 2015 at 23:32
12
$\begingroup$

Here is one example of how to create a neo4j connection using JDBC driver from this git.

The steeps are:

1- Put the JAR file (neo4j-jdbc-2.0.1-SNAPSHOT-jar-with-dependencies.jar) together with other jar drivers in: FileNameJoin@{ $InstallationDirectory, "Contents/SystemFiles/Links/DatabaseLink/Java"}

2- Restart you Mathematica

3- Now, using Movie Graph as example (that comes with neo4j).

Needs["DatabaseLink`"]
conn=OpenSQLConnection[ JDBC["org.neo4j.jdbc.Driver", "jdbc:neo4j://localhost:7474"]];
sql="
    MATCH (tom:Person {name: \"Tom Hanks\"})-[:ACTED_IN]->(movies)
    RETURN tom.name,movies.title
    LIMIT 10
";
SQLExecute[conn,sql,"ShowColumnHeadings"-> True]//TableForm
CloseSQLConnection[conn];

enter image description here

UPDATE

The maven repository sounds to be the best place to download the latest JDBC jar version. https://maven-repository.com/artifact/org.neo4j/neo4j-jdbc

I retested the 2.1.4 file (with dependency), and worked just nice.

$\endgroup$
1
  • $\begingroup$ I get this working fine and also get @faysou example working but cannot get a mongodb connection using OpenSQLConnection. I know the driver is fine because I can reproduce faysou example. $\endgroup$ Dec 13, 2015 at 3:25
12
$\begingroup$

I have an open-source MongoDB Mathematica driver here: https://github.com/zbjornson/MongoDBLink. There are quick setup instructions and an overview in the readme.

It's built on the official Java driver, like @faysou's answer, but it's optimized to reduce the JLink overhead when serializing/deserializing many documents. It also transparently maps some of the Wolfram Dataset/Query operations to MongoDB aggregation operations, so you can do things like myCollection[Total, "myKey"].

If you have any issues or feature requests, please open a GitHub issue, as it's much easier to discuss there than on StackExchange :).

$\endgroup$
9
  • $\begingroup$ Nice! Couchbase is very good also especially since the introduction of N1QL, I switched to it recently from MongoDb. In case you happen to use it, I wish to see a similar driver for it! Thanks again $\endgroup$
    – faysou
    Jan 1, 2016 at 20:28
  • $\begingroup$ Are their any simple instructions on how to upgrade to use this with Mongo V3.2. I downloaded the latest driver but presumably the class files needs updating ?? $\endgroup$ Jan 13, 2016 at 5:42
  • $\begingroup$ The java will need to be recompiled. It doesn't look like any methods I used are removed so it should be easy, will do it in the morning. Github issue for this. $\endgroup$
    – ZachB
    Jan 13, 2016 at 8:32
  • $\begingroup$ @MikeHoneychurch evidently didn't need recompiling. All tests pass with 3.2 driver; re-download the package from github to update. $\endgroup$
    – ZachB
    Jan 13, 2016 at 20:02
  • $\begingroup$ @ZachB I had already updated the driver yesterday but continued to get errors which is why I thought other parts of the package needed updating. So today I downloaded Github again and I still get errors when running your tests. FindDocuments does not work for me. I am having the same problems as this user here: mathematica.stackexchange.com/questions/103925/… $\endgroup$ Jan 13, 2016 at 22:49
6
$\begingroup$

Wolfram Language version 11.3 now has support for interfacing with MongoDB.

You can connect to a MongoDB database via MongoConnect. For example, connecting to a database running locally:

In[1]:= Needs["MongoLink`"]
client = MongoConnect[];
MongoGetDatabaseNames[client]

Out[2]= {"Test", "admin"}
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.