Tell me more ×
Mathematica Stack Exchange is a question and answer site for users of Mathematica. It's 100% free, no registration required.
up vote 7 down vote favorite
4

I'm struggling to import huge datasets (up to a couple million rows) from a MYSQL database into Mathematica. I already tried http requests, direct sql calls, as well as implementing a web service, However, it all seemed to be quite slow and not elegant. What is the "standard way" of achieving something like this?

share|improve this question
2  
Did you use DatabaseLink for your SQL queries? – Leonid Shifrin Feb 7 at 20:41
Thanks for your comments - yes i did try DatabaseLink, however it has some downsides for me, as i would like to preprocess some of the data on the server side depending on the requests and i'm also not happy with providing remote database access. It also seems to be quite slow... – user871784 Feb 7 at 21:29
DatabaseLink uses JDBC under the hood, so shouldn't be too slow. If it is too slow for your purposes, you could probably create a database dump in some format (CSV, for example), and then there are ways to read such a file in Mathematica rather quickly. Of course, this is not a good option if you want to frequently update your dataset in the database, or if you use each single dataset only once. – Leonid Shifrin Feb 7 at 21:51
if you preprocess on the server side -- and you consider this a downside -- then this will be a problem regardless of whatever method you use to get the data from database to Mma. But I'm surprised that you are finding this to be a problem. It can often be more efficient to do some preprocessing in SQL prior to getting the data into Mma. – Mike Honeychurch Feb 8 at 6:37
Thank you guys for your comments - i now ended up setting up a remote kernel on my server with a direct database connection, so there is no need to transfer all the data to the client anymore. – user871784 Feb 28 at 9:12

Know someone who can answer? Share a link to this question via email, Google+, Twitter, or Facebook.

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Browse other questions tagged or ask your own question.