19
$\begingroup$

I would like to geolocate a few hundred IP addresses (as part of analysing server logs). What is the best way to do this using Mathematica?

Does Mathematica have any builtin functionality for this?

Is the best approach to try to query an online service instead? Is there one which is easy to use from Mathematica and would be able to handle up to 1000 addresses?

Should I try to find a database and use it offline from Mathematica?

I am not necessarily looking for working code. A general high-level guide on the best approach will be sufficient.


Update: For this application, performance is important. Here's a dataset to use as benchmark:

websites = 
  WolframAlpha[
   "top websites", {{"OrdinalRankings", 1}, "ComputableData"}, 
   PodStates -> {"OrdinalRankings__More", "OrdinalRankings__More", 
     "OrdinalRankings__More", "OrdinalRankings__More", 
     "OrdinalRankings__More"}];

Length[websites]
(* 160 *)

iplist = First /@ 
   Flatten@DeleteMissing[HostLookup /@ websites[[All, 2]]];

Length[iplist]
(* 355 *)
$\endgroup$
  • 2
    $\begingroup$ You can use URLExecute["freegeoip.net/json/someIP"]. They allow 15000 api calls per hour, although using the above takes approx 0.7s on my computer/connection per call. Maybe you can use URLSaveAsynchronous as in this answer $\endgroup$ – Sascha Sep 5 '17 at 9:28
  • 1
    $\begingroup$ You can also use FindGeoLocation. It's very slow though. $\endgroup$ – b3m2a1 Sep 5 '17 at 15:40
  • $\begingroup$ @b3m2a1 Somehow I did not notice this function. It's worth an answer, even if it is slow. $\endgroup$ – Szabolcs Sep 5 '17 at 15:50
  • $\begingroup$ I'll put in a quick one $\endgroup$ – b3m2a1 Sep 5 '17 at 15:53
10
$\begingroup$

Batch mode API call, for best performance

From the documentation at http://ip-api.com/docs/api:batch "Batch JSON", a batch processing with the ability to query multiple IP addresses in one HTTP request, significantly faster than submitting individual queries. A batch request requires a POST request to http://ip-api.com/batch with a Body string in JSON array format, containing up to 100 objects. Therefore, here the ipLongList arguments is Partition into as many ipShortList as necessary.

Free for non-commercial use only!

BatchIP2Location[ipLongList : List[_String ..]] := 
 AssociationThread[ipLongList,
  Flatten@Table[
    Query[Values, GeoPosition][
     ImportString[
      URLRead[
       HTTPRequest[
        "http://ip-api.com/batch",
        <|
         Method -> "POST",
         "Query" -> {"fields" -> "lat,lon"},
         "Body" -> 
          ExportString[
           Map[{"query" -> #} &, ipShortList], 
           "JSON"]
         |>]
       , "Body"], {"RawJSON"}]
     ],
    {ipShortList, Partition[ipLongList, UpTo[99]]}
    ]]

Performance

Length@iplist
(* 358 *)
First@AbsoluteTiming@BatchIP2Location@iplist
(* 0.525389 *)
$\endgroup$
16
$\begingroup$

It seems to be possible to get this data directly in Mathematica using entities. Here's an example:

Let's get a list of IP addresses to work with:

ips = 
 First@*First@*HostLookup /@ {"wolfram.com", "stackexchange.com", 
   "google.com", "baidu.com", "www.elysee.fr", "global.jaxa.jp", 
   "qwant.com", "eso.org"}
(* {"206.123.112.238", "151.101.1.69", "216.58.198.206", \
"220.181.57.217", "207.123.33.126", "202.32.9.55", "194.187.168.99", \
"134.171.75.1"} *)

Transform them into entities:

entityList = Entity["IPAddress", #] & /@ ips;

The coordinates are available as a property:

EntityProperties@First[entityList]
(* {EntityProperty["IPAddress", "Address"], 
 EntityProperty["IPAddress", "Country"], 
 EntityProperty["IPAddress", "FullIPv6Address"], 
 EntityProperty["IPAddress", "HostCoordinates"], 
 EntityProperty["IPAddress", "HostLocation"], 
 EntityProperty["IPAddress", "HostOrganization"], 
 EntityProperty["IPAddress", "HostSegment"], 
 EntityProperty["IPAddress", "IPv4Address"], 
 EntityProperty["IPAddress", "IPv6Address"], 
 EntityProperty["IPAddress", "Name"]} *)

Let's get them all at once:

coords = 
   EntityValue[entityList, "HostCoordinates"]; // AbsoluteTiming
(* {0.976854, Null} *)

And plot them:

GeoGraphics[GeoMarker@GeoPosition[coords], GeoRange -> "World"]

Mathematica graphics

Optionally use DeleteMissing[coords] to remove failed lookups.


Timing using the benchmark dataset:

coord = 
   EntityValue[Entity["IPAddress", #] & /@ iplist, "HostCoordinates"]; // AbsoluteTiming
(* {3.68313, Null} *)
$\endgroup$
  • $\begingroup$ Unfortunately, this does not work properly with {"wolfram.com","stackexchange.com","google.com","lute.lviv.ua","global.jaxa.jp","qwant.com","eso.org"} though 7 addresses {"140.177.205.134", "151.101.193.69", "172.217.20.206", \ "194.44.248.4", "202.32.9.55", "194.187.168.99", "134.171.75.1"} are produced. $\endgroup$ – user64494 Sep 5 '17 at 11:16
  • $\begingroup$ @user64494 Seems to work for me ... $\endgroup$ – Szabolcs Sep 5 '17 at 11:20
  • $\begingroup$ But not properly. There are six points instead of seven points in dropbox.com/s/2m3wbg4pzmlpo1j/geo.pdf?dl=0 . $\endgroup$ – user64494 Sep 5 '17 at 11:33
  • 1
    $\begingroup$ @user64494 I can confirm that it works properly for me, too. It seems that for you, the addresses get resolved to different IPs, two of which have coordinates very close together. You can check this by looking at coords or by making the map bigger (the leftmost marker is actually two) $\endgroup$ – Lukas Lang Sep 5 '17 at 12:52
  • $\begingroup$ You are completely right. Your original example demonstrates difficulties a Mathematica user may face. $\endgroup$ – user64494 Sep 5 '17 at 13:56
11
$\begingroup$

Do not apply this method to long lists, where Batch mode would be more efficient and less abusive to the API provider. See this other answer.

3 API Integrated solution (One API call per IP)

ClearAll[IP2Location];
Options[IP2Location] = {Method -> Automatic};
SetAttributes[IP2Location, Listable]
IP2Location[ip_String | IPAddress[ip_String], OptionsPattern[]] := 
 Block[{url, response, methodlist, method, latlon},
  methodlist = {"nekudo", "freegeoip", "ip-api"};
  method = If[
    OptionValue[Method] === Automatic || FreeQ[methodlist, OptionValue[Method]],
    RandomChoice[methodlist],
    OptionValue[Method]
    ];
  url = URLBuild@Switch[method,
     "nekudo", <|"Scheme" -> "http", "Domain" -> "geoip.nekudo.com", "Path" -> {"api", ip}|>,
     "freegeoip", <|"Scheme" -> "http", "Domain" -> "freegeoip.net", "Path" -> {"json", ip}|>,
     "ip-api", <|"Scheme" -> "http", "Domain" -> "ip-api.com", "Path" -> {"json", ip}|>
     ];
  response = Import[url, "RawJSON"];
  latlon = ToExpression@Values[
     Switch[method,
       "nekudo", Query["location", {"latitude", "longitude"}],
       "freegeoip", Query[{"latitude", "longitude"}],
       "ip-api", Query[{"lat", "lon"}]
       ][response]
     ];
  (*Echo[response,method];*)
  GeoLocation[latlon]
  ]

Performance

First@AbsoluteTiming@IP2Location[iplist]
(*37.6867*)

Failed in 7 sites.

$\endgroup$
  • 2
    $\begingroup$ Might be a good idea to just have one routine with a Method option taking whatever geolocation services are implemented (e.g. IPToLocation[ip, Method -> "FreeGeoIP"]). $\endgroup$ – J. M. will be back soon Sep 5 '17 at 10:15
  • 2
    $\begingroup$ I find it amusing that Automatic corresponds to a random choice... :D $\endgroup$ – J. M. will be back soon Sep 5 '17 at 10:44
  • $\begingroup$ @J.M. Beware that not all services return the same coordinates! IP2Location["206.123.112.238", Method -> #] & /@ {"nekudo", "freegeoip", "ip-api"} (*{GeoLocation[{28.5383, -81.3792}], GeoLocation[{28.5383, -81.3792}], GeoLocation[{38.6143, -90.4444}]}*) $\endgroup$ – rhermans Sep 5 '17 at 10:48
  • 1
    $\begingroup$ If you are going to put in this much work, you should wrap it up into a package and put it on GitHub. $\endgroup$ – Szabolcs Sep 5 '17 at 11:38
  • 1
    $\begingroup$ @rhermans OP is around so you don't need mod to delete obsolete comments. $\endgroup$ – Kuba Sep 6 '17 at 9:03
9
$\begingroup$

Maybe the most concise way is through FindGeoLocation.

It's easy to use, but crushingly slow:

FindGeoLocation /@ HostLookup["www.wolfram.com"] // RepeatedTiming

{0.43, {GeoPosition[{40.1125, -88.2426}], 
  GeoPosition[{28.5383, -81.3792}]}}
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.