How to use URLFetch for a https secure automated login?

Question

UPDATE 9 March 2017: I added a bounty to this question, I think it is a hard and challenging problem and can prove that Mathematica is capable of almost anything.

Sadly user1473011's answer does not work. I came back to this issue for many times, so far all my attempts to handle this from within Mathematica failed. Although I have a working solution in the form of a Java program I would rather have it done inside Mathematica.

With Mathematica 11.0.1 on Windows 10 I want to use URLFetch to do a so called 'Non-interactive/Bot login' at Betfair. The Betfair API manual gives this example in Python:

#!/usr/bin/env python

import requests

payload = 'username=myusername&password=password'
headers = {'X-Application': 'SomeKey', 'Content-Type': 'application/x-www-form-urlencoded'}

resp = requests.post('https://identitysso.betfair.com/api/certlogin', data=payload, cert=('client-2048.crt', 'client-2048.key'), headers=headers)

if resp.status_code == 200:
  resp_json = resp.json()
  print resp_json['loginStatus']
  print resp_json['sessionToken']
else:
  print "Request failed."

'client-2048.crt' and 'client-2048.key' are files which I have generated with win-openssl and are stored on disk. I have a username and password.

There are also examples for C# and Java in the API Manual

I came as far the following URLFetch but I get a HTTP 405 return code, which was to be expected because I did not know how to handle the certification file, I generated earlier, basically the question is how to translate the python to URLfetch with an emphasis on the cert file.

URLFetch["https://identitysso.betfair.com/api/certlogin", {"StatusCode",    "Headers", "Content"}, "Headers" -> {"X-Application" -> "nilostep", "Accept" ->"application/json"}, "Parameters" -> {"Username" -> "myusername", "Password" -> "mypassword"}];

I have successfully used the API before but I copied a 'session token' for access which you can get only after manually logging in. Related question: How to access a web API with HTTP POST and JSON

(EDIT: Any example of a https secure login using certification keys using Mathematica will help.)

(EDIT-2: Might this help?: Can anyone explain how to use Mathematica's OAuth package? ? )

Answer 1

This api isn't accessible in the US, which makes it difficult to post a working code.

Try:

myCert = Import["path/to/my.crt"];

file = FileNameJoin[{
                     Replace["Location",PacletInformation["HTTPClient"]],
                     "SSL",
                     "cacert.pem"
                     }];

(*Append your certificate to the existing cacert.pem file*)
o = OpenAppend[file];
WriteString[o, "\n" <> myCert];
Close[o];

URLFetch[ "https://identitysso.betfair.com/api/certlogin",
    { "StatusCode", "Headers", "Content" },
    Method->"POST",
    "Headers" -> {
        "X-Application" -> "nilostep",
        "Content-Type" -> "application/x-www-form-urlencoded",
        "Accept" -> "application/json", 
        "Username" -> "myusername",
        "Password" -> "mypassword"
    }
]

Answer 2

On SSL and cert files

So this seems to fundamentally be a question of doing SSL certification. Going from here there's some discussion of adding SSL certificates in a standard location. Here's a way to find your equivalent of that file:

$CAFile =
  First@FileNames["*.pem", $InstallationDirectory, \[Infinity]];

Mine lives here:

FileNameJoin@{$InstallationDirectory, "SystemFiles", "Links", 
  "CURLLink", "SSL", "cacert.pem"}

And as URLRead calls URLFetch, which in turn calls the CURLLink system I think this is the right place to append my credentials.

Here's a wholly untested way to do this (as I have no .pem files to test it with) without copying it in by hand:

CAAppend[cert_String?(Not@*FileExistsQ)] :=

  If[Not@StringContainsQ[Import[$CAFile, "Text"], cert],
   CopyFile[$CAFile,
    FileNameJoin@{DirectoryName@$CAFile,
      FileBaseName@$CAFile <> "_backup." <> FileExtension@$CAFile
      }];
   With[{append = OpenAppend[$CAFile]},
    WriteString[append, cert];
    Close@append;
    ]
   ];
CAAppend[cert_String?FileExistsQ] :=
  CAAppend@Import[cert, "Text"];

Obviously the best way would be to find some way to pass the necessary SSL cert data in the headers of an http request, but that'll take more research.

Original posting

Since you mention you're on 11. have you tried formatting an HTTPRequest instead? I've had success using it to get an application-only access token off of Twitter. Basically I followed their guide and ended up with this:

HTTPRequest[
  <|
   "Scheme" -> "https",
   "Domain" -> "api.twitter.com",
   "Path" -> {"oauth2", "token"}
   |>,
  <|
   "Host" -> "api.twitter.com",
   "Body" ->
    {"grant_type" -> "client_credentials"},
   "UserAgent" ->
    $consumerName,
   "Headers" -> {
     "Authorization" ->
      "Basic " <> $bearerToken
     }
   |>
  ] // Import

then translating that over to the template you had I ran this (note that when you have your real $cert files you should uncomment File/@ in $cert.

$cert =
  (*File/@*){"client-2048.crt", "client-2048.key"};
$username =
  "myusername";
$password =
  "mypassword";
$application =
  "SomeKey";
HTTPRequest[
  <|
   "Scheme" -> "https",
   "Domain" -> "identitysso.w-con.betfair.com",
   "Path" -> {"api", "certlogin"}
   |>,
  <|
   "Headers" -> {
     "X-Application" ->
      $application
     },
   "Body" -> {
     "username" ->
      $username,
     "password" ->
      $password,
     "cert" ->
      $cert
     }
   |>
  ] // Import[#, "JSON"] &

Executing that returns:

{"loginStatus" -> "ACCOUNT_ALREADY_LOCKED"}

Which tells me that something is happening. Try dropping in your credentials and see what happens.

Answer 3

The following solution is a working workaround.

Needs["JLink`"]
BOTA = "D:\\Dropbox\\DATA\\MyJavaProjects\\bota\\out\\artifacts\\bota_jar\\bota.jar";
ReinstallJava[CommandLine -> "C:\\Progra~1\\Java\\jdk1.8.0_45\\jre\\bin\\java", ClassPath -> BOTA];
conn = JavaNew["org.nilostep.bota.HttpClientSSO"]
conn@getSessionKey[]

bota.jar contains the class HttpClientSSO as described in the API manual ( see question ) with an additional method that returns the sessionkey as a string.

Classpath is the path to bota.jar

getSessionKey() returns a String and is a new method in HttpClientSSO

The above should be possible with just URLFetch though.