Divergent import behavior on URL and HTTPRequest

Question

I'm playing with the IMSLP API (such as it is) and was running it through my generalized API interface which formats calls to HTTPRequest for reasons of flexibility, but it kept failing where it shouldn't. Specifically, this is the issue:

In[879]:= HTTPRequest[
  "http://imslp.org/imslpscripts/API.ISCR.php?account=worklist/disclaimer=accepted/sort=id/type=1/start=0/retformat=json"
  ]//Import
Out[879]= "Unknown account. Please see <a href='\/wiki\/IMSLP:API'>IMSLP:API<\/a> for documentation."

And yet this works fine if I strip the HTTPRequest head:

In[880]:= "http://imslp.org/imslpscripts/API.ISCR.php?account=worklist/disclaimer=accepted/sort=id/type=1/start=0/retformat=json"//
Lookup[Import[#,"JSON"],"0"]&
Out[880]= {type->1,id->Category:(van Luiken) Bakker, Jeroen,parent->,permlink->http://imslp.org/wiki/Category:(van_Luiken)_Bakker,_Jeroen}

Does anyone know why this is? What headers would Import be sending that HTTPRequest isn't?

Can confirm that URLRead fails here, but URLFetch succeeds. Of course, URLRead calls URLFetch so this just makes it weirder.

Edit

I figured out why this happens:

URLUtilities`PackageScope`doURLFetch uses HTTPRequest[url]["URL"] which splits the URL with URLParse, then interprets it and splices it back together with URLBuild and so applies something like URLEncode to the "Query" string so the "/" characters get converted into a form that the IMSLP API can't deal with.

Which means I'm in the market for a work-around. I would prefer to continue to use HTTPRequest if possible, but will probably just have to swap back to a URLFetch structure.

Answer 1

I was able to get around this by providing an override for URLQueryEncode. Since the strings that gave me trouble were those that I'd already built using a special "QuerySeparator" I was able to get away with this override:

Internal`InheritedBlock[{URLQueryEncode},
  Unprotect@URLQueryEncode;
  URLQueryEncode[{k_ -> 
      s_String?(StringContainsQ[a["QuerySeparator"]])}] :=

   k <> "=" <> s;
  URLRead[HTTPRequest@w, args]
  ];

The w there is an API call object, but it formats out to a url and metainfo for an HTTPRequest. Using this I'm able to use it with URLRead and friends.