Questions tagged [web-access]
Questions about using Mathematica for accessing web sites and using such functions as URLFetch and URLSave.
266 questions
8
votes
1
answer
979
views
Import the table from an html website
I would like to import the data (table) from this website: http://dr.ittf.com/ittf_ranking/ittf_team_ranking.html
I don't know how to import and automatically change the date/year. Let's say, I want ...
- 71.2k
1
vote
0
answers
109
views
Deal with AJAX when importing websites
I'm currently planning to create a small package to enable Mathematica to manipulate WeChat, a extremely famous chatting software in China.
The first step is to get the QRCode on the website, but the ...
- 9,660
2
votes
0
answers
25
views
URLFetch fails where import doesn't [duplicate]
So as part of breaking down how to write a service connection I need to use URLFetch as opposed to URLRead. Unfortunately this ...
- 62.3k
4
votes
1
answer
927
views
Can't get a POST URL working with HTTPRequest [duplicate]
I'm puzzled by the syntax of building a URL with HTTPRequest.
I can build a URL POST request with postman and that works perfectly.
The HTTP code that is build with Postman is:
POST /rest/login-...
- 4,064
7
votes
1
answer
1k
views
How to change proxy settings programmatically?
I'm using version 10.2 on Linux, and need to access some web services via proxy server sometimes. Is there a way to change the proxy settings, which can be modified manually in the "Preferences" AFAIK,...
- 62.3k
7
votes
0
answers
666
views
How to crawl some website needed username and password [duplicate]
In version 11,if we want to crawl public website like wiki,we can do it like this:
URLRead["https://en.wikipedia.org/wiki/Wiki", "Body"]
But I don't know how ...
- 37.4k
3
votes
1
answer
237
views
How to omit the authentification dialog during URLRead?
When I use URLRead
Mathematica will popup a window like:
Actually I have given the parameters Password and ...
CommunityBot
- 1
10
votes
1
answer
428
views
What is the equivalent of curl?
The URL related functions have confused me yet again (I have v11.1)!
What is the Mathematica equivalent for the following generic curl command?
...
- 111
1
vote
0
answers
48
views
Has anyone successfully connected to Google Contacts using ServiceConnect in V11?
When I call: googlecontacts = ServiceConnect["GoogleContacts"]
The "Wolfram Connector" widget pops up and I click on "Sign in to Google Contacts" which directs me ...
- 7,075
3
votes
1
answer
188
views
Import::fmterr URLExecute wont import JSON
I'm trying to perform a multiple element GET request from an alfresco database of elements. The way we have it set up is if a GET request is made with a body, it will return the elements listed in the ...
- 138k
7
votes
1
answer
216
views
How to get StatusCode without fetching Body/Content in the background
I only need to know whether a specific call returns 200 but:
...
- 17.4k
5
votes
1
answer
673
views
Is it possible to grab all the links at once on a webpage?
I'm working with some webscraping using Mathematica and have run into a question: is it possible to grab all the links on a webpage that scrolls down? The page I am trying to scrape is this: "http://...
- 71.2k
2
votes
1
answer
703
views
How to call Imgur API
My image host returns a link starting with http://. It seems this format is not supported in SE anymore. To avoid this, I want to change the image host to Imgur. ...
CommunityBot
- 1
6
votes
1
answer
1k
views
How to access a web API with HTTP POST and JSON
I want to get data using HTTPS from
POST https://developers.betfair.com/api.betfair.com/exchange/betting/json-rpc/v1 HTTP/1.1
using JSON
...
2
votes
2
answers
574
views
Using URLFetch/URLExecute to post to an internal server
I've been looking up how to do this for a while and I haven't seen any thorough explanations, just hints. Is there like a secret repository of clearly defined Mathematica code somewhere that I cant ...
- 8,727
3
votes
1
answer
128
views
URLDownload aborts prematurely
So I've been trying to automate the download and build of various large packages for hooking Mathematica into extensions but the downloads continually finish prematurely.
Trying to get the Psi4 ...
- 62.3k
6
votes
1
answer
96
views
Inconsistent HTTPResponse handling
It seems that sources of URL* related features are not initialized properly which leads to problems in many cases.
Please follow those steps to reproduce the ...
- 138k
0
votes
1
answer
192
views
How to extract an answer on Mathematica.SE as an image [duplicate]
When I try to get this answer from Mathematica.SE by evaluating
Import["https://mathematica.stackexchange.com/a/95725/21532"]
I just get some text without any ...
CommunityBot
- 1
3
votes
1
answer
85
views
Divergent import behavior on URL and HTTPRequest
I'm playing with the IMSLP API (such as it is) and was running it through my generalized API interface which formats calls to HTTPRequest for reasons of flexibility,...
CommunityBot
- 1
14
votes
3
answers
2k
views
How to use URLFetch for a https secure automated login?
UPDATE 9 March 2017:
I added a bounty to this question, I think it is a hard and challenging problem and can prove that Mathematica is capable of almost anything.
Sadly user1473011's answer does not ...
CommunityBot
- 1
1
vote
1
answer
149
views
How do I use Mathematica with a web-based tool?
I use Mathematica to generate lists of genes from Saccharomyces Cerevisiae that I then throw into the GO enrichment analysis tool here: http://pantherdb.org/webservices/go/overrep.jsp. I know how to ...
- 8,727
6
votes
3
answers
340
views
Possible bug with URLRead
Version 11 introduces URLRead (and others) to replace URLFetch, but I have run into a strange problem while trying to switch to ...
- 1,101
12
votes
1
answer
1k
views
More complete "MutipartData" POSTs using URLFetch
I'm trying to upload a file to a wiki using Mathematica's URLFetch. Some preliminary work is required - one needs to log in to the wiki, obtain an edit token, and ...
- 1,210
4
votes
1
answer
91
views
Import file from website using process indiciator
Would it be possible to have some sort of progress indicator for downloading files?
For example if we would like to import:
...
- 236k
44
votes
1
answer
4k
views
Web Browsing Automation with Mathematica
I want to build a bot to automate web browsing, this mean something like:
filling forms
press "submit" buttons
find certain text inside pages
and so on...
How can I do this with Mathematica?
The <...
- 9,831
4
votes
1
answer
147
views
URLSave and bitbucket
I'm trying to download a file hosted on bitbucket via Mathematica. The direct link
https://bitbucket.org/feynmanIntegrals/fire/downloads/FIRE5.2.tar.gz
works fine with Firefox and wget. However, ...
- 15.1k
1
vote
1
answer
241
views
How to create string from {"jsonrpc" -> "2.0", "method" -> method, "params" -> {"filter" -> "{}"}}
Consider the following function:
...
- 9,687
4
votes
0
answers
317
views
4
votes
2
answers
350
views
How to call Picture Bed API
I want to call this Picture bed API to upload my picture and return its link.I have translated the official instructional the page into English.And there is a example succeed to use it with Python. I ...
- 346
6
votes
1
answer
399
views
How to use URLDownload if the url doesn't have the correct filetype extension?
I'm using the new (in V11) function UrlDownload[]
I have a url I'd like to save to a local file, but the url is missing a (or has the incorrect) filetype extension....
-5
votes
1
answer
57
views
Passing the name of an url to ma [closed]
I have the following command
How to circonvent this weird comportment. Perhaps, it is not possible to open a pdf file from mathematica. But this is not the full problem.
4
votes
0
answers
134
views
Use Mathematica inside web app?
My python code running in a Flask web application needs to return results from Mathematica, what is the best way to achieve this? Are there any examples of this having been done successfully?
- 3,820
6
votes
1
answer
233
views
How to use the new function FindCookies?
This is a further question of this post. Actually this code from SqRoots.And this code has completed the task.
(I assume you are using Mozilla Firefox and MMA version 11.0)
...
CommunityBot
- 1
1
vote
0
answers
183
views
Make Mathematica input stuffs on websites
Main Question
I'm currently making a web crawler, but I'll need to deal with login windows sometimes. All simple web crawlers created with Mathematica I've seen before cannot deal with input boxes or ...
CommunityBot
- 1
3
votes
0
answers
60
views
Ideas on source to learn the ins and outs of all Web Operations?
Since beginning to learn Mathematica a few months ago, I have been amazed by its capabilities. I have recently come across basic web operations such as URLFetch and I am interested in learning more. ...
- 31
7
votes
0
answers
175
views
How to `URLEncode` spaces properly?
URLEncode encodes spaces as plus symbols, but this isn't what we want:
The encoding used by default is based on a very early version of the general URI percent-...
CommunityBot
- 1
1
vote
0
answers
117
views
URLFetch meet the problem of 502
I'm using Mathematica10.0.2 on Win7 to write a crawler. Here is a piece of my code:
...
- 2,745
5
votes
1
answer
1k
views
How to download pictures from webpage automatically?
Today, I read a good web page about love which owns many beautiful pictures. It is fussy for me to download every picture hand-by-hand, so I'd like to use Mathematica to do this case automatically.
I ...
CommunityBot
- 1
30
votes
2
answers
878
views
Reading from a socket stream
I noticed an undocumented socket package (SocketLink) in Mathematica (Using version 10, not sure when it was actually added) and wrote a barebones HTTP responder ...
- 62.1k
7
votes
2
answers
583
views
Web Browsing Automation: how to expand all comments under a blog post?
I wish to expand all the comments under a LiveJournal post for the purposes of saving the complete thread of comments. According to LiveJournal's FAQ when a post gets more than 50 comments they are ...
CommunityBot
- 1
2
votes
0
answers
140
views
Web crawler: want to ignore login windows [closed]
I'm trying to make web crawler with Mathematica.
...
- 108k
10
votes
3
answers
821
views
How to make a picture grid of all Nobel Laureates in physics?
I want to make a picture grid of all Nobel Laureates in physics using mathematica. Later I wanna print this picture grid on a big wall!
The first thought I came up with is to exploit the powerful ...
CommunityBot
- 1
3
votes
3
answers
570
views
URLFetch functionality for a Persian website
I want to fetch the content of a Persian website. I tried
URLFetch["http://www.tsetmc.com/"]
Why is Mathematica unable to recode the characters in the results ...
5
votes
1
answer
485
views
Is there a library or API for interacting with WebSockets in Mathematica?
I'm trying to build a Mathematica model that can retrieve and send data to a service that uses a WebSocket protocol. Is there a library or set of APIs for MMA that interace to WebSockets?
CommunityBot
- 1
1
vote
0
answers
106
views
Two-step web data submission
I am trying to automate the process of downloading data from my univ. webpage. I need to submit the form in the URL below.
http://202.88.252.21/CuPbhavan/res_newregentry.php?id=4789
An inspection ...
0
votes
1
answer
129
views
Getting error from CommunityGraphPlot
Situation
I'd like analyse my web site with CommunityGraphPlot as follows:
Step 1: Define a function to scrape the all the webpages from expected website.
...
- 108k
1
vote
0
answers
205
views
Import using POST method and web-page drill down
A response to a previous question about invoking the POST method via Mathematica was most informative in trying to further understand the intricacies of using the POST Method to obtain tabular data ...
CommunityBot
- 1
16
votes
2
answers
1k
views
How do you submit an Input type of "file" using URLExecute?
I have to submit a form like as follows:
<input type="file" name="jpgFile">
I can't seem to find any instructions on sending this in URLExecute. If I ...
CommunityBot
- 1
2
votes
1
answer
334
views
How to download tables from a url?
I'm trying to import all the tables on a website, this one in particular:
https://aws.amazon.com/ec2/pricing/
For local files that are already tabular, ...
CommunityBot
- 1
1
vote
1
answer
743
views
How to scrape image search engines?
I'd like to download highest resolution images possible from the search results like these:
...
- 27.6k