Questions tagged [web-access]
Questions about using Mathematica for accessing web sites and using such functions as URLFetch and URLSave.
69 questions
44
votes
1
answer
4k
views
Web Browsing Automation with Mathematica
I want to build a bot to automate web browsing, this mean something like:
filling forms
press "submit" buttons
find certain text inside pages
and so on...
How can I do this with Mathematica?
The <...
- 571
34
votes
3
answers
10k
views
Who is to blame: parsing UTF8 encoded JSON HTTPResponse fails
Bug introduced in 11.0 or earlier and persists through 12.0
(confirmed by WRI Support but read summary below)
Answers summary
I think we need a quick summary here to justify keeping bug.
Original ...
- 138k
30
votes
3
answers
4k
views
How to manipulate web pages on Mathematica?
I wanted to input some data on this page through Mathematica. Then I would press the "buscar" button also through Mathematica coding and get the results on Mathematica. The most similar post I've ...
- 5,573
59
votes
9
answers
3k
views
Is it possible to invoke the OEIS from Mathematica?
I had always wondered if there might be a way to write a function, which I'll call OEISData[], that more or less works as a curated data function for The On-Line ...
12
votes
1
answer
1k
views
More complete "MutipartData" POSTs using URLFetch
I'm trying to upload a file to a wiki using Mathematica's URLFetch. Some preliminary work is required - one needs to log in to the wiki, obtain an edit token, and ...
- 313
30
votes
3
answers
4k
views
Export to Google Spreadsheet with OAuthAuthentication
Context
I'm trying to setup a connection between Mathematica and Google Docs. Specifically, I would like to be able to export data to a Google Spreadsheet (Note: if the spreadsheet is public it's ...
- 3,328
30
votes
2
answers
878
views
Reading from a socket stream
I noticed an undocumented socket package (SocketLink) in Mathematica (Using version 10, not sure when it was actually added) and wrote a barebones HTTP responder ...
- 566
7
votes
1
answer
526
views
curl in Mathematica, google link-shortener
I want to automatically get short links equivalents for things I CloudDeploy. I'd appreciate some tips on integrating command line workflow below into Mathematica ...
- 6,697
5
votes
1
answer
524
views
Decoding GZIP encoded Body, BodyBytes (ByteArray) and BodyBytesArray from URLRead
When calling the Stack Exchange API v2.2 using the new (Mma11) function URLRead we get a GZIP encoded Body in alternative formats: ...
- 37.4k
2
votes
1
answer
838
views
Is there a better way to download images in parallel?
Is this the best way to batch save a list of image urls in parallel?
...
- 31.8k
46
votes
4
answers
5k
views
How to do http post in Mathematica?
The Import command lets you do a HTTP GET from a URL, like this
Import @ "http://merkosoncampus.com/wp-content/uploads/2011/05/higgs-boson.jpg"
This sends an ...
- 8,763
19
votes
2
answers
2k
views
Encrypt the password used for SendMail
I want Mathematica to send me an email when the computation is finished. To that end, I have a cell with SendMail[...] which I place behind my computation in the ...
- 333
16
votes
2
answers
1k
views
How do you submit an Input type of "file" using URLExecute?
I have to submit a form like as follows:
<input type="file" name="jpgFile">
I can't seem to find any instructions on sending this in URLExecute. If I ...
- 731
14
votes
1
answer
977
views
How to implement a free version of WebImageSearch?
I really don't like functions that I have to pay for, if they can be implemented for free.
Is there a way to get around this pay-wall for WebImageSearch? Perhaps using some other free api or service?
...
- 31.8k
11
votes
2
answers
275
views
How to show Plots or Graphics in mini MMA HTTP Server?
I learned some codes to make a mini HTTP server in MMA way here.
...
- 2,509
11
votes
1
answer
602
views
Using URLExecute to login to website
One of my students is doing summer research for a local company to help analyze some of their data. Trouble is, the web interface they provide for downloading the data only allows one to download one ...
- 1,939
10
votes
2
answers
2k
views
How can I commit files to Github by Mathematica?
How can I publish Notebooks to Github via using Mathematica?
I'm on windows 8.1 x64, Mathematica 9.
How can I upload the current notebook to Github?
My function expected is: ...
- 8,649
9
votes
1
answer
251
views
Authentication dialog for URLDownload / URLSave
Bug introduced in 11.0 and persists through 11.3
The original problem is was fixed in V11.2 but since then it returns File[path/to/file/that/does/not/exits] instead ...
- 138k
8
votes
1
answer
926
views
Programmatic browser interaction: How to click on web-page and get new URL and list of links
In Mathematica it's easy to Import data from the web and perform all sort of Web Operations. But the later doesn't allow to simulate a more interactive browser ...
- 37.4k
3
votes
0
answers
436
views
URLFetch parameters interfere with POST
(orig: URLFetch Multipart POST fails to post Body?)
Update 2:
So after much weeping and gnashing of teeth Mathematica deemed me worthy of figuring out (some of) what was going on.
It turns out the ...
- 47.1k
3
votes
1
answer
237
views
How to omit the authentification dialog during URLRead?
When I use URLRead
Mathematica will popup a window like:
Actually I have given the parameters Password and ...
- 27.2k
53
votes
1
answer
3k
views
Google Map in Notebooks?
GoogleMaps are implemented as HTML and Javascript code which can be viewed by a web browser. Is it possible to load this map into the Mathematica notebook, which obviously means for example that
<...
- 1,583
16
votes
3
answers
838
views
Which external services are compatible with GeoServer?
Using the GeoServer option, it is possible to load map tiles from external services.
Which services are compatible with ...
- 236k
12
votes
1
answer
410
views
How to work with ElasticSearch in Mathematica?
Steps to install and run ElasticSearch:
install Java (64-bit): https://www.java.com/en/download/manual.jsp
download ElasticSearch (https://www.elastic.co/start) and extract archive in ...
- 9,621
12
votes
1
answer
459
views
A tour of AWSLink?
Seems like there are a lot of useful functions in this undocumented package AWSLink:
Any developers on here that can demo it a bit?
- 3,820
10
votes
1
answer
506
views
Import data from dynamically generated webpage
I'm trying to get data from a web-page
data=Import["https://stats.nba.com/team/1610612738/traditional/", "Data"]
It returns only a textual information. ...
- 143
9
votes
1
answer
418
views
Import citation data for BibTeX using DOI?
I recently discovered that Digital Object Identifiers can be used for a lot more than just a reliable link to an article or dataset, and in some cases the DOI url's support semantic queries.
This ...
- 4,341
6
votes
1
answer
239
views
Export Data from several tabs in a website
Here is the website from the official federation:
http://results.ittf.link/index.php?option=com_fabrik&view=list&listid=36&Itemid=158
I want to export the Data (the table). There are ...
- 1,771
4
votes
2
answers
350
views
How to call Picture Bed API
I want to call this Picture bed API to upload my picture and return its link.I have translated the official instructional the page into English.And there is a example succeed to use it with Python. I ...
- 27.2k
3
votes
1
answer
770
views
How can I determine `Parameters` for use in `URLFetch`?
I'm trying to download data from website automatically using Mathematica, and URLFetch seems like it should be suited to the task. I'm interested in its "passing ...
- 2,745
3
votes
3
answers
570
views
URLFetch functionality for a Persian website
I want to fetch the content of a Persian website. I tried
URLFetch["http://www.tsetmc.com/"]
Why is Mathematica unable to recode the characters in the results ...
- 585
23
votes
2
answers
2k
views
Can Mathematica act like a web server?
I have a Mathematica program that modifies images that I would like to make available for other people to use through the web. The input is only one parameter. So, for example, I could run it as a ...
- 4,102
17
votes
3
answers
430
views
How do I put my packages into Dropbox
I don't want to necessarily put my packages in Mathematica's $UserBaseDirectory, which is on my local hard drive. Can I somehow tell Mathematica to look for ...
- 19.8k
14
votes
2
answers
557
views
Does Mathematica support deep links?
So GitHub sent me on quite the journey, after it opened its desktop app from my browser. Copying the link I found something looking like: "github-mac://...".
I ...
- 47.1k
10
votes
0
answers
169
views
URL functions overload
All these functions seem capable of doing the same things: Import, URLFetch, URLSave, ...
- 31.8k
10
votes
1
answer
1k
views
How to execute JavaScipt on a webpage and then import the result on OSX?
I would like to import data from this page through Mathematica,
http://www.oddsportal.com/soccer/netherlands/eredivisie/ .
I tried importing through the default Import function
...
- 841
10
votes
3
answers
821
views
How to make a picture grid of all Nobel Laureates in physics?
I want to make a picture grid of all Nobel Laureates in physics using mathematica. Later I wanna print this picture grid on a big wall!
The first thought I came up with is to exploit the powerful ...
- 17.4k
9
votes
1
answer
133
views
HTTPRequest syntax problem
Bug introduced in V11.0 and persisting through 11.3
[CASE:3981157] confirmed
...
- 138k
8
votes
1
answer
346
views
How can I use multiple overlaid map layers (e.g. basemap, labels) in GeoGraphics?
We can use GeoServer to use map tiles from an external service. There are many compatible services:
Which external services are compatible with GeoServer?
These ...
- 236k
8
votes
1
answer
178
views
Inconsistent Authorization Dialog (401) handling
Bug introduced in V11 and fixed in 11.3
[CASE:3855554] was created
[...] It does appear that the URLRead is not behaving correctly. I have forwarded an incident report to our developers with ...
- 138k
8
votes
1
answer
2k
views
Download Youtube Videos
Is there a way to display or download youtube videos in Mathematica.
Youtube is HTML5 enabled so I am hopeful there might be an easy way.
- 81
8
votes
1
answer
461
views
Importing images from the web using ImportString & URLFetch
I can bring images from the web using the Import command like this:
Import["http://blog.wolfram.com/wp-content/uploads/2008/06/se-30.jpg"]
Sometimes the web ...
- 8,727
8
votes
1
answer
271
views
How to programatically disable internet connectivity
Internet connectivity may cause slows evaluations or cause unexpected problems (particularly in GUIs) when working in an environment with slow Internet or no Internet access at all. I know how to ...
- 1,707
7
votes
2
answers
583
views
Web Browsing Automation: how to expand all comments under a blog post?
I wish to expand all the comments under a LiveJournal post for the purposes of saving the complete thread of comments. According to LiveJournal's FAQ when a post gets more than 50 comments they are ...
- 62.3k
7
votes
1
answer
216
views
How to get StatusCode without fetching Body/Content in the background
I only need to know whether a specific call returns 200 but:
...
- 138k
7
votes
0
answers
666
views
How to crawl some website needed username and password [duplicate]
In version 11,if we want to crawl public website like wiki,we can do it like this:
URLRead["https://en.wikipedia.org/wiki/Wiki", "Body"]
But I don't know how ...
- 27.2k
7
votes
1
answer
418
views
How to upload an image to the Stack Exchange Imgur account?
This is not a duplicate of How to call Imgur API.
I tried to upload to imgur using its API, but (as my friends test) the API of imgur cannot be called in Chinese mainland. Then I found that ...
- 27.2k
7
votes
2
answers
419
views
Curl analog with URLRead/Execute?
I have a working curl command:
curl https://myapp.herokuapp.com/doit -F filedata=@path/to/img.jpg
How do I write this in the modern way with ...
- 31.8k
6
votes
1
answer
233
views
How to use the new function FindCookies?
This is a further question of this post. Actually this code from SqRoots.And this code has completed the task.
(I assume you are using Mozilla Firefox and MMA version 11.0)
...
- 27.2k
6
votes
1
answer
96
views
Inconsistent HTTPResponse handling
It seems that sources of URL* related features are not initialized properly which leads to problems in many cases.
Please follow those steps to reproduce the ...
- 138k