Questions tagged [web-access]
Questions about using Mathematica for accessing web sites and using such functions as URLFetch and URLSave.
266 questions
10
votes
0
answers
169
views
URL functions overload
All these functions seem capable of doing the same things: Import, URLFetch, URLSave, ...
- 31.8k
-5
votes
1
answer
57
views
Passing the name of an url to ma [closed]
I have the following command
How to circonvent this weird comportment. Perhaps, it is not possible to open a pdf file from mathematica. But this is not the full problem.
- 4,632
4
votes
0
answers
134
views
Use Mathematica inside web app?
My python code running in a Flask web application needs to return results from Mathematica, what is the best way to achieve this? Are there any examples of this having been done successfully?
- 3,820
6
votes
1
answer
399
views
How to use URLDownload if the url doesn't have the correct filetype extension?
I'm using the new (in V11) function UrlDownload[]
I have a url I'd like to save to a local file, but the url is missing a (or has the incorrect) filetype extension....
- 3,820
12
votes
0
answers
268
views
Is it possible to search the web by image from Mathematica?
Is it possible to perform a web search based on an image input?
There are several services that provide this: Google, Bing, TinEye, and maybe others. I am not sure which can be accessed ...
- 236k
6
votes
1
answer
233
views
How to use the new function FindCookies?
This is a further question of this post. Actually this code from SqRoots.And this code has completed the task.
(I assume you are using Mozilla Firefox and MMA version 11.0)
...
- 27.2k
23
votes
2
answers
2k
views
Can Mathematica act like a web server?
I have a Mathematica program that modifies images that I would like to make available for other people to use through the web. The input is only one parameter. So, for example, I could run it as a ...
- 4,102
7
votes
0
answers
666
views
How to crawl some website needed username and password [duplicate]
In version 11,if we want to crawl public website like wiki,we can do it like this:
URLRead["https://en.wikipedia.org/wiki/Wiki", "Body"]
But I don't know how ...
- 27.2k
1
vote
0
answers
183
views
Make Mathematica input stuffs on websites
Main Question
I'm currently making a web crawler, but I'll need to deal with login windows sometimes. All simple web crawlers created with Mathematica I've seen before cannot deal with input boxes or ...
- 9,660
3
votes
0
answers
60
views
Ideas on source to learn the ins and outs of all Web Operations?
Since beginning to learn Mathematica a few months ago, I have been amazed by its capabilities. I have recently come across basic web operations such as URLFetch and I am interested in learning more. ...
- 31
7
votes
0
answers
175
views
How to `URLEncode` spaces properly?
URLEncode encodes spaces as plus symbols, but this isn't what we want:
The encoding used by default is based on a very early version of the general URI percent-...
- 31.8k
1
vote
0
answers
117
views
URLFetch meet the problem of 502
I'm using Mathematica10.0.2 on Win7 to write a crawler. Here is a piece of my code:
...
- 2,745
0
votes
1
answer
192
views
How to extract an answer on Mathematica.SE as an image [duplicate]
When I try to get this answer from Mathematica.SE by evaluating
Import["https://mathematica.stackexchange.com/a/95725/21532"]
I just get some text without any ...
- 27.2k
7
votes
2
answers
583
views
Web Browsing Automation: how to expand all comments under a blog post?
I wish to expand all the comments under a LiveJournal post for the purposes of saving the complete thread of comments. According to LiveJournal's FAQ when a post gets more than 50 comments they are ...
- 62.3k
2
votes
0
answers
140
views
Web crawler: want to ignore login windows [closed]
I'm trying to make web crawler with Mathematica.
...
4
votes
1
answer
214
views
How do I use Mathematica to upload to Clyp?
Clyp is a service for hosting sound files that (as of now) doesn't require registration.
I'm wondering if it's possible to write a function that'll upload a Sound ...
- 41
10
votes
3
answers
821
views
How to make a picture grid of all Nobel Laureates in physics?
I want to make a picture grid of all Nobel Laureates in physics using mathematica. Later I wanna print this picture grid on a big wall!
The first thought I came up with is to exploit the powerful ...
- 17.4k
14
votes
3
answers
2k
views
How to use URLFetch for a https secure automated login?
UPDATE 9 March 2017:
I added a bounty to this question, I think it is a hard and challenging problem and can prove that Mathematica is capable of almost anything.
Sadly user1473011's answer does not ...
- 9,687
6
votes
1
answer
1k
views
How to access a web API with HTTP POST and JSON
I want to get data using HTTPS from
POST https://developers.betfair.com/api.betfair.com/exchange/betting/json-rpc/v1 HTTP/1.1
using JSON
...
- 9,687
2
votes
1
answer
506
views
Uploading an image to a CloudDeploy Wolfram Language web app
I want to do image processing using the Wolfram Language on a web app hosted on the Wolfram Cloud.
The user must be able to upload a PNG image to the Wolfram Cloud, how can I set this up?
I've ...
- 2,403
1
vote
0
answers
106
views
Two-step web data submission
I am trying to automate the process of downloading data from my univ. webpage. I need to submit the form in the URL below.
http://202.88.252.21/CuPbhavan/res_newregentry.php?id=4789
An inspection ...
7
votes
1
answer
1k
views
How to change proxy settings programmatically?
I'm using version 10.2 on Linux, and need to access some web services via proxy server sometimes. Is there a way to change the proxy settings, which can be modified manually in the "Preferences" AFAIK,...
- 73
5
votes
1
answer
485
views
Is there a library or API for interacting with WebSockets in Mathematica?
I'm trying to build a Mathematica model that can retrieve and send data to a service that uses a WebSocket protocol. Is there a library or set of APIs for MMA that interace to WebSockets?
- 471
0
votes
1
answer
129
views
Getting error from CommunityGraphPlot
Situation
I'd like analyse my web site with CommunityGraphPlot as follows:
Step 1: Define a function to scrape the all the webpages from expected website.
...
- 1,320
1
vote
0
answers
205
views
Import using POST method and web-page drill down
A response to a previous question about invoking the POST method via Mathematica was most informative in trying to further understand the intricacies of using the POST Method to obtain tabular data ...
- 1,975
16
votes
2
answers
1k
views
How do you submit an Input type of "file" using URLExecute?
I have to submit a form like as follows:
<input type="file" name="jpgFile">
I can't seem to find any instructions on sending this in URLExecute. If I ...
- 731
10
votes
2
answers
398
views
Use Mathematica to save all citations from Google Scholar search
How can I save all citations from a Google Scholar search? For instance, in a search for "Radon transformation", there are about 35,500 results. I want to download all citations into a string - is ...
- 1,939
2
votes
1
answer
334
views
How to download tables from a url?
I'm trying to import all the tables on a website, this one in particular:
https://aws.amazon.com/ec2/pricing/
For local files that are already tabular, ...
- 31.8k
3
votes
3
answers
191
views
Why does Import[#, "ImageLinks"]& fail?
I'm having difficulty with Importing urls embedded in a site (with and without pagination). This command:
...
- 31.8k
2
votes
1
answer
398
views
URLFetch Problem with the Size of HTTP Message Body [closed]
I am trying to transmit this string as an HTTP Message BODY
...
- 1,301
3
votes
1
answer
73
views
How to access the string of the cell I am typing?
I'm trying to map a keyboard shortcut to google search the contents of the cell I am typing. Here is my code so far:
...
- 31.8k
1
vote
1
answer
743
views
How to scrape image search engines?
I'd like to download highest resolution images possible from the search results like these:
...
- 31.8k
2
votes
1
answer
838
views
Is there a better way to download images in parallel?
Is this the best way to batch save a list of image urls in parallel?
...
- 31.8k
5
votes
1
answer
1k
views
How to download pictures from webpage automatically?
Today, I read a good web page about love which owns many beautiful pictures. It is fussy for me to download every picture hand-by-hand, so I'd like to use Mathematica to do this case automatically.
I ...
- 655
9
votes
1
answer
418
views
Import citation data for BibTeX using DOI?
I recently discovered that Digital Object Identifiers can be used for a lot more than just a reliable link to an article or dataset, and in some cases the DOI url's support semantic queries.
This ...
- 4,341
2
votes
0
answers
118
views
Retrieving a Portion of Webpage Code using URLFetch
The URLFetch command is very useful for pulling the entire html source of a given webpage, however, if one wanted to speed up the process, by only requesting a ...
- 901
30
votes
2
answers
878
views
Reading from a socket stream
I noticed an undocumented socket package (SocketLink) in Mathematica (Using version 10, not sure when it was actually added) and wrote a barebones HTTP responder ...
- 566
8
votes
1
answer
2k
views
Download Youtube Videos
Is there a way to display or download youtube videos in Mathematica.
Youtube is HTML5 enabled so I am hopeful there might be an easy way.
- 81
2
votes
0
answers
106
views
How to get the subdirectory of a ftp weblink?
For example, I can use
FileNames["*",dir]
to get all subdirectory under dir if dir is some directory of my local computer. Suppose that I want to get all ...
- 2,745
3
votes
1
answer
770
views
How can I determine `Parameters` for use in `URLFetch`?
I'm trying to download data from website automatically using Mathematica, and URLFetch seems like it should be suited to the task. I'm interested in its "passing ...
- 2,745
5
votes
0
answers
250
views
Dropbox datastore
Dropbox seems to have a feature where it can be used as a database.
This would be really cool to use for memoized functions, that can be restored
on several computers and sessions.
I have a function ...
- 2,499
10
votes
1
answer
1k
views
How to execute JavaScipt on a webpage and then import the result on OSX?
I would like to import data from this page through Mathematica,
http://www.oddsportal.com/soccer/netherlands/eredivisie/ .
I tried importing through the default Import function
...
- 841
8
votes
2
answers
596
views
Lookup in Inverse Symbolic Calculator+ from Mathematica
There is a useful feature in WolframAlpha that allows to find a possible closed form for approximate numeric values. It can be used, for example, to guess the value of an integral that Mathematica ...
- 7,333
12
votes
1
answer
1k
views
More complete "MutipartData" POSTs using URLFetch
I'm trying to upload a file to a wiki using Mathematica's URLFetch. Some preliminary work is required - one needs to log in to the wiki, obtain an edit token, and ...
- 313
4
votes
1
answer
1k
views
Using a web-crawler to search website for occurrences of keywords
Given this list of cities:
...
- 339
4
votes
1
answer
2k
views
11
votes
1
answer
1k
views
Parse website data that requires interaction with a form (POST)
I want to analyse Chilean elections data available in the official website. What I have returns the data for president (numbers are strings, to be parsed later).
...
- 37.4k
2
votes
1
answer
224
views
Problem with importing a JCAMP-DX IR spectrum file
I wrote a little function to import JCAMP-DX electron ionization or mass spectral files from the NIST webbook web site (e.g. mass spectrum of hydrogen chloride at
http://webbook.nist.gov/cgi/cbook.cgi?...
- 897
11
votes
2
answers
1k
views
Look up info associated with a given CAS chemical identifier from the NIST webbook & NCI CADD Chemical Identifier Resolver
I would like to retrieve information associated with a given CAS registry number (Chemical Abstracts Service nr) from the NIST webbook web site in Mathematica, using the provided API (e.g. for ...
17
votes
3
answers
430
views
How do I put my packages into Dropbox
I don't want to necessarily put my packages in Mathematica's $UserBaseDirectory, which is on my local hard drive. Can I somehow tell Mathematica to look for ...
- 19.8k