0
$\begingroup$

I'm working with some scraping and have run into a bit of a problem. The website I am scraping is this: https://seekingalpha.com/symbol/CRM/stocktalks I am essentially trying to scrape the "stock talks", but that is not the problem. Using the developer tools on my browser, I've noticed that as I scroll down the page and more Tweet like stock talks show up, the URL that initiates them look like this: https://seekingalpha.com/symbol/CRM/more_stocktalks?user_id=47745202&time=3&force_filter_symbol%5B%5D=crm&force_filter_subject_types%5B%5D=StockTalks The scroll pages are all the same except for one number that goes "&time=". In the URL above it is the third reload page, and it keeps going on and on.

Opening the first 14 of these URLs is no problem, and I simply do this with:

Import["https://seekingalpha.com/symbol/CRM/more_stocktalks?user_id=47745202&time=14&force_filter_symbol%5B%5D=crm&force_filter_subject_types%5B%5D=StockTalks","JSON"]

That opens up the JSON for the 14th website. But when I start trying to pull the 15th website and URLs beyond that, I get this error: FetchURL::httperr: The request to URL https://seekingalpha.com/symbol/CRM/more_stocktalks?user_id=47745202&time=20&force_filter_symbol%5B%5D=crm&force_filter_subject_types%5B%5D=StockTalks was not successful. The server returned the HTTP status code 403 ("Forbidden").

I've read somewhere that this might be because you can't have spaces in your urls, but I don't see the difference between the 14th URL and the 15th. Can anyone help?

Improve this question
$\endgroup$
4
  • $\begingroup$ It works for me... $\endgroup$
    – C. E.
    Jun 30, 2017 at 14:08
  • $\begingroup$ Strange, I tried it in a new notebook and it works, but doesn't when I run it on the same one as pages previous to 15 $\endgroup$
    – Vic Jongmin Youn
    Jun 30, 2017 at 16:23
  • $\begingroup$ I don't know about this particular web site, but some sites actively prevent web scraping by returning errors like 403 FORBIDDEN if too many requests are received within a short time period. $\endgroup$
    – WReach
    Jul 1, 2017 at 5:16
  • $\begingroup$ @WReach, yeah that seems to be what's happening. If I space out the scraper to different time intervals I'm able to retrieve several pages at a time $\endgroup$
    – Vic Jongmin Youn
    Jul 3, 2017 at 13:04

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Browse other questions tagged or ask your own question.