How to asynchronously download 100000 urls in an efficient way

Question

URLSaveAsynchronous can download one url page, how to download 100000 urls in an efficient way?

And better to with an whole progress bar.

I've downloaded 90000 htmls a web-dictionary in mathematica this way, I think it's interesting.

progress= 0.;
progFunction[_, "progress", {dlnow_, dltotal_, _, _}]:= Quiet[progress = dlnow/dltotal]
Dynamic[ProgressIndicator[progress]]

cells = Table[Defer[URLSaveAsynchronous][urls[[i]], names[[i]], progFunction, 
"Progress" -> True], {i, 10000}];

Here is the working Document.

CreateDocument[ExpressionCell[#, "Input"] & /@ cells]

Maybe can speed up it?

Another aspect is how to do the whole job in Mathematica more comfortably.

To build the download tasks took so much time in the working document which looks like I'm not downloading in an asynchronous way.

Answer 1

This is a complicated task and I believe that Mathematica is not the best tool to do it. If you want to do it only once, just go ahead with the method described in your question. Otherwise, if it is a frequent task, try to select a better tool such as Apache Nutch, which has a built in crawler. Not so easy to get used to, but it saves a lot of time afterwards.

Anyways, if you want to choose Mathematica, here is some guidelines that increase the efficiency. Writing a high-performance crawler requires a good knowledge of the TCP/IP protocol stack. Specifically, you should be familiar with the details of TCP and HTTP protocols. I am not going to write the code since testing the code requires the database of links and takes a huge amount of time.

I assume that you want to fetch the urls using a single machine. I also assume your machine has enough power so that processing/storage is not the bottleneck. There are at least two bottlenecks involved in the crawling process then. One is your network throughput (the connection between your machine and the outside world) and the other one is the server(s).

Fetching the urls one by one is not a good idea since because of the latency, the throughput of your crawler would be very low. Fetching all of the urls in parallel at once is a bad idea too. There are limits on the number of connections that any machine can handle at the same time (because of the limited amount of memory and processing power available). Servers also disallow their clients to fetch too many pages in a short period of time to prevent denial-of-service attacks. Therefore, there must be an optimal value, say $k_t$, for the number of parallel fetches at any given time. And, this number, because of the variable condition of your network, is changing during the time.

Guideline 1: Do not open too many connections to a single server. Open at most 3-5 connections. Instead, use multiple servers to have more connections open. For example, if $k_t=40$, select 10 servers and open at most 4 connections to each of them. If you have more than 10 servers in your list, you can choose more servers, each with fewer connections.

Guideline 2: To find the optimal value of $k_t$, I propose you to use a closed loop controller. Set an initial value for $k_0$, say 10. Increase the amount of $k_t$ once every few minutes as long as increasing the amount, increases the throughput as well. Otherwise, decrease it. Knowing how TCP controls congestion helps a lot.

Guideline 3: There is another trick used by crawlers and browsers to speed up the download process of multiple files. But, I am not sure if there is a straightforward way to implement it in Mathematica (e.g., without using Java). Here is the trick: Download multiple files, back to back, with a single TCP connection to the server. I think, the URLSaveAsynchronous function by default does not do such a thing and you need a bit more effort to implement it. The reason behind this technique is that making a connection to the server takes time and has a huge overhead. Therefore, reusing the connection for several downloads, shares the overhead.