Writing a basic webcrawler is quite easy using constructs like
Import["http://www.wolfram.com/", "Hyperlinks"]
and then recursively calling Import on the result. Writing a good webcrawler, however, that performs large crawls, correctly resolves various possible URL forms, and respects the robots exclusion standard would be quite a lot of work. I seriously doubt that you'll be able to get Mathematica to crawl forms, press buttons, and what-not.
On the other hand, good web-crawlers have already been written - why not use one of those? If you want to do your analysis in Mathematica, simply dump the results and read them in. Here's a webpage that indicates how to do this with an open-source webcrawler called Nutch:
http://facstaff.unca.edu/mcmcclur/class/Seminar/Pagerank/nutch/
Importworks). The question is very general in this form, and I'd be inclined to say Mathematica is not the right tool for this (you'll end up using JLink or .NETLink anyway). But if you can give a very specific example, we can think about how to implement it in Mathematica (or will be able to say with more confidence that it's not possible without external libraries) – Szabolcs Feb 15 '12 at 10:32wget. You can include it in Mma code by usingRun. About to go to bed but do a search cause posted something here in answer to another question a couple of weeks ago. ...here it is: mathematica.stackexchange.com/questions/1186/… – Mike Honeychurch Feb 15 '12 at 11:26curl: stackoverflow.com/a/6977128/695132 – Szabolcs Feb 15 '12 at 12:08