Writing a basic webcrawler is quite easy using constructs like
and then recursively calling
Import on the result. Writing a good webcrawler, however, that performs large crawls, correctly resolves various possible URL forms, and respects the robots exclusion standard would be quite a lot of work. I seriously doubt that you'll be able to get Mathematica to crawl forms, press buttons, and what-not.
On the other hand, good web-crawlers have already been written - why not use one of those? If you want to do your analysis in Mathematica, simply dump the results and read them in. Here's a webpage that indicates how to do this with an open-source webcrawler called Nutch: