Simple fact of The Google Economy: people can’t find stuff if it’s not indexed in major search engines. A slow site might not seem as bad as blocking the crawlers that search engines use to index your content, but it does seriously affect the depth and frequency of crawling they do.
The above is Google’s report of their crawling activity on a site I’ve been trying to optimize server performance on. Comparing the response time in the bottom graph to the number of pages crawled at the top offers some clear evidence of the relationship between the two. Most people seem to think server performance only matters if your site gets Dugg (old people say Slashdotted). Now even I’ll think again.