Ran into this at a customer's site. You untar the Drupal stuff into it's directory, but when you surf to the site, you get an error along the lines of:
The requested URL /cgi-bin/install.php was not found on this server.
Although there are a lot of results when you search the internet for this exact error, I thought I'd add it here, for my own reference.
The solution is setting $base_url in sites/default/settings.php. Very simple. Credits go to misterfancypants at the Drupal forum.