The Spider of Doom
Josh Breckman worked for a company that landed a contract to
develop a content management system for a fairly large government
website. Much of the project involved developing a content management
system so that employees would be able to build and maintain the
ever-changing content for their site.
Because they already had an existing website with a lot of content,
the customer wanted to take the opportunity to reorganize and upload
all the content into the new site before it went live. As you might
imagine, this was a fairly time consuming process. But after a few
months, they had finally put all the content into the system and
opened it up to the Internet.
Things went pretty well for a few days after going live. But, on day
six, things went not-so-well: all of the content on the website had
completely vanished and all pages led to the default “please enter
content” page. Whoops.
[…]
As it turns out, Google’s spider doesn’t use cookies, which means that
it can easily bypass a check for the “isLoggedOn” cookie to be
“false”. It also doesn’t pay attention to Javascript, which would
normally prompt and redirect users who are not logged on. It does,
however, follow every hyperlink on every page it finds, including
those with “Delete Page” in the title.
Whoops.
source: The Spider of Doom
