The bots are beginning to behave like they should. To begin with they had a malfunction regarding the robots.txt. If the server was busy or a timeout occurred it simply disregarded it – big mistake! Now you can control the crawler using normal robots directives.