The bots are beginning to behave like they should. To begin with they had a malfunction regarding the robots.txt. If the server was busy or a timeout occurred it simply disregarded it – big mistake! Now you can control the crawler using normal robots directives.

The system has been heavily updated on the hardware level. Now it should be able to handle more languages and more data. Right now, there are over 54 million pages in the system. All are Danish.

The project is still running in private mode. But, you can try a public beta version of the system at Zooka

NB: Right now, it is running in Danish only. More languages will be added soon.