Crawler

Crawlers are nothing more than programmes created by search engines to automatically index individual web pages. The indexing robots ‘read’ the text content on the site, including its coding, and also move on to further pages through the links contained therein. According to crawler is to create copies of documents by indexing them in search engine or database results.

External service providers, such as tools used in the daily work of SEO specialists, also have their own robots. Such crawlers make it much easier to control the structure of a website and its optimisation.

It is worth knowing that it is possible to block access of specific robots to a website or some of its resources. This can be done by appropriately modifying the robots.txt file. If you do not want your website to be indexed, its source code should contain the instruction noindex.

Back to glossary