Crawlers (or bots) are used to gather data obtainable on the internet. By utilizing web site navigation menus, and studying inner and external hyperlinks, the bots begin to grasp the context of a page. Of course, the words, photographs, and different knowledge on pages also help search engines like google and yahoo understand what they’re about.