Web crawler

A Web crawler is a computer program that browses the World Wide Web in a methodical, automated manner. Other terms for Web crawlers are ants, automatic indexers, bots, and worms or Web spider, Web robot, or—especially in the FOAF community—Web scutter. This process is called Web crawling or spidering. Many sites, in particular search engines, use spidering as a means of...
Read article at Wikipedia
Edit and Show details

Add or delete facts, expose empty fields, download data in JSON or RDF formats, and explore topic metadata.

Freebase Logo
What is Freebase?

Freebase is a huge collection of facts, built by people like you. Freebase connects facts in ways other sites can't, giving you new ways to explore millions of subjects.
You can help improve it!

Flag this Topic
Why do you want to flag this topic?
Freebase Attribution

Freebase data is free for use under the CC-BY license.

The original description for this topic was automatically generated from the Wikipedia article "Web crawler" licensed under the GNU Free Documentation License .
Learn more about Freebase licensing and attribution