Crawlar yanar gizo shiri ne wanda ke bincika da tsarin shafukan yanar gizo da kuma yin jeri ta hanyar bin hanyoyi daga shafi zuwa shafi, yawanci ana amfani da shi don gano da kuma yin jeri abun ciki a cikin shafin ko a fadin yanar gizo. Injunan bincike suna amfani da crawlers don gina jerin su, kuma kamfanonin AI suna kara amfani da crawlers don tattara bayanan horo da kuma dawo da bayanai.

Crawlers suna bambanta daga masu tsinkaye na musamman a cikin fadi: mai tsinkaye yawanci yana fitar da bayanai na musamman daga shafukan da aka sani, yayin da aikin crawlers shine gano, yana nemo shafuka da ba su riga sun san su ba ta hanyar bin hanyoyi zuwa waje.

Crawlar mai kyau yana duba fayil ɗin robots.txt na shafin kafin ya fara crawling kuma yana tsallake duk hanyoyin da aka hana, duk da haka RFC 9309 yana bayyana cewa waɗannan dokokin ba su da wani nau'i na izinin shiga kuma ba su maye gurbin ainihin matakan tsaro.

Kalmar da ta shafi