Cikakken mai binciken yanar gizo na AI shine mai bincike da kamfanin AI ke gudanarwa don tattara abun ciki na yanar gizo don horar da samfuran harshe ko kuma don karfafa samun bayanai a lokaci guda, wanda ya sha bamban da mai binciken injin bincike na gargajiya a dalili. Misalai sun hada da GPTBot na OpenAI da ClaudeBot na Anthropic; Google ma yana bayar da Google-Extended, wani token na robots.txt wanda ke ba wa masu wallafa iko akan ko abun ciki da masu binciken Google ke tattarawa na iya amfani da shi don horar da samfuran Gemini.
OpenAI, Anthropic da Perplexity suna gudanar da bots da dama tare da ayyuka daban-daban, don haka "mai binciken AI" yana rufe nau'ikan zirga-zirga guda uku: masu bincike da ke tattara bayanan horo, masu bincike da ke gina jeri don binciken AI, da masu ziyartar shafi saboda mai amfani ya tambayi tambaya.
Nau'ikan Bots na AI Uku
- Masu binciken horo: suna tattara shafuka da za a iya amfani da su don horar da samfuran, kamar GPTBot da ClaudeBot.
- Masu binciken bincike: suna jera shafuka don su iya bayyana a cikin amsoshin binciken AI, kamar OAI-SearchBot, Claude-SearchBot da PerplexityBot. Perplexity na cewa PerplexityBot ba a yi amfani da shi don tattara bayanan horo.
- Masu ziyartar shafi bisa ga umarnin mai amfani: suna loda shafi a cikin lokaci na gaske lokacin da wani ya tambayi mai taimako wani abu, kamar ChatGPT-User, Claude-User da Perplexity-User.
Bots na AI ta Kamfani
| Kamfani | Horo | Bincike | Mai ziyartar bisa ga umarnin |
|---|---|---|---|
| OpenAI | GPTBot | OAI-SearchBot | ChatGPT-User |
| Anthropic | ClaudeBot | Claude-SearchBot | Claude-User |
| Perplexity | (babu wanda aka lissafa) | PerplexityBot | Perplexity-User |
Google yana daukar wata dabara daban. Google-Extended ba shi da mai amfani na daban: yana da token na iko wanda kuma ke rufe tushe, inda abun cikin Binciken Google ke ba da izini ga samfuran Gemini a lokacin tambaya. Google na cewa ba ya shafi shigarwa ko matsayi a cikin Binciken Google.
Hana Horo Ba Ya Dole Ya Hana Binciken AI
Kowane kamfani yana daukar waɗannan bots a matsayin canje-canje masu zaman kansu:
- OpenAI: na cewa kowanne saitin yana da 'yanci, don haka shafin yanar gizo na iya ba da izinin OAI-SearchBot ya bayyana a cikin binciken ChatGPT yayin da yake hana GPTBot.
- Anthropic: na cewa hana Claude-SearchBot ko Claude-User na iya rage ganin shafin a cikin sakamakon binciken mai amfani.
- Masu ziyartar bisa ga umarnin suna bambanta: OpenAI na cewa ka'idojin robots.txt na iya zama ba su shafi ChatGPT-User ba, kuma Perplexity na cewa Perplexity-User yawanci yana watsi da su, saboda mai amfani ya nemi ziyartar. Anthropic na cewa masu mallakar shafin na iya amfani da robots.txt don sarrafa Claude-User.
Ga ƙungiyoyin SEO da ke bibiyar ambaton AI, doka ta gama "hana dukkan bots na AI" na iya cire shafin daga amsoshin AI, ba kawai daga bayanan horo ba.
Yaya Yawan Binciken AI Akwai
Cloudflare ya bayar da rahoto a watan Yuli 2025 cewa rabo na GPTBot na zirga-zirgar mai binciken AI kawai ya tashi daga 5% zuwa 30% tsakanin Mayu 2024 da Mayu 2025, tare da ClaudeBot a 21%. A ranar guda, Cloudflare ya sanar da cewa zai hana masu binciken AI samun abun ciki ba tare da izini ba ta tsohuwa, kuma ya tambayi kowanne sabon yanki ko ya ba su izinin.
Menene Wannan Ke Nufi Ga Cire Bayanai Daga Yanar Gizo
- Sunan bot ba ya tabbatar da komai: kowanne abokin ciniki na iya aika sunan mai binciken AI a cikin User-Agent header, don haka shafukan da ke da mahimmanci suna duba fiye da wannan rubutun.
- Karanta ka'idojin don token naka: robots.txt na iya saita ka'idoji daban-daban ga kowanne mai bincike, don haka duba rukuni da ya shafi mai amfani da kai, ba kawai bots na AI ba.
- Yi tsammanin karin toshe kan zirga-zirgar da ta shafi AI: tare da toshewa ta tsohuwa a kan hanyoyin sadarwa kamar Cloudflare, hanyoyin bayanai don ayyukan AI ya kamata su shirya don bukatun da aka hana da kuma girmama su.