403, 429, me ngā Hē CAPTCHA: Me Pehea te Tirotiro i ngā Pūkete Proxy

I tōna wā, i te haere ngā mahi a tō crawler. I tēnei wā, kei te titiro koe ki ngā 403, 429, me ngā CAPTCHAs e kore e mutu. Ko ia tono i whakakorehia e piki ana i ngā utu, e tango ana i ngā wā, ā, e pokanoa ana i ngā KPI. Ko tēnei aratohu he hīkoi whaihua mō te rapu raru i ngā aukati proxy, i te mea ka taea e koe te whakahoki i te tere me te kounga raraunga i te tere.

Ko ngā mea ka whiwhi koe: he pukapuka mārama hei tirotiro i ngā hapa, hei mapi i ngā take taketake, hei whiriwhiri i te pēhanga IP tika, me te tirotiro i ngā hua me ngā tohu whakaputa.

Mēnā kei te whiwhi koe i ngā whakautu 403, 429, rānei CAPTCHA, tuatahi tirohia mēnā ko te aukati he pā ana ki te IP, te whanonga, rānei te pēhanga. Me te ine i te tere tono me te pākiki, whakamātauria he wāhanga ma, whakarereke i ngā upoko hei whakakotahi ki ngā pūtirotiro tūturu, ā, whakamātauria ngā momo IP rerekē (rūma noho vs pokanga raraunga). Whakaritehia te maha o ngā tono, tāpiri i te pākiki, me te pākiki kaha, me te mau tonu i ngā wāhanga. Whakamātauria ngā whakatikatika me te reiti aukati me te reiti angitu pātea.

Tirohia ngā tohu: 403 vs 429 vs CAPTCHA

• Ko te 403 Forbidden e tohu ana kāore te tūmau e whakaae ki te uru. Ko ngā take noa ko ngā rārangi IP i aukatihia, ngā geo i aukatihia, te hātepe takiuru, rānei ngā pēhanga bot.
• Ko te 429 Too Many Requests he whakatūpato mō te reiti. I pā te pākiki, te maha o ngā tono, ki ngā taumata i te IP, i te wāhanga rānei.
• Ko te CAPTCHA he wero whakauka tangata. Ka whakaohohia tēnei i te nuinga o te wā i muri i ngā tauira whanonga, i ngā pēhanga e tohu ana i te aunoa.

He aha tēnei e pā ana: e tohu ana ia tohu ki tētahi ara whakatikatika rerekē. Ko te whakakotahi i ngā rongoā ka pau te wā. Ka tere ake te whakahoki mēnā ka whakakotahi koe i te whānau hapa ki te take pea, ā, ka whakamātauria ngā whakatikatika i roto i ngā whakamātautau iti, e whakahaerehia ana.

Mapihia ngā aukati ki tō whakamahinga

Kaore ngā paetukutuku e aukati i te katoa i te ara kotahi. Ko tētahi bot tirotiro utu, he kaiwhakahaere SERP haerenga, me te kaiwhakamahi kāri kua takiuru ka pā ki ngā tiaki rerekē. Mapihia ō ngā rerenga me ngā momo ihirangi hei whakarite i ngā whakatikatika ki ngā tauira kaiwhakamahi tūturu.

Mō te tirohanga whānui ki te pehea e hanga ai ngā tīma i ngā rerenga tango i runga i te whāinga, tirohia ngā whakamahinga proxy noa; ka awhina ēnei ki te whakarite i te rautaki IP, te tere, me te hoahoa wāhanga ki ngā putanga pakihi. Tirohia ēnei tauira o ngā whakamahinga proxy noa.

Te Rapu Raru i ngā Aukati Proxy: He Pukapuka Whakaputa

Tīmatahia i ngā mea māmā, kātahi ka haere ki ngā mea hōhonu mēnā ka rerekē ngā whakatau.

1. Whakamārama me te wehewehe:

• Tautuhia te ara e hāngai ana, te tikanga HTTP, me te uiui he tika mai i tētahi pūtirotiro noa.
• Tirohia te tono kotahi me mehemea he proxy, hei whakamōhio mēnā he pā ana te aukati ki te IP.

2. Tākina ngā tohu tika:

• Hopukina ngā waehere āhua, ngā wā e whakahoki ana, ngā upoko tūmau, me ngā huihuinga set-cookie.
• Tākina te tauira tono: ngā tono ia hēkona, te pākiki, me te rōpū i ia rohe.

3. Tirohia te whanonga i mua i te tuakiri:

• Whakaritehia te maha o ngā tono, me tāpiri i ngā pākiki rerekē (jitter) hei kite mēnā ka heke ngā 429/soft CAPTCHA.
• Whakamahia te pākiki (ETag/If-None-Match, If-Modified-Since) hei whakaiti i ngā pāhitanga takirua.

4. Whakaritehia tō pēhanga kiritaki:

• Whakamahia tētahi pūtirotiro tūturu, he kōtahi-kore-ā-ringa me ngā upoko pūmau me ngā whakaaetanga e whakaaetia ana.
• Purihia ngā kuki me ngā whare rokiroki i ia wāhanga. Whakaritehia ngā kaiwhakamahi i ngā wā kāore e taea; ka taea te tākaro i te āhua rerekē.

5. Whakamātauria ngā whakaritenga IP me te geo:

• Tirohia tētahi rōpū iti me tētahi ASN rerekē, rānei te momo IP.
• Whakamātauria te tika o te geo mēnā ka whakaritea te paetukutuku, e hāngai ana rānei ki te rohe.

6. Whakaritehia ngā whakamātautau iti:

• Whakaritehia tētahi rerekētanga kotahi i te wā, ā, whakahaerehia ngā tono 100–500.
• Tirohia ngā pātea e rua: te reiti aukati me te reiti angitu pātea (CPSR). CPSR = (ngā whārangi angitu kāore he pākiki) / (ngā whakamātau katoa). I ngā kupu māmā: te pēhea e pā ana te whārangi e hiahia ana koe kāore he pākiki.

Ngā whāinga tauira hei whakamātautau i tētahi whakamātautau:

• Te reiti aukati i raro i te 5–10% i ngā whārangi kāwai.
• CPSR i runga i te 85% i ngā ihirangi tūmatanui.
• Te pumau o te wāhanga i runga i te 30 meneti mō ngā rerenga takiuru.

7. Whakaritehia te whakatikatika:

• Whakaritehia ngā whakaritenga tere, te mau tonu o ngā wāhanga, me te whakamātau/whakatūpato ki tō kiritaki.
• Rūhia ngā IP e mōhiotia ana hei wera me ngā kuki wāhanga mō ngā ara nui ake.

Mēnā ka piki ngā hapa 403, 429, me CAPTCHA ahakoa i ngā tere iti, tērā pea ko te rongoā o tō IP, te ingoa ASN rānei e pā ana. Ko te tapanga IP e pā ana ki te wahi i ahu mai ai ngā IP me te ahua o ngā IP i runga i te ipurangi. He mea nui tēnei i te wā e rapu ana i ngā whāinga uaua.

• Ko ngā IP kāinga e ahu mai ana i ngā ISP kaihoko. Ka piri rātou ki ngā rerenga kaiwhakamahi noa, ā, ka taea te pā te WAF kaha me ngā tirohanga geo. He nui ake te utu, ā, ka taea te noho pōturi, engari ka whakaheke i ngā aukati kaha i ngā paetukutuku e pā ana ki ngā kaihoko.
• Ko ngā IP pūkoro e mahi ana i te ahua o ngā rerenga whatunga pūkoro, ā, ka taea te awhina i te wā kāore e taea e ngā kāinga. He nui ake te utu, ā, he uaua ki te whakahaere.
• Ko ngā IP pokanga he tere, he utu whaihua. Ka pai rātou ki ngā ihirangi kāore e tiakina, engari he māmā ki te tango i ngā tapanga me te aukati.

Mēnā e whakaaro ana koe he kaha ngā ture WAF, he pākiki hoki te whakaritenga geo, whakaarohia te whakamātautau i tētahi rōpū iti mā ngā proxy kāinga i mua i te whakahou i tō scraper. Whakamahia rātou i ngā wā e nui ake ai te kounga me te uru i te mea he nui ake te mea i te tere.

Whakaritehia te tere me te pāhekoheko hei whakaiti i ngā 429

Ko ngā 429 he pā ana ki te pēhanga, kāore ki te tuakiri. Ko te rongoā ko te whakakāhoretanga i tō rerenga kia uru ki ngā pānga e kitea ana e te paetukutuku.

• Whakaritehia ngā pānga pāhekoheko i runga i te IP. Tīmatahia me te 1–3 ngā tono pāhekoheko i ia rohe, ā, whakatūpato.
• Tāpiritia te hokinga ā-ringa i muri i ngā 429, i ngā CAPTCHA māmā (hei tauira, 30–120 hēkona), ā, tāpiri i te hēkona rārangi.
• Whakaritehia te uta i ngā wā e pā ana ki te wā me te whakahirahira i ngā wā e pā ana ki ngā pooti.
• Whakaora i ngā raraunga me te whakakore i ngā URL hei karo i ngā tono rorirori.

I te wā e pākiki ana te whāinga, ā, ko tō bottleneck ko te tere, ka taea e ngā IP pokanga te tuku tere i te rahi. Whakamahia he huarahi rerekē e whakahaere ana i ngā rawa pūmau, i ngā whārangi kāore e pā ana ki ngā mōhiohio i runga i ngā proxy pokanga i te wā e pupuri ana ngā pūnaha ngoikore i ngā IP kaha.

Ngā taputapu me te tirohanga ka taea e koe te whakawhirinaki

Kaore e taea e koe te whakatika i ngā mea kāore e taea e koe te kite. Tāpiritia he rongoā pūkoro ki te iti te pānga, ā, tirohia i ia rohe.

• Ngā pānga matua: te reiti aukati e pā ana ki te whānau waehere (403/429/CAPTCHA), CPSR, te wa toharite ki te pātuhi tuatahi, te roa o te huihuinga, me te tika geo.
• Ngā whakairoiro: ngā upoko tono/whakautu katoa mō ngā tauira, te momo wero CAPTCHA, me ngā ID whakahe e wātea ana i te wā e wātea ana.
• Te whakatū i ngā pānga: whakatūhia i te wā e te reiti aukati > X% rānei ko te CPSR < Y% mō te nui atu i te Z meneti.

Mō ngā tauira reo motuhake me ngā tauira hono, tirohia ngā tuhinga kaiwhakawhanake mō te whakauru proxy me te whakarite ki tō pūnaha (ngā tono, Playwright, Puppeteer, curl, rānei ngā kaihoko HTTP ritenga).

Ngā take matua me ngā whakatikatika whaihua e pā ana ki te tohu

403 Forbidden: ngā aukati tuakiri, kaupapa

Ngā whakaoho noa:

• Ngā rongoā IP, ngā aukati ASN.
• Ngā aukati geo, ngā upoko kāore e wātea ana.
• Ngā ihirangi e hiahia ana ki te rehita me te whakahaere huihuinga tika.
• Ngā tapanga bot: te raupapa upoko rerekē, ngā tohu TLS, rānei ngā upoko whakaae kāore e rite ana.

Ngā whakatikatika hei whakamātautau:

• Huri i te momo IP/ASN me te whakarite i te geo ki te rohe e hiahiatia ana.
• Whakaritehia ngā huihuinga me te whakahoki i ngā pooti; kāore e taea te tango i ngā whārangi e tiakina ana.
• Whakaritehia ngā upoko me te whakamahi i tētahi kaiwhakamahi hou, e mau ana.
• Whakatika i ngā whārangi me tētahi pūtirotiro ā-hanga i te wā e whakawhirinaki ana te ihirangi ki te JS.

429 Too Many Requests: ngā whakahaere reiti me te pīkau

Ngā whakaoho noa:

• Te pāhekoheko teitei mai i tētahi IP, i tētahi huihuinga.
• Ngā tauira pīkau, pēnā i te 20 ngā tono i roto i te 1 hēkona, ā, kātahi ka noho pōturi.

Ngā whakatikatika hei whakamātautau:

• Ngā pānga pāhekoheko i runga i te IP me ngā pākete tohu i ia rohe.
• Te hokinga ā-ringa i muri i ngā whakautu me ngā CAPTCHA.
• Te whakaora me te If-None-Match/If-Modified-Since hei whakaiti i ngā pānga kāore e hiahiatia ana.

CAPTCHA: te whanonga me te tapanga

Ngā whakaoho noa:

• Te haere tere, ngā pooti puka, rānei ngā whakamātautau rehita.
• Ngā kaiwhakamahi rerekē me ngā pooti kāore e wātea ana.
• Ngā tapanga ā-hanga, ā, ngā tapanga aunoa.

Ngā whakatikatika hei whakamātautau:

• Whakaritehia ngā huihuinga pūmau me ngā huarahi haere e pā ana ki te tangata.
• Whakaitihia te tere pātea/whakatika, ā, tāpiritia te wā whakaaro.
• Whakamahia ngā āhua pūtirotiro huna me ngā pūrere/whakatika e wātea ana.
• Mō ngā CAPTCHA kaha e mau ana, whakakotahi i te kounga IP, rānei whakaitihia te pāhekoheko anō.

Kia tūpato i tēnei

• Te whai i ngā whakatikatika kotahi: ko te huri i ngā kaiwhakamahi 100 wa ka kore e whakatika i te 429.
• Te huri i ngā IP: ko ngā IP hou i ia tono ka pātea i ngā rerenga kua rehita.
• Te noho kāore e aro ki te geo: ka 403 te pae i te Iākarania anake i te rerenga heke mai i te rohe kāore e tika.
• Te pānui i ngā upoko kāinga: ko te whak doubling i tō tono ka tono ki ngā here mō te kore painga.
• Te whakakotahi i ngā tauira pūkoro me ngā rorohiko: ko ngā huri taputapu i waenganui i te wā ka āta tirohia.

He mātai tere

Ngā āhuatanga o te ao

Āhuatanga 1: He kaiwhakahaere haerenga e kite ana i ngā 403 i ngā whārangi utu ahakoa te tere iti. Ko te huri ki te pūnaha IP kāinga e hāngai ana ka heke ngā 403, engari ka noho tonu ngā CAPTCHA. Ko te pātea i ngā kuki i ia huarahi me te whakatika i ngā upoko ka whakaiti i ngā wero. Ka piki te CPSR ki runga i te 85% o te whāinga rōpū.

Āhuatanga 2: He kaiwhakamahi eCommerce e pā ana ki ngā whārangi hua me ngā tono 20 i te wā kotahi i ia IP ka inundate i ngā 429. Ka mutu te rōpū ki te 2 i ia IP, tāpiri i te 100–400 ms hihiri, me te whakahohe i te kāinga ETag. Ka heke te reiti aukati ki raro i te 8%, ka noho tonu te pātea i te tohatoha i te uta i runga i ngā IP maha.

Ngā Pātai Ka Pūtā

Q1: Me pēhea e taea ai e au te mōhio mēnā he aukati e pā ana ki te IP, ki te pā ana ki te whanonga? A: Tirohia te tono kotahi me me kāore he pūnaha. Mēnā ka mahi kāore he pūnaha engari kāore e mahi me te pūnaha, he mea pea ko te IP, ko te geo. Mēnā e rua e hē ana i muri i ngā tono tere, he mea pea ko te whanonga, ko te tohu. Whakamahia ngā whakamātautau iti me te huri i te kotahi i te wā.

Q2: Me whakamahi au i ngā IP kāinga, i ngā IP pokanga mō ngā pae e tiakina ana? A: Mō ngā WAF tino, ngā rerenga inihua, he mea pea ka pātea ngā IP kāinga i ngā whakamātautau i ngā tere iti. Mō ngā ara tūmatanui, pūmau, rānei he iti te pānga, he tere, he iti te utu ngā IP pokanga. He maha ngā rōpū e whakakotahi ana i te rua i runga i te pānga o te pito.

Q3: He aha te pātea tika i ia IP hei karo i ngā 429? A: Ka rerekē i runga i te pae. Hei tīmatanga, whakamātauria te 1–3 ngā tono i ia IP i ia rohe me te tāpiri i te hihiri. Piki haere i te āta tirohia te reiti aukati me te CPSR. Whakamātauria ngā here i te whakamātautau i mua i te piki.

Q4: Me pēhea e taea ai e au te whakaiti i ngā CAPTCHA kāore e whakatika i a rātou i te rahi? A: Whakatū i tō wā (ngā kuki, te rokiroki), whakaroa i te tirohanga ki te wā e pā ana ki te tangata, me te whakamahi i te pūtirotiro huna. Mēnā ka noho tonu ngā CAPTCHA i ngā tere iti, whakamātauria he pai ake te tohu IP me te tirohia te geo tika. Whakamahia ngā whakaritenga uaua mō ngā pito matua anake.

Q5: He aha ngā pānga e tino pā ana mō te tirohanga tonu? A: Tirohia te reiti aukati e wehe ana i te 403/429/CAPTCHA, CPSR, te roa o te wā, me te tika o te geo. Tāpiri i ngā pānui mō ngā pikinga i runga i ngā paerewa mō ngā wā e mau tonu ana. Purihia ngā tuhinga tauira o ngā upoko katoa me ngā whārangi wero hei tere te mātai.

Q6: Me pēhea e taea ai e au te pupuri i ngā utu i raro i te whakahaere i te whakapai ake i te uru? A: Whakamahia te kāinga me te whakakore i ngā tono katoa. Whakamahia ngā IP pokanga mō ngā pito e manawanui ana, me te pupuri i ngā IP kāinga, pūkoro mō ngā ara he uaua. Tīpako i te pātea tika i te wā e kore e huri ki te nui atu o ngā IP.

Q7: He raru e pā ana ki te whakakī i ngā pūnaha i muri i ngā pūnaha? A: Ka raru te raru i runga i ngā tikanga, te momo raraunga, me te mana. Mahi ki ngā kaiwhakawā, te here i ngā raraunga e pā ana, me te tuhinga i te whakamahinga e hiahia ana. Whakamahia ngā here reiti me te whakaute i ngā robots me ngā rohe whakaaetanga hei whakatau kaupapa mō tō rōpū.

Ngā Huarahi e whai ake

Ko te mōhio matua he māmā: whakakotahi tō whakatikatika ki te tohu. Ka tohu ngā 403 ki te tuakiri me te kaupapa. Ka tohu ngā 429 ki te pēhanga. Ka noho ngā CAPTCHA i waenganui i te whanonga me te tohu. Ko te whakawhiti ko te tere ki te huna—mēnā ka he te taurite ka piki ngā utu me te kore e pai ake te uru.

Tīmatahia he whakamātautau raru mō te aukati proxy. Whakamātauria tō tapawhā IP, te geo, me te hoahoa wā, kātahi ka whakaritea te rahi o te whakaritenga me te hihiri. Whakamahia te CPSR, te reiti aukati, me te pūmau wā kia taea ai e koe te whakaatu i ngā painga. Mō ngā tauira hōhonu me ngā taipitopito whakatinana, tirohia ngā aratohu me ngā rauemi hangarau e pā ana ki te SquidProxies.