f$ ddlZddlZddlZdgZejddZGddZGddZGdd Z dS) NRobotFileParser RequestRatezrequests secondscZeZdZ ddZdZdZdZdZdZdZ d Z d Z d Z d Z d ZdS)rcg|_g|_d|_d|_d|_||d|_dS)NFr)entriessitemaps default_entry disallow_all allow_allset_url last_checkedselfurls 9/opt/alt/python311/lib64/python3.11/urllib/robotparser.py__init__zRobotFileParser.__init__sG  !! Sc |jSN)rrs rmtimezRobotFileParser.mtime%s   rcB ddl}||_dS)Nr)timer)rrs rmodifiedzRobotFileParser.modified.s(    IIKKrc~ ||_tj|dd\|_|_dS)N)rurllibparseurlparsehostpathrs rr zRobotFileParser.set_url6s7:%|44S99!A#> 4999rc tj|j}|}||ddS#tjj $rK}|j dvrd|_ n)|j dkr|j dkrd|_ Yd}~dSYd}~dSYd}~dSYd}~dSd}~wwxYw)Nzutf-8)iiTii) rrequesturlopenrreadr decode splitlineserror HTTPErrorcoder r )rfrawerrs rr'zRobotFileParser.read;sB 9&&tx00A&&((C JJszz'**5577 8 8 8 8 8|% & & &x:%%$(!!SSX^^!%!"!!!!!%3^^^^^ &s$A77C .CCcpd|jvr|j ||_dSdS|j|dSN*) useragentsr rappend)rentrys r _add_entryzRobotFileParser._add_entryHsM %" " "!)%*"""*) L   & & & & &rc> d}t}||D]V}|sB|dkrt}d}n+|dkr%||t}d}|d}|dkr |d|}|}|s|dd}t |dkr|d|d<tj |d|d<|ddkrM|dkr#||t}|j |dd}o|ddkr8|dkr0|j t|ddd}|dd kr8|dkr0|j t|dd d}|dd krP|dkrH|drt!|d|_d}S|dd kr|dkr|dd }t |dkr|drg|dr;t%t!|dt!|d|_d}*|ddkr |j |dX|dkr||dSdS)Nrr#:z user-agentdisallowFallowTz crawl-delayz request-rate/sitemap)Entryrr6findstripsplitlenlowerrr unquoter3r4 rulelinesRuleLineisdigitintdelayrreq_rater )rlinesstater5lineinumberss rr zRobotFileParser.parseQsO  7 27 2D A::!GGEEEaZZOOE***!GGEE #AAvvBQBx::< 4   5\**6<+?+?+D+DEE l%%r"Z_  j. 0C'EFFl  %% C\ , ,E ** ,s+++++ ,   5%//44 4trc|sdS|jD] }||r |jcS!|jr |jjSdSr)rrrWrJr rrYr5s r crawl_delayzRobotFileParser.crawl_delaysmzz|| 4\ # #E ** #{""" #   ,%+ +trc|sdS|jD] }||r |jcS!|jr |jjSdSr)rrrWrKr r]s r request_ratezRobotFileParser.request_ratesmzz|| 4\ & &E ** &~%%% &   /%. .trc"|jsdS|jSr)r rs r site_mapszRobotFileParser.site_mapss} 4}rc|j}|j ||jgz}dtt|S)Nz )rr joinmapstr)rrs r__str__zRobotFileParser.__str__s>,   )!3 44G{{3sG,,---rN)r)__name__ __module__ __qualname__rrrr r'r6r r[r^r`rbrgrrrrs !!!(((??? 9 9 9'''G#G#G#R: .....rc"eZdZ dZdZdZdS)rGc|dkr|sd}tjtj|}tj||_||_dS)NrT)rr rRr!rVr#rX)rr#rXs rrzRuleLine.__init__s[ 2::i:I|&&v|'<'9zTADIMMrN)rhrirjrrWrgrkrrrGrGsN1###BBBNNNNNrrGc(eZdZ dZdZdZdZdS)r?c>g|_g|_d|_d|_dSr)r3rFrJrKrs rrzEntry.__init__s"  rc|g}|jD]}|d||j|d|j|j,|j}|d|jd|j|tt|j d |S)Nz User-agent: z Crawl-delay: zRequest-rate: r= ) r3r4rJrKrequestssecondsextendrerfrFrd)rretagentrates rrgz Entry.__str__s_ / /E JJ-e-- . . . . : ! JJ3tz33 4 4 4 = $=D JJF FF FF G G G 3sDN++,,,yy~~rc |dd}|jD]&}|dkrdS|}||vrdS'dS)Nr=rr2TF)rBrDr3)rrYr|s rrWzEntry.applies_tosq@OOC((+1133 _  E||ttKKMME !!tt"urcX |jD] }||r |jcS!dS)NT)rFrWrX)rrprNs rrXzEntry.allowance sF %N & &Dx(( &~%%% &trN)rhrirjrrgrWrXrkrrr?r?sSI      rr?) collections urllib.parserurllib.request__all__ namedtuplerrrGr?rkrrrs   $k$]4FGG ~.~.~.~.~.~.~.~.BNNNNNNNN$((((((((((r