Oՠp d Z ddlZddlZddlZdgZ ej dd Z G d d Z G d d Z G d d
Z
dS ) a% robotparser.py
Copyright (C) 2000 Bastian Kleineidam
You can choose between two licenses when using this package:
1) GNU GPLv2
2) PSF license for Python 2.2
The robots.txt Exclusion Protocol is implemented as specified in
http://www.robotstxt.org/norobots-rfc.txt
NRobotFileParser RequestRatezrequests secondsc \ e Zd ZdZddZd Zd Zd Zd Zd Z d Z
d
Z d Z d Z
d
Zd ZdS )r zs This class provides a set of methods to read, parse and answer
questions about a single robots.txt file.
c g | _ g | _ d | _ d| _ d| _ | | d| _ d S )NFr )entriessitemaps
default_entry disallow_all allow_allset_url last_checkedselfurls 9/opt/alt/python311/lib64/python3.11/urllib/robotparser.py__init__zRobotFileParser.__init__ sG
! ! S c | j S )zReturns the time the robots.txt file was last fetched.
This is useful for long-running web spiders that need to
check for new robots.txt files periodically.
)r r s r mtimezRobotFileParser.mtime% s
r c @ ddl }| | _ dS )zYSets the time the robots.txt file was last fetched to the
current time.
r N)timer )r r s r modifiedzRobotFileParser.modified. s#
IIKKr c | || _ t j | dd \ | _ | _ dS )z,Sets the URL referring to a robots.txt file. N)r urllibparseurlparsehostpathr s r r
zRobotFileParser.set_url6 s4 %| 4 4S 9 9!A# > 4999r c t j | j }| }| | d dS # t j j $ rK}|j
dv rd| _ n)|j
dk r|j
dk rd| _ Y d}~dS Y d}~dS Y d}~dS Y d}~dS d}~ww xY w)z4Reads the robots.txt URL and feeds it to the parser.zutf-8)i i Ti i N)
r requesturlopenr readr decode
splitlineserror HTTPErrorcoder r )r frawerrs r r& zRobotFileParser.read; s 9&&tx00A &&((C JJszz'**5577 8 8 8 8 8 |% & & &x:%%$(!!S SX^^!% ! "!!!!!%3^^^^^ &s $A6 6C
.CCc p d|j v r| j || _ d S d S | j | d S N*)
useragentsr
r append)r entrys r
_add_entryzRobotFileParser._add_entryH sM %" " "!)%*""" *)
L & & & & &r c < d}t }| |D ]V}|sB|dk rt }d}n+|dk r%| | t }d}| d }|dk r
|d| }| }|s| dd }t
| dk r|d |d< t j
|d |d< |d dk rM|dk r#| | t }|j |d d}o|d dk r8|dk r0|j
t |d d d}|d d
k r8|dk r0|j
t |d d d}|d d k rP|dk rH|d rt! |d |_ d}S|d d
k r|dk r|d d }t
| dk r|d rg|d r;t% t! |d t! |d |_ d}*|d dk r | j |d X|dk r| | dS dS )zParse the input lines from a robots.txt file.
We allow that a user-agent: line is not preceded by
one or more blank lines.
r r #N:z
user-agentdisallowFallowTz crawl-delayz request-rate/sitemap)Entryr r5 findstripsplitlenlowerr r unquoter2 r3 rulelinesRuleLineisdigitintdelayr req_rater )r linesstater4 lineinumberss r r zRobotFileParser.parseQ sJ
7 2 7 2D
A:: !GGE EEaZZOOE*** !GGE E #AAvvBQBx::< 4
5 \**6<+?+?+D+DEE
l%%r"Z_ j .
0C'E F Fl %% C\ , ,E **
, s+++++
, 5%//44 4tr c | sd S | j D ] }| | r |j c S !| j r | j j S d S N)r r rV rI r
r rX r4 s r crawl_delayzRobotFileParser.crawl_delay sm zz|| 4\ # #E **
# {"""
# ,%+ +tr c | sd S | j D ] }| | r |j c S !| j r | j j S d S r\ )r r rV rJ r
r] s r request_ratez RobotFileParser.request_rate sm zz|| 4\ & &E **
& ~%%%
& /%. .tr c " | j sd S | j S r\ )r r s r site_mapszRobotFileParser.site_maps s } 4} r c | j }| j || j gz }d t t | S )Nz
)r r
joinmapstr)r r s r __str__zRobotFileParser.__str__ s> , ) !3 44G{{3sG,,---r N)r )__name__
__module__ __qualname____doc__r r r r
r& r5 r rZ r^ r` rb rg r r r r s
! ! !( ( (? ? ?
9 9 9' ' 'G# G# G#R :
. . . . .r c $ e Zd ZdZd Zd Zd ZdS )rF zoA rule line is a single "Allow:" (allowance==True) or "Disallow:"
(allowance==False) followed by a path.c |dk r|sd}t j t j | }t j | | _ || _ d S )Nr T)r r rQ r rU r" rW )r r" rW s r r zRuleLine.__init__ s[ 2::i: I|&&v|'<'9zTADIMMr N)rh ri rj rk r rV rg rl r r rF rF sS 1 1# # #B B BN N N N Nr rF c * e Zd ZdZd Zd Zd Zd ZdS )r> z?An entry has one or more user-agents and zero or more rulelinesc > g | _ g | _ d | _ d | _ d S r\ )r2 rE rI rJ r s r r zEntry.__init__ s"
r c | g }| j D ]}| d| | j | d| j | j ,| j }| d|j d|j | t t | j d
| S )Nz User-agent: z
Crawl-delay: zRequest-rate: r<
) r2 r3 rI rJ requestssecondsextendre rf rE rd )r retagentrates r rg z
Entry.__str__ s _ / /E JJ-e-- . . . . : ! JJ3tz33 4 4 4 = $=D JJF
FF FF G G G
3sDN++,,,yy~~ r c | d d }| j D ]&}|dk r dS | }||v r dS 'dS )z2check if this entry applies to the specified agentr< r r1 TF)rA rC r2 )r rX r} s r rV zEntry.applies_to sp OOC((+1133 _ E||ttKKMME !!tt "ur c V | j D ] }| | r |j c S !dS )zZPreconditions:
- our agent applies to this entry
- filename is URL decodedT)rE rV rW )r rq rM s r rW zEntry.allowance
sA N & &Dx((
&~%%%
&tr N)rh ri rj rk r rg rV rW rl r r r> r> sV II
r r> ) rk collections urllib.parser urllib.request__all__
namedtupler r rF r> rl r r r s
$k$]4FGG ~. ~. ~. ~. ~. ~. ~. ~.BN N N N N N N N$( ( ( ( ( ( ( ( ( (r