2c6r D d Z ddlZddlZddlZddlZddlZddlZddlZddlZ ddl
Z ddl Z ddl
mZ ddlmZmZmZmZmZmZmZmZ ddlmZmZ ddlm Z ddl m Z m Z ddl m!Z! dd l"m#Z# dd
l$m%Z% dd l&m'Z' dd l(m)Z) dd
l*m+Z+ ddl,m-Z-m.Z. ddl/m0Z0 ddl1m2Z2m3Z3m4Z4 ej5 e6 Z7e j8 j9 j: Z;ee G d de? Z@de ddfdZA G d de? ZBdeZV G d? d@e ZW G dA dB ZXdS )EzO
The main purpose of this module is to expose LinkCollector.collect_sources().
N)Values)CallableIterableListMutableMapping
NamedTupleOptionalSequenceUnion)html5librequests)Response)
RetryErrorSSLError)NetworkConnectionError)Link) SearchScope)
PipSession)raise_for_status)is_archive_file)pairwiseredact_auth_from_url)vcs )CandidatesFromPage
LinkSource build_sourceurlreturnc t j D ]D}| | r| t | dv r|c S EdS )zgLook for VCS schemes in the URL.
Returns the matched VCS scheme, or None if there's no match.
z+:N)r schemeslower
startswithlen)r schemes /builddir/build/BUILDROOT/alt-python311-pip-21.3.1-4.el9.x86_64/opt/alt/python311/lib/python3.11/site-packages/pip/_internal/index/collector.py_match_vcs_schemer' 0 sW
+ 99;; ! !& ) ) c#f++.>$.F.FMMM 4 c , e Zd Zdededdf fd Z xZS )_NotHTML content_type request_descr Nc h t || || _ || _ d S N)super__init__r+ r, )selfr+ r, __class__s r& r0 z_NotHTML.__init__< s3
|444 ( (r( )__name__
__module__ __qualname__strr0
__classcell__)r2 s @r& r* r* ; sR )S ) ) ) ) ) ) ) ) ) ) ) )r( r* responsec | j dd }| d st || j j dS )zCheck the Content-Type header to ensure the response contains HTML.
Raises `_NotHTML` if the content type is not text/html.
Content-Type text/htmlN)headersgetr" r# r* requestmethod)r8 r+ s r& _ensure_html_headerrA B s^
#'';;L * *; 7 7 >|X%5%<===> >r( c e Zd ZdS )_NotHTTPN)r3 r4 r5 r( r& rC rC L s Dr( rC sessionc t j | \ }}}}}|dvrt | | d }t | t
| dS )zSend a HEAD request to the URL, and ensure the response contains HTML.
Raises `_NotHTTP` if the URL is not available for a HEAD request, or
`_NotHTML` if the content type is not text/html.
> httphttpsT)allow_redirectsN)urllibparseurlsplitrC headr rA )r rE r% netlocpathqueryfragmentresps r& _ensure_html_responserS P sq -3L,A,A#,F,F)FFD%
&&&jj <<T< 2 2DT r( c * t t | j rt | | t dt
| | | ddd }t | t | |S )a Access an HTML page with GET, and return the response.
This consists of three parts:
1. If the URL looks suspiciously like an archive, send a HEAD first to
check the Content-Type is HTML, to avoid downloading a large file.
Raise `_NotHTTP` if the content type cannot be determined, or
`_NotHTML` if it is not HTML.
2. Actually perform the request. Raise HTTP exceptions on network failures.
3. Check the Content-Type header to make sure we got HTML, and raise
`_NotHTML` otherwise.
rE zGetting page %sr< z max-age=0)Acceptz
Cache-Control)r= )
r r filenamerS loggerdebugr r> r rA )r rE rR s r& _get_html_responserZ ` s tCyy)** 4 c73333
LL"$8$=$=>>> ;; ! )
D( T Kr( r= c d | r-d| v r)t j | d \ }}d|v r|d S dS )z=Determine if we have any encoding information in our headers.r: charsetN)cgi parse_header)r= r+ paramss r& _get_encoding_from_headersr` sI %>W,, " /0G H H f )$ $ 4r( documentpage_urlc l | d D ] }| d }||c S |S )a Determine the HTML document's base URL.
This looks for a ```` tag in the HTML document. If present, its href
attribute denotes the base URL of anchor tags in the document. If there is
no such tag (or if it does not have a valid href attribute), the HTML
file's URL is used as the base URL.
:param document: An HTML document representation. The current
implementation expects the result of ``html5lib.parse()``.
:param page_url: The URL of the HTML document.
z.//basehref)findallr> )ra rb baserd s r& _determine_base_urlrg sK ++ xx KKK Or( partc z t j t j | S )zP
Clean a "part" of a URL path (i.e. after splitting on "@" characters).
)rJ rK quoteunquoterh s r& _clean_url_path_partrm s*
< fl 2 24 8 8 9 99r( c z t j t j | S )z
Clean the first part of a URL path that corresponds to a local
filesystem path (i.e. the first part after splitting on "@" characters).
)rJ r? pathname2url url2pathnamerl s r& _clean_file_url_pathrq s* > & &v~'B'B4'H'H I IIr( z(@|%2F)rO
is_local_pathc b |rt }nt }t | }g }t t j |dg D ]J\ }}| || | | Kd | S )z*
Clean the path portion of a URL.
r; )
rq rm _reserved_chars_resplitr itertoolschainappendupperjoin)rO rr
clean_funcparts
cleaned_partsto_cleanreserveds r& _clean_url_pathr s *)
)
$ $T * *EM &yurd'C'C D D / /( ZZ 1 1222 X^^ - -....
77= ! !!r( c t j | }|j }t |j | }t j | | S )z
Make sure a link is fully quoted.
For example, if ' ' occurs in the URL, it will be replaced with "%20",
and without double-quoting other characters.
)rr )rO )rJ rK urlparserN r rO
urlunparse_replace)r resultrr rO s r& _clean_linkr s[ \
"
"3
'
'F
%M 6;m D D DD < " "6???#=#= > >>r( anchorbase_urlc | d }|sdS t t j || }| d }| d }t |||| }|S )zJ
Convert an anchor element in a simple repository page to a Link.
rd Nzdata-requires-pythonz data-yanked)
comes_fromrequires_python
yanked_reason)r> r rJ rK urljoinr )r rb r rd r pyrequirer links r& _create_link_from_elementr s ::f D t
fl**8T::
;
;C
122IJJ}--M !# D Kr( c 2 e Zd Zd dZdedefdZdefdZdS )
CacheablePageContentpageHTMLPager Nc & |j sJ || _ d S r. )cache_link_parsingr )r1 r s r& r0 z CacheablePageContent.__init__ s &&&& r( otherc p t |t | o| j j |j j k S r. )
isinstancetyper r )r1 r s r& __eq__zCacheablePageContent.__eq__ s* %d,,P %*.1PPr( c 4 t | j j S r. )hashr r r1 s r& __hash__z CacheablePageContent.__hash__ s DIM"""r( )r r r N) r3 r4 r5 r0 objectboolr intr rD r( r& r r sk QF Qt Q Q Q Q## # # # # # #r( r fnr c t j d dt dt t f fd t j dddt t f fd }|S ) z
Given a function that parses an Iterable[Link] from an HTMLPage, cache the
function's result (keyed by CacheablePageContent), unless the HTMLPage
`page` has `page.cache_link_parsing == False`.
N)maxsizecacheable_pager c > t | j S r. )listr )r r s r& wrapperz'with_cached_html_pages..wrapper s BB~*++,,,r( r r c r | j r t | S t | S r. )r r r )r r r s r& wrapper_wrapperz/with_cached_html_pages..wrapper_wrapper s; " 77/5566 6BBtHH~~ r( ) functools lru_cacher r r wraps)r r r s ` @r& with_cached_html_pagesr s &&&- 4 -d - - - - - '&- _R j T$Z
r( r c # K t j | j | j d }| j }t || }| d D ]}t ||| }||V dS )zP
Parse an HTML document, and yield its anchor elements as Link objects.
F)transport_encodingnamespaceHTMLElementsz.//a)rb r N)r rK contentencodingr rg re r )r ra r r r r s r& parse_linksr ( s
~ = # H (C"8S11H""6** (
<
r( c J e Zd ZdZ d dedee dededdf
d Zdefd
Z dS ) r z'Represents one page, along with its URLTr r r r r Nc > || _ || _ || _ || _ dS )am
:param encoding: the encoding to decode the given content.
:param url: the URL from which the HTML was downloaded.
:param cache_link_parsing: whether links parsed from this page's url
should be cached. PyPI index urls should
have this set to False, for example.
N)r r r r )r1 r r r r s r& r0 zHTMLPage.__init__C s'
"4 r( c * t | j S r. )r r r s r& __str__zHTMLPage.__str__V s #DH---r( T)
r3 r4 r5 __doc__bytesr r6 r r0 r rD r( r& r r @ s 11 $( 5 55 3-5 5
! 5
5 5 5 5&. . . . . . .r( r reasonmeth).Nc < | t j } |d| | d S )Nz%Could not fetch URL %s: %s - skipping)rX rY )r r r s r& _handle_get_page_failr Z s+
||D 0$?????r( Tr c d t | j }t | j || j | S )N)r r r )r` r= r r r )r8 r r s r& _make_html_pager d s: )(*:;;H L- r( c |t d | j dd d }t | }|r t d|| d S t j | \ }}}}}}|dk rt j
t j
| rU| d s|dz
}t j |d }t d | t# ||
}t% || j S # t( $ r t d | Y n
t* $ r1}t d
| |j |j Y d }~nd }~wt0 $ r}t3 | | Y d }~nd }~wt4 $ r}t3 | | Y d }~nd }~wt6 $ r:}d} | t9 | z
} t3 | | t j Y d }~nRd }~wt< j $ r }t3 | d| Y d }~n(d }~wt< j $ r t3 | d Y nw xY wd S )Nz?_get_html_page() missing 1 required keyword argument: 'session'#r r zICannot look at %s URL %s because it does not support lookup as web pages.file/z
index.htmlz# file: URL is directory, getting %srU )r z`Skipping page %s because it looks like an archive, and cannot be checked by a HTTP HEAD request.ziSkipping page %s because the %s request got Content-Type: %s.The only supported Content-Type is text/htmlz4There was a problem confirming the ssl certificate: )r zconnection error: z timed out)! TypeErrorr ru r' rX warningrJ rK r osrO isdirr? rp endswithr rY rZ r r rC r* r, r+ r r r r r6 infor
ConnectionErrorTimeout)
r rE r
vcs_schemer% _rO rR excr s
r& _get_html_pager n s M
(..a
#C #3''J W
t &| 4 4S 9 9 FAtQ1
BGMM&.*E*Ed*K*K L L ||C 3JCl""3 55 :C@@@ Q!#w7778 t8OPPPP7
1
;
" ) ) ) dC(((((((( ) ) ) dC(((((((( > > >G#c(( dF========= # @ @ @ d$>$>$>???????? 1 1 1 dK000001 4sT E %I9/ I98'F$$
I91G
I9G((
I950H**I9<I"I98I9c T e Zd ZU eee ed< eee ed< dS )CollectedSources
find_links
index_urlsN)r3 r4 r5 r
r r __annotations__rD r( r& r r s> *-....*-......r( r c
e Zd ZdZdededdfdZe ddeded e dd fd
Z
e de e
fd Zd edee fd
Zde
dedefdZdS )
LinkCollectorz
Responsible for collecting Link objects from all configured locations,
making network requests as needed.
The class's main method is its collect_sources() method.
rE search_scoper Nc " || _ || _ d S r. )r rE )r1 rE r s r& r0 zLinkCollector.__init__ s
) r( Foptionssuppress_no_indexc |j g|j z }|j r<|s:t dd d |D g }|j pg }t j || }t || }|S )z
:param session: The Session to use to make requests.
:param suppress_no_index: Whether to ignore the --no-index option
when constructing the SearchScope object.
zIgnoring indexes: %s,c 3 4 K | ]}t | V d S r. )r ).0r s r& z'LinkCollector.create.. s+ IIs-c22IIIIIIr( r r )rE r )
index_urlextra_index_urlsno_indexrX rY rz r r creater )clsrE r r r r r link_collectors r& r zLinkCollector.create s '(7+CC
$5 LL&IIjIIIII
J '-2
")!!
'%
r( c | j j S r. )r r r s r& r zLinkCollector.find_links s ++r( locationc . t || j S )z>
Fetch an HTML page containing package links.
rU )r rE )r1 r s r&
fetch_pagezLinkCollector.fetch_page s h ====r( project_namecandidates_from_pagec t t j fd j | D }t j fd j D }t t j red t j || D }t | d| dg|z }t
d | t t! | t! | S )Nc 3 T K | ]"}t |j j d d V #dS )Fr page_validator
expand_dirr Nr rE is_secure_originr locr r1 s r& r z0LinkCollector.collect_sources.. s\ 4
4
%9 #| < #(
4
4
4
4
4
4
r( c 3 T K | ]"}t |j j d d V #dS )Tr Nr r s r& r z0LinkCollector.collect_sources.. s\ 5
5
%9 #| < #'
5
5
5
5
5
5
r( c 4 g | ]}||j d|j S )Nz* )r )r ss r&
z1LinkCollector.collect_sources..
s6 =QV%7 QV
%7%7%7r( z' location(s) to search for versions of :
r ) collections OrderedDictr get_index_urls_locationsvaluesr rX isEnabledForloggingDEBUGrv rw r$ rY rz r r )r1 r r index_url_sourcesfind_links_sourcesliness ` ` r& collect_sourcesz LinkCollector.collect_sources s ( 3 4
4
4
4
4
(AA,OO 4
4
4
&(( ) 4 5
5
5
5
5
5
5
5
&(( w} - -
+ " );=NOO E u:: 3 3#/3 3 3 E
LL5)) * * * .// -..
r( )F)r3 r4 r5 r r r r0 classmethodr r r propertyr r6 r r r r r r r r rD r( r& r r s! "
#(
[ @ ,DI , , , X,>4 >HX,> > > > > *
*
1*
*
*
*
*
*
*
r( r r. r )Yr r] r r rv r r re urllib.parserJ urllib.requestxml.etree.ElementTreexmloptparser typingr r r r r r r
r pip._vendorr r
pip._vendor.requestsr pip._vendor.requests.exceptionsr r pip._internal.exceptionsr pip._internal.models.linkr !pip._internal.models.search_scoper pip._internal.network.sessionr pip._internal.network.utilsr pip._internal.utils.filetypesr pip._internal.utils.miscr r pip._internal.vcsr sourcesr r r getLoggerr3 rX etree ElementTreeElement HTMLElementr6 ResponseHeadersr' Exceptionr* rA rC rS rZ r` rg rm rq compile
IGNORECASErt r r r r r r r r r r r r r rD r( r& r( s9
+ * * * * * * * ) ) ) ) ) ) @ @ @ @ @ @ @ @ ; ; ; ; ; ; * * * * * * 9 9 9 9 9 9 4 4 4 4 4 4 8 8 8 8 8 8 9 9 9 9 9 9 C C C C C C C C ! ! ! ! ! ! A A A A A A A A A A 8 $ $i#+ c*3 8C= ) ) ) ) )y ) ) )>( >t > > > > y
s
Z
D
/C /* / / / / /d HSM + &:s :s : : : :
Js
Js
J
J
J
J RZ 2=99 "# "d "s " " " ", ?S ?S ? ? ? ? d^ 4 # # # # # # # #* x~-.
zlDJ&' , j Xd^ .. . . . . . . .: +/@ @
@ #y. !@ 8I&
'@
@ @ @ @ h D H 15< <
<!*-<
j< < < <~/ / / / /z / / /
f
f
f
f
f
f
f
f
f
f
r(