ر4z d Z ddlmZ ddlZddlZddlZddlZddlZddl Z ddl
Z
ddl Z ddl
Z ddlmZmZmZ ddlmZ ddlmZ ddlmZ ddlmZmZmZ dd l m Z dd
l m Z dd l m!Z!m"Z" dd l#m$Z$ dd
l%m&Z& ddl'm(Z( ddl)m*Z* ddl+m,Z, ddl-m.Z. ddl/m0Z0 ddl1m2Z2 ddl3m4Z4m5Z5m6Z6 e j7 e8 Z9ee:e:f Z;dIdZ< G d d e= Z>dJd Z? G d! d"e= Z@dKd%ZAdLd&ZBdMd)ZC G d* d+ ZD G d, d-e ZEdNd/ZFeFdOd3 ZG ed45 G d6 d1 ZH G d7 d8e ZI dPdQd?ZJ dRdSdBZKdTdDZL G dE dFe ZM G dG dH ZNdS )UzO
The main purpose of this module is to expose LinkCollector.collect_sources().
) annotationsN)IterableMutableMappingSequence) dataclass)
HTMLParser)Values)Callable
NamedTupleProtocol)requests)Response)
RetryErrorSSLError)NetworkConnectionError)Link) SearchScope)
PipSession)raise_for_status)is_archive_fileredact_auth_from_url)vcs )CandidatesFromPage
LinkSource build_sourceurlstrreturn
str | Nonec t j D ]D}| | r| t | dv r|c S EdS )zgLook for VCS schemes in the URL.
Returns the matched VCS scheme, or None if there's no match.
z+:N)r schemeslower
startswithlen)r schemes o/builddir/build/BUILD/cloudlinux-venv-1.0.10/venv/lib/python3.11/site-packages/pip/_internal/index/collector.py_match_vcs_schemer) . sW
+ 99;; ! !& ) ) c#f++.>$.F.FMMM 4 c e Zd Zd fd Z xZS )_NotAPIContent content_typer request_descr Nonec h t || || _ || _ d S N)super__init__r- r. )selfr- r. __class__s r( r3 z_NotAPIContent.__init__: s3
|444 ( (r* )r- r r. r r r/ )__name__
__module__ __qualname__r3
__classcell__r5 s @r( r, r, 9 s= ) ) ) ) ) ) ) ) ) )r* r, responser r/ c | j dd }| }| d rdS t || j j )z
Check the Content-Type header to ensure the response contains a Simple
API Response.
Raises `_NotAPIContent` if the content type is not a valid content-type.
Content-TypeUnknown)z text/htmlz#application/vnd.pypi.simple.v1+html#application/vnd.pypi.simple.v1+jsonN)headersgetr$ r% r, requestmethod)r; r- content_type_ls r( _ensure_api_headerrE @ sg #'' BBL!''))N
x'7'>
?
??r* c e Zd ZdS )_NotHTTPN)r6 r7 r8 r* r( rG rG V s Dr* rG sessionr c t j | \ }}}}}|dvrt | | d }t | t
| dS )z
Send a HEAD request to the URL, and ensure the response contains a simple
API Response.
Raises `_NotHTTP` if the URL is not available for a HEAD request, or
`_NotAPIContent` if the content type is not a valid content type.
> httphttpsT)allow_redirectsN)urllibparseurlsplitrG headr rE )r rI r' netlocpathqueryfragmentresps r( _ensure_api_responserW Z sq -3L,A,A#,F,F)FFD%
&&&jj <<T< 2 2DTt r* c t t | j rt | | t dt
| | | d g d dd }t | t | t dt
| |j d d
|S ) aY Access an Simple API response with GET, and return the response.
This consists of three parts:
1. If the URL looks suspiciously like an archive, send a HEAD first to
check the Content-Type is HTML or Simple API, to avoid downloading a
large file. Raise `_NotHTTP` if the content type cannot be determined, or
`_NotAPIContent` if it is not HTML or a Simple API.
2. Actually perform the request. Raise HTTP exceptions on network failures.
3. Check the Content-Type header to make sure we got a Simple API response,
and raise `_NotAPIContent` otherwise.
rI zGetting page %sz, )r? z*application/vnd.pypi.simple.v1+html; q=0.1ztext/html; q=0.01z max-age=0)Acceptz
Cache-Control)r@ zFetched page %s as %sr= r> ) r r filenamerW loggerdebugr rA joinr rE r@ )r rI rV s r( _get_simple_responser_ l s tCyy)** 3 S'2222
LL"$8$=$=>>> ;; ii ( )+
D4 T t
LL S!! 33 Kr* r@ ResponseHeadersc | rSd| v rOt j }| d |d< | d }|rt | S dS )z=Determine if we have any encoding information in our headers.r= z content-typecharsetN)emailmessageMessage get_paramr )r@ mrb s r( _get_encoding_from_headersrh s` >W,, M ! ! # # #N 3.++i(( w<< 4r* c & e Zd Zd
dZdd Zdd Zd S )CacheablePageContentpage IndexContentr r/ c & |j sJ || _ d S r1 )cache_link_parsingrk r4 rk s r( r3 z CacheablePageContent.__init__ s &&&& r* otherobjectboolc p t |t | o| j j |j j k S r1 )
isinstancetyperk r )r4 rp s r( __eq__zCacheablePageContent.__eq__ s* %d,,P %*.1PPr* intc 4 t | j j S r1 )hashrk r r4 s r( __hash__z CacheablePageContent.__hash__ s DIM"""r* N)rk rl r r/ )rp rq r rr )r rw )r6 r7 r8 r3 rv r{ rH r* r( rj rj sT Q Q Q Q# # # # # #r* rj c e Zd ZddZdS )
ParseLinksrk rl r Iterable[Link]c d S r1 rH ro s r( __call__zParseLinks.__call__ s r* Nrk rl r r~ )r6 r7 r8 r rH r* r( r} r} s AAAAAAr* r} fnc | t j d fd t j d
fd }|S ) z
Given a function that parses an Iterable[Link] from an IndexContent, cache the
function's result (keyed by CacheablePageContent), unless the IndexContent
`page` has `page.cache_link_parsing == False`.
cacheable_pagerj r
list[Link]c > t | j S r1 )listrk )r r s r( wrapperz*with_cached_index_content..wrapper s BB~*++,,,r* rk rl c r | j r t | S t | S r1 )rn rj r )rk r r s r( wrapper_wrapperz2with_cached_index_content..wrapper_wrapper s; " 77/5566 6BBtHH~~ r* )r rj r r )rk rl r r ) functoolscachewraps)r r r s ` @r( with_cached_index_contentr sr _- - - - - _- _R
r* rk rl r~ c # K | j }| d rUt j | j }| dg D ]#}t j || j }| |V $dS t | j }| j pd}| | j
| | j }|j p|}|j D ] } t j | || }| |V !dS )z\
Parse a Simple API's Index Content, and yield its anchor elements as Link objects.
r? filesNzutf-8)page_urlbase_url)r- r$ r% jsonloadscontentrA r from_jsonr HTMLLinkParserencodingfeeddecoder anchors from_element)
rk rD datafilelinkparserr r r anchors
r( parse_linksr s! &,,..N !FGG z$,''HHWb)) D>$11D|JJJJ
DH
%
%F}'H
KK ##H--...
(C%#H. #III <
r* T)frozenc R e Zd ZU dZded< ded< ded< ded< d Zd
ed < dd
ZdS )rl a Represents one response (or page), along with its URL.
:param encoding: the encoding to decode the given content.
:param url: the URL from which the HTML was downloaded.
:param cache_link_parsing: whether links parsed from this page's url
should be cached. PyPI index urls should
have this set to False, for example.
bytesr r r- r! r r Trr rn r c * t | j S r1 )r r rz s r( __str__zIndexContent.__str__ s #DH---r* N)r r )r6 r7 r8 __doc____annotations__rn r rH r* r( rl rl sr NNN HHH #####. . . . . .r* c 4 e Zd ZdZd
fd Zdd
Zdd Z xZS )r zf
HTMLParser that keeps the first base HREF and a list of all anchor
elements' attributes.
r r r r/ c v t d || _ d | _ g | _ d S )NT)convert_charrefs)r2 r3 r r r )r4 r r5 s r( r3 zHTMLLinkParser.__init__ s7
$///$(
46 r* tagattrs list[tuple[str, str | None]]c |dk r)| j "| | }| || _ d S d S |dk r)| j t | d S d S )Nbasea)r get_hrefr appenddict)r4 r r hrefs r( handle_starttagz HTMLLinkParser.handle_starttag sn &==T] 2==''D $
CZZ L U , , , , , Zr* r! c * |D ]\ }}|dk r|c S d S )Nr rH )r4 r namevalues r( r zHTMLLinkParser.get_href! s1 KD%v~~ tr* )r r r r/ )r r r r r r/ )r r r r! )r6 r7 r8 r r3 r r r9 r: s @r( r r so
7 7 7 7 7 7- - - - r* r r r reasonstr | ExceptionmethCallable[..., None] | Nonec < | t j } |d| | d S )Nz%Could not fetch URL %s: %s - skipping)r\ r] )r r r s r( _handle_get_simple_failr ( s+
||D 0$?????r* rn rr c | t | j }t | j | j d || j | S )Nr= )r r rn )rh r@ rl r r )r; rn r s r( _make_index_contentr 2 sG *(*:;;H ( L- r* IndexContent | Nonec | j dd d }t | }|r t d|| d S t
j | \ }}}}}}|dk rt j
t
j | rU|
d s|dz
}t
j |d }t d| t! || }t# || j
S # t& $ r t d | Y n
t( $ r1}t d | |j |j Y d }~nd }~wt. $ r}t1 | | Y d }~nd }~wt2 $ r}t1 | | Y d }~nd }~wt4 $ r:}d
} | t7 | z
} t1 | | t j Y d }~nRd }~wt: j $ r }t1 | d| Y d }~n(d }~wt: j $ r t1 | d Y nw xY wd S )N#r r zICannot look at %s URL %s because it does not support lookup as web pages.r /z
index.htmlz# file: URL is directory, getting %srY )rn z`Skipping page %s because it looks like an archive, and cannot be checked by a HTTP HEAD request.zSkipping page %s because the %s request got Content-Type: %s. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/htmlz4There was a problem confirming the ssl certificate: )r zconnection error: z timed out) r splitr) r\ warningrN rO urlparseosrS isdirrB url2pathnameendswithurljoinr] r_ r rn rG r, r. r- r r r r r infor
ConnectionErrorTimeout)
r rI r
vcs_schemer' _rS rV excr s
r( _get_index_contentr ? s
(..a
#C #3''J W
t &| 4 4S 9 9 FAtQ1
BGMM&.*E*Ed*K*K L L ||C 3JC
l""3 55 :C@@@ U#C999: #4Dz'LinkCollector.create.. s+ IIs-c22IIIIIIr* )r r no_index)rI r )
index_urlextra_index_urlsr r\ r] r^ r r creater )clsrI r r r r r link_collectors r( r zLinkCollector.create s '(7+CC
$5 LL&IIjIIIII
J '-2
")!! %
'%
r* list[str]c | j j S r1 )r r rz s r( r zLinkCollector.find_links s ++r* locationr r c . t || j S )z>
Fetch an HTML page containing package links.
rY )r rI )r4 r s r( fetch_responsez LinkCollector.fetch_response s "(DLAAAAr* project_namer candidates_from_pager r c z t j fd j D }t j fd j D }t t j red t j || D }t | d dg|z }t
d | t t! | t! | S )Nc 3 V K | ]#}t |j j d d V $dS )Fr page_validator
expand_dirrn r Nr rI is_secure_originr locr r r4 s r( r z0LinkCollector.collect_sources.. s_
4
4
%9 #| < #( )
4
4
4
4
4
4
r* c 3 V K | ]#}t |j j d d V $dS )Tr Nr r s r( r z0LinkCollector.collect_sources.. s_
5
5
%9 #| < #' )
5
5
5
5
5
5
r* c 4 g | ]}||j d|j S )Nz* )r )r ss r(
z1LinkCollector.collect_sources.. s6 =QV%7 QV
%7%7%7r* z' location(s) to search for versions of :
)r r ) collections OrderedDictr get_index_urls_locationsvaluesr r\ isEnabledForloggingDEBUG itertoolschainr& r] r^ r r )r4 r r index_url_sourcesfind_links_sourcesliness ``` r( collect_sourcesz LinkCollector.collect_sources s ( 3
4
4
4
4
4
4
(AA,OO
4
4
4
&(( ) 4
5
5
5
5
5
5
5
5
5
&(( w} - -
+ " );=NOO E u:: 3 3#/3 3 3 E
LL5)) * * * .// -..
r* N)rI r r r r r/ )F)rI r r r r rr r r )r r )r r r r )r r r r r r ) r6 r7 r8 r r3 classmethodr propertyr r r
rH r* r( r r s
#( [ B , , , X,B B B B ,
,
,
,
,
,
r* r )r r r r! )r; r r r/ )r r rI r r r/ )r r rI r r r )r@ r` r r! )r r} r r} r r1 )r r r r r r r r/ )T)r; r rn rr r rl )r r rI r r r )Or
__future__r r
email.messagerc r r r r r urllib.parserN urllib.requestcollections.abcr r r dataclassesr html.parserr optparser typingr
r r pip._vendorr
pip._vendor.requestsr pip._vendor.requests.exceptionsr r pip._internal.exceptionsr pip._internal.models.linkr !pip._internal.models.search_scoper pip._internal.network.sessionr pip._internal.network.utilsr pip._internal.utils.filetypesr pip._internal.utils.miscr pip._internal.vcsr sourcesr r r getLoggerr6 r\ r r` r) Exceptionr, rE rG rW r_ rh rj r} r r rl r r r r r r rH r* r( r' s # " " " " " > > > > > > > > > > ! ! ! ! ! ! " " " " " " ! ) ) ) ) ) ) @ @ @ @ @ @ @ @ ; ; ; ; ; ; * * * * * * 9 9 9 9 9 9 4 4 4 4 4 4 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 9 9 ! ! ! ! ! ! A A A A A A A A A A 8 $ $ c* ) ) ) ) )Y ) ) )@ @ @ @, y $< < < <~ # # # # # # # #B B B B B B B B ( 8 $. . . . . . . .( Z > (,@ @ @ @ @ 48
: : : :z, , , , ,z , , ,
h
h
h
h
h
h
h
h
h
h
r*