= d Z ddlmZ ddlZddlZddlZddlZddlZddl Z ddl
Z
ddl Z ddl
mZmZmZ ddlmZ ddlmZ ddlmZ ddlmZmZmZ dd lm Z dd
l m Z dd l m Z m!Z! dd l"m#Z# dd
l$m%Z% ddl&m'Z' ddl(m)Z) ddl*m+Z+ ddl,m-Z- ddl.m/Z/ ddl0m1Z1 ddl2m3Z3 ddl4m5Z5m6Z6m7Z7 e j8 e9 Z:ee;e;f Z Z?dKd!Z@ G d" d#e> ZAdLd&ZBdMd'ZCdNd*ZD G d+ d, ZE G d- d.e ZFdOd0ZGeGdPd4 ZH ed56 G d7 d2 ZI G d8 d9e ZJ dQdRd@ZK dSdTdCZLdUdEZM G dF dGe ZN G dH dI ZOdS )VzO
The main purpose of this module is to expose LinkCollector.collect_sources().
) annotationsN)IterableMutableMappingSequence) dataclass)
HTMLParser)Values)Callable
NamedTupleProtocol)requests)Response)
RetryErrorSSLError)NetworkConnectionError)Link) SearchScope)
PipSession)raise_for_status)is_archive_fileredact_auth_from_url) url_to_path)vcs )CandidatesFromPage
LinkSource build_sourceurlstrreturn
str | Nonec t j D ]D}| | r| t | dv r|c S EdS )zgLook for VCS schemes in the URL.
Returns the matched VCS scheme, or None if there's no match.
z+:N)r schemeslower
startswithlen)r schemes }/builddir/build/BUILD/imunify360-venv-2.6.2/opt/imunify360/venv/lib/python3.11/site-packages/pip/_internal/index/collector.py_match_vcs_schemer* . sW
+ 99;; ! !& ) ) c#f++.>$.F.FMMM 4 c e Zd Zd fd Z xZS )_NotAPIContent content_typer request_descr! Nonec h t || || _ || _ d S N)super__init__r. r/ )selfr. r/ __class__s r) r4 z_NotAPIContent.__init__: s3
|444 ( (r+ )r. r r/ r r! r0 )__name__
__module__ __qualname__r4
__classcell__r6 s @r) r- r- 9 s= ) ) ) ) ) ) ) ) ) )r+ r- responser r0 c | j dd }| }| d rdS t || j j )z
Check the Content-Type header to ensure the response contains a Simple
API Response.
Raises `_NotAPIContent` if the content type is not a valid content-type.
Content-TypeUnknown)z text/htmlz#application/vnd.pypi.simple.v1+html#application/vnd.pypi.simple.v1+jsonN)headersgetr% r& r- requestmethod)r< r. content_type_ls r) _ensure_api_headerrF @ sg #'' BBL!''))N
x'7'>
?
??r+ c e Zd ZdS )_NotHTTPN)r7 r8 r9 r+ r) rH rH V s Dr+ rH sessionr c t j | \ }}}}}|dvrt | | d }t | t
| dS )z
Send a HEAD request to the URL, and ensure the response contains a simple
API Response.
Raises `_NotHTTP` if the URL is not available for a HEAD request, or
`_NotAPIContent` if the content type is not a valid content type.
> httphttpsT)allow_redirectsN)urllibparseurlsplitrH headr rF )r rJ r( netlocpathqueryfragmentresps r) _ensure_api_responserX Z sq -3L,A,A#,F,F)FFD%
&&&jj <<T< 2 2DTt r+ c t t | j rt | | t dt
| | | d g d dd }t | t | t dt
| |j d d
|S ) aY Access an Simple API response with GET, and return the response.
This consists of three parts:
1. If the URL looks suspiciously like an archive, send a HEAD first to
check the Content-Type is HTML or Simple API, to avoid downloading a
large file. Raise `_NotHTTP` if the content type cannot be determined, or
`_NotAPIContent` if it is not HTML or a Simple API.
2. Actually perform the request. Raise HTTP exceptions on network failures.
3. Check the Content-Type header to make sure we got a Simple API response,
and raise `_NotAPIContent` otherwise.
rJ zGetting page %sz, )r@ z*application/vnd.pypi.simple.v1+html; q=0.1ztext/html; q=0.01z max-age=0)Acceptz
Cache-Control)rA zFetched page %s as %sr> r? ) r r filenamerX loggerdebugr rB joinr rF rA )r rJ rW s r) _get_simple_responser` l s tCyy)** 3 S'2222
LL"$8$=$=>>> ;; ii ( )+
D4 T t
LL S!! 33 Kr+ rA ResponseHeadersc | rSd| v rOt j }| d |d< | d }|rt | S dS )z=Determine if we have any encoding information in our headers.r> z content-typecharsetN)emailmessageMessage get_paramr )rA mrc s r) _get_encoding_from_headersri s` >W,, M ! ! # # #N 3.++i(( w<< 4r+ c & e Zd Zd
dZdd Zdd Zd S )CacheablePageContentpage IndexContentr! r0 c & |j sJ || _ d S r2 )cache_link_parsingrl r5 rl s r) r4 z CacheablePageContent.__init__ s &&&& r+ otherobjectboolc p t |t | o| j j |j j k S r2 )
isinstancetyperl r )r5 rq s r) __eq__zCacheablePageContent.__eq__ s* %d,,P %*.1PPr+ intc 4 t | j j S r2 )hashrl r r5 s r) __hash__z CacheablePageContent.__hash__ s DIM"""r+ N)rl rm r! r0 )rq rr r! rs )r! rx )r7 r8 r9 r4 rw r| rI r+ r) rk rk sT Q Q Q Q# # # # # #r+ rk c e Zd ZddZdS )
ParseLinksrl rm r! Iterable[Link]c d S r2 rI rp s r) __call__zParseLinks.__call__ s r+ Nrl rm r! r )r7 r8 r9 r rI r+ r) r~ r~ s AAAAAAr+ r~ fnc | t j d fd t j d
fd }|S ) z
Given a function that parses an Iterable[Link] from an IndexContent, cache the
function's result (keyed by CacheablePageContent), unless the IndexContent
`page` has `page.cache_link_parsing == False`.
cacheable_pagerk r!
list[Link]c > t | j S r2 )listrl )r r s r) wrapperz*with_cached_index_content..wrapper s BB~*++,,,r+ rl rm c r | j r t | S t | S r2 )ro rk r )rl r r s r) wrapper_wrapperz2with_cached_index_content..wrapper_wrapper s; " 77/5566 6BBtHH~~ r+ )r rk r! r )rl rm r! r ) functoolscachewraps)r r r s ` @r) with_cached_index_contentr sr _- - - - - _- _R
r+ rl rm r c # K | j }| d rUt j | j }| dg D ]#}t j || j }| |V $dS t | j }| j pd}| | j
| | j }|j p|}|j D ] } t j | || }| |V !dS )z\
Parse a Simple API's Index Content, and yield its anchor elements as Link objects.
r@ filesNzutf-8)page_urlbase_url)r. r% r& jsonloadscontentrB r from_jsonr HTMLLinkParserencodingfeeddecoder anchors from_element)
rl rE datafilelinkparserr r r anchors
r) parse_linksr s! &,,..N !FGG z$,''HHWb)) D>$11D|JJJJ
DH
%
%F}'H
KK ##H--...
(C%#H. #III <
r+ T)frozenc R e Zd ZU dZded< ded< ded< ded< d Zd
ed < dd
ZdS )rm a Represents one response (or page), along with its URL.
:param encoding: the encoding to decode the given content.
:param url: the URL from which the HTML was downloaded.
:param cache_link_parsing: whether links parsed from this page's url
should be cached. PyPI index urls should
have this set to False, for example.
bytesr r r. r" r r Trs ro r! c * t | j S r2 )r r r{ s r) __str__zIndexContent.__str__ s #DH---r+ N)r! r )r7 r8 r9 __doc____annotations__ro r rI r+ r) rm rm sr NNN HHH #####. . . . . .r+ c 4 e Zd ZdZd
fd Zdd
Zdd Z xZS )r zf
HTMLParser that keeps the first base HREF and a list of all anchor
elements' attributes.
r r r! r0 c v t d || _ d | _ g | _ d S )NT)convert_charrefs)r3 r4 r r r )r5 r r6 s r) r4 zHTMLLinkParser.__init__ s7
$///$(
46 r+ tagattrs list[tuple[str, str | None]]c |dk r)| j "| | }| || _ d S d S |dk r)| j t | d S d S )Nbasea)r get_hrefr appenddict)r5 r r hrefs r) handle_starttagz HTMLLinkParser.handle_starttag sn &==T] 2==''D $
CZZ L U , , , , , Zr+ r" c * |D ]\ }}|dk r|c S d S )Nr rI )r5 r namevalues r) r zHTMLLinkParser.get_href! s1 KD%v~~ tr+ )r r r! r0 )r r r r r! r0 )r r r! r" )r7 r8 r9 r r4 r r r: r; s @r) r r so
7 7 7 7 7 7- - - - r+ r r r reasonstr | ExceptionmethCallable[..., None] | Nonec < | t j } |d| | d S )Nz%Could not fetch URL %s: %s - skipping)r] r^ )r r r s r) _handle_get_simple_failr ( s+
||D 0$?????r+ ro rs c | t | j }t | j | j d || j | S )Nr> )r r ro )ri rA rm r r )r< ro r s r) _make_index_contentr 2 sG *(*:;;H ( L- r+ IndexContent | Nonec | j dd d }t | }|r t d|| d S | d rt j t | rU|
d s|dz
}t j
|d }t d| t || }t! || j
S # t$ $ r t d | Y n
t& $ r1}t d | |j |j Y d }~nd }~wt, $ r}t/ | | Y d }~nd }~wt0 $ r}t/ | | Y d }~nd }~wt2 $ r:}d
}|t5 | z
}t/ | |t j Y d }~nRd }~wt8 j $ r }t/ | d| Y d }~n(d }~wt8 j $ r t/ | d Y nw xY wd S )N#r r zICannot look at %s URL %s because it does not support lookup as web pages.zfile:/z
index.htmlz# file: URL is directory, getting %srZ )ro z`Skipping page %s because it looks like an archive, and cannot be checked by a HTTP HEAD request.zSkipping page %s because the %s request got Content-Type: %s. The only supported Content-Types are application/vnd.pypi.simple.v1+json, application/vnd.pypi.simple.v1+html, and text/htmlz4There was a problem confirming the ssl certificate: )r zconnection error: z timed out) r splitr* r] warningr& osrT isdirr endswithrO rP urljoinr^ r` r ro rH r- r/ r. r r r r r infor
ConnectionErrorTimeout)r rJ r
vcs_schemerW excr s r) _get_index_contentr ? s
(..a
#C #3''J W
t ~~g
A27==S1A1A#B#B
A ||C 3JC
l""3 55 :C@@@ U#C999: #4Dz'LinkCollector.create.. s+ IIs-c22IIIIIIr+ )r r no_index)rJ r )
index_urlextra_index_urlsr r] r^ r_ r r creater )clsrJ r r r r r link_collectors r) r zLinkCollector.create s '(7+CC
$5 LL&IIjIIIII
J '-2
")!! %
'%
r+ list[str]c | j j S r2 )r r r{ s r) r zLinkCollector.find_links s ++r+ locationr r c . t || j S )z>
Fetch an HTML page containing package links.
rZ )r rJ )r5 r s r) fetch_responsez LinkCollector.fetch_response s "(DLAAAAr+ project_namer candidates_from_pager r c z t j fd j D }t j fd j D }t t j red t j || D }t | d dg|z }t
d | t t! | t! | S )Nc 3 V K | ]#}t |j j d d V $dS )Fr page_validator
expand_dirro r Nr rJ is_secure_originr locr r r5 s r) r z0LinkCollector.collect_sources.. s_
4
4
%9 #| < #( )
4
4
4
4
4
4
r+ c 3 V K | ]#}t |j j d d V $dS )Tr Nr r s r) r z0LinkCollector.collect_sources.. s_
5
5
%9 #| < #' )
5
5
5
5
5
5
r+ c 4 g | ]}||j d|j S )Nz* )r )r ss r)
z1LinkCollector.collect_sources.. s6 =QV%7 QV
%7%7%7r+ z' location(s) to search for versions of :
)r r ) collections OrderedDictr get_index_urls_locationsvaluesr r] isEnabledForloggingDEBUG itertoolschainr' r^ r_ r r )r5 r r index_url_sourcesfind_links_sourcesliness ``` r) collect_sourcesz LinkCollector.collect_sources s ( 3
4
4
4
4
4
4
(AA,OO
4
4
4
&(( ) 4
5
5
5
5
5
5
5
5
5
&(( w} - -
+ " );=NOO E u:: 3 3#/3 3 3 E
LL5)) * * * .// -..
r+ N)rJ r r r r! r0 )F)rJ r r r r rs r! r )r! r )r r r! r )r r r r r! r ) r7 r8 r9 r r4 classmethodr propertyr r r rI r+ r) r r s
#( [ B , , , X,B B B B ,
,
,
,
,
,
r+ r )r r r! r" )r< r r! r0 )r r rJ r r! r0 )r r rJ r r! r )rA ra r! r" )r r~ r! r~ r r2 )r r r r r r r! r0 )T)r< r ro rs r! rm )r r rJ r r! r )Pr
__future__r r
email.messagerd r r r r r urllib.parserO collections.abcr r r dataclassesr html.parserr optparser typingr
r r pip._vendorr
pip._vendor.requestsr pip._vendor.requests.exceptionsr r pip._internal.exceptionsr pip._internal.models.linkr !pip._internal.models.search_scoper pip._internal.network.sessionr pip._internal.network.utilsr pip._internal.utils.filetypesr pip._internal.utils.miscr pip._internal.utils.urlsr pip._internal.vcsr sourcesr r r getLoggerr7 r] r ra r* Exceptionr- rF rH rX r` ri rk r~ r r rm r r r r r r rI r+ r) r% s # " " " " " > > > > > > > > > > ! ! ! ! ! ! " " " " " " ! ) ) ) ) ) ) @ @ @ @ @ @ @ @ ; ; ; ; ; ; * * * * * * 9 9 9 9 9 9 4 4 4 4 4 4 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 9 9 0 0 0 0 0 0 ! ! ! ! ! ! A A A A A A A A A A 8 $ $ c* ) ) ) ) )Y ) ) )@ @ @ @, y $< < < <~ # # # # # # # #B B B B B B B B ( 8 $. . . . . . . .( Z > (,@ @ @ @ @ 48
9 9 9 9x, , , , ,z , , ,
h
h
h
h
h
h
h
h
h
h
r+