3
Re@W @ s d Z ddlZddlZddlZddlZddlZddlZddlZddlm Z ddl
m Z m Z ddl
mZ ddlmZmZ ddlmZ ddlmZ dd lmZ dd
lmZ dd lm Z dd l m Z dd
l m Z ddl!m"Z" ddl#m$Z$m%Z% ddl&m'Z' ddl(m)Z)m*Z* ddl+m,Z,m-Z- e'rddl.m/Z/ ddl0m1Z1m2Z2m3Z3m4Z4m5Z5m6Z6m7Z7m8Z8 ddl9Z:ddl;mZ> e:j?j@jAZBe4eCeCf ZDejEeFZGdd ZHdd ZIG dd d eJZKd d ZLG d d d eJZMd!d" ZNd#d$ ZOd%d& ZPd'd( ZQd)d* ZRd+d, ZSejTd-ejUZVd.d/ ZWd0d1 ZXd2d3 ZYG d4d5 d5eZZ[d6d7 Z\e\d8d9 Z]G d:d; d;eZZ^dLd rB rA ) urllib_parseurlsplitr@ headr
r? )r& sessionr' netlocpathqueryfragmentrespr( r( r) _ensure_html_responsef s rM c C sL t | rt| |d tjdt| |j| dddd}t| t| |S )a Access an HTML page with GET, and return the response.
This consists of three parts:
1. If the URL looks suspiciously like an archive, send a HEAD first to
check the Content-Type is HTML, to avoid downloading a large file.
Raise `_NotHTTP` if the content type cannot be determined, or
`_NotHTML` if it is not HTML.
2. Actually perform the request. Raise HTTP exceptions on network failures.
3. Check the Content-Type header to make sure we got HTML, and raise
`_NotHTML` otherwise.
)rG zGetting page %sz text/htmlz max-age=0)Acceptz
Cache-Control)r; )r. rM loggerdebugr r< r
r? )r& rG rL r( r( r) _get_html_responsew s rQ c C s2 | r.d| kr.t j| d \}}d|kr.|d S dS )zBDetermine if we have any encoding information in our headers.
z Content-TypecharsetN)cgi parse_header)r; r2 paramsr( r( r) _get_encoding_from_headers s
rV c C s. x(| j dD ]}|jd}|dk r |S q W |S )a Determine the HTML document's base URL.
This looks for a ```` tag in the HTML document. If present, its href
attribute denotes the base URL of anchor tags in the document. If there is
no such tag (or if it does not have a valid href attribute), the HTML
file's URL is used as the base URL.
:param document: An HTML document representation. The current
implementation expects the result of ``html5lib.parse()``.
:param page_url: The URL of the HTML document.
z.//basehrefN)findallr< )documentpage_urlbaserW r( r( r) _determine_base_url s
r\ c C s t jt j| S )zP
Clean a "part" of a URL path (i.e. after splitting on "@" characters).
)rD quoteunquote)partr( r( r) _clean_url_path_part s r` c C s t jt j| S )z
Clean the first part of a URL path that corresponds to a local
filesystem path (i.e. the first part after splitting on "@" characters).
)urllib_request pathname2url url2pathname)r_ r( r( r) _clean_file_url_path s rd z(@|%2F)c C sb |r
t }nt}tj| }g }x:ttj|dgD ]$\}}|j|| |j|j q0W dj |S )z*
Clean the path portion of a URL.
r: )
rd r` _reserved_chars_resplitr itertoolschainappendupperjoin)rI
is_local_path
clean_funcparts
cleaned_partsto_cleanreservedr( r( r) _clean_url_path s
rr c C s2 t j| }|j }t|j|d}t j|j|dS )z
Make sure a link is fully quoted.
For example, if ' ' occurs in the URL, it will be replaced with "%20",
and without double-quoting other characters.
)rl )rI )rD urlparserH rr rI
urlunparse_replace)r& resultrl rI r( r( r) _clean_link s
rw c C sf | j d}|sdS ttj||}| j d}|r8t|nd}| j d}|rRt|}t||||d}|S )zJ
Convert an anchor element in a simple repository page to a Link.
rW Nzdata-requires-pythonz data-yanked)
comes_fromrequires_python
yanked_reason)r< rw rD urljoinr r )anchorrZ base_urlrW r& pyrequirerz linkr( r( r) _create_link_from_element s
r c @ s$ e Zd Zdd Zdd Zdd ZdS )CacheablePageContentc C s |j s
t|| _d S )N)cache_link_parsingAssertionErrorpage)r4 r r( r( r) r1 , s
z CacheablePageContent.__init__c C s t |t| o | jj|jjkS )N)
isinstancetyper r& )r4 otherr( r( r) __eq__1 s zCacheablePageContent.__eq__c C s t | jjS )N)hashr r& )r4 r( r( r) __hash__6 s z CacheablePageContent.__hash__N)r6 r7 r8 r1 r r r( r( r( r) r + s r c s2 t dd fddtj fdd}|S )z
Given a function that parses an Iterable[Link] from an HTMLPage, cache the
function's result (keyed by CacheablePageContent), unless the HTMLPage
`page` has `page.cache_link_parsing == False`.
N)maxsizec s t | jS )N)listr )cacheable_page)fnr( r) wrapperE s z'with_cached_html_pages..wrapperc s | j rt| S t | S )N)r r r )r )r r r( r) wrapper_wrapperJ s z/with_cached_html_pages..wrapper_wrapper)r functoolswraps)r r r( )r r r) with_cached_html_pages; s
r c c sZ t j| j| jdd}| j}t||}x0|jdD ]"}t|||d}|dkrLq0|V q0W dS )zP
Parse an HTML document, and yield its anchor elements as Link objects.
F)transport_encodingnamespaceHTMLElementsz.//a)rZ r} N)r r contentencodingr& r\ rX r )r rY r& r} r| r r( r( r) parse_linksT s
r c @ s" e Zd ZdZdddZdd ZdS ) HTMLPagez'Represents one page, along with its URLTc C s || _ || _|| _|| _dS )am
:param encoding: the encoding to decode the given content.
:param url: the URL from which the HTML was downloaded.
:param cache_link_parsing: whether links parsed from this page's url
should be cached. PyPI index urls should
have this set to False, for example.
N)r r r& r )r4 r r r& r r( r( r) r1 p s zHTMLPage.__init__c C s
t | jS )N)r r& )r4 r( r( r) __str__ s zHTMLPage.__str__N)T)r6 r7 r8 __doc__r1 r r( r( r( r) r m s
r c C s |d krt j}|d| | d S )Nz%Could not fetch URL %s: %s - skipping)rO rP )r reasonmethr( r( r) _handle_get_page_fail s r Tc C s t | j}t| j|| j|dS )N)r r& r )rV r; r r r& )r> r r r( r( r) _make_html_page s
r c
C s |d krt d| jjddd }t|}|r@tjd|| d S tj|\}}}}}}|dkrtj j
t j |r|j
ds|d7 }tj|d}tjd | yt||d
}W nF tk
r tjd | Y n4 tk
r } ztjd | |j|j W Y d d }~X n tk
r0 } zt| | W Y d d }~X n tk
r\ } zt| | W Y d d }~X n tk
r } z&d
} | t|7 } t| | tjd W Y d d }~X nd tj k
r } zt| dj | W Y d d }~X n0 tj k
r t| d Y nX t || j dS d S )Nz?_get_html_page() missing 1 required keyword argument: 'session'# r zICannot look at %s URL %s because it does not support lookup as web pages.file/z
index.htmlz# file: URL is directory, getting %s)rG z`Skipping page %s because it looks like an archive, and cannot be checked by a HTTP HEAD request.ziSkipping page %s because the %s request got Content-Type: %s.The only supported Content-Type is text/htmlz4There was a problem confirming the ssl certificate: )r zconnection error: {}z timed out)r )! TypeErrorr& rf r* rO warningrD rs osrI isdirra rc r, r{ rP rQ r@ r/ r3 r2 r
r r r strinfor ConnectionErrorformatTimeoutr r )
r rG r&
vcs_schemer' _rI rL excr r( r( r) _get_html_page sP
r c C s t tj| S )zQ
Return a list of links, with duplicates removed and ordering preserved.
)r r fromkeys)linksr( r( r) _remove_duplicate_links s r Fc s g g fdd}x| D ]}t jj|}|jd}|s>|r|rH|}nt|}t jj|r|rt jj|}xBt j|D ]}|t jj|| qxW q|rj | qt
j d| qt jj |r|| qt
j d| q t
|rj | q t
j d| q W fS )z
Divide a list of locations into two groups: "files" (archives) and "urls."
:return: A pair of lists (files, urls).
c s8 t | }tj|ddd dkr*j| n
j| d S )NF)strictr z text/html)r mimetypes
guess_typeri )rI r& )filesurlsr( r) sort_path s z"group_locations..sort_pathzfile:z(Path '%s' is ignored: it is a directory.z:Url '%s' is ignored: it is neither a file nor a directory.zQUrl '%s' is ignored. It is either a non-existing path or lacks a specific scheme.)r rI existsr$ r r realpathlistdirrk ri rO r isfiler ) locations
expand_dirr r& rl is_file_urlrI itemr( )r r r) group_locations s<
r c @ s e Zd ZdZdd ZdS )CollectedLinksa
Encapsulates the return value of a call to LinkCollector.collect_links().
The return value includes both URLs to project pages containing package
links, as well as individual package Link objects collected from other
sources.
This info is stored separately as:
(1) links from the configured file locations,
(2) links from the configured find_links, and
(3) urls to HTML project pages, as described by the PEP 503 simple
repository API.
c C s || _ || _|| _dS )z
:param files: Links from file locations.
:param find_links: Links from find_links.
:param project_urls: URLs to HTML project pages, as described by
the PEP 503 simple repository API.
N)r
find_links project_urls)r4 r r r r( r( r) r1 , s
zCollectedLinks.__init__N)r6 r7 r8 r r1 r( r( r( r) r s r c @ sB e Zd ZdZdd ZedddZedd Zd d
Z d d Z
d
S )
LinkCollectorz
Responsible for collecting Link objects from all configured locations,
making network requests as needed.
The class's main method is its collect_links() method.
c C s || _ || _d S )N) search_scoperG )r4 rG r r( r( r) r1 G s zLinkCollector.__init__Fc C sb |j g|j }|jr:| r:tjddjdd |D g }|jpBg }tj||d}t ||d}|S )z
:param session: The Session to use to make requests.
:param suppress_no_index: Whether to ignore the --no-index option
when constructing the SearchScope object.
zIgnoring indexes: %s,c s s | ]}t |V qd S )N)r ).0r& r( r( r) \ s z'LinkCollector.create..)r
index_urls)rG r )
index_urlextra_index_urlsno_indexrO rP rk r r creater )clsrG optionssuppress_no_indexr r r link_collectorr( r( r) r P s
zLinkCollector.createc C s | j jS )N)r r )r4 r( r( r) r k s zLinkCollector.find_linksc C s t || jdS )z>
Fetch an HTML page containing package links.
)rG )r rG )r4 locationr( r( r)
fetch_pagep s zLinkCollector.fetch_pagec
s j }|j|}t|\}}t jdd\}}dd tj||D }dd jD } fddtjdd |D d d |D D }
t|
}
d
jt|
|g} x |
D ]} | j d j| qW t
j d j | t
|| |
d
S )zFind all available links for the given project name.
:return: All the Link objects (unfiltered), as a CollectedLinks object.
T)r c S s g | ] }t |qS r( )r )r r& r( r( r)
s z/LinkCollector.collect_links..c S s g | ]}t |d qS )z-f)r )r r& r( r( r) r s c s g | ]} j j|r|qS r( )rG is_secure_origin)r r )r4 r( r) r s c s s | ]}t |d dV qdS )F)r N)r )r r& r( r( r) r s z.LinkCollector.collect_links..c s s | ]}t |V qd S )N)r )r r& r( r( r) r s z,{} location(s) to search for versions of {}:z* {}
)r r r )r get_index_urls_locationsr r rg rh r r r% ri rO rP rk r )
r4 project_namer index_locationsindex_file_loc
index_url_loc fl_file_loc
fl_url_loc
file_linksfind_link_links
url_locationslinesr r( )r4 r)
collect_linksw s(
zLinkCollector.collect_linksN)F) r6 r7 r8 r r1 classmethodr propertyr r r r( r( r( r) r > s r )N)T)N)F)fr rS r rg loggingr r re collectionsr pip._vendorr r pip._vendor.distlib.compatr Z pip._vendor.requests.exceptionsr r Z pip._vendor.six.moves.urllibr rD r ra pip._internal.exceptionsr
pip._internal.models.linkr !pip._internal.models.search_scoper pip._internal.network.utilsr
pip._internal.utils.compatr pip._internal.utils.filetypesr pip._internal.utils.miscr r pip._internal.utils.typingr pip._internal.utils.urlsr r pip._internal.vcsr r optparser typingr r r r r r r r xml.etree.ElementTreexmlZpip._vendor.requestsr pip._internal.network.sessionr! etree ElementTreeElementZ HTMLElementr ZResponseHeaders getLoggerr6 rO r* r. Exceptionr/ r? r@ rM rQ rV r\ r` rd compile
IGNORECASEre rr rw r objectr r r r r r r r r r r r( r( r( r) st (
3
9
;#