B
*äM3¹r ã @ s2 d dl Z d dlmZ d dlZd dlZd dlZd dlZd dlZy d dlZW n e k
rd d dl
ZY nX d dl Z ddl m
Z
ddlmZmZmZmZmZmZmZmZmZmZmZmZm Z m Z ddl m Z m Z m!Z! ddl"m#Z#m$Z$ ddl%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z- dd l.m/Z/m0Z0 dd
l1m2Z2m3Z3 e 4e5¡Z6e 7d ¡Z8e 7d ej9¡Z:e 7d
¡Z;dZG dd„ de?ƒZ@G dd„ de@ƒZAG dd„ de@ƒZBG dd„ de?ƒZCG dd „ d e@ƒZDG d d „ d e@ƒZEG d d „ d e@ƒZFG d!d"„ d"e@ƒZGG d#d$„ d$e@ƒZHeHeFƒ eDd%d&d'd(d)ZIeIjJZJG d*d+„ d+e?ƒZKdS )-é N)ÚBytesIOé )ÚDistlibException)ÚurljoinÚurlparseÚ
urlunparseÚ url2pathnameÚ pathname2urlÚqueueÚquoteÚunescapeÚ build_openerÚHTTPRedirectHandlerÚ text_typeÚRequestÚ HTTPErrorÚURLError)Ú DistributionÚDistributionPathÚ make_dist)ÚMetadataÚMetadataInvalidError)Úcached_propertyÚ ensure_slashÚsplit_filenameÚget_project_dataÚparse_requirementÚparse_name_and_versionÚ ServerProxyÚnormalize_name)Ú
get_schemeÚUnsupportedVersionError)ÚWheelÚ
is_compatiblez^(\w+)=([a-f0-9]+)z;\s*charset\s*=\s*(.*)\s*$z text/html|application/x(ht)?mlzhttps://pypi.org/pypic C s2 | dkr t } t| dd}z| ¡ S |dƒƒ X dS )z•
Return all distribution names known by an index.
:param url: The URL of the index.
:return: A list of all known distribution names.
Ng @)ÚtimeoutÚclose)Ú
DEFAULT_INDEXr Ú
list_packages)ÚurlÚclient© r* úA/opt/alt/python37/lib/python3.7/site-packages/distlib/locators.pyÚget_all_distribution_names) s r, c @ s$ e Zd ZdZdd„ Ze Z ZZdS )ÚRedirectHandlerzE
A class to work around a bug in some Python 3.2.x releases.
c C s„ d }x dD ]}||kr
|| }P q
W |d kr0d S t |ƒ}|jdkrpt| ¡ |ƒ}t|dƒrh| ||¡ n|||< t | |||||¡S )N)ÚlocationZuriÚ Úreplace_header)r Úschemer Z get_full_urlÚhasattrr0 ÚBaseRedirectHandlerÚhttp_error_302) ÚselfÚreqÚfpÚcodeÚmsgÚheadersZnewurlÚkeyZurlpartsr* r* r+ r4 @ s
z RedirectHandler.http_error_302N)Ú__name__Ú
__module__Ú __qualname__Ú__doc__r4 Zhttp_error_301Zhttp_error_303Zhttp_error_307r* r* r* r+ r- 7 s r- c @ s¶ e Zd ZdZdZdZdZdZed Zd)dd „Z d
d „ Z
d d
„ Z dd„ Z dd„ Z
dd„ Zee
eƒZdd„ Zdd„ Zdd„ Zdd„ Zd d „ Zd d „ Zd d!„ Zd"d#„ Zd$d%„ Zd*d'd(„ZdS )+ÚLocatorzG
A base class for locators - things that locate distributions.
)z.tar.gzz.tar.bz2z.tarz.zipz.tgzz.tbz)z.eggz.exez.whl)z.pdfN)z.whlÚdefaultc C s, i | _ || _ttƒ ƒ| _d| _t ¡ | _dS )a^
Initialise an instance.
:param scheme: Because locators look for most recent versions, they
need to know the version scheme to use. This specifies
the current PEP-recommended scheme - use ``'legacy'``
if you need to support existing distributions on PyPI.
N) Ú_cacher1 r
r- ÚopenerÚmatcherr
ÚQueueÚerrors)r5 r1 r* r* r+ Ú__init__f s
zLocator.__init__c C sX g }xN| j ¡ sRy| j d¡}| |¡ W n | j jk
rD wY nX | j ¡ qW |S )z8
Return any errors which have occurred.
F)rF ÚemptyÚgetÚappendZEmptyÚ task_done)r5 ÚresultÚer* r* r+ Ú
get_errorsy s zLocator.get_errorsc C s | ¡ dS )z>
Clear any errors which may have been logged.
N)rN )r5 r* r* r+ Ú clear_errors‡ s zLocator.clear_errorsc C s | j ¡ d S )N)rB Úclear)r5 r* r* r+ Ú clear_cacheŽ s zLocator.clear_cachec C s | j S )N)Ú_scheme)r5 r* r* r+ Ú _get_scheme‘ s zLocator._get_schemec C s
|| _ d S )N)rR )r5 Úvaluer* r* r+ Ú _set_scheme” s zLocator._set_schemec C s t dƒ‚dS )a=
For a given project, get a dictionary mapping available versions to Distribution
instances.
This should be implemented in subclasses.
If called from a locate() request, self.matcher will be set to a
matcher for the requirement to satisfy, otherwise it will be None.
z Please implement in the subclassN)ÚNotImplementedError)r5 Únamer* r* r+ Ú _get_project™ s
zLocator._get_projectc C s t dƒ‚dS )zJ
Return all the distribution names known to this locator.
z Please implement in the subclassN)rV )r5 r* r* r+ Úget_distribution_names¥ s z Locator.get_distribution_namesc C sL | j dkr| |¡}n2|| j kr,| j | }n | ¡ | |¡}|| j |< |S )zÖ
For a given project, get a dictionary mapping available versions to Distribution
instances.
This calls _get_project to do all the work, and just implements a caching layer on top.
N)rB rX rO )r5 rW rL r* r* r+ Ú get_project« s
zLocator.get_projectc C s^ t |ƒ}t |j¡}d}| d¡}| | j¡}|rBtt|ƒ| jƒ}|j dkd|j
k||||fS )zu
Give an url a score which can be used to choose preferred URLs
for a given project release.
Tz.whlÚhttpszpypi.org) r Ú posixpathÚbasenameÚpathÚendswithÚdownloadable_extensionsr# r" Ú
wheel_tagsr1 Únetloc)r5 r( Útr] Z
compatibleZis_wheelZis_downloadabler* r* r+ Ú score_url¼ s
zLocator.score_urlc C sR |}|rN| |¡}| |¡}||kr(|}||kr@t d||¡ nt d||¡ |S )a{
Choose one of two URLs where both are candidates for distribution
archives for the same version of a distribution (for example,
.tar.gz vs. zip).
The current implementation favours https:// URLs over http://, archives
from PyPI over those from other locations, wheel compatibility (if a
wheel) and then the archive name.
zNot replacing %r with %rzReplacing %r with %r)rd ÚloggerÚdebug)r5 Úurl1Úurl2rL Ús1Ús2r* r* r+ Ú
prefer_urlË s
zLocator.prefer_urlc C s
t ||ƒS )zZ
Attempt to split a filename in project name, version and Python version.
)r )r5 ÚfilenameÚ project_namer* r* r+ r á s zLocator.split_filenamec C s dd„ }d}t |ƒ\}}}}} }
|
¡ d¡ry~t |ƒ}t
|| j ƒs°t d
|¡ nX|dkr¾d }n ||j |ƒ}|r|j |j
|jt||||| d fƒd
dd„ |jD ƒ¡dœ}W n0 tk
r: } zt d|¡ W dd}~X Y nX nÄ| | j¡sZt d|¡ n¨t |¡ }}x˜| jD ]Ž}| |¡rp|dt|ƒ … }| ||¡}|s²t d|¡ nH|\}}}|rÎ|||ƒrú|||t||||| d fƒdœ}|rú||d< P qpW |r| r|
|d| < |S )a
See if a URL is a candidate for a download URL for a project (the URL
has typically been scraped from an HTML page).
If it is, a dictionary is returned with keys "name", "version",
"filename" and "url"; otherwise, None is returned.
c S s t | ƒt |ƒkS )N)r )Zname1Zname2r* r* r+ Ú same_projectï s z:Locator.convert_url_to_download_info..same_projectNzegg=z %s: version hint in fragment: %r)NNéÿÿÿÿú/z.whlzWheel not compatible: %sTr/ z, c S s" g | ]}d t|dd… ƒ¡‘qS )Ú.é N)ÚjoinÚlist)Ú.0Úvr* r* r+ ú
s z8Locator.convert_url_to_download_info..)rW Úversionrl r( zpython-versionzinvalid path for wheel: %szNot downloadable: %sz No match for project/version: %s)rW rx rl r( zpython-versionz %s_digest)r ÚlowerÚ
startswithre rf Ú HASHER_HASHÚmatchÚgroupsr_ r" r# ra rW rx rl r rs ÚpyverÚ ExceptionÚwarningr` r\ r] Úlenr )r5 r( rm rn rL r1 rb r^ ÚparamsÚqueryÚfragÚmÚalgoÚdigestZorigpathÚwheelÚincluderM rl Úextrc rW rx r~ r* r* r+ Ú convert_url_to_download_infoç sj
z$Locator.convert_url_to_download_infoc C sl d}d|kr8|d }x"dD ]}||kr||| f}P qW |shx*dD ]"}d| }||krB||| f}P qBW |S )zò
Get a digest from a dictionary by looking at a "digests" dictionary
or keys of the form 'algo_digest'.
Returns a 2-tuple (algo, digest) if found, else None. Currently
looks only for SHA256, then MD5.
NÚdigests)Zsha256Úmd5z %s_digestr* )r5 ÚinforL rŒ r† r; r* r* r+ Ú _get_digest1 s
zLocator._get_digestc C s® | d¡}| d¡}||kr,|| }|j}nt||| jd}|j}| |¡ |_}|d }||d |< |j|d krœ| |j|¡|_|d |t ƒ ¡
|¡ | |_ |||< dS )zñ
Update a result dictionary (the final result from _get_project) with a
dictionary for a specific version, which typically holds information
gleaned from a filename or URL for an archive for the distribution.
rW rx )r1 r( rŒ ÚurlsN) ÚpopÚmetadatar r1 r r‡ Ú
source_urlrk Ú
setdefaultÚsetÚaddÚlocator) r5 rL rŽ rW rx ÚdistÚmdr‡ r( r* r* r+ Ú_update_version_dataH s
z Locator._update_version_dataFc C sŠ d}t |ƒ}|dkr td| ƒ‚t| jƒ}| |j¡ | _}t d|t|ƒj ¡ |
|j ¡}t |ƒdkrg }|j
} xb|D ]Z}
|
dkrˆqzy(| |
¡s–n|s¤| |
ƒjs®| |
¡ W qz tk
rÒ t d||
¡ Y qzX qzW t |ƒdkròt||jd}|rt d |¡ |d
} || }|r€|jr*|j|_| d i ¡ | tƒ ¡|_i } | d i ¡}
x&|jD ] }||
krZ|
| | |< qZW | |_d| _|S )
a
Find the most recent distribution which matches the given
requirement.
:param requirement: A requirement of the form 'foo (1.0)' or perhaps
'foo (>= 1.0, < 2.0, != 1.3)'
:param prereleases: If ``True``, allow pre-release versions
to be located. Otherwise, pre-release versions
are not returned.
:return: A :class:`Distribution` instance, or ``None`` if no such
distribution could be located.
NzNot a valid requirement: %rzmatcher: %s (%s)rr )r rŒ zerror matching %s with %rr )r; zsorted list: %sro r rŒ )r r r r1 rD Ú requirementre rf Útyper< rZ rW r Z
version_classr| Z
is_prereleaserJ r r€ Úsortedr; ZextrasrI r• Ú
download_urlsrŒ )r5 r› Ú prereleasesrL Úrr1 rD ÚversionsZslistZvclsÚkrx ÚdÚsdr( r* r* r+ Úlocate_ sP
zLocator.locate)rA )F)r< r= r> r? Úsource_extensionsÚbinary_extensionsÚexcluded_extensionsra r` rG rN rO rQ rS rU Úpropertyr1 rX rY rZ rd rk r r‹ r rš r¥ r* r* r* r+ r@ V s.
Jr@ c s0 e Zd ZdZ‡ fdd„Zdd„ Zdd„ Z‡ ZS )ÚPyPIRPCLocatorz‘
This locator uses XML-RPC to locate distributions. It therefore
cannot be used with simple mirrors (that only mirror file content).
c s* t t| ƒjf |Ž || _t|dd| _dS )z—
Initialise an instance.
:param url: The URL to use for XML-RPC.
:param kwargs: Passed to the superclass constructor.
g @)r$ N)Úsuperrª rG Úbase_urlr r) )r5 r( Úkwargs)Ú __class__r* r+ rG s zPyPIRPCLocator.__init__c C s t | j ¡ ƒS )zJ
Return all the distribution names known to this locator.
)r• r) r' )r5 r* r* r+ rY « s z%PyPIRPCLocator.get_distribution_namesc C s i i dœ}| j |d¡}xê|D ]â}| j ||¡}| j ||¡}t| jd}|d |_|d |_| d¡|_ | dg ¡|_
| d¡|_ t |ƒ}|r |d } | d
|_
| | ¡|_| |_|||< xB|D ]:} | d
}
| | ¡} |d |tƒ ¡ |
¡ | |d |
< qÂW q W |S )
N)r rŒ T)r1 rW rx ÚlicenseÚkeywordsÚsummaryr r( r rŒ )r) Zpackage_releasesZ release_urlsZ release_datar r1 rW rx rI r¯ r° r± r r“ r r‡ r— r” r• r– ) r5 rW rL r¡ rv r Údatar’ r˜ rŽ r( r‡ r* r* r+ rX ± s0
zPyPIRPCLocator._get_project)r< r= r> r? rG rY rX Ú
__classcell__r* r* )r® r+ rª › s rª c s0 e Zd ZdZ‡ fdd„Zdd„ Zdd„ Z‡ ZS )ÚPyPIJSONLocatorzw
This locator uses PyPI's JSON interface. It's very limited in functionality
and probably not worth using.
c s t t| ƒjf |Ž t|ƒ| _d S )N)r« r´ rG r r¬ )r5 r( r )r® r* r+ rG Ð s zPyPIJSONLocator.__init__c C s t dƒ‚dS )zJ
Return all the distribution names known to this locator.
z Not available from this locatorN)rV )r5 r* r* r+ rY Ô s z&PyPIJSONLocator.get_distribution_namesc
C s i i dœ}t | jdt|ƒ ƒ}y°| j |¡}| ¡ ¡ }t |¡}t | j
d}|d }|d |_ |d |_ |
d¡|_|
dg ¡|_|
d ¡|_t|ƒ}| |_|d
} |||j < x`|d
D ]T}
|
d }|j |¡ | |
¡|j|< |d
|j tƒ ¡ |¡ | |
¡|d |< q¼W x¸|d
¡ D ]¨\} } | |j kr:q"t | j
d}
|j |
_ | |
_ t|
ƒ}| |_||| < x\| D ]T}
|
d }|j |¡ | |
¡|j|< |d
| tƒ ¡ |¡ | |
¡|d |< qpW q"W W n@ tk
r } z | j t |ƒ¡ t d|¡ W d d }~X Y nX |S )N)r rŒ z%s/json)r1 rŽ rW rx r¯ r° r± r r( rŒ ZreleaseszJSON fetch failed: %s) r r¬ r rC ÚopenÚreadÚdecodeÚjsonÚloadsr r1 rW rx rI r¯ r° r± r r— rž r– r rŒ r” r• Úitemsr rF Úputr re Ú exception)r5 rW rL r( Úrespr² r£ r™ r˜ r rŽ rx ZinfosZomdÚodistrM r* r* r+ rX Ú sT
" z PyPIJSONLocator._get_project)r< r= r> r? rG rY rX r³ r* r* )r® r+ r´ Ë s r´ c @ s` e Zd ZdZe dejejB ejB ¡Z e dejejB ¡Z
dd„ Z e dej¡Z e
dd„ ƒZd S )
ÚPagez4
This class represents a scraped HTML page.
zÝ
(rel\s*=\s*(?:"(?P[^"]*)"|'(?P[^']*)'|(?P[^>\s
]*))\s+)?
href\s*=\s*(?:"(?P[^"]*)"|'(?P[^']*)'|(?P[^>\s
]*))
(\s+rel\s*=\s*(?:"(?P[^"]*)"|'(?P[^']*)'|(?P[^>\s
]*)))?
z!]+)c C s4 || _ | | _| _| j | j ¡}|r0| d¡| _dS )zk
Initialise an instance with the Unicode page contents and the URL they
came from.
r N)r² r¬ r( Ú_baseÚsearchÚgroup)r5 r² r( r… r* r* r+ rG s
z
Page.__init__z[^a-z0-9$&+,/:;=?@.#%_\\|-]c C s dd„ }t ƒ }xœ| j | j¡D ]Š}| d¡}|d pZ|d pZ|d pZ|d pZ|d pZ|d }|d
pr|d pr|d }t| j|ƒ}t|ƒ}| j d
d„ |¡}|
||f¡ q W t |dd„ dd}|S )zâ
Return the URLs of all the links on a page together with information
about their "rel" attribute, for determining which ones to treat as
downloads and which ones to queue for further scraping.
c S s, t | ƒ\}}}}}}t||t|ƒ|||fƒS )zTidy up an URL.)r r r )r( r1 rb r^ r‚ rƒ r„ r* r* r+ Úclean4 s zPage.links..cleanr/ Zrel1Zrel2Zrel3Zrel4Zrel5Zrel6rg rh Zurl3c S s dt | d¡ƒ S )Nz%%%2xr )Úordr )r… r* r* r+ ÚB ó z Page.links..c S s | d S )Nr r* )rc r* r* r+ rÅ F rÆ T)r; Úreverse) r• Ú_hrefÚfinditerr² Ú groupdictr r¬ r Ú _clean_reÚsubr– r )r5 rà rL r| r£ Úrelr( r* r* r+ Úlinks- s
z
Page.linksN)r< r= r> r? ÚreÚcompileÚIÚSÚXrÈ rÀ rG rË r rÎ r* r* r* r+ r¿ s r¿ c s˜ e Zd ZdZejdd„ dd„ dœZd ‡ fdd „ Zd
d „ Zd d
„ Z dd„ Z
e de j
¡Zdd„ Zdd„ Zdd„ Zdd„ Zdd„ Ze d¡Zd d „ Z‡ ZS ) ÚSimpleScrapingLocatorzã
A locator which scrapes HTML pages to locate downloads for a distribution.
This runs multiple threads to do the I/O; performance is at least as good
as pip's PackageFinder, which works in an analogous fashion.
c C s t jt| ƒd ¡ S )N)Úfileobj)ÚgzipZGzipFiler r¶ )Úbr* r* r+ rÅ T rÆ z SimpleScrapingLocator.c C s | S )Nr* )r× r* r* r+ rÅ U rÆ )ÚdeflaterÖ ZnoneNé
c sl t t| ƒjf |Ž t|ƒ| _|| _i | _tƒ | _t
¡ | _ tƒ | _ d| _
|| _t ¡ | _t ¡ | _d| _dS )a¤
Initialise an instance.
:param url: The root URL to use for scraping.
:param timeout: The timeout, in seconds, to be applied to requests.
This defaults to ``None`` (no timeout specified).
:param num_workers: The number of worker threads you want to do I/O,
This defaults to 10.
:param kwargs: Passed to the superclass.
FN)r« rÔ rG r r¬ r$ Ú _page_cacher• Ú_seenr
rE Ú _to_fetchÚ
_bad_hostsÚskip_externalsÚ num_workersÚ threadingÚRLockÚ_lockÚ_gplockÚplatform_check)r5 r( r$ rß r )r® r* r+ rG X s
z SimpleScrapingLocator.__init__c C sF g | _ x:t| jƒD ],}tj| jd}d|_| ¡ | j |¡ qW dS )z¾
Threads are created only when get_project is called, and terminate
before it returns. They are there primarily to parallelise I/O (i.e.
fetching web pages).
)ÚtargetTN) Ú_threadsÚrangerß rà ÚThreadÚ_fetchÚdaemonÚstartrJ )r5 Úirc r* r* r+ Ú_prepare_threadss s z&SimpleScrapingLocator._prepare_threadsc C s> x| j D ]}| j d¡ qW x| j D ] }| ¡ q$W g | _ dS )zu
Tell all the threads to terminate (by sending a sentinel value) and
wait for them to do so.
N)ræ rÜ r» rs )r5 rc r* r* r+ Ú
_wait_threads€ s
z#SimpleScrapingLocator._wait_threadsc C s’ i i dœ}| j x || _|| _t| jdt|ƒ ƒ}| j ¡ | j ¡ | ¡ z&t
d|¡ | j
|¡ | j ¡ W d | ¡ X | `W d Q R X |S )N)r rŒ z%s/z Queueing %s)rã rL rm r r¬ r rÛ rP rÚ rí re rf rÜ r» rs rî )r5 rW rL r( r* r* r+ rX s
z"SimpleScrapingLocator._get_projectz<\b(linux_(i\d86|x86_64|arm\w+)|win(32|_amd64)|macosx_?\d+)\bc C s | j |¡S )zD
Does an URL refer to a platform-specific download?
)Úplatform_dependentrÁ )r5 r( r* r* r+ Ú_is_platform_dependent¢ s z,SimpleScrapingLocator._is_platform_dependentc C sZ | j r| |¡rd}n| || j¡}t d||¡ |rV| j | | j|¡ W dQ R X |S )a%
See if an URL is a suitable download for a project.
If it is, register information in the result dictionary (for
_get_project) about the specific version it's for.
Note that the return value isn't actually used other than as a boolean
value.
Nzprocess_download: %s -> %s) rä rð r‹ rm re rf râ rš rL )r5 r( rŽ r* r* r+ Ú_process_download¨ s
z'SimpleScrapingLocator._process_downloadc
C sÄ t |ƒ\}}}}}}| | j| j | j ¡r2d}n|| jrJ| | j¡sJd}nd| | j¡s\d}nR|dkrjd}nD|dkrxd}n6| |¡rˆd}n&| dd¡d } |
¡ dkrªd}nd}t d ||||¡ |S )
z‘
Determine whether a link URL from a referring page and with a
particular "rel" attribute should be queued for scraping.
F)ZhomepageZdownload)Zhttpr[ Zftpú:r r Z localhostTz#should_queue: %s (%s) from %s -> %s)
r r_ r¦ r§ r¨ rÞ rz r¬ rð Úsplitry re rf )
r5 ÚlinkZreferrerrÍ r1 rb r^ Ú_rL Úhostr* r* r+ Ú
_should_queue¼ s*
z#SimpleScrapingLocator._should_queuec
C sò xì| j ¡ }zÊy’|rž| |¡}|dkr(wxt|jD ]j\}}|| jkr0yB| j |¡ | |¡s‚| |||¡r‚t d||¡ | j
|¡ W q0 t k
r˜ Y q0X q0W W n2 t k
rÒ } z| j
t|ƒ¡ W dd}~X Y nX W d| j ¡ X |sP qW dS )z×
Get a URL to fetch from the work queue, get the HTML page, examine its
links for download candidates and candidates for further scraping.
This is a handy method to run in a thread.
NzQueueing %s from %s)rÜ rI Úget_pagerÎ rÛ r– rñ r÷ re rf r» r r rF r rK )r5 r( Úpagerô rÍ rM r* r* r+ ré Ù s,
& z SimpleScrapingLocator._fetchc C sX t |ƒ\}}}}}}|dkr:tj t|ƒ¡r:tt|ƒdƒ}|| jkr`| j| }t d||¡ nô|
dd¡d }d}|| j krt d||¡ nÄt |d d
id }z¤yât d |¡ | j
j|| jd
} t d|¡ | ¡ }
|
dd¡} t | ¡r„| ¡ } | ¡ }
|
d¡}|r"| j| }||
ƒ}
d}t | ¡}|r@| d¡}y|
|¡}
W n tk
rn |
d¡}
Y nX t |
| ƒ}|| j| < W n¼ t k
rÄ } z |j dkr´t d||¡ W dd}~X Y n€ t k
r } z0t d||¡ | j! | j "|¡ W dQ R X W dd}~X Y n2 t#k
rB } zt d||¡ W dd}~X Y nX W d|| j|< X |S )a
Get the HTML for an URL, possibly from an in-memory cache.
XXX TODO Note: this cache is never actually cleared. It's assumed that
the data won't get stale over the lifetime of a locator instance (not
necessarily true for the default_locator).
Úfilez
index.htmlzReturning %s from cache: %srò r r Nz Skipping %s due to bad host %szAccept-encodingZidentity)r: z Fetching %s)r$ z
Fetched %sz Content-Typer/ zContent-Encodingzutf-8zlatin-1i” zFetch failed: %s: %s)$r Úosr^ Úisdirr r r rÚ re rf ró rÝ r rC rµ r$ rŽ rI ÚHTML_CONTENT_TYPEr| Zgeturlr¶ ÚdecodersÚCHARSETrÁ r r· Ú UnicodeErrorr¿ r r8 r¼ r râ r– r )r5 r( r1 rb r^ rõ rL rö r6 r½ r: Ú content_typeZ final_urlr² ÚencodingÚdecoderr… rM r* r* r+ rø ú sZ
&$ z SimpleScrapingLocator.get_pagez]*>([^<]+) r? ÚzlibÚ
decompressrþ rG rí rî rX rÏ rÐ rÑ rï rð rñ r÷ ré rø r rY r³ r* r* )r® r+ rÔ J s"
!;
rÔ c s8 e Zd ZdZ‡ fdd„Zdd„ Zdd„ Zdd „ Z‡ ZS )
ÚDirectoryLocatorz?
This class locates distributions in a directory tree.
c sN | dd¡| _tt| ƒjf |Ž tj |¡}tj |¡sDt d| ƒ‚|| _
dS )aŒ
Initialise an instance.
:param path: The root of the directory tree to search.
:param kwargs: Passed to the superclass constructor,
except for:
* recursive - if True (the default), subdirectories are
recursed into. If False, only the top-level directory
is searched,
Ú recursiveTzNot a directory: %rN) r‘ r r« r rG rû r^ Úabspathrü r Úbase_dir)r5 r^ r )r® r* r+ rG H s
zDirectoryLocator.__init__c C s | | j¡S )zá
Should a filename be considered as a candidate for a distribution
archive? As well as the filename, the directory which contains it
is provided, though not used by the current implementation.
)r_ r` )r5 rl Úparentr* r* r+ Úshould_includeY s z DirectoryLocator.should_includec C s– i i dœ}x†t | j¡D ]v\}}}xb|D ]Z}| ||¡r(t j ||¡}tddtt j |¡ƒdddfƒ}| ||¡}|r(|
||¡ q(W | j sP qW |S )N)r rŒ rú r/ ) rû Úwalkr
r r^ rs r r r r‹ rš r ) r5 rW rL ÚrootÚdirsÚfilesÚfnr( rŽ r* r* r+ rX a s
z DirectoryLocator._get_projectc C s” t ƒ }xˆt | j¡D ]x\}}}xd|D ]\}| ||¡r$tj ||¡}tddttj |¡ƒdddfƒ}|
|d¡}|r$| |d ¡ q$W | j sP qW |S )zJ
Return all the distribution names known to this locator.
rú r/ NrW )
r• rû r
r
r r^ rs r r r r‹ r– r )r5 rL r r r r r( rŽ r* r* r+ rY q s
z'DirectoryLocator.get_distribution_names) r< r= r> r? rG r rX rY r³ r* r* )r® r+ r C s
r c @ s e Zd ZdZdd„ Zdd„ ZdS )Ú JSONLocatora
This locator uses special extended metadata (not available on PyPI) and is
the basis of performant dependency resolution in distlib. Other locators
require archive downloads before dependencies can be determined! As you
might imagine, that can be slow.
c C s t dƒ‚dS )zJ
Return all the distribution names known to this locator.
z Not available from this locatorN)rV )r5 r* r* r+ rY ‹ s z"JSONLocator.get_distribution_namesc C sÞ i i dœ}t |ƒ}|rÚxÂ| dg ¡D ]²}|d dks$|d dkrBq$t|d |d | d d
¡| jd }|j}|d |_d
|kr’|d
r’d|d
f|_| di ¡|_| di ¡|_|||j < |d
|j t ƒ ¡ |d ¡ q$W |S )N)r rŒ r ZptypeÚsdistZ pyversionÚsourcerW rx r± zPlaceholder for summary)r± r1 r( r‡ r Z requirementsÚexportsr )
r rI r r1 r’ r“ r‡ Z dependenciesr rx r” r• r– )r5 rW rL r² rŽ r˜ r™ r* r* r+ rX ‘ s&
"zJSONLocator._get_projectN)r< r= r> r? rY rX r* r* r* r+ r „ s r c s( e Zd ZdZ‡ fdd„Zdd„ Z‡ ZS )ÚDistPathLocatorz‚
This locator finds installed distributions in a path. It can be useful for
adding to an :class:`AggregatingLocator`.
c s* t t| ƒjf |Ž t|tƒs t‚|| _dS )zs
Initialise an instance.
:param distpath: A :class:`DistributionPath` instance to search.
N)r« r rG Ú
isinstancer ÚAssertionErrorÚdistpath)r5 r r )r® r* r+ rG ¯ s zDistPathLocator.__init__c C sP | j |¡}|d kr i i dœ}n,|j|d|jt|jgƒid|jtd gƒii}|S )N)r rŒ r rŒ )r Zget_distributionrx r• r“ )r5 rW r˜ rL r* r* r+ rX ¹ s z DistPathLocator._get_project)r< r= r> r? rG rX r³ r* r* )r® r+ r ª s
r c sR e Zd ZdZ‡ fdd„Z‡ fdd„Zdd„ Zeej j
eƒZ dd „ Z d
d „ Z ‡ Z
S ) ÚAggregatingLocatorzI
This class allows you to chain and/or merge a list of locators.
c s* | dd¡| _|| _tt| ƒjf |Ž dS )aÏ
Initialise an instance.
:param locators: The list of locators to search.
:param kwargs: Passed to the superclass constructor,
except for:
* merge - if False (the default), the first successful
search from any of the locators is returned. If True,
the results from all locators are merged (this can be
slow).
ÚmergeFN)r‘ r Úlocatorsr« r rG )r5 r r )r® r* r+ rG Ê s zAggregatingLocator.__init__c s* t t| ƒ ¡ x| jD ] }| ¡ qW d S )N)r« r rQ r )r5 r— )r® r* r+ rQ Ú s z AggregatingLocator.clear_cachec C s || _ x| jD ]
}||_qW d S )N)rR r r1 )r5 rT r— r* r* r+ rU ß s z AggregatingLocator._set_schemec C sô i }xê| j D ]à}| |¡}|r | jr®| di ¡}| di ¡}| |¡ | d¡}|r|rx6| ¡ D ]*\}} ||kr„|| | O < qb| ||< qbW | d¡}
|rì|
rì|
|¡ q | jd kr¾d} n$d} x |D ]}| j |¡rÈd} P qÈW | r |}P q W |S )Nr rŒ TF)r rZ r rI Úupdaterº rD r| ) r5 rW rL r— r£ r rŒ Zdfr¢ rv ÚddÚfoundr* r* r+ rX æ s8
z AggregatingLocator._get_projectc C s@ t ƒ }x4| jD ]*}y|| ¡ O }W q tk
r6 Y qX qW |S )zJ
Return all the distribution names known to this locator.
)r• r rY rV )r5 rL r— r* r* r+ rY s
z)AggregatingLocator.get_distribution_names)r< r= r> r? rG rQ rU r© r@ r1 ÚfgetrX rY r³ r* r* )r® r+ r Æ s ,r zhttps://pypi.org/simple/g @)r$ Zlegacy)r1 c @ sL e Zd ZdZddd„Zdd„ Zdd„ Zd d
„ Zd d „ Zd
d„ Z ddd„Z
dS )ÚDependencyFinderz0
Locate dependencies for distributions.
Nc C s |pt | _t| jjƒ| _dS )zf
Initialise an instance, using the specified locator
to locate distributions.
N)Údefault_locatorr— r r1 )r5 r— r* r* r+ rG / s
zDependencyFinder.__init__c C sv t d|¡ |j}|| j|< || j||jf< xD|jD ]:}t|ƒ\}}t d|||¡ | j |t
ƒ ¡ ||f¡ q4W dS )z¨
Add a distribution to the finder. This will update internal information
about who provides what.
:param dist: The distribution to add.
zadding distribution %szAdd to provided: %s, %s, %sN) re rf r; Ú
dists_by_nameÚdistsrx Úprovidesr Úprovidedr” r• r– )r5 r˜ rW Úprx r* r* r+ Úadd_distribution7 s
z!DependencyFinder.add_distributionc C s| t d|¡ |j}| j|= | j||jf= xN|jD ]D}t|ƒ\}}t d|||¡ | j| }| ||f¡ |s0| j|= q0W dS )z°
Remove a distribution from the finder. This will update internal
information about who provides what.
:param dist: The distribution to remove.
zremoving distribution %sz Remove from provided: %s, %s, %sN)
re rf r; r# r$ rx r% r r& Úremove)r5 r˜ rW r' rx Úsr* r* r+ Úremove_distributionF s
z$DependencyFinder.remove_distributionc C sB y| j |¡}W n, tk
r< | ¡ d }| j |¡}Y nX |S )zÞ
Get a version matcher for a requirement.
:param reqt: The requirement
:type reqt: str
:return: A version matcher (an instance of
:class:`distlib.version.Matcher`).
r )r1 rD r! ró )r5 ÚreqtrD rW r* r* r+ Ú get_matcherX s z DependencyFinder.get_matcherc C sv | |¡}|j}tƒ }| j}||krrxL|| D ]@\}}y| |¡}W n tk
r\ d}Y nX |r.| |¡ P q.W |S )zÓ
Find the distributions which can fulfill a requirement.
:param reqt: The requirement.
:type reqt: str
:return: A set of distribution which can fulfill the requirement.
F)r- r; r• r& r| r! r– ) r5 r, rD rW rL r& rx Úproviderr| r* r* r+ Úfind_providersh s
z DependencyFinder.find_providersc C s¦ | j | }tƒ }x,|D ]$}| |¡}| |j¡s| |¡ qW |r^| d||t|ƒf¡ d}nD| |¡ | j |= x"|D ]}| j |tƒ ¡ |¡ qvW | |¡ d}|S )aŠ
Attempt to replace one provider with another. This is typically used
when resolving dependencies from multiple sources, e.g. A requires
(B >= 1.0) while C requires (B >= 1.1).
For successful replacement, ``provider`` must meet all the requirements
which ``other`` fulfills.
:param provider: The provider we are trying to replace with.
:param other: The provider we're trying to replace.
:param problems: If False is returned, this will contain what
problems prevented replacement. This is currently
a tuple of the literal string 'cantreplace',
``provider``, ``other`` and the set of requirements
that ``provider`` couldn't fulfill.
:return: True if we can replace ``other`` with ``provider``, else
False.
Z cantreplaceFT)
Úreqtsr• r- r| rx r– Ú frozensetr+ r” r( ) r5 r. ÚotherÚproblemsZrlistZ unmatchedr* rD rL r* r* r+ Útry_to_replace€ s"
z DependencyFinder.try_to_replaceFc C s
i | _ i | _i | _i | _t|p g ƒ}d|krH| d¡ |tdddgƒO }t|tƒrh| }}t d|¡ n4| j
j ||d }}|dkrt d| ƒ‚t d |¡ d
|_
tƒ }t|gƒ}t|gƒ}xø|r¶| ¡ }|j} | | jkrê| |¡ n"| j| }
|
|kr | ||
|¡ |j|jB } |j} tƒ }
|rh||krhx2d D ]*}d | }||kr:|
t|d
| ƒO }
q:W | | B |
B }x<|D ]2}| |¡}|sRt d|¡ | j
j ||d}|dkrÎ|sÎ| j
j |d
d}|dkrôt d|¡ | d|f¡ n^|j|j }}||f| jkr | |¡ | |¡ || krR||krR| |¡ t d|j¡ xZ|D ]R}|j} | | jkr†| j |tƒ ¡ |¡ n"| j| }
|
|krX| ||
|¡ qXW q|W qÀW t| j ¡ ƒ}x.|D ]&}||k|_ |j rÌt d|j¡ qÌW t d|¡ ||fS )aŽ
Find a distribution and all distributions it depends on.
:param requirement: The requirement specifying the distribution to
find, or a Distribution instance.
:param meta_extras: A list of meta extras such as :test:, :build: and
so on.
:param prereleases: If ``True``, allow pre-release versions to be
returned - otherwise, don't return prereleases
unless they're all that's available.
Return a set of :class:`Distribution` instances and a set of
problems.
The distributions returned should be such that they have the
:attr:`required` attribute set to ``True`` if they were
from the ``requirement`` passed to ``find()``, and they have the
:attr:`build_time_dependency` attribute set to ``True`` unless they
are post-installation dependencies of the ``requirement``.
The problems should be a tuple consisting of the string
``'unsatisfied'`` and the requirement which couldn't be satisfied
by any distribution known to the locator.
z:*:z:test:z:build:z:dev:zpassed %s as requirement)rŸ NzUnable to locate %rz
located %sT)ÚtestÚbuildÚdevz:%s:z %s_requireszNo providers found for %rzCannot satisfy %rZ unsatisfiedzAdding %s to install_distsz#%s is a build-time dependency only.zfind done for %s) r& r$ r# r0 r• r) r r re rf r— r¥ r Z requestedr‘ r; r( r4 Z run_requiresZ
meta_requiresZbuild_requiresÚgetattrr/ r– rx Zname_and_versionr” ÚvaluesZbuild_time_dependency)r5 r› Z meta_extrasrŸ r˜ r¾ r3 ZtodoZ
install_distsrW r2 ZireqtsZsreqtsZereqtsr; rM Z all_reqtsr Z providersr. Únrv r' r$ r* r* r+ Úfind¨ sŒ
zDependencyFinder.find)N)NF) r< r= r> r? rG r( r+ r- r/ r4 r; r* r* r* r+ r! * s
(r! )N)LrÖ Úior r¸ Úloggingrû r\ rÏ rà Ú ImportErrorZdummy_threadingr r/ r Úcompatr r r r r r
r r r
r r3 r r r r Zdatabaser r r r’ r r Úutilr r r r r r r r rx r r! rˆ r" r# Ú getLoggerr< re rÐ r{ rÑ rÿ rý r& r, r- Úobjectr@ rª r´ r¿ rÔ r r r r r" r¥ r! r* r* r* r+ Ú sX @(
G0E: zA& [