B
;Ôð‰'B¢Sã @ s d Z dZddlZddlZddlZddlZddlZddlZddlZddl m
Z
ddl m Z m
Z
mZ ddlmZ ddlmZmZ dd lmZ d!d d „ZG d
d„ deƒZdd„ Zdd„ Zdd„ Zdd„ Z dd„ Z G dd„ deƒZ G dd „ d e ƒZ G d d „ d eƒZ!G d d „ d e ƒZ"dS )"zÛRefactoring framework.
Used as a main program, this can refactor any number of files and/or
recursively descend down directories. Imported as a module, this
provides infrastructure to write your own refactoring tool.
z#Guido van Rossum é N)Úchainé )ÚdriverÚtokenizeÚtoken)Ú find_root)ÚpytreeÚpygram)Ú btm_matcherTc C sX t | g g dgƒ}g }x>t |j¡D ].\}}}| d¡r"|rF|dd… }| |¡ q"W |S )zEReturn a sorted list of all available fix names in the given package.Ú*Úfix_é N)Ú
__import__ÚpkgutilZ iter_modulesÚ__path__Ú
startswithÚappend)Z fixer_pkgZ
remove_prefixZpkgZ fix_namesÚfinderÚnameZispkg© r ú5/opt/alt/python37/lib64/python3.7/lib2to3/refactor.pyÚget_all_fix_names s
r c @ s e Zd ZdS )Ú
_EveryNodeN)Ú__name__Ú
__module__Ú __qualname__r r r r r + s r c C sš t | tjtjfƒr(| jdkr t‚| jhS t | tjƒrH| jrDt| jƒS t‚t | tj ƒrŠt
ƒ }x*| jD ] }x|D ]}| t|ƒ¡ qlW qbW |S t d| ƒ‚dS )zf Accepts a pytree Pattern Node and returns a set
of the pattern types which will match first. Nz$Oh no! I don't understand pattern %s)
Ú
isinstancer Z NodePatternZ LeafPatternÚtyper ZNegatedPatternZcontentÚ_get_head_typesZWildcardPatternÚsetÚupdateÚ Exception)ZpatÚrÚpÚxr r r r / s
r c C sÊ t t¡}g }x„| D ]|}|jrjyt|jƒ}W n tk
rJ | |¡ Y qX xB|D ]}|| |¡ qRW q|jdk r†||j |¡ q| |¡ qW x,tt j
j ¡ t j
j
ƒD ]}|| |¡ q¬W t|ƒS )z^ Accepts a list of fixers and returns a dictionary
of head node type --> fixer list. N)Ú collectionsÚ defaultdictÚlistÚpatternr r r Z _accept_typer r Úpython_grammarZ
symbol2numberÚvaluesÚtokensÚextendÚdict)Z
fixer_listZ
head_nodesZeveryÚfixerZheadsZ node_typer r r Ú_get_headnode_dictK s"
r/ c s ‡ fdd„t ˆ dƒD ƒS )zN
Return the fully qualified names for fixers in the package pkg_name.
c s g | ]}ˆ d | ‘qS )Ú.r )Ú.0Úfix_name)Úpkg_namer r ú
h s z+get_fixers_from_package..F)r )r3 r )r3 r Úget_fixers_from_packaged s
r5 c C s | S )Nr )Úobjr r r Ú _identityk s r7 c sV d}t t | ¡j¡‰ ‡ fdd„}ttjt jtj hƒ}t
ƒ }yúxô|ƒ \}}||krVq@q@|tj krl|rfP d}q@|tj kr.|dkr.|ƒ \}}|tj ksž|dkr P |ƒ \}}|tj ks¼|dkr¾P |ƒ \}}|tj
krä|dkrä|ƒ \}}xJ|tj kr*| |¡ |ƒ \}}|tj
ks |d kr P |ƒ \}}qæW q@P q@W W n tk
rL Y nX t|ƒS )
NFc s t ˆ ƒ} | d | d fS )Nr r )Únext)Útok)Úgenr r Úadvancer s z(_detect_future_features..advanceTÚfromZ
__future__Úimportú(ú,)r Úgenerate_tokensÚioÚStringIOÚreadlineÚ frozensetr ÚNEWLINEÚNLÚCOMMENTr ÚSTRINGÚNAMEÚOPÚaddÚ
StopIteration)ÚsourceZhave_docstringr; ÚignoreÚfeaturesÚtpÚvaluer )r: r Ú_detect_future_featureso sD
rR c @ s e Zd ZdZdS )Ú
FixerErrorz A fixer could not be loaded.N)r r r Ú__doc__r r r r rS — s rS c @ sä e Zd ZdddœZdZdZd4dd„Zdd „ Zd
d „ Zd d
„ Z dd„ Z
dd„ Z d5dd„Z d6dd„Z
dd„ Zd7dd„Zdd„ Zd8d d „Zd d „ Zd d!„ Zd9d"d#„Zd:d$d%„Zd&Zd'Zd(d)„ Zd*d+„ Zd,d-„ Zd.d/„ Zd0d1„ Z d2d3„ Z dS );ÚRefactoringToolF)Úprint_functionÚwrite_unchanged_filesZFixr Nc C s2 || _ |p g | _| j ¡ | _|dk r0| j |¡ | jd rDtj| _ntj | _| j
d¡| _ g | _ t
d¡| _g | _d| _tj| jtj| jd| _| ¡ \| _| _g | _t ¡ | _ g | _ g | _ xXt | j| jƒD ]F}|j rä| j !|¡ qÌ|| jkrü| j "|¡ qÌ|| jkrÌ| j "|¡ qÌW t#| j ƒ| _$t#| j ƒ| _%dS )zÑInitializer.
Args:
fixer_names: a list of fixers to import
options: a dict with configuration.
explicit: a list of fixers to run even if they are explicit.
NrV rW rU F)ÚconvertÚlogger)&ÚfixersÚexplicitÚ_default_optionsÚcopyÚoptionsr r Ú!python_grammar_no_print_statementÚgrammarr) ÚgetrW ÚerrorsÚloggingZ getLoggerrY Ú fixer_logÚwroter ZDriverr rX Ú
get_fixersÚ pre_orderÚ
post_orderÚfilesÚbmZ
BottomMatcherÚBMZ
bmi_pre_orderZbmi_post_orderr Z
BM_compatibleZ add_fixerr r/ Úbmi_pre_order_headsÚbmi_post_order_heads)ÚselfZ fixer_namesr^ r[ r. r r r Ú__init__£ s<
zRefactoringTool.__init__c C s^ g }g }x(| j D ] }t|i i dgƒ}| dd¡d }| | j¡rV|t| jƒd… }| d¡}| jd dd „ |D ƒ¡ }yt ||ƒ}W n& t
k
r® t d
||f ƒd‚Y nX || j | j
ƒ} | jræ| jd k ræ|| jkræ| d |¡ q| d
|¡ | jdkr
| | ¡ q| jdkr"| | ¡ qt d| j ƒ‚qW t d¡}
|j|
d |j|
d ||fS )a Inspects the options to load the requested patterns and handlers.
Returns:
(pre_order, post_order), where pre_order is the list of fixers that
want a pre-order AST traversal, and post_order is the list that want
post-order traversal.
r r0 r éÿÿÿÿNÚ_Ú c S s g | ] }| ¡ ‘qS r )Útitle)r1 r# r r r r4 ç s z.RefactoringTool.get_fixers..zCan't find %s.%sTzSkipping optional fixer: %szAdding transformation: %sZpreZpostzIllegal fixer order: %rZ run_order)Úkey)rZ r Úrsplitr Ú FILE_PREFIXÚlenÚsplitÚ CLASS_PREFIXÚjoinÚgetattrÚAttributeErrorrS r^ rd r[ Ú log_messageÚ log_debugÚorderr ÚoperatorÚ
attrgetterÚsort) rn Zpre_order_fixersZpost_order_fixersZ fix_mod_pathÚmodr2 ÚpartsÚ
class_nameZ fix_classr. Zkey_funcr r r rf × s8
zRefactoringTool.get_fixersc O s ‚ dS )z Called when an error occurs.Nr )rn ÚmsgÚargsÚkwdsr r r Ú log_errorÿ s zRefactoringTool.log_errorc G s |r || }| j |¡ dS )zHook to log a message.N)rY Úinfo)rn r† r‡ r r r r} s zRefactoringTool.log_messagec G s |r || }| j |¡ d S )N)rY Údebug)rn r† r‡ r r r r~ s zRefactoringTool.log_debugc C s dS )zTCalled with the old version, new version, and filename of a
refactored file.Nr )rn Úold_textÚnew_textÚfilenameÚequalr r r Ú print_output s z RefactoringTool.print_outputc C s< x6|D ].}t j |¡r&| |||¡ q| |||¡ qW dS )z)Refactor a list of files and directories.N)ÚosÚpathÚisdirÚ refactor_dirÚ
refactor_file)rn ÚitemsÚwriteÚ
doctests_onlyZ dir_or_filer r r Úrefactor s
zRefactoringTool.refactorc
C s¢ t jd }x’t |¡D ]„\}}}| d|¡ | ¡ | ¡ xF|D ]>}| d¡sBt j |¡d |krBt j ||¡} | | ||¡ qBW dd„ |D ƒ|dd…< qW dS )zÄDescends down a directory and refactor every Python file found.
Python files are assumed to have a .py extension.
Files and subdirectories starting with '.' are skipped.
ÚpyzDescending into %sr0 r c S s g | ]}| d ¡s|‘qS )r0 )r )r1 Zdnr r r r4 . s z0RefactoringTool.refactor_dir..N)
r‘ ÚextsepÚwalkr~ r‚ r r’ Úsplitextrz r• )
rn Zdir_namer— r˜ Zpy_extÚdirpathZdirnamesÚ filenamesr Úfullnamer r r r” s
z RefactoringTool.refactor_dirc
C sŠ yt |dƒ}W n0 tk
r> } z| d||¡ dS d}~X Y nX zt |j¡d }W d| ¡ X tj |d|dd}| ¡ |fS Q R X dS ) zG
Do our best to decode a Python source file correctly.
ÚrbzCan't open %s: %s)NNNr r" rr )ÚencodingÚnewline) ÚopenÚOSErrorr‰ r Údetect_encodingrC ÚcloserA Úread)rn rŽ ÚfÚerrr¢ r r r Ú_read_python_source0 s
z#RefactoringTool._read_python_sourcec C sº | |¡\}}|dkrdS |d7 }|rn| d|¡ | ||¡}| jsL||kr`| |||||¡ q¶| d|¡ nH| ||¡}| jsŠ|rª|jrª| jt|ƒdd… |||d n | d|¡ dS )zRefactors a file.NÚ
zRefactoring doctests in %szNo doctest changes in %srp )r— r¢ zNo changes in %s)r« r~ Úrefactor_docstringrW Úprocessed_fileÚrefactor_stringÚ was_changedÚstr)rn rŽ r— r˜ Úinputr¢ ÚoutputÚtreer r r r• @ s z RefactoringTool.refactor_filec
C s– t |ƒ}d|krtj| j_zLy| j |¡}W n6 tk
rb } z| d||jj |¡ dS d}~X Y nX W d| j| j_X ||_
| d|¡ | ||¡ |S )aF Refactor a given input string.
Args:
data: a string holding the code to be refactored.
name: a human-readable name for use in error/log messages.
Returns:
An AST corresponding to the refactored input stream; None if
there were errors during the parse.
rV zCan't parse %s: %s: %sNzRefactoring %s)
rR r r_ r r` Z parse_stringr! r‰ Ú __class__r Úfuture_featuresr~ Ú
refactor_tree)rn Údatar rO r´ rª r r r r¯ W s
z RefactoringTool.refactor_stringc C sŒ t j ¡ }|rN| d¡ | |d¡}| js2||krB| |d|¡ qˆ| d¡ n:| |d¡}| jsj|r~|jr~| t |ƒd|¡ n
| d¡ d S )Nz Refactoring doctests in stdinzzNo doctest changes in stdinzNo changes in stdin)
ÚsysÚstdinr¨ r~ r rW r® r¯ r° r± )rn r˜ r² r³ r´ r r r Úrefactor_stdinr s
z RefactoringTool.refactor_stdinc
C sü x"t | j| jƒD ]}| ||¡ qW | | j| ¡ ¡ | | j| ¡ ¡ | j | ¡ ¡}xtt
| ¡ ƒrÎx^| jj D ]P}||krv|| rv|| j
tjjdd |jr¼|| j
tjjd xt|| ƒD ]ø}||| krê|| |¡ y t|ƒ W n tk
r wÌY nX |jr&||jkr&qÌ| |¡}|rÌ| ||¡}|dk rÌ| |¡ x,| ¡ D ] }|jsng |_|j |¡ q\W | j | ¡ ¡}x2|D ]*} | |kr¬g || < || || ¡ q–W qÌW qvW q\W x$t | j| jƒD ]}| ||¡ qàW |j S )aÏ Refactors a parse tree (modifying the tree in place).
For compatible patterns the bottom matcher module is
used. Otherwise the tree is traversed node-to-node for
matches.
Args:
tree: a pytree.Node instance representing the root of the tree
to be refactored.
name: a human-readable name for this tree.
Returns:
True if the tree was modified, False otherwise.
T)rt Úreverse)rt N) r rg rh Z
start_treeÚ traverse_byrl rm rk ÚrunZleavesÚanyr* rZ r‚ r ZBaseZdepthZkeep_line_orderZ
get_linenor' Úremover Ú
ValueErrorZfixers_appliedÚmatchÚ transformÚreplacer r, Z finish_treer° )
rn r´ r r. Z match_setÚnodeÚresultsÚnewZ new_matchesZfxrr r r r· ‚ sJ
$z RefactoringTool.refactor_treec C s^ |sdS xP|D ]H}xB||j D ]4}| |¡}|r | ||¡}|dk r | |¡ |}q W qW dS )a Traverse an AST, applying a set of fixers to each node.
This is a helper method for refactor_tree().
Args:
fixers: a list of fixer instances.
traversal: a generator that yields AST nodes.
Returns:
None
N)r r rà rÄ )rn rZ Z traversalrÅ r. rÆ rÇ r r r r½ Ñ s
zRefactoringTool.traverse_byc C s† | j |¡ |dkr.| |¡d }|dkr.dS ||k}| ||||¡ |r`| d|¡ | js`dS |rv| ||||¡ n | d|¡ dS )zR
Called when a file has been refactored and there may be changes.
Nr zNo changes to %szNot writing changes to %s)ri r r« r r~ rW Ú
write_file)rn r rŽ rŒ r— r¢ r r r r r® è s z RefactoringTool.processed_filec C s® yt j|d|dd}W n0 tk
rF } z| d||¡ dS d}~X Y nX |F y| |¡ W n0 tk
rŒ } z| d||¡ W dd}~X Y nX W dQ R X | d|¡ d| _dS ) zÑWrites a string to a file.
It first shows a unified diff between the old text and the new text, and
then rewrites the file; the latter is only done if the write option is
set.
Úwrr )r¢ r£ zCan't create %s: %sNzCan't write %s: %szWrote changes to %sT)rA r¤ r¥ r‰ r— r~ re )rn r rŽ rŒ r¢ Úfprª r r r rÈ ý s * zRefactoringTool.write_filez>>> z... c
C s g }d}d}d}d}xØ|j ddD ]È}|d7 }| ¡ | j¡r€|dk r\| | ||||¡¡ |}|g}| | j¡} |d| … }q"|dk rº| || j ¡s®||| j ¡ d krº| |¡ q"|dk rØ| | ||||¡¡ d}d}| |¡ q"W |dk r| | ||||¡¡ d
|¡S )aË Refactors a docstring, looking for doctests.
This returns a modified version of the input string. It looks
for doctests, which start with a ">>>" prompt, and may be
continued with "..." prompts, as long as the "..." is indented
the same as the ">>>".
(Unfortunately we can't use the doctest module's parser,
since, like most parsers, it is not geared towards preserving
the original source.)
Nr T)Úkeependsr r¬ rr ) Ú
splitlinesÚlstripr ÚPS1r, Úrefactor_doctestÚfindÚPS2Úrstripr rz )
rn r² rŽ ÚresultÚblockZ block_linenoÚindentÚlinenoÚlineÚir r r r s:
z"RefactoringTool.refactor_docstringc
s yˆ ||ˆ ¡}W nf tk
rx } zHˆj tj¡rRx |D ]}ˆ d| d¡¡ q8W ˆ d|||j j
|¡ |S d}~X Y nX ˆ ||¡r t |ƒj
dd}|d|d … ||d d… } }|d d¡sØ|d d7 < ˆ ˆj | d ¡ g}|r |‡ ‡fd
d „|D ƒ7 }|S ) zÞRefactors one doctest.
A doctest is given as a block of lines, the first of which starts
with ">>>" (possibly indented), while the remaining lines start
with "..." (identically indented).
z
Source: %sr¬ z+Can't parse docstring in %s line %s: %s: %sNT)rË r rp r c s g | ]}ˆ ˆj | ‘qS r )rÑ )r1 r× )rÕ rn r r r4 Z s z4RefactoringTool.refactor_doctest..)Ú parse_blockr! rY Z isEnabledForrc ÚDEBUGr~ rÒ r‰ rµ r r· r± rÌ ÚendswithrÎ Úpop)
rn rÔ rÖ rÕ rŽ r´ rª r× rÇ Zclippedr )rÕ rn r rÏ @ s$
"z RefactoringTool.refactor_doctestc C sÒ | j r d}nd}| js$| d|¡ n&| d|¡ x| jD ]}| |¡ q8W | jrt| d¡ x| jD ]}| |¡ qbW | jrÎt| jƒdkr”| d¡ n| dt| jƒ¡ x&| jD ] \}}}| j|f|ž|Ž q®W d S ) NÚwerez
need to bezNo files %s modified.zFiles that %s modified:z$Warnings/messages while refactoring:r zThere was 1 error:zThere were %d errors:)re ri r} rd rb rw )rn rÝ ÚfileÚmessager† r‡ rˆ r r r Ú summarize] s$
zRefactoringTool.summarizec C s" | j | |||¡¡}tƒ |_|S )z³Parses a block into a tree.
This is necessary to get correct line number / offset information
in the parser diagnostics and embedded into the parse tree.
)r Z parse_tokensÚ wrap_toksrD r¶ )rn rÔ rÖ rÕ r´ r r r rÙ t s zRefactoringTool.parse_blockc c sh t | ||¡j¡}xN|D ]F\}}\}}\} }
} ||d 7 }| |d 7 } ||||f| |
f| fV qW dS )z;Wraps a tokenize stream to systematically modify start/end.r N)r r@ Ú gen_linesÚ__next__) rn rÔ rÖ rÕ r+ r rQ Zline0Zcol0Zline1Zcol1Z line_textr r r rá ~ s
zRefactoringTool.wrap_toksc c s€ || j }|| j }|}xV|D ]N}| |¡r@|t|ƒd… V n(|| ¡ d krXdV ntd||f ƒ‚|}q W x
dV qrW dS )z–Generates lines as expected by tokenize from a list of lines.
This strips the first len(indent + self.PS1) characters off each line.
Nr¬ zline=%r, prefix=%rrr )rÎ rÑ r rw rÒ ÚAssertionError)rn rÔ rÕ Úprefix1Zprefix2Úprefixr× r r r râ Œ s
zRefactoringTool.gen_lines)NN)FF)FF)FF)F)NFN)N) r r r r\ ry rv ro rf r‰ r} r~ r r™ r” r« r• r¯ r» r· r½ r® rÈ rÎ rÑ r rÏ rà rÙ rá râ r r r r rU › s:
4(
O
+
rU c @ s e Zd ZdS )ÚMultiprocessingUnsupportedN)r r r r r r r rç s rç c sB e Zd Z‡ fdd„Zd ‡ fdd„ Z‡ fdd„Z‡ fd d
„Z‡ ZS ) ÚMultiprocessRefactoringToolc s" t t| ƒj||Ž d | _d | _d S )N)Úsuperrè ro ÚqueueÚ output_lock)rn r‡ Úkwargs)rµ r r ro ¦ s z$MultiprocessRefactoringTool.__init__Fr c
s |dkr t tˆƒ |||¡S y dd l‰ W n tk
r@ t‚Y nX ˆjd k rTtdƒ‚ˆ ¡ ˆ_ˆ ¡ ˆ_
‡ ‡fdd„t |ƒD ƒ}z.x|D ] }| ¡ qˆW t tˆƒ |||¡ W d ˆj
¡ x t |ƒD ]}ˆj d ¡ qÄW x |D ]}| ¡ rÞ|
¡ qÞW d ˆ_X d S )Nr r z already doing multiple processesc s g | ]}ˆ j ˆjd ‘qS ))Útarget)ZProcessÚ_child)r1 rØ )Úmultiprocessingrn r r r4 ¸ s z8MultiprocessRefactoringTool.refactor..)ré rè r™ rï Ú ImportErrorrç rê Ú RuntimeErrorZ
JoinableQueueZLockrë ÚrangeÚstartrz ÚputZis_alive)rn r– r— r˜ Z
num_processesZ processesr# rØ )rµ )rï rn r r™ « s2
z$MultiprocessRefactoringTool.refactorc sR | j ¡ }xB|d k rL|\}}ztt| ƒj||Ž W d | j ¡ X | j ¡ }q W d S )N)rê ra ré rè r• Z task_done)rn Ztaskr‡ rì )rµ r r rî È s
z"MultiprocessRefactoringTool._childc s2 | j d k r | j ||f¡ ntt| ƒj||ŽS d S )N)rê rô ré rè r• )rn r‡ rì )rµ r r r• Ó s
z)MultiprocessRefactoringTool.refactor_file)FFr )r r r ro r™ rî r• Ú
__classcell__r r )rµ r rè ¤ s
rè )T)#rT Ú
__author__rA r‘ r r¹ rc r€ r% Ú itertoolsr Zpgen2r r r Z
fixer_utilr rr r r r
rj r r! r r r/ r5 r7 rR rS ÚobjectrU rç rè r r r r Ú s8
(