a
XC?hsk ã @ s d Z dZddlZddlZddlZddlZddlZddlZddlZddl m
Z
ddl m Z m
Z
mZ ddlmZ ddlmZmZ dd lmZ d!d d „ZG d
d„ deƒZdd„ Zdd„ Zdd„ Zdd„ Z dd„ Z G dd„ deƒZ G dd „ d e ƒZ G d d „ d eƒZ!G d d „ d e ƒZ"dS )"zÛRefactoring framework.
Used as a main program, this can refactor any number of files and/or
recursively descend down directories. Imported as a module, this
provides infrastructure to write your own refactoring tool.
z#Guido van Rossum é N)Úchainé )ÚdriverÚtokenizeÚtoken)Ú find_root)ÚpytreeÚpygram)Ú btm_matcherTc C sT t | g g dgƒ}g }t |j¡D ].\}}}| d¡r |rD|dd… }| |¡ q |S )zEReturn a sorted list of all available fix names in the given package.Ú*Úfix_é N)Ú
__import__ÚpkgutilZ iter_modulesÚ__path__Ú
startswithÚappend)Z fixer_pkgZ
remove_prefixZpkgZ fix_namesÚfinderÚnameZispkg© r ú5/opt/alt/python39/lib64/python3.9/lib2to3/refactor.pyÚget_all_fix_names s
r c @ s e Zd ZdS )Ú
_EveryNodeN©Ú__name__Ú
__module__Ú __qualname__r r r r r + s r c C s’ t | tjtjfƒr(| jdu r t‚| jhS t | tjƒrH| jrDt| jƒS t‚t | tj ƒr‚t
ƒ }| jD ] }|D ]}| t|ƒ¡ qhq`|S t d| ƒ‚dS )zf Accepts a pytree Pattern Node and returns a set
of the pattern types which will match first. Nz$Oh no! I don't understand pattern %s)
Ú
isinstancer Z NodePatternZ LeafPatternÚtyper ZNegatedPatternZcontentÚ_get_head_typesZWildcardPatternÚsetÚupdateÚ Exception)ZpatÚrÚpÚxr r r r / s
r c C s¼ t t¡}g }| D ]v}|jrbzt|jƒ}W n tyF | |¡ Y qˆ0 |D ]}|| |¡ qLq|jdur~||j |¡ q| |¡ qtt j
j ¡ t j
j
ƒD ]}|| |¡ q t|ƒS )z^ Accepts a list of fixers and returns a dictionary
of head node type --> fixer list. N)Ú collectionsÚ defaultdictÚlistÚpatternr r r Z _accept_typer r Úpython_grammarZ
symbol2numberÚvaluesÚtokensÚextendÚdict)Z
fixer_listZ
head_nodesZeveryÚfixerZheadsZ node_typer r r Ú_get_headnode_dictK s$
ÿr0 c s ‡ fdd„t ˆ dƒD ƒS )zN
Return the fully qualified names for fixers in the package pkg_name.
c s g | ]}ˆ d | ‘qS ©Ú.r )Ú.0Úfix_name©Zpkg_namer r Ú
h s ÿz+get_fixers_from_package..F)r r5 r r5 r Úget_fixers_from_packaged s
ÿr7 c C s | S ©Nr )Úobjr r r Ú _identityk s r: c sV d}t t | ¡j¡‰ ‡ fdd„}ttjt jtj hƒ}t
ƒ }zü|ƒ \}}||v rTq>q>|tj krl|rfq6d}q>|tj kr6|dkr6|ƒ \}}|tj ksž|dkr¢q6|ƒ \}}|tj ks¾|dkrÂq6|ƒ \}}|tj
krè|dkrè|ƒ \}}|tj kr4| |¡ |ƒ \}}|tj
ks.|d kr"q4|ƒ \}}qèq>q6q>W n tyL Y n0 t|ƒS )
NFc s t ˆ ƒ} | d | d fS )Nr r )Únext)Útok©Úgenr r Úadvancer s z(_detect_future_features..advanceTÚfromZ
__future__Úimportú(ú,)r Úgenerate_tokensÚioÚStringIOÚreadlineÚ frozensetr ÚNEWLINEÚNLÚCOMMENTr ÚSTRINGÚNAMEÚOPÚaddÚ
StopIteration)ÚsourceZhave_docstringr? ÚignoreÚfeaturesÚtpÚvaluer r= r Ú_detect_future_featureso sB
rV c @ s e Zd ZdZdS )Ú
FixerErrorz A fixer could not be loaded.N)r r r Ú__doc__r r r r rW — s rW c @ sæ e Zd ZddddœZdZdZd4dd„Zdd „ Zd
d „ Zd d
„ Z dd„ Z
dd„ Z d5dd„Z d6dd„Z
dd„ Zd7dd„Zdd„ Zd8d d „Zd d „ Zd d!„ Zd9d"d#„Zd:d$d%„Zd&Zd'Zd(d)„ Zd*d+„ Zd,d-„ Zd.d/„ Zd0d1„ Z d2d3„ Z dS );ÚRefactoringToolF)Úprint_functionÚ
exec_functionÚwrite_unchanged_filesZFixr Nc C sJ || _ |p g | _| j ¡ | _|dur0| j |¡ tj ¡ | _| jd rR| jj d= n| jd rf| jj d= | j
d¡| _ g | _ t
d¡| _g | _d| _tj| jtj| jd | _| ¡ \| _| _g | _t ¡ | _ g | _ g | _ t | j| jƒD ]H}|j rü| j !|¡ qä|| jv r| j "|¡ qä|| jv rä| j "|¡ qät#| j ƒ| _$t#| j ƒ| _%dS )
zÑInitializer.
Args:
fixer_names: a list of fixers to import
options: a dict with configuration.
explicit: a list of fixers to run even if they are explicit.
NrZ Úprintr[ Úexecr\ rY F)ÚconvertÚlogger)&ÚfixersÚexplicitÚ_default_optionsÚcopyÚoptionsr! r r* ÚgrammarÚkeywordsÚgetr\ ÚerrorsÚloggingZ getLoggerr` Ú fixer_logÚwroter ZDriverr r_ Ú
get_fixersÚ pre_orderÚ
post_orderÚfilesÚbmZ
BottomMatcherÚBMZ
bmi_pre_orderZbmi_post_orderr Z
BM_compatibleZ add_fixerr r0 Úbmi_pre_order_headsÚbmi_post_order_heads)ÚselfZ fixer_namesre rb r/ r r r Ú__init__¤ sB
þ
zRefactoringTool.__init__c C sV g }g }| j D ]}t|i i dgƒ}| dd¡d }| | j¡rR|t| jƒd… }| d¡}| jd dd „ |D ƒ¡ }zt ||ƒ}W n$ t
y¨ t d
||f ƒd‚Y n0 || j | j
ƒ} | jrà| jd urà|| jvrà| d |¡ q| d
|¡ | jdkr| | ¡ q| jdkr | | ¡ qt d| j ƒ‚qt d¡}
|j|
d |j|
d ||fS )a Inspects the options to load the requested patterns and handlers.
Returns:
(pre_order, post_order), where pre_order is the list of fixers that
want a pre-order AST traversal, and post_order is the list that want
post-order traversal.
r r2 r éÿÿÿÿNÚ_Ú c S s g | ] }| ¡ ‘qS r )Útitle)r3 r$ r r r r6 ë ó z.RefactoringTool.get_fixers..zCan't find %s.%sTzSkipping optional fixer: %szAdding transformation: %sZpreZpostzIllegal fixer order: %rZ run_order©Úkey)ra r Úrsplitr Ú FILE_PREFIXÚlenÚsplitÚ CLASS_PREFIXÚjoinÚgetattrÚAttributeErrorrW re rk rb Ú log_messageÚ log_debugÚorderr ÚoperatorÚ
attrgetterÚsort) ru Zpre_order_fixersZpost_order_fixersZ fix_mod_pathÚmodr4 ÚpartsÚ
class_nameZ fix_classr/ Zkey_funcr r r rm Û s:
ÿ
zRefactoringTool.get_fixersc O s ‚ dS )z Called when an error occurs.Nr )ru ÚmsgÚargsÚkwdsr r r Ú log_error s zRefactoringTool.log_errorc G s |r || }| j |¡ dS )zHook to log a message.N)r` Úinfo©ru r r r r r r† s zRefactoringTool.log_messagec G s |r || }| j |¡ d S r8 )r` Údebugr” r r r r‡
s zRefactoringTool.log_debugc C s dS )zTCalled with the old version, new version, and filename of a
refactored file.Nr )ru Úold_textÚnew_textÚfilenameÚequalr r r Ú print_output s z RefactoringTool.print_outputc C s8 |D ].}t j |¡r$| |||¡ q| |||¡ qdS )z)Refactor a list of files and directories.N)ÚosÚpathÚisdirÚ refactor_dirÚ
refactor_file)ru ÚitemsÚwriteÚ
doctests_onlyZ dir_or_filer r r Úrefactor s zRefactoringTool.refactorc
C sš t jd }t |¡D ]€\}}}| d|¡ | ¡ | ¡ |D ]>}| d¡s>t j |¡d |kr>t j ||¡} | | ||¡ q>dd„ |D ƒ|dd…< qdS )zÄDescends down a directory and refactor every Python file found.
Python files are assumed to have a .py extension.
Files and subdirectories starting with '.' are skipped.
ÚpyzDescending into %sr2 r c S s g | ]}| d ¡s|‘qS r1 )r )r3 Zdnr r r r6 2 r{ z0RefactoringTool.refactor_dir..N)
r› ÚextsepÚwalkr‡ r‹ r rœ Úsplitextrƒ rŸ )
ru Zdir_namer¡ r¢ Zpy_extÚdirpathZdirnamesÚ filenamesr Úfullnamer r r rž s
ÿz RefactoringTool.refactor_dirc
C s® zt |dƒ}W n4 tyB } z | d||¡ W Y d}~dS d}~0 0 z t |j¡d }W | ¡ n
| ¡ 0 tj |d|dd }| ¡ |fW d ƒ S 1 s 0 Y dS ) zG
Do our best to decode a Python source file correctly.
ÚrbzCan't open %s: %sN)NNr r# ry ©ÚencodingÚnewline) ÚopenÚOSErrorr’ r Údetect_encodingrG ÚcloserE Úread)ru r˜ ÚfÚerrr r r r Ú_read_python_source4 s z#RefactoringTool._read_python_sourcec C sº | |¡\}}|du rdS |d7 }|rn| d|¡ | ||¡}| jsL||kr`| |||||¡ q¶| d|¡ nH| ||¡}| jsŠ|rª|jrª| jt|ƒdd… |||d n | d|¡ dS )zRefactors a file.NÚ
zRefactoring doctests in %szNo doctest changes in %srw )r¡ r zNo changes in %s)r¶ r‡ Úrefactor_docstringr\ Úprocessed_fileÚrefactor_stringÚ was_changedÚstr)ru r˜ r¡ r¢ Úinputr ÚoutputÚtreer r r rŸ D s ÿz RefactoringTool.refactor_filec
C s° t |ƒ}d|v rtj| j_zfz| j |¡}W nF tyr } z.| d||jj |¡ W Y d}~W | j| j_dS d}~0 0 W | j| j_n | j| j_0 ||_
| d|¡ | ||¡ |S )aF Refactor a given input string.
Args:
data: a string holding the code to be refactored.
name: a human-readable name for use in error/log messages.
Returns:
An AST corresponding to the refactored input stream; None if
there were errors during the parse.
rZ zCan't parse %s: %s: %sNzRefactoring %s)
rV r Z!python_grammar_no_print_statementr rf Z parse_stringr" r’ Ú __class__r Úfuture_featuresr‡ Ú
refactor_tree)ru Údatar rS r¿ rµ r r r rº [ s"
ÿ
þ z RefactoringTool.refactor_stringc C sŒ t j ¡ }|rN| d¡ | |d¡}| js2||krB| |d|¡ qˆ| d¡ n:| |d¡}| jsj|r~|jr~| t |ƒd|¡ n
| d¡ d S )Nz Refactoring doctests in stdinzzNo doctest changes in stdinzNo changes in stdin)
ÚsysÚstdinr³ r‡ r¸ r\ r¹ rº r» r¼ )ru r¢ r½ r¾ r¿ r r r Úrefactor_stdinv s
z RefactoringTool.refactor_stdinc
C sÚ t | j| jƒD ]}| ||¡ q| | j| ¡ ¡ | | j| ¡ ¡ | j | ¡ ¡}t
| ¡ ƒr²| jj D ]B}||v rj|| rj|| j
tjjdd |jr°|| j
tjjd t|| ƒD ]ð}||| v rÚ|| |¡ z t|ƒ W n t yþ Y q¼Y n0 |jr||jv rq¼| |¡}|r¼| ||¡}|dur¼| |¡ | ¡ D ] }|js\g |_|j |¡ qJ| j | ¡ ¡}|D ]*} | |vr–g || < || || ¡ q€q¼qjqTt | j| jƒD ]}| ||¡ qÀ|j S )aÏ Refactors a parse tree (modifying the tree in place).
For compatible patterns the bottom matcher module is
used. Otherwise the tree is traversed node-to-node for
matches.
Args:
tree: a pytree.Node instance representing the root of the tree
to be refactored.
name: a human-readable name for this tree.
Returns:
True if the tree was modified, False otherwise.
T)r} Úreverser| N) r rn ro Z
start_treeÚ traverse_byrs rt rr ÚrunZleavesÚanyr+ ra r‹ r ZBaseZdepthZkeep_line_orderZ
get_linenor( Úremover Ú
ValueErrorZfixers_appliedÚmatchÚ transformÚreplacer r- Z finish_treer» )
ru r¿ r r/ Z match_setÚnodeÚresultsÚnewZ new_matchesZfxrr r r r † sJ
z RefactoringTool.refactor_treec C sV |sdS |D ]D}||j D ]4}| |¡}|r| ||¡}|dur| |¡ |}qq dS )a Traverse an AST, applying a set of fixers to each node.
This is a helper method for refactor_tree().
Args:
fixers: a list of fixer instances.
traversal: a generator that yields AST nodes.
Returns:
None
N)r rÍ rÎ rÏ )ru ra Z traversalrÐ r/ rÑ rÒ r r r rÈ Õ s
zRefactoringTool.traverse_byc C s† | j |¡ |du r.| |¡d }|du r.dS ||k}| ||||¡ |r`| d|¡ | js`dS |rv| ||||¡ n | d|¡ dS )zR
Called when a file has been refactored and there may be changes.
Nr zNo changes to %szNot writing changes to %s)rp r r¶ rš r‡ r\ Ú
write_file)ru r— r˜ r– r¡ r r™ r r r r¹ ì s z RefactoringTool.processed_filec C sÈ zt j|d|dd}W n4 tyJ } z | d||¡ W Y d}~dS d}~0 0 |R z| |¡ W n2 ty’ } z| d||¡ W Y d}~n
d}~0 0 W d ƒ n1 s¨0 Y | d|¡ d| _dS ) zÑWrites a string to a file.
It first shows a unified diff between the old text and the new text, and
then rewrites the file; the latter is only done if the write option is
set.
Úwry r¬ zCan't create %s: %sNzCan't write %s: %szWrote changes to %sT)rE r¯ r° r’ r¡ r‡ rl )ru r— r˜ r– r Úfprµ r r r rÓ s B zRefactoringTool.write_filez>>> z... c
C s g }d}d}d}d}|j ddD ]È}|d7 }| ¡ | j¡r~|durZ| | ||||¡¡ |}|g}| | j¡} |d| … }q |dur¸| || j ¡s¬||| j ¡ d kr¸| |¡ q |durÖ| | ||||¡¡ d}d}| |¡ q |dur
| | ||||¡¡ d
|¡S )aË Refactors a docstring, looking for doctests.
This returns a modified version of the input string. It looks
for doctests, which start with a ">>>" prompt, and may be
continued with "..." prompts, as long as the "..." is indented
the same as the ">>>".
(Unfortunately we can't use the doctest module's parser,
since, like most parsers, it is not geared towards preserving
the original source.)
Nr T©Úkeependsr r· ry ) Ú
splitlinesÚlstripr ÚPS1r- Úrefactor_doctestÚfindÚPS2Úrstripr rƒ )
ru r½ r˜ ÚresultÚblockZ block_linenoÚindentÚlinenoÚlineÚir r r r¸ sD ÿ ÿþ ÿ
ÿz"RefactoringTool.refactor_docstringc
s zˆ ||ˆ ¡}W nh tyz } zPˆj tj¡rL|D ]}ˆ d| d¡¡ q4ˆ d|||j j
|¡ |W Y d}~S d}~0 0 ˆ ||¡rt |ƒj
dd}|d|d … ||d d… } }|d d¡sÚ|d d7 < ˆ ˆj | d ¡ g}|r|‡ ‡fd
d „|D ƒ7 }|S ) zÞRefactors one doctest.
A doctest is given as a block of lines, the first of which starts
with ">>>" (possibly indented), while the remaining lines start
with "..." (identically indented).
z
Source: %sr· z+Can't parse docstring in %s line %s: %s: %sNTrÖ r rw r c s g | ]}ˆ ˆj | ‘qS r )rÝ )r3 rã ©rá ru r r r6 ^ r{ z4RefactoringTool.refactor_doctest..)Ú parse_blockr" r` Z isEnabledForrj ÚDEBUGr‡ rÞ r’ rÀ r r r¼ rØ ÚendswithrÚ Úpop)
ru rà râ rá r˜ r¿ rµ rã rÒ Zclippedr rå r rÛ D s& ÿ"z RefactoringTool.refactor_doctestc C sÌ | j r d}nd}| js$| d|¡ n"| d|¡ | jD ]}| |¡ q6| jrl| d¡ | jD ]}| |¡ q\| jrÈt| jƒdkrŒ| d¡ n| dt| jƒ¡ | jD ]"\}}}| j|g|¢R i |¤Ž q¤d S ) NÚwerez
need to bezNo files %s modified.zFiles that %s modified:z$Warnings/messages while refactoring:r zThere was 1 error:zThere were %d errors:)rl rp r† rk ri r€ )ru rê ÚfileÚmessager r r‘ r r r Ú summarizea s$
zRefactoringTool.summarizec C s" | j | |||¡¡}tƒ |_|S )z³Parses a block into a tree.
This is necessary to get correct line number / offset information
in the parser diagnostics and embedded into the parse tree.
)r Z parse_tokensÚ wrap_toksrH rÁ )ru rà râ rá r¿ r r r ræ x s zRefactoringTool.parse_blockc c sd t | ||¡j¡}|D ]F\}}\}}\} }
} ||d 7 }| |d 7 } ||||f| |
f| fV qdS )z;Wraps a tokenize stream to systematically modify start/end.r N)r rD Ú gen_linesÚ__next__) ru rà râ rá r, r rU Zline0Zcol0Zline1Zcol1Z line_textr r r rî ‚ s
zRefactoringTool.wrap_toksc c sx || j }|| j }|}|D ]N}| |¡r>|t|ƒd… V n(|| ¡ d krVdV ntd||f ƒ‚|}q dV qldS )z–Generates lines as expected by tokenize from a list of lines.
This strips the first len(indent + self.PS1) characters off each line.
Nr· zline=%r, prefix=%rry )rÚ rÝ r r€ rÞ ÚAssertionError)ru rà rá Úprefix1Zprefix2Úprefixrã r r r rï s
zRefactoringTool.gen_lines)NN)FF)FF)FF)F)NFN)N) r r r rc r‚ r rv rm r’ r† r‡ rš r£ rž r¶ rŸ rº rÆ r rÈ r¹ rÓ rÚ rÝ r¸ rÛ rí ræ rî rï r r r r rY › s@ þ
7(
O ÿ
+
rY c @ s e Zd ZdS )ÚMultiprocessingUnsupportedNr r r r r rô ¤ s rô c sB e Zd Z‡ fdd„Zd ‡ fdd„ Z‡ fdd„Z‡ fd d
„Z‡ ZS ) ÚMultiprocessRefactoringToolc s& t t| ƒj|i |¤Ž d | _d | _d S r8 )Úsuperrõ rv ÚqueueÚ output_lock©ru r Úkwargs©rÀ r r rv ª s z$MultiprocessRefactoringTool.__init__Fr c
s> |dkr t tˆƒ |||¡S z dd l‰ W n ty> t‚Y n0 ˆjd urRtdƒ‚ˆ ¡ ˆ_ˆ ¡ ˆ_
‡ ‡fdd„t |ƒD ƒ}zn|D ] }| ¡ q„t tˆƒ |||¡ W ˆj
¡ t |ƒD ]}ˆj d ¡ qº|D ]}| ¡ rÐ|
¡ qÐd ˆ_nLˆj
¡ t |ƒD ]}ˆj d ¡ q |D ]}| ¡ r|
¡ qd ˆ_0 d S )Nr r z already doing multiple processesc s g | ]}ˆ j ˆjd ‘qS ))Útarget)ZProcessÚ_child)r3 rä ©Úmultiprocessingru r r r6 ¼ s ÿz8MultiprocessRefactoringTool.refactor..)rö rõ r£ rÿ Ú ImportErrorrô r÷ Ú RuntimeErrorZ
JoinableQueueZLockrø ÚrangeÚstartrƒ ÚputZis_alive)ru r r¡ r¢ Z
num_processesZ processesr$ rä rû rþ r r£ ¯ sF
ÿ
ÿ
ÿ
ú
z$MultiprocessRefactoringTool.refactorc s\ | j ¡ }|d urX|\}}z$tt| ƒj|i |¤Ž W | j ¡ n | j ¡ 0 | j ¡ }q
d S r8 )r÷ rh rö rõ rŸ Z task_done)ru Ztaskr rú rû r r rý Ì s
ÿÿz"MultiprocessRefactoringTool._childc s6 | j d ur | j ||f¡ ntt| ƒj|i |¤ŽS d S r8 )r÷ r rö rõ rŸ rù rû r r rŸ × s
ÿÿz)MultiprocessRefactoringTool.refactor_file)FFr )r r r rv r£ rý rŸ Ú
__classcell__r r rû r rõ ¨ s ÿ rõ )T)#rX Ú
__author__rE r› r rÄ rj r‰ r& Ú itertoolsr Zpgen2r r r Z
fixer_utilr ry r r r
rq r r" r r r0 r7 r: rV rW ÚobjectrY rô rõ r r r r Ú s8
(