o
0¦isk ã @ s d Z dZddlZddlZddlZddlZddlZddlZddlZddl m
Z
ddl m Z m
Z
mZ ddlmZ ddlmZmZ dd lmZ d!d d „ZG d
d„ deƒZdd„ Zdd„ Zdd„ Zdd„ Z dd„ Z G dd„ deƒZ G dd „ d e ƒZ G d d „ d eƒZ!G d d „ d e ƒZ"dS )"zÛRefactoring framework.
Used as a main program, this can refactor any number of files and/or
recursively descend down directories. Imported as a module, this
provides infrastructure to write your own refactoring tool.
z#Guido van Rossum é N)Úchainé )ÚdriverÚtokenizeÚtoken)Ú find_root)ÚpytreeÚpygram)Ú btm_matcherTc C sT t | g g dgƒ}g }t |j¡D ]\}}}| d¡r'|r"|dd… }| |¡ q|S )zEReturn a sorted list of all available fix names in the given package.Ú*Úfix_é N)Ú
__import__ÚpkgutilZ iter_modulesÚ__path__Ú
startswithÚappend)Z fixer_pkgZ
remove_prefixZpkgZ fix_namesÚfinderÚnameZispkg© r ú7/opt/alt/python310/lib64/python3.10/lib2to3/refactor.pyÚget_all_fix_names s
€r c @ ó e Zd ZdS )Ú
_EveryNodeN©Ú__name__Ú
__module__Ú __qualname__r r r r r + ó r c C sŽ t | tjtjfƒr| jdu rt‚| jhS t | tjƒr$| jr"t| jƒS t‚t | tj ƒrAt
ƒ }| jD ]}|D ] }| t|ƒ¡ q4q0|S t d| ƒ‚)zf Accepts a pytree Pattern Node and returns a set
of the pattern types which will match first. Nz$Oh no! I don't understand pattern %s)
Ú
isinstancer Z NodePatternZ LeafPatternÚtyper ZNegatedPatternZcontentÚ_get_head_typesZWildcardPatternÚsetÚupdateÚ Exception)ZpatÚrÚpÚxr r r r! / s
ÿ r! c C s¼ t t¡}g }| D ];}|jr1zt|jƒ}W n ty# | |¡ Y q w |D ] }|| |¡ q&q |jdur?||j |¡ q | |¡ q tt j
j ¡ t j
j
ƒD ] }|| |¡ qPt|ƒS )z^ Accepts a list of fixers and returns a dictionary
of head node type --> fixer list. N)Ú collectionsÚ defaultdictÚlistÚpatternr! r r Z _accept_typer r Úpython_grammarZ
symbol2numberÚvaluesÚtokensÚextendÚdict)Z
fixer_listZ
head_nodesZeveryÚfixerZheadsZ node_typer r r Ú_get_headnode_dictK s(
ÿÿ
ÿr2 c s ‡ fdd„t ˆ dƒD ƒS )zN
Return the fully qualified names for fixers in the package pkg_name.
c s g | ]}ˆ d | ‘qS ©Ú.r )Ú.0Úfix_name©Zpkg_namer r Ú
h s ÿz+get_fixers_from_package..F)r r7 r r7 r Úget_fixers_from_packaged s
ÿr9 c C s | S ©Nr )Úobjr r r Ú _identityk s r< c sj d}t t | ¡j¡‰ ‡ fdd„}ttjt jtj hƒ}t
ƒ }zŠ |ƒ \}}||v r*q |tj kr5|r2ncd}n_|tj kr“|dkr“|ƒ \}}|tj ksL|dkrMnR|ƒ \}}|tj ks[|dkr\nH|ƒ \}}|tj
kro|dkro|ƒ \}}|tj kr’| |¡ |ƒ \}}|tj
ks‡|d krˆn
|ƒ \}}|tj kstnnq W t|ƒS W t|ƒS W t|ƒS W t|ƒS ty´ Y t|ƒS w )
NFc s t ˆ ƒ} | d | d fS )Nr r )Únext)Útok©Úgenr r Úadvancer s z(_detect_future_features..advanceTÚfromZ
__future__Úimportú(ú,)r Úgenerate_tokensÚioÚStringIOÚreadlineÚ frozensetr ÚNEWLINEÚNLÚCOMMENTr" ÚSTRINGÚNAMEÚOPÚaddÚ
StopIteration)ÚsourceZhave_docstringrA ÚignoreÚfeaturesÚtpÚvaluer r? r Ú_detect_future_featureso s\
û€çýïò þþrX c @ s e Zd ZdZdS )Ú
FixerErrorz A fixer could not be loaded.N)r r r Ú__doc__r r r r rY — s rY c @ sê e Zd ZddddœZdZdZd4dd„Zdd „ Zd
d „ Zd d
„ Z dd„ Z
dd„ Z d5dd„Z d5dd„Z
dd„ Zd5dd„Zdd„ Zd6d d „Zd d „ Zd d!„ Z d7d"d#„Zd8d$d%„Zd&Zd'Zd(d)„ Zd*d+„ Zd,d-„ Zd.d/„ Zd0d1„ Z d2d3„ Z dS )9ÚRefactoringToolF)Úprint_functionÚ
exec_functionÚwrite_unchanged_filesZFixr Nc C sH || _ |pg | _| j ¡ | _|dur| j |¡ tj ¡ | _| jd r)| jj d= n
| jd r3| jj d= | j
d¡| _ g | _ t
d¡| _g | _d| _tj| jtj| jd | _| ¡ \| _| _g | _t ¡ | _ g | _ g | _ t | j| jƒD ]#}|j r~| j !|¡ qr|| jv rŠ| j "|¡ qr|| jv r•| j "|¡ qrt#| j ƒ| _$t#| j ƒ| _%dS )
zÑInitializer.
Args:
fixer_names: a list of fixers to import
options: a dict with configuration.
explicit: a list of fixers to run even if they are explicit.
Nr\ Úprintr] Úexecr^ r[ F)ÚconvertÚlogger)&ÚfixersÚexplicitÚ_default_optionsÚcopyÚoptionsr# r r, ÚgrammarÚkeywordsÚgetr^ ÚerrorsÚloggingZ getLoggerrb Ú fixer_logÚwroter ZDriverr ra Ú
get_fixersÚ pre_orderÚ
post_orderÚfilesÚbmZ
BottomMatcherÚBMZ
bmi_pre_orderZbmi_post_orderr Z
BM_compatibleZ add_fixerr r2 Úbmi_pre_order_headsÚbmi_post_order_heads)ÚselfZ fixer_namesrg rd r1 r r r Ú__init__¤ sD
þ
€ zRefactoringTool.__init__c C sJ g }g }| j D ]ˆ}t|i i dgƒ}| dd¡d }| | j¡r(|t| jƒd… }| d¡}| jd dd „ |D ƒ¡ }zt ||ƒ}W n t
yQ t d
||f ƒd‚w || j | j
ƒ} | jrm| jd urm|| jvrm| d |¡ q| d
|¡ | jdkr~| | ¡ q| jdkr‰| | ¡ qt d| j ƒ‚t d¡}
|j|
d |j|
d ||fS )a Inspects the options to load the requested patterns and handlers.
Returns:
(pre_order, post_order), where pre_order is the list of fixers that
want a pre-order AST traversal, and post_order is the list that want
post-order traversal.
r r4 r éÿÿÿÿNÚ_Ú c S s g | ]}| ¡ ‘qS r )Útitle)r5 r& r r r r8 ë s z.RefactoringTool.get_fixers..zCan't find %s.%sTzSkipping optional fixer: %szAdding transformation: %sZpreZpostzIllegal fixer order: %rZ run_order©Úkey)rc r Úrsplitr Ú FILE_PREFIXÚlenÚsplitÚ CLASS_PREFIXÚjoinÚgetattrÚAttributeErrorrY rg rm rd Ú log_messageÚ log_debugZorderr ÚoperatorÚ
attrgetterÚsort) rw Zpre_order_fixersZpost_order_fixersZ fix_mod_pathÚmodr6 ÚpartsÚ
class_nameZ fix_classr1 Zkey_funcr r r ro Û s:
ÿ
zRefactoringTool.get_fixersc O s ‚ )z Called when an error occurs.r )rw ÚmsgÚargsÚkwdsr r r Ú log_error s zRefactoringTool.log_errorc G s |r|| }| j |¡ dS )zHook to log a message.N)rb Úinfo©rw r r r r r r‡ s zRefactoringTool.log_messagec G s |r|| }| j |¡ d S r: )rb Údebugr” r r r rˆ
s zRefactoringTool.log_debugc C s dS )zTCalled with the old version, new version, and filename of a
refactored file.Nr )rw Úold_textÚnew_textÚfilenameÚequalr r r Ú print_output s z RefactoringTool.print_outputc C s8 |D ]}t j |¡r| |||¡ q| |||¡ qdS )z)Refactor a list of files and directories.N)ÚosÚpathÚisdirÚ refactor_dirÚ
refactor_file)rw ÚitemsÚwriteÚ
doctests_onlyZ dir_or_filer r r Úrefactor s
üzRefactoringTool.refactorc
C sš t jd }t |¡D ]@\}}}| d|¡ | ¡ | ¡ |D ] }| d¡s>t j |¡d |kr>t j ||¡} | | ||¡ q dd„ |D ƒ|dd…< q
dS )zÄDescends down a directory and refactor every Python file found.
Python files are assumed to have a .py extension.
Files and subdirectories starting with '.' are skipped.
ÚpyzDescending into %sr4 r c S s g | ] }| d ¡s|‘qS r3 )r )r5 Zdnr r r r8 2 ó z0RefactoringTool.refactor_dir..N)
r› ÚextsepÚwalkrˆ r‹ r rœ Úsplitextr„ rŸ )
rw Zdir_namer¡ r¢ Zpy_extÚdirpathZdirnamesÚ filenamesr Úfullnamer r r rž s
€öz RefactoringTool.refactor_dirc
C s® zt |dƒ}W n ty! } z| d||¡ W Y d}~dS d}~ww zt |j¡d }W | ¡ n| ¡ w tj |d|dd}| ¡ |fW d ƒ S 1 sPw Y dS ) zG
Do our best to decode a Python source file correctly.
ÚrbzCan't open %s: %sN©NNr r% r{ ©ÚencodingÚnewline) ÚopenÚOSErrorr’ r Údetect_encodingrI ÚcloserG Úread)rw r˜ ÚfÚerrr¯ r r r Ú_read_python_source4 s €þ
$ÿz#RefactoringTool._read_python_sourcec C sÀ | |¡\}}|du r
dS |d7 }|r9| d|¡ | ||¡}| js&||kr1| |||||¡ dS | d|¡ dS | ||¡}| jsG|rX|jrX| jt|ƒdd… |||d dS | d|¡ dS )zRefactors a file.NÚ
zRefactoring doctests in %szNo doctest changes in %sry )r¡ r¯ zNo changes in %s)r¸ rˆ Úrefactor_docstringr^ Úprocessed_fileÚrefactor_stringÚ was_changedÚstr)rw r˜ r¡ r¢ Úinputr¯ ÚoutputÚtreer r r rŸ D s
ÿz RefactoringTool.refactor_filec
C s° t |ƒ}d|v r
tj| j_z3z| j |¡}W n# ty9 } z| d||jj |¡ W Y d}~W | j| j_dS d}~ww W | j| j_n| j| j_w ||_
| d|¡ | ||¡ |S )aF Refactor a given input string.
Args:
data: a string holding the code to be refactored.
name: a human-readable name for use in error/log messages.
Returns:
An AST corresponding to the refactored input stream; None if
there were errors during the parse.
r\ zCan't parse %s: %s: %sNzRefactoring %s)
rX r Z!python_grammar_no_print_statementr rh Z parse_stringr$ r’ Ú __class__r Úfuture_featuresrˆ Ú
refactor_tree)rw Údatar rU rÁ r· r r r r¼ [ s&
ÿ €ûÿ z RefactoringTool.refactor_stringc C s’ t j ¡ }|r)| d¡ | |d¡}| js||kr"| |d|¡ d S | d¡ d S | |d¡}| js7|rB|jrB| t |ƒd|¡ d S | d¡ d S )Nz Refactoring doctests in stdinzzNo doctest changes in stdinzNo changes in stdin)
ÚsysÚstdinrµ rˆ rº r^ r» r¼ r½ r¾ )rw r¢ r¿ rÀ rÁ r r r Úrefactor_stdinv s
z RefactoringTool.refactor_stdinc
C sÌ t | j| jƒD ]}| ||¡ q| | j| ¡ ¡ | | j| ¡ ¡ | j | ¡ ¡}t
| ¡ ƒrÓ| jj D ]˜}||v rÌ|| rÌ|| j
tjjdd |jrV|| j
tjjd t|| ƒD ]o}||| v rk|| |¡ zt|ƒ W n tyz Y q\w |jr„||jv r„q\| |¡}|rË| ||¡}|durË| |¡ | ¡ D ]}|js¦g |_|j |¡ qž| j | ¡ ¡}|D ]} | |vrÁg || < || || ¡ q·q\q4t
| ¡ ƒs0t | j| jƒD ]}| ||¡ qÚ|j S )aÏ Refactors a parse tree (modifying the tree in place).
For compatible patterns the bottom matcher module is
used. Otherwise the tree is traversed node-to-node for
matches.
Args:
tree: a pytree.Node instance representing the root of the tree
to be refactored.
name: a human-readable name for this tree.
Returns:
True if the tree was modified, False otherwise.
T)r~ Úreverser} N) r rp rq Z
start_treeÚ traverse_byru rv rt ÚrunZleavesÚanyr- rc r‹ r ZBaseZdepthZkeep_line_orderZ
get_linenor* Úremover Ú
ValueErrorZfixers_appliedÚmatchÚ transformÚreplacer r/ Z finish_treer½ )
rw rÁ r r1 Z match_setÚnodeÚresultsÚnewZ new_matchesZfxrr r r rÄ † sP ý
€ Ñ1z RefactoringTool.refactor_treec C sV |sdS |D ]"}||j D ]}| |¡}|r'| ||¡}|dur'| |¡ |}q
qdS )a Traverse an AST, applying a set of fixers to each node.
This is a helper method for refactor_tree().
Args:
fixers: a list of fixer instances.
traversal: a generator that yields AST nodes.
Returns:
None
N)r rÏ rÐ rÑ )rw rc Z traversalrÒ r1 rÓ rÔ r r r rÊ Õ s
€úÿzRefactoringTool.traverse_byc C sˆ | j |¡ |du r| |¡d }|du rdS ||k}| ||||¡ |r0| d|¡ | js0dS |r<| ||||¡ dS | d|¡ dS )zR
Called when a file has been refactored and there may be changes.
Nr zNo changes to %szNot writing changes to %s)rr r r¸ rš rˆ r^ Ú
write_file)rw r— r˜ r– r¡ r¯ r™ r r r r» ì s z RefactoringTool.processed_filec C sÈ z t j|d|dd}W n ty% } z| d||¡ W Y d}~dS d}~ww |) z| |¡ W n tyI } z
| d||¡ W Y d}~nd}~ww W d ƒ n1 sTw Y | d|¡ d| _dS ) zÑWrites a string to a file.
It first shows a unified diff between the old text and the new text, and
then rewrites the file; the latter is only done if the write option is
set.
Úwr{ r® zCan't create %s: %sNzCan't write %s: %szWrote changes to %sT)rG r± r² r’ r¡ rˆ rn )rw r— r˜ r– r¯ Úfpr· r r r rÕ s$ €þ€ÿ€ ý
zRefactoringTool.write_filez>>> z... c
C s g }d}d}d}d}|j ddD ]d}|d7 }| ¡ | j¡r?|dur-| | ||||¡¡ |}|g}| | j¡} |d| … }q|dur\| || j ¡sV||| j ¡ d kr\| |¡ q|durk| | ||||¡¡ d}d}| |¡ q|dur„| | ||||¡¡ d
|¡S )aË Refactors a docstring, looking for doctests.
This returns a modified version of the input string. It looks
for doctests, which start with a ">>>" prompt, and may be
continued with "..." prompts, as long as the "..." is indented
the same as the ">>>".
(Unfortunately we can't use the doctest module's parser,
since, like most parsers, it is not geared towards preserving
the original source.)
Nr T©Úkeependsr r¹ r{ ) Ú
splitlinesÚlstripr ÚPS1r/ Úrefactor_doctestÚfindÚPS2Úrstripr r„ )
rw r¿ r˜ ÚresultÚblockZ block_linenoÚindentÚlinenoÚlineÚir r r rº sB ÿ ÿ ÿ ÿ
z"RefactoringTool.refactor_docstringc
s z ˆ ||ˆ ¡}W n4 ty= } z(ˆj tj¡r&|D ] }ˆ d| d¡¡ qˆ d|||j j
|¡ |W Y d}~S d}~ww ˆ ||¡r…t |ƒj
dd}|d|d … ||d d… } }|d d¡sl|d d7 < ˆ ˆj | d ¡ g}|r…|‡ ‡fd
d „|D ƒ7 }|S ) zÞRefactors one doctest.
A doctest is given as a block of lines, the first of which starts
with ">>>" (possibly indented), while the remaining lines start
with "..." (identically indented).
z
Source: %sr¹ z+Can't parse docstring in %s line %s: %s: %sNTrØ r ry r c s g | ] }ˆ ˆj | ‘qS r )rß )r5 rå ©rã rw r r r8 ^ r¥ z4RefactoringTool.refactor_doctest..)Ú parse_blockr$ rb Z isEnabledForrl ÚDEBUGrˆ rà r’ r r rÄ r¾ rÚ ÚendswithrÜ Úpop)
rw râ rä rã r˜ rÁ r· rå rÔ Zclippedr rç r rÝ D s* ÿ€ú "z RefactoringTool.refactor_doctestc C sÐ | j rd}nd}| js| d|¡ n| d|¡ | jD ]}| |¡ q| jr6| d¡ | jD ]}| |¡ q.| jrdt| jƒdkrF| d¡ n | dt| jƒ¡ | jD ]\}}}| j|g|¢R i |¤Ž qRd S d S ) NÚwerez
need to bezNo files %s modified.zFiles that %s modified:z$Warnings/messages while refactoring:r zThere was 1 error:zThere were %d errors:)rn rr r‡ rm rk r )rw rì ÚfileÚmessager r r‘ r r r Ú summarizea s(
úzRefactoringTool.summarizec C s" | j | |||¡¡}tƒ |_|S )z³Parses a block into a tree.
This is necessary to get correct line number / offset information
in the parser diagnostics and embedded into the parse tree.
)r Z parse_tokensÚ wrap_toksrJ rà )rw râ rä rã rÁ r r r rè x s zRefactoringTool.parse_blockc c sf t | ||¡j¡}|D ]#\}}\}}\} }
} ||d 7 }| |d 7 } ||||f| |
f| fV q
dS )z;Wraps a tokenize stream to systematically modify start/end.r N)r rF Ú gen_linesÚ__next__) rw râ rä rã r. r rW Zline0Zcol0Zline1Zcol1Z line_textr r r rð ‚ s € øzRefactoringTool.wrap_toksc c sx || j }|| j }|}|D ]'}| |¡r |t|ƒd… V n|| ¡ d kr,dV ntd||f ƒ‚|}q dV q8)z–Generates lines as expected by tokenize from a list of lines.
This strips the first len(indent + self.PS1) characters off each line.
Nr¹ zline=%r, prefix=%rTr{ )rÜ rß r r rà ÚAssertionError)rw râ rã Úprefix1Zprefix2Úprefixrå r r r rñ s €
ÿzRefactoringTool.gen_linesr )FF)F)NFNr: ) r r r re rƒ r€ rx ro r’ r‡ rˆ rš r£ rž r¸ rŸ r¼ rÈ rÄ rÊ r» rÕ rÜ rß rº rÝ rï rè rð rñ r r r r r[ › sB þ
7(
O
ÿ
+
r[ c @ r )ÚMultiprocessingUnsupportedNr r r r r rö ¤ r rö c sF e Zd Z‡ fdd„Z d ‡ fdd„ Z‡ fdd„Z‡ fd d
„Z‡ ZS ) ÚMultiprocessRefactoringToolc s& t t| ƒj|i |¤Ž d | _d | _d S r: )Úsuperr÷ rx ÚqueueÚ output_lock©rw r Úkwargs©r r r rx ª s
z$MultiprocessRefactoringTool.__init__Fr c
s2 |dkrt tˆƒ |||¡S zdd l‰ W n ty t‚w ˆjd ur'tdƒ‚ˆ ¡ ˆ_ˆ ¡ ˆ_
‡ ‡fdd„t |ƒD ƒ}z8|D ]}| ¡ q@t tˆƒ |||¡ W ˆj
¡ t |ƒD ]}ˆj d ¡ q[|D ]
}| ¡ rp|
¡ qfd ˆ_d S ˆj
¡ t |ƒD ]}ˆj d ¡ q|D ]
}| ¡ r”|
¡ qŠd ˆ_w )Nr r z already doing multiple processesc s g | ] }ˆ j ˆjd ‘qS ))Útarget)ZProcessÚ_child)r5 ræ ©Úmultiprocessingrw r r r8 ¼ s ÿz8MultiprocessRefactoringTool.refactor..)rø r÷ r£ r Ú ImportErrorrö rù Ú RuntimeErrorZ
JoinableQueueZLockrú ÚrangeÚstartr„ ÚputZis_alive)rw r r¡ r¢ Z
num_processesZ processesr& ræ rý r r r£ ¯ sL
ÿ ÿ
ÿ
ÿ
€
ú €z$MultiprocessRefactoringTool.refactorc sf | j ¡ }|d ur1|\}}ztt| ƒj|i |¤Ž W | j ¡ n| j ¡ w | j ¡ }|d us d S d S r: )rù rj rø r÷ rŸ Z task_done)rw Ztaskr rü rý r r rÿ Ì s
ÿÿ
ùz"MultiprocessRefactoringTool._childc s4 | j d ur| j ||f¡ d S tt| ƒj|i |¤ŽS r: )rù r rø r÷ rŸ rû rý r r rŸ × s
ÿÿz)MultiprocessRefactoringTool.refactor_file)FFr )r r r rx r£ rÿ rŸ Ú
__classcell__r r rý r r÷ ¨ s ÿ r÷ )T)#rZ Ú
__author__rG r› r rÆ rl r‰ r( Ú itertoolsr Zpgen2r r r Z
fixer_utilr r{ r r r
rs r r$ r r! r2 r9 r< rX rY Úobjectr[ rö r÷ r r r r Ú s:
(