ݺ d Z ddlZddlZddlZddlmZmZ ddlmZ ddl m
Z
m Z m Z m
Z
mZ ddlmZmZmZmZmZmZ ddlmZ g dZ ej d Zg d
Z e d Z G d d
e Z G d de Z G d de Z! G d de" Z# G d d Z$ e$ Z% G d de& Z' G d d Z(d Z) G d d Z* e* Z+d Z, G d d Z- G d! d"e Z. G d# d$e Z/ G d% d&e e/ Z0 G d' d( Z1 G d) d*e0 Z2d+ Z3 G d, d-e/ Z4 G d. d/e0e4 Z5dS )0z
pygments.lexer
~~~~~~~~~~~~~~
Base lexer classes.
:copyright: Copyright 2006-2025 by the Pygments team, see AUTHORS.
:license: BSD, see LICENSE for details.
N)
apply_filtersFilter)get_filter_by_name)ErrorTextOther
Whitespace
_TokenType) get_bool_opt get_int_opt get_list_optmake_analysatorFuture guess_decode) regex_opt)
Lexer
RegexLexerExtendedRegexLexerDelegatingLexer LexerContextincludeinheritbygroupsusingthisdefaultwordsline_rez.*?
))s utf-8)s zutf-32)s zutf-32be)s zutf-16)s zutf-16bec dS )N )xs l/builddir/build/BUILD/cloudlinux-venv-1.0.10/venv/lib/python3.11/site-packages/pip/_vendor/pygments/lexer.pyr% " s # c e Zd ZdZd ZdS ) LexerMetaz
This metaclass automagically converts ``analyse_text`` methods into
static methods which always return float values.
c t d|v rt |d |d< t | ||| S )N analyse_text)r type__new__)mcsnamebasesds r$ r, zLexerMeta.__new__+ s< Q /.0A B BAn ||Cua000r& N)__name__
__module__ __qualname____doc__r, r" r& r$ r( r( % s-
1 1 1 1 1r& r( c b e Zd ZdZdZg Zg Zg Zg ZdZ dZ
dZ dZ d Z
d Zd Zd Zd Zd d
Zd ZdS )
r a"
Lexer for a specific language.
See also :doc:`lexerdevelopment`, a high-level guide to writing
lexers.
Lexer classes have attributes used for choosing the most appropriate
lexer based on various criteria.
.. autoattribute:: name
:no-value:
.. autoattribute:: aliases
:no-value:
.. autoattribute:: filenames
:no-value:
.. autoattribute:: alias_filenames
.. autoattribute:: mimetypes
:no-value:
.. autoattribute:: priority
Lexers included in Pygments should have two additional attributes:
.. autoattribute:: url
:no-value:
.. autoattribute:: version_added
:no-value:
Lexers included in Pygments may have additional attributes:
.. autoattribute:: _example
:no-value:
You can pass options to the constructor. The basic options recognized
by all lexers and processed by the base `Lexer` class are:
``stripnl``
Strip leading and trailing newlines from the input (default: True).
``stripall``
Strip all leading and trailing whitespace from the input
(default: False).
``ensurenl``
Make sure that the input ends with a newline (default: True). This
is required for some lexers that consume input linewise.
.. versionadded:: 1.3
``tabsize``
If given and greater than 0, expand tabs in the input (default: 0).
``encoding``
If given, must be an encoding name. This encoding will be used to
convert the input string to Unicode, if it is not already a Unicode
string (default: ``'guess'``, which uses a simple UTF-8 / Locale /
Latin1 detection. Can also be ``'chardet'`` to use the chardet
library, if it is installed.
``inencoding``
Overrides the ``encoding`` if given.
Nr c || _ t |dd | _ t |dd | _ t |dd | _ t |dd | _ | dd | _ | d
p| j | _ g | _ t |d d D ]}| | d
S )a
This constructor takes arbitrary options as keyword arguments.
Every subclass must first process its own options and then call
the `Lexer` constructor, since it processes the basic
options like `stripnl`.
An example looks like this:
.. sourcecode:: python
def __init__(self, **options):
self.compress = options.get('compress', '')
Lexer.__init__(self, **options)
As these options must all be specifiable as strings (due to the
command line usage), there are various utility functions
available to help with that, see `Utilities`_.
stripnlTstripallFensurenltabsizer encodingguess
inencodingfiltersr" N) optionsr r7 r8 r9 r r: getr; r> r
add_filter)selfr? filter_s r$ __init__zLexer.__init__ s & #GY== $Wj%@@
$Wj$??
"7Iq99 J88
L11BT]
#GY;; % %G OOG $ $ $ $ % %r& c ` | j rd| j j d| j dS d| j j dS )Nz)r? __class__r1 rB s r$ __repr__zLexer.__repr__ sC < BWt~'>WWdlWWW WAt~'>AAA Ar& c ~ t |t s
t |fi |}| j | dS )z8
Add a new stream filter to this lexer.
N)
isinstancer r r> append)rB rC r? s r$ rA zLexer.add_filter sG '6** =(<.streamer sC 66t<<
1ad
r& )rc r r> )rB rN
unfilteredrj streams `` r$
get_tokenszLexer.get_tokens sd ++D11 ?"64<>>F
r& c t )aS
This method should process the text and return an iterable of
``(index, tokentype, value)`` tuples where ``index`` is the starting
position of the token within the input text.
It must be overridden by subclasses. It is recommended to
implement it as a generator to maximize effectiveness.
)NotImplementedError)rB rN s r$ rg z Lexer.get_tokens_unprocessed s
"!r& )F)r1 r2 r3 r4 r. aliases filenamesalias_filenames mimetypespriorityurl
version_added_examplerD rI rA r* rc rm rg r" r& r$ r r 1 s 8 8v D G
I O I H C M H % % %<B B B % % % "/ / /b 0 " " " " "r& r ) metaclassc " e Zd ZdZefdZd ZdS )r a
This lexer takes two lexer as arguments. A root lexer and
a language lexer. First everything is scanned using the language
lexer, afterwards all ``Other`` tokens are lexed using the root
lexer.
The lexers from the ``template`` lexer package use this base lexer.
c l |di || _ |di || _ || _ t j | fi | d S Nr" )
root_lexerlanguage_lexerneedler rD )rB _root_lexer_language_lexer_needler? s r$ rD zDelegatingLexer.__init__- sV %+0000 -o 8 8 8 8
t''w'''''r& c d}g }g }| j | D ]U\ }}}|| j u r.|r&| t | |f g }||z
}=| |||f V|r$| t | |f t || j | S )N )r} rg r~ rL rX
do_insertionsr| )rB rN buffered
insertions
lng_bufferirh ri s r$ rg z&DelegatingLexer.get_tokens_unprocessed3 s
*AA$GG - -GAq!DK $ %%s8}}j&ABBB!#JA
!!1a),,,, ; s8}}j 9 : : : Z !_ C CH M MO O Or& N)r1 r2 r3 r4 r rD rg r" r& r$ r r # sL >C ( ( ( ( O O O O Or& r c e Zd ZdZdS )r zI
Indicates that a state should include rules from another state.
Nr1 r2 r3 r4 r" r& r$ r r J s Dr& r c e Zd ZdZd ZdS )_inheritzC
Indicates the a state should inherit from its superclass.
c dS )Nr r" rH s r$ rI z_inherit.__repr__U s yr& N)r1 r2 r3 r4 rI r" r& r$ r r Q s- r& r c e Zd ZdZd Zd ZdS )combinedz:
Indicates a state combined from multiple states.
c 8 t | | S rf )tupler, )clsargss r$ r, zcombined.__new__` s }}S$'''r& c d S rf r" )rB r s r$ rD zcombined.__init__c s r& N)r1 r2 r3 r4 r, rD r" r& r$ r r [ s< ( ( (
r& r c < e Zd ZdZd Zd dZd dZd dZd Zd Z dS )
_PseudoMatchz:
A pseudo match object constructed from a string.
c " || _ || _ d S rf )_text_start)rB startrN s r$ rD z_PseudoMatch.__init__m s
r& Nc | j S rf )r rB args r$ r z_PseudoMatch.startq s
{r& c : | j t | j z S rf )r rX r r s r$ endz_PseudoMatch.endt s {S__,,r& c 2 |rt d | j S )Nz
No such group)
IndexErrorr r s r$ groupz_PseudoMatch.groupw s . _-- -zr& c | j fS rf )r rH s r$ groupsz_PseudoMatch.groups| s
} r& c i S rf r" rH s r$ groupdictz_PseudoMatch.groupdict s r& rf )
r1 r2 r3 r4 rD r r r r r r" r& r$ r r h s - - - -
r& r c d fd }|S )zL
Callback that yields multiple actions for each group in the match.
Nc
3 K t D ]\ }}|t | t u r8| |dz }|r | |dz ||fV V| |dz }|Y|r | |dz |_ || t
| |dz | | D ]}|r|V |r| |_ d S d S )N ) enumerater+ r
r r posr r )lexermatchctxr actiondataitemr s r$ callbackzbygroups..callback s* "4 ' 'IAv~f ++ {{1q5)) ; ++a!e,,fd:::: {{1q5))# 5"'++a!e"4"4 &u'3EKKA4F4F'M'Ms!T !T ' ' '"&JJJ "iikkCGGG " "r& rf r" )r r s ` r$ r r s( " " " " " "& Or& c e Zd ZdZdS )_ThiszX
Special singleton used for indicating the caller class.
Used by ``using``.
Nr r" r& r$ r r s r& r c i dv r> d }t |t t f r|d< nd|fd< t u rdfd }nd fd }|S )a
Callback that processes the match with a different lexer.
The keyword arguments are forwarded to the lexer, except `state` which
is handled separately.
`state` specifies the state that the new lexer will start in, and can
be an enumerable such as ('root', 'inline', 'string') or a simple
string which is assumed to be on top of the root state.
Note: For that to work, `_other` must not be an `ExtendedRegexLexer`.
statestackrootNc 3 ( K r( | j | j di }n| }| } |j | fi D ]\ }}}||z ||fV |r| |_ d S d S r{ )updater? rG r rg r r r )
r r r lxsr rh ri gt_kwargskwargss
r$ r zusing..callback s
em,,,$U_..v..
A424U[[]]PPiPP
"
"1a!eQk!!!!
& ))++
&
&r& c 3 K
| j di
}| } |j | fi D ]\ }}}||z ||fV |r| |_ d S d S r{ )r r? r rg r r r ) r r r r r r rh ri _otherr r s r$ r zusing..callback s MM%- ( ( (!!&!!B
A424U[[]]PPiPP
"
"1a!eQk!!!!
& ))++
&
&r& rf )poprK listr r )r r r r r s `` @r$ r r s I& JJw a$ ' ' -!"Ig "(!Ig
~~
&
&
&
&
&
&
&
& & & & & & & & & Or& c e Zd ZdZd ZdS )r z
Indicates a state or state action (e.g. #pop) to apply.
For example default('#pop') is equivalent to ('', Token, '#pop')
Note that state tuples may be used as well.
.. versionadded:: 2.0
c || _ d S rf )r )rB r s r$ rD zdefault.__init__ s
r& N)r1 r2 r3 r4 rD r" r& r$ r r s- r& r c e Zd ZdZddZd ZdS )r z
Indicates a list of literal words that is transformed into an optimized
regex that matches any of the words.
.. versionadded:: 2.0
r c 0 || _ || _ || _ d S rf )r prefixsuffix)rB r r r s r$ rD zwords.__init__ s
r& c D t | j | j | j S )Nr r )r r r r rH s r$ r@ z words.get s DK LLLLr& N)r r )r1 r2 r3 r4 rD r@ r" r& r$ r r sF
M M M M Mr& r c > e Zd ZdZd Zd Zd Zd Zd
dZd Z d Z
dS ) RegexLexerMetazw
Metaclass for RegexLexer, creates the self._tokens attribute from
self.tokens on the first instantiation.
c t |t r| }t j || j S )zBPreprocess the regular expression component of a token definition.)rK r r@ recompiler )r regexrflagsr s r$ _process_regexz RegexLexerMeta._process_regex s6 eV $ $ IIKKEz%((..r& c j t | t u s t | s
J d| |S )z5Preprocess the token component of a token definition.z0token type must be simple type or callable, not )r+ r
callable)r tokens r$ _process_tokenz RegexLexerMeta._process_token s< E{{j((HUOO(( Hu H H )(; r& c 2 t |t rJ|dk rdS ||v r|fS |dk r|S |dd dk rt |dd S J d| t |t rfd | j z }| xj d
z
c_ g }|D ]?}||k s
J d | | | ||| @|||< |fS t |t r |D ]}||v s|d v s
J d|z |S J d
| )z=Preprocess the state transition action of a token definition.#pop#pushN z#pop:Fzunknown new state z_tmp_%dr zcircular state ref )r r zunknown new state def )rK rT intr _tmpnameextend_process_stater )r new_state unprocessed processed tmp_stateitokensistates r$ _process_new_statez!RegexLexerMeta._process_new_state s i % % AF""rk))!|#g%% 2A2 '))IabbM****@ @9 @ @@@u
8
,
, A!CL0I LLA LLG#
F
F ***,L&,L,L***s 1 1+2;V E E F F F F#*Ii <
5
)
) A#
2
2 +-- "3333(61 434 @@9@@ @ @5r& c 0 t |t s
J d| |d dk s
J d| ||v r|| S g x}||< | j }|| D ]}t |t rK||k s
J d| | | ||t | ct |t ryt |t rL| |j || }|
t j d j
d|f t | t u s
J d| | |d || }n4# t" $ r'} t% d |d d
|d | d | | d} ~ ww xY w| |d
}
t) | dk rd}n | |d || }|
||
|f |S )z%Preprocess a single state definition.zwrong state name r #zinvalid state name zcircular state reference r Nzwrong rule def zuncompilable regex z
in state z of z: r )rK rT flagsr r r r r r r rL r r r r+ r r Exception
ValueErrorr rX ) r r r r tokensr tdefr rexerrr s r$ r z RegexLexerMeta._process_state) sr %%%DD'D5'D'DDD%Qx3 ?e ? ? I U# #$&&5! & 3 3D$((
u}}}&K%&K&K}}}
c 0 0i14T < < = = =$))
$((
2 24:{I V V
rz"~~ 3T9 EFFF::&&&(B$(B(B&&&
r((a&%@@
r
r
r !gtAw!g!gE!g!gY\!g!gbe!g!ghhnqq
r &&tAw//E4yyA~~ 2 2473> K K
MM3y1 2 2 2 2
s E::
F+"F&&F+Nc i x}| j |< |p | j | }t | D ]}| ||| |S )z-Preprocess a dictionary of token definitions.) _all_tokensr r r )r r. tokendefsr r s r$ process_tokendefz RegexLexerMeta.process_tokendefT sZ ,.. COD) 1D!1 )__ < >(B//D $
C
C u!::e,,#
%*F5M!&+kk'&:&: % ! ! ! !)4K& )ooeT : : & 7<[]23C #(++g"6"6K *5{)BK&& " D3
C<
s$ A::
BB4C
C$#C$c d| j vrSi | _ d| _ t | d r| j rn-| d| | _ t j | g|R i |S )z:Instantiate cls after preprocessing its token definitions._tokensr token_variantsr )
r r r hasattrr r r r r+ __call__)r r kwdss r$ r zRegexLexerMeta.__call__ s CL ( ( CO CLs ,--
L#2D
L ! 2 22s7H7H7J7J K K }S040004000r& rf ) r1 r2 r3 r4 r r r r r r r r" r& r$ r r s
/ / / !A !A !AF) ) )V / / /b 1 1 1 1 1r& r c , e Zd ZdZej Zi ZddZdS )r z
Base for simple stateful regular expression-based lexers.
Simplifies the lexing process so that you need only
provide a list of states and regular expressions.
r c # K d}| j }t | }||d } |D ]p\ }}} ||| }
|
rZ|Bt | t u r|||
fV n || |
E d{V |
}| t
| t rk| D ]g} | dk r(t | dk r| 0| dk r |
|d R|
| hnpt
| t r,t | t | k r|dd= n5|| d= n/| dk r |
|d n
J d| ||d } nVr || d k r d
g}|d
}|t d fV |dz
}|t || fV |dz
}n# t $ r Y dS w xY w) z~
Split ``text`` into (tokentype, text) pairs.
``stack`` is the initial stack (default: ``['root']``)
r r r Nr r Fwrong state def: rR r )r r r+ r
r r rK r rX r rL r absr r r ) rB rN r r r
statestack statetokensrexmatchr r mr s r$ rg z!RegexLexer.get_tokens_unprocessed s L %[[
2/ 1 /: 0
0
+&) HT3'' ) <<:55"%vqwwyy"8 8 8 8 8'-vdA 6 6 6 6 6 6 6%%''C ,%i77 L)2 = =#(F??'*:':':(2(8(8(8%*g%5%5$.$5$5jn$E$E$E$E$.$5$5e$<$<$<$< = ( 3 7 7 L #9~~Z @ @$.qrrNN$.yzz$:$: &' 1 1 & - -jn = = = = K*Ki*K*K K K5&/
2&? E? F CyD((&,X
&/&7 !:t 3333q ud3i////1HCC! EEa1 s (G! G! !
G/.G/Nr ) r1 r2 r3 r4 r MULTILINEr r rg r" r& r$ r r sB
LE0 F; ; ; ; ; ;r& r c e Zd ZdZddZd ZdS )r z9
A helper object that holds lexer position data.
Nc b || _ || _ |pt | | _ |pdg| _ d S )Nr )rN r rX r r )rB rN r r r s r$ rD zLexerContext.__init__ s4 ##d))&vh
r& c 8 d| j d| j d| j dS )Nz
LexerContext(z, ))rN r r rH s r$ rI zLexerContext.__repr__ s) KtyKKdhKKDJKKKKr& NN)r1 r2 r3 r4 rD rI r" r& r$ r r sF ' ' ' ' L L L L Lr& r c e Zd ZdZddZdS )r zE
A RegexLexer that uses a context object to store its state.
Nc # ` K | j }|st |d }|d }n |}||j d }|j } |D ]\ }}} |||j |j } | r|vt
| t u r8|j || fV | |_ n( || | | E d{V |s||j d }|5t |t r|D ]}
|
dk r2t |j dk r|j :|
dk r&|j
|j d f|j
|
nt |t r;t | t |j k r |j dd= nD|j |d= n9|dk r&|j
|j d n
J d | ||j d } n |j |j k rdS ||j d
k r3dg|_ |d }|j t d
fV |xj dz
c_ /|j t" ||j fV |xj dz
c_ n# t$ $ r Y dS w xY wn) z
Split ``text`` into (tokentype, text) pairs.
If ``context`` is given, use this lexer context instead.
r r r r Nr r Fr
rR )r r r rN r r r+ r
r rK r rX r rL r r r r r ) rB rN contextr r r
r r r r r s r$ rg z)ExtendedRegexLexer.get_tokens_unprocessed s
L tQ''C#F+KKC#CIbM2K8D3 /: 2
2
+&) HT37CG44 ! ) <<:55"%'617799"< < < <&'eeggCGG'-vdAs';'; ; ; ; ; ; ; ;#, G.7 "
.F ,%i77 L)2 < <#(F??'*39~~'9'9(+
%*g%5%5$'I$4$4SYr]$C$C$C$C$'I$4$4U$;$;$;$; < ( 3 7 7 L "9~~SY ? ?$'IabbMM$'Iijj$9$9 &' 1 1 I , ,SYr] ; ; ; ; K*Ki*K*K K K5&/ "
&> EC!F
w#')) CG},,%+H &/&7 !gtT 11111 '5$sw-7777GGqLGGG! EEe3 s J ,AJ 0,J
J+*J+r )r1 r2 r3 r4 rg r" r& r$ r r s8 @ @ @ @ @ @r& r c # K t | } t | \ }}n# t $ r |E d{V Y dS w xY wd}d}|D ]\ }}}||}d} |r|t | z |k r|| ||z
}
|
r|||
fV |t |
z
}|D ] \ } } }
|| |
fV |t |
z
} ||z
} t | \ }}n# t $ r d}Y n w xY w|r|t | z |k | t | k r$|||| d fV |t | | z
z
}|rQ|pd}|D ] \ }}}|||fV |t | z
} t | \ }}n# t $ r d}Y dS w xY w|OdS dS )ag
Helper for lexers which must combine the results of several
sublexers.
``insertions`` is a list of ``(index, itokens)`` pairs.
Each ``itokens`` iterable should be inserted at position
``index`` into the token stream given by the ``tokens``
argument.
The result is a combined token stream.
TODO: clean up the code here.
NTr F)iternext
StopIterationrX )r r r r realposinsleftr rh ri olditmpvalit_indexit_tokenit_valueps r$ r r S s{ j!!J j))ww
GG % %1a ?G
!c!ff*--tEAI~&F
' q&((((3v;;&07
)
),(H x11113x==(19D
!%j!1!1ww
!c!ff*-- #a&&==1ah& & & & s1vv} $G
,Q GAq!1a- s1vv GG !* - -NE77 G EE
s0 & <<9C CCE* * E:9E:c e Zd ZdZd ZdS )ProfilingRegexLexerMetaz>Metaclass for ProfilingRegexLexer, collects regex timing info.c t |t r"t |j |j |j n|t j | t j f fd }|S )Nr c j d
fddg }t j } | || }t j }|dxx dz
cc< |dxx ||z
z
cc< |S )Nr r r! r )
_prof_data
setdefaulttimer ) rN r endposinfot0rest1r compiledr r s r$
match_funcz:ProfilingRegexLexerMeta._process_regex..match_func s >"%00% 3xHHDB..sF33CB GGGqLGGG GGGrBw GGGJr& ) rK r r r r r r sysmaxsize)r r r r r3 r2 r s ` ` @@r$ r z&ProfilingRegexLexerMeta._process_regex s eU # # EK #(<1 1 1CC C:c6**), r& N)r1 r2 r3 r4 r r" r& r$ r' r' s) HH r& r' c " e Zd ZdZg ZdZddZdS )ProfilingRegexLexerzFDrop-in replacement for RegexLexer that does profiling of its regexes. r c # ~ K j j i t || E d {V j j }t
d | D fdd }t d |D }t t d j j
t | |fz t d t dd z t d
|D ]}t d |z t d d S ) Nc 3 K | ]Y\ \ }}\ }}|t | d dd dd |d|z d|z |z fV ZdS )zu'z\\\NA i )reprr[ rQ ).0r rnrh s r$ z=ProfilingRegexLexer.get_tokens_unprocessed.. s @ @+FQFQ 477==//77EEcrcJ4!8TAX\3 @ @ @ @ @ @r& c | j S rf )_prof_sort_index)r# rB s r$ r% z. s Ad&;$< r& T)keyreversec 3 & K | ] }|d V
dS ) Nr" )r> r# s r$ rA z=ProfilingRegexLexer.get_tokens_unprocessed.. s& ++!++++++r& z2Profiling result for %s lexing %d chars in %.3f mszn==============================================================================================================z$%-20s %-64s ncalls tottime percall)r r zn--------------------------------------------------------------------------------------------------------------z%-20s %-65s %5d %8.4f %8.4f) rG r* rL r rg r sortedr sumprintr1 rX )rB rN r rawdatar sum_totalr0 s ` r$ rg z*ProfilingRegexLexer.get_tokens_unprocessed so !((,,, 44T4GGGGGGGGG.+//11 @ @/6}}@ @ @ =<<< " $ $ $
++d+++++
B~&D 9=> ? ? ?
i
47IIJJJ
i 5 5A /!3 4 4 4 4
ir& Nr )r1 r2 r3 r4 r* rC rg r" r& r$ r7 r7 s9 PPJ r& r7 )6r4 r r4 r, pip._vendor.pygments.filterr r pip._vendor.pygments.filtersr pip._vendor.pygments.tokenr r r r r
pip._vendor.pygments.utilr r r
r r r pip._vendor.pygments.regexoptr __all__r r rV staticmethod_default_analyser+ r( r r rT r r r r r r r r r r r r r r r r r r' r7 r" r& r$ rU sg
= = = = = = = = ; ; ; ; ; ; Q Q Q Q Q Q Q Q Q Q Q Q Q Q* * * * * * * * * * * * * * * * 3 3 3 3 3 3 * * * "*W
, , ,
<
.. 1 1 1 1 1 1 1 1o" o" o" o" o"i o" o" o" o"d O O O O Oe O O ON c (**
u
6 4
uww/ / /d
M
M
M
M
MF
M
M
M d1 d1 d1 d1 d1Y d1 d1 d1N^ ^ ^ ^ ^. ^ ^ ^ ^B L L L L L L L L E E E E E E E EP= = =@ n , *0G r&