V?vx d Z ddlZddlZddlZddlmZmZmZ ddlm Z ddl
m Z ddl m
Z
mZmZ ddlmZ dd lmZ dd
lmZ dd lmZ dd lmZ dd
lmZ ddl m Z G d d Z dS )a
Module containing the UniversalDetector detector class, which is the primary
class a user of ``chardet`` should use.
:author: Mark Pilgrim (initial port to Python)
:author: Shy Shalom (original C code)
:author: Dan Blanchard (major refactoring for 3.0)
:author: Ian Cordasco
N)ListOptionalUnion )CharSetGroupProber)
CharSetProber)
InputStateLanguageFilter ProbingState)EscCharSetProber) Latin1Prober)MacRomanProber)MBCSGroupProber)
ResultDict)SBCSGroupProber)
UTF1632Proberc X e Zd ZdZdZ ej d Z ej d Z ej d Z dddd d
d d d
dZ
dddd ddddZ e j
dfde deddfdZedefd Zedefd Zedee fd Zd!d Zd eeef ddfd Zde fd Z dS )"UniversalDetectoraq
The ``UniversalDetector`` class underlies the ``chardet.detect`` function
and coordinates all of the different charset probers.
To get a ``dict`` containing an encoding and its confidence, you can simply
run:
.. code::
u = UniversalDetector()
u.feed(some_bytes)
u.close()
detected = u.result
g?s [-]s (|~{)s [-]z Windows-1252z Windows-1250z Windows-1251z Windows-1256z Windows-1253z Windows-1255z Windows-1254z Windows-1257)
iso-8859-1z
iso-8859-2z
iso-8859-5z
iso-8859-6z
iso-8859-7z
iso-8859-8
iso-8859-9z iso-8859-13z ISO-8859-11GB18030CP949UTF-16)asciir ztis-620r gb2312zeuc-krzutf-16leF lang_filtershould_rename_legacyreturnNc d | _ d | _ g | _ d dd d| _ d| _ d| _ t j | _ d| _ || _
t j t | _ d| _ || _ | d S )N encoding
confidencelanguageF )_esc_charset_prober_utf1632_prober_charset_probersresultdone _got_datar
PURE_ASCII _input_state
_last_charr logging getLogger__name__logger_has_win_bytesr reset)selfr r s k/builddir/build/BUILD/cloudlinux-venv-1.0.10/venv/lib/python3.11/site-packages/chardet/universaldetector.py__init__zUniversalDetector.__init__d s
@D 8< 57 #
#
& 1 & '11 #$8!
r% c | j S N)r- r5 s r6 input_statez UniversalDetector.input_state{ s r% c | j S r9 )r3 r: s r6
has_win_bytesz UniversalDetector.has_win_bytes s ""r% c | j S r9 )r( r: s r6 charset_probersz!UniversalDetector.charset_probers s $$r% c 2 dddd| _ d| _ d| _ d| _ t j | _ d| _ | j r| j | j
r| j
| j D ]}| dS )z
Reset the UniversalDetector and all of its probers back to their
initial states. This is called by ``__init__``, so you only need to
call this directly in between analyses of different documents.
Nr r! Fr% ) r) r* r+ r3 r r, r- r. r& r4 r' r( )r5 probers r6 r4 zUniversalDetector.reset s $(sMM # & 1 # - $ * * , , , ) & & ( ( (+ F LLNNNN r% byte_strc p | j rdS |sdS t |t st | }| j s| t
j r dddd| _ n| t
j t
j f r dddd| _ nx| d r dddd| _ nW| d r d
ddd| _ n6| t
j
t
j f r d ddd| _ d | _ | j d
d | _ dS | j t j k rt| j | rt j | _ nH| j t j k r3| j | j |z rt j | _ |dd | _ | j st- | _ | j j t0 j k r]| j | t0 j k r5| j j | j dd| _ d | _ dS | j t j k r| j st? | j | _ | j | t0 j k r?| j j | j | j j! d| _ d | _ dS dS | j t j k r'| j" stG | j g| _" | j tH j% z r&| j" & tO | j" & tQ | j" & tS | j" D ]U}| | t0 j k r0|j | |j! d| _ d | _ nV| j* | r d | _+ dS dS dS )a
Takes a chunk of a document and feeds it through all of the relevant
charset probers.
After calling ``feed``, you can check the value of the ``done``
attribute to see if you need to continue feeding the
``UniversalDetector`` more data, or if it has made a prediction
(in the ``result`` attribute).
.. note::
You should always call ``close`` when you're done feeding in your
document if ``done`` is not already ``True``.
Nz UTF-8-SIG ? r! zUTF-32s zX-ISO-10646-UCS-4-3412s zX-ISO-10646-UCS-4-2143r Tr" ),r*
isinstance bytearrayr+
startswithcodecsBOM_UTF8r) BOM_UTF32_LE BOM_UTF32_BEBOM_LEBOM_BEr- r r, HIGH_BYTE_DETECTORsearch HIGH_BYTE ESC_DETECTORr. ESC_ASCIIr' r stater DETECTINGfeedFOUND_IT charset_nameget_confidencer& r r r$ r( r r
NON_CJKappendr r
r WIN_BYTE_DETECTORr3 )r5 rB rA s r6 rW zUniversalDetector.feed s] 9 F F(I.. + **H ~ % ""6?33
X !,"% "
$$f&96;N%OPP
X ,43TV W W $$%899
X !9"% " $$%899
X !9"% " $$fmV]%CDD
X ,43TV W W !DN{:&2
5 5 5&--h77
9$.$8!!!Z%:::%,,T_x-GHH ; %/$8!"233- # 3#0??D % )? ? ?#((22l6KKK $ 4 A"&"6"E"E"G"G "
!
4 4 4+
N+;D>> /
;;x((L,AAA$*$7&,&;&;&=&=$*O# #DK
!%DIE B %,,X66
+&*###% 7
6"
+
+r% c | j r| j S d| _ | j s | j d n%| j t j k r
dddd| _ n| j t j k rd}d}d}| j D ]#}|s|
}||k r|}|}$|r|| j k r|j }|J |
}|
}| d r"| j r| j || }| j r/| j |pd
| }|||j d| _ | j t, j k r| j d
| j d | j D ]}|st1 |t2 rD|j D ];}| j d |j |j |
<^| j d |j |j |
| j S )
z
Stop analyzing the current document and come up with a final
prediction.
:returns: The ``result`` attribute, a ``dict`` with the keys
`encoding`, `confidence`, and `language`.
Tzno data received!r rD rE r! Nr ziso-8859r" z no probers hit minimum thresholdz%s %s confidence = %s)r* r) r+ r2 debugr- r r, rR r( rZ MINIMUM_THRESHOLDrY lowerrI r3 ISO_WIN_MAPgetr
LEGACY_MAPr$ getEffectiveLevelr/ DEBUGrG r probers) r5 prober_confidencemax_prober_confidence
max_proberrA rY lower_charset_namer# group_probers r6 closezUniversalDetector.close s 9 ; ~ ( K 1 2 2 2 2
*"7
7
7'.crRRDKK
*"6
6
6 $ $' ! J /
(
( $*$9$9$;$;!$'<<<,=)!'J
4t7M M M ) 6 #///%1%7%7%9%9" ' 6 6 8 8
&00<< * '+'7';'; . ( ( , #'?#6#6%+2244l$ $L !-", * 3 ; ( ( * *gm ; ;{:&. !!"DEEE$($9 L' ! !,0BCC &2&: F K - - 7 & 3 & & 5 5 7 7 )) 3 ( 5 ( 1 ( 7 7 9 9 {r% )r N) r1
__module__ __qualname____doc__r` recompilerP rS r] rb rd r
ALLboolr7 propertyintr; r= r r r? r4 r bytesrH rW r rm r% r6 r r 8 s #N33 2:l++L"
>22$$$$$$$% K $ $ J '5&8%* # #
. !S ! ! ! X! #t # # # X# %m!4 % % % X% &A+U5)#3 4 A+ A+ A+ A+ A+FMz M M M M M Mr% r ) rp rJ r/ rq typingr r r charsetgroupproberr
charsetproberr enumsr r
r escproberr latin1proberr
macromanproberr mbcsgroupproberr
resultdictr sbcsgroupproberr
utf1632proberr r rx r% r6 r sF 8
( ( ( ( ( ( ( ( ( ( 2 2 2 2 2 2 ( ( ( ( ( ( ; ; ; ; ; ; ; ; ; ; ' ' ' ' ' ' & & & & & & * * * * * * , , , , , , " " " " " " , , , , , , ( ( ( ( ( (r r r r r r r r r rr%