.b٢ d Z ddlZddlZddlZddlmZ ddlmZmZm Z ddl
m Z ddl m
Z
ddlmZ dd lmZ G d
d e ZdS ) a
Module containing the UniversalDetector detector class, which is the primary
class a user of ``chardet`` should use.
:author: Mark Pilgrim (initial port to Python)
:author: Shy Shalom (original C code)
:author: Dan Blanchard (major refactoring for 3.0)
:author: Ian Cordasco
N )CharSetGroupProber)
InputStateLanguageFilter ProbingState)EscCharSetProber) Latin1Prober)MBCSGroupProber)SBCSGroupProberc e Zd ZdZdZ ej d Z ej d Z ej d Z dddd d
d d d
dZ
e j fdZ
d Zd Zd ZdS )UniversalDetectoraq
The ``UniversalDetector`` class underlies the ``chardet.detect`` function
and coordinates all of the different charset probers.
To get a ``dict`` containing an encoding and its confidence, you can simply
run:
.. code::
u = UniversalDetector()
u.feed(some_bytes)
u.close()
detected = u.result
g?s [-]s (|~{)s [-]z Windows-1252z Windows-1250z Windows-1251z Windows-1256z Windows-1253z Windows-1255z Windows-1254z Windows-1257)z
iso-8859-1z
iso-8859-2z
iso-8859-5z
iso-8859-6z
iso-8859-7z
iso-8859-8z
iso-8859-9z iso-8859-13c d | _ g | _ d | _ d | _ d | _ d | _ d | _ || _ t j t | _ d | _ |
d S )N)_esc_charset_prober_charset_probersresultdone _got_data _input_state
_last_char lang_filterlogging getLogger__name__logger_has_win_bytesreset)selfr s /builddir/build/BUILDROOT/alt-python311-pip-21.3.1-4.el9.x86_64/opt/alt/python311/lib/python3.11/site-packages/pip/_vendor/chardet/universaldetector.py__init__zUniversalDetector.__init__Q si #' " & '11 "
c dddd| _ d| _ d| _ d| _ t j | _ d| _ | j r| j | j
D ]}| dS )z
Reset the UniversalDetector and all of its probers back to their
initial states. This is called by ``__init__``, so you only need to
call this directly in between analyses of different documents.
N encoding
confidencelanguageFr ) r r r r r
PURE_ASCIIr r r r r )r probers r r zUniversalDetector.reset^ s $(sMM # & 1 # - $ * * , , ,+ F LLNNNN r c | j rdS t | sdS t |t st | }| j s| t j r dddd| _ n| t j t j
f r dddd| _ nx| d r dddd| _ nW| d r d
ddd| _ n6| t j t j f r d ddd| _ d | _ | j d
d | _ dS | j
t j k rt| j | rt j | _
nH| j
t j k r3| j | j |z rt j | _
|dd | _ | j
t j k r| j st/ | j | _ | j | t4 j k r?| j j | j | j j d| _ d | _ dS dS | j
t j k r| j sztA | j g| _ | j tB j" z r&| j # tI | j # tK | j D ]U}| | t4 j k r0|j | |j d| _ d | _ nV| j& | r d | _' dS dS dS )a
Takes a chunk of a document and feeds it through all of the relevant
charset probers.
After calling ``feed``, you can check the value of the ``done``
attribute to see if you need to continue feeding the
``UniversalDetector`` more data, or if it has made a prediction
(in the ``result`` attribute).
.. note::
You should always call ``close`` when you're done feeding in your
document if ``done`` is not already ``True``.
Nz UTF-8-SIG ? r# zUTF-32s zX-ISO-10646-UCS-4-3412s zX-ISO-10646-UCS-4-2143zUTF-16Tr$ )(r len
isinstance bytearrayr
startswithcodecsBOM_UTF8r BOM_UTF32_LE BOM_UTF32_BEBOM_LEBOM_BEr r r' HIGH_BYTE_DETECTORsearch HIGH_BYTE ESC_DETECTORr ESC_ASCIIr r r feedr FOUND_IT charset_nameget_confidencer& r r
r NON_CJKappendr r WIN_BYTE_DETECTORr )r byte_strr( s r r< zUniversalDetector.feedo s 9 F8}} F(I.. + **H ~ " ""6?33
/+6-0+- / / $$f&9&,&9&; < <
/ ,4-0+- / / $$%899
/+C-0+- / / $$%899
/+C-0+- / / $$fmV]%CDD
/ ,4-0+- / / "DN{:&2
5 5 5&--h77
9$.$8!!"j&;;;%,,T_x-GHH <$.$8!"233-
4 4 4+
N+;D lowerr0 r ISO_WIN_MAPgetr& getEffectiveLevelr DEBUGr. r probers) r prober_confidencemax_prober_confidence
max_proberr( r> lower_charset_namer% group_probers r closezUniversalDetector.close s 9 ; ~ ! @ K 1 2 2 2 2
*"7
7
7'.),')+ +DKK
*"6
6
6 $ $' ! J /
(
( $*$9$9$;$;!$'<<<,=)!'J
@ 4t7M M M ) 6 %/%<%B%B%D%D" ' 6 6 8 8
&00<< J* J'+'7';'; @ @
; ( ( * *gm ; ;{:&. !!"DEEE$($9
I
IL' ! !,0BCC
I&2&: G GF K - -.E.4.A.4o.4.C.C.E.E G G G GG ))*A*6*C*6*?*6*E*E*G*GI I I I {r N)r
__module__ __qualname____doc__rG recompiler7 r: rB rI r ALLr r r< rS r r r
r
3 s #N33 2:l++L"
>22!/!/!/!/!/!/!/"02 2K $2#5 "k+ k+ k+ZB B B B Br r
)rV r1 r rW charsetgroupproberr enumsr r r escproberr latin1proberr mbcsgroupproberr
sbcsgroupproberr objectr
rZ r r rb s 8
2 2 2 2 2 2 ; ; ; ; ; ; ; ; ; ; ' ' ' ' ' ' & & & & & & , , , , , , , , , , , ,k k k k k k k k k kr