)"^ d dl Zn# e$ r d dlZY nw xY wd dlZd dlZd dlmZ d dlmZ d dl m
Z
d dl m Z d dl
mZmZmZmZmZ d dlmZ dd lmZmZmZmZmZmZ e
e
d e d e fd
Z e
e
d e d e fd Z e
e
d e d ee fd Z e
e
d e d e fd Z!d e d e fdZ" e
e
d e d e fd Z# e
e
d e d e fd Z$ e
e
d e d e fd Z% e
e
d e d e fd Z& e
e
d e d e fd Z'd e d e fdZ( e
e
d e d e fd Z) e
e
d e d e fd Z* e
e
d e d e fd Z+ e
e
d e d e fd Z, e
e
d e d e fd Z- e
e.e
d e d e fd Z/d8d e0d!e1d ee fd"Z2 e
d#
d$e d e fd% Z3d e0d eee e0f fd&Z4d'e d e fd(Z5d9d*e d+e d e fd,Z6d-e d ee fd.Z7d/e d0e d e8fd1Z9d/e d0e d e fd2Z:d3ej; d4fd$e d5e1d6e d dfd7Z!@!@
I I % % & 4r+ c X t j | }n# t $ r Y dS w xY wd|v S )NFLATINr r s r is_latinr5 C sF !&y11 uu k !!r c V | d n# t $ r Y dS w xY wdS )NasciiFT)encodeUnicodeEncodeErrorr s r is_asciir; L sE !!!! uu 4s
&&c d t j | }d|v rdS t | }|dS d|v S )NPTF Punctuationr categoryr2 r character_categorycharacter_ranges r is_punctuationrD T sG $-i88
t#I..O u O ++r+ c l t j | }d|v sd|v rdS t | }|dS d|v S )NSNTFFormsr? rA s r is_symbolrI c sR $-i88
C+=$=$=t#I..O u o %%r+ c 0 t | }|dS d|v S )NF Emoticons)r2 )r rC s r is_emoticonrL r s% #I..O u / ))r+ c f | s| dv rdS t j | }d|v S )N> ｜+,;<>TZ)isspacer r@ r rB s r is_separatorrW | sC i+K K Kt$-i88 $ $$r+ c V | | k S N)islowerisupperr: s r is_case_variabler\ s% )"3"3"5"5 55r+ c 6 t j | }|dk S )NCo)r r@ rV s r is_private_use_onlyr_ s $-i88 %%r+ c X t j | }n# t $ r Y dS w xY wd|v S )NFCJKr r character_names r is_cjkrd sH $))44 uu N ""r c X t j | }n# t $ r Y dS w xY wd|v S )NFHIRAGANAr rb s r is_hiraganarg H $))44 uu ''r c X t j | }n# t $ r Y dS w xY wd|v S )NFKATAKANAr rb s r is_katakanark rh r c X t j | }n# t $ r Y dS w xY wd|v S )NFHANGULr rb s r is_hangulrn sH $))44 uu ~ %%r c X t j | }n# t $ r Y dS w xY wd|v S )NFTHAIr rb s r is_thairq sH $))44 uu ^ ##r r0 c D t fdt D S )Nc 3 K | ]}|v V d S rY ).0keywordr0 s r z-is_unicode_range_secondary.. s( TTw*$TTTTTTr+ )anyr )r0 s `r is_unicode_range_secondaryry s' TTTT4STTT T TTr+ sequence search_zonec t | t st t | }t t
| dt
|| dd }t | dk rdS |D ][}| dd }t j D ] \ }}||k r|c c S ||k r|c c S \dS )zW
Extract using ASCII-only decoder any specified encoding in the first n-bytes.
Nr7 ignoreerrorsr -_)
isinstancebytes TypeErrorlenr r mindecodelowerreplacer r. )r{ r| seq_lenresultsspecified_encodingencoding_alias
encoding_ianas r any_specified_encodingr s h & & (mmG',3w ,,,-44WX4NN G
7||qt % % % / 5 5 7 7 ? ?S I I-4]__ % % )NM !333$$$$$$ 222$$$$$$ 3 % 4r+ r c | dv p>t t j d | j t
S )zQ
Verify is a specific encoding is a multi byte one based on it IANA name
> utf_7utf_8utf_16utf_32 utf_16_be utf_16_le utf_32_be utf_32_le utf_8_sig encodings.{})
issubclass importlib
import_moduleformatr r )r s r is_multi_byte_encodingr sL
5 5d ; ;<<O#
r+ c t D ]I}t | }t |t r|g}|D ] }| | r||fc c S JdS )z9
Identify and extract SIG/BOM in given sequence.
)Nr+ )r r r
startswith)r{
iana_encodingmarksmarks r identify_sig_or_bomr s
( + +
}- eU # # GE + +D""4((
+$d******
+ + 9r+ r c
| dvS )N> r r rt )r s r should_strip_sig_or_bomr s 4 44r+ Tcp_namestrictc | dd } t j D ]\ }}| ||fv r|c S |r"t d | | S )Nr r z Unable to retrieve IANA for '{}')r r r r. r r )r r r r s r iana_namer s mmoo%%c3//G)0 ! !%
~}5 5 5 6 M;BB7KKLLL Nr+ decoded_sequencec t }| D ])}t | }|| | *t | S rY )setr2 addlist)r rangesr rC s r
range_scanr sQ
UUF% $ $ ' 22 "
?#### <<r+ iana_name_a iana_name_bc t | st | rdS t j d | j }t j d | j } |d } |d }d}t d D ]C}t
|g }| | | | k r|dz
}D|dz S ) Ng r r~ r r r
)r r r r r ranger r ) r r decoder_a decoder_bid_aid_bcharacter_match_counti
to_be_decodeds r
cp_similarityr + s k** .D[.Q.Q s'(=(=k(J(JKK^I'(=(=k(J(JKK^I 9H % % %D 9H % % %D
3ZZ ' ' qc
;;} % %])C)C C C !Q & ! 3 &&r+ c 2 | t v o|t | v S )z
Determine if two code page are at least 80% similar. IANA_SUPPORTED_SIMILAR dict was generated using
the function cp_similarity.
)r )r r s r
is_cp_similarr @ s% -- ? 1+> >r+ charset_normalizerz)%(asctime)s | %(levelname)s | %(message)slevel
format_stringc t j | }| | t j }| t j | | | d S rY )logging getLoggersetLevel
StreamHandler setFormatter Formatter
addHandler)r r r loggerhandlers r set_logging_handlerr K sm
t
$
$F
OOE#%%G *=99:::
g r+ )rz )T)= unicodedata2r ImportErrorr r codecsr encodings.aliasesr functoolsr rer typingr r r r
r _multibytecodecr constantr r r r r r strboolr r* r2 r5 r; rD rI rL rW r\ r_ rd rg rk rn rq r ry r r' r r r r r r floatr r INFOr rt r+ r r s &&&&& % % % % % % % % % % % % 4 4 4 4 4 4 4 4 4 4 4 4 4 4 7 7 7 7 7 7 *+++ c d ,+ *+++"S "S " " " ,+" *+++
S
Xc]
,+
*+++" " " " " ,+" *+++ ,c ,d , , , ,+ , *+++ & & & & & ,+ & *+++*3 *4 * * * ,+* *+++%C %D % % % ,+% *+++6 6 6 6 6 ,+6&3 &4 & & & & *+++#c #d # # # ,+# *+++(3 (4 ( ( ( ,+( *+++(3 (4 ( ( ( ,+( *+++& & & & & ,+& *+++$s $t $ $ $ ,+$ 33.//000U3 U4 U U U 10U U QT
: 3 (% E(3-2F,G $53 54 5 5 5 5
s
D
C
c 's ' ' ' ' ' '*s % D
s