qW d dl Z d dlmZmZ d dlmZmZmZmZ d dl m
Z
n
# e $ r e Z
Y nw xY wddl
mZmZmZmZ ddlmZmZmZmZ ddlmZ dd lmZmZ dd
l m Z m Z m Z m Z m!Z!m"Z" e j# d Z$ e j% Z&e&' e j( d d!de)de*de*de+dee dee de,de,defdZ- d!d ede*de*de+dee dee de,de,defd Z. d!d e
de*de*de+dee dee de,de,defd Z/ d"d e
de*de*de+dee dee de,defd Z0dS )# N)basenamesplitext)BinaryIOListOptionalSet)PathLike )coherence_ratioencoding_languagesmb_encoding_languagesmerge_coherence_ratios)IANA_SUPPORTEDTOO_BIG_SEQUENCETOO_SMALL_SEQUENCETRACE)
mess_ratio) CharsetMatchCharsetMatches)any_specified_encoding iana_nameidentify_sig_or_bom
is_cp_similaris_multi_byte_encodingshould_strip_sig_or_bomcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s 皙?TF sequencessteps
chunk_size threshold cp_isolation cp_exclusionpreemptive_behaviourexplainreturnc h t | t t f s/t d t | |rJt j }t t t
t t | } | dk rt
d |rEt t t
|p t j t# t% | dddg d g S |At t d d
| d |D }ng }|At t d d
| d
|D }ng }| ||z k r't t d||| d}| }|dk r| |z |k rt+ | |z }t | t, k }
t | t. k } |
r4t t d | n5| r3t t d | g } |rt1 | nd}
|
6| |
t t d|
t5 }g }g }d}d}d}t# }t7 | \ }}|D| | t t dt | | | d d| vr| d | t8 z D ]}|r||vr
|r||v r||v r| | d}||k }|ot= | }|dv r$|s"t t d| l t? | }n8# t@ tB f$ r$ t t d| Y w xY w | rS|du rOtE |du r| dt+ d n#| t | t+ d | n,tE |du r| n| t | d | }nx# tF tH f$ rd} t | tH s/t t d|tE | | | Y d} ~ d} ~ ww xY wd} |D ]} tK || rd} n| r$t t d || tM |sdnt | | t+ | |z } |o|duot | | k } | r!t t d | t+ t | d z }!tO |!d }!d}"d}#g }$g }%| D ]m}&|&|z | d z k r| |&|&|z }'|r |du r||'z }' |'( ||rd!nd"# }(nK# tF $ r>} t t d$|tE | |!}"d}#Y d} ~ nd} ~ ww xY w|r|&dk r~| |& d%k rrtS |d& })|r`|(d|) |vrTtM |&|&d z
d' D ]?}*| |*|&|z }'|r |du r||'z }'|'( |d!# }(|(d|) |v r n@|$ |( |% tU |(| |%d' |k r|"dz
}"|"|!k s|r|du r no|#s| r|s | t+ d( d ( |d"# n\# tF $ rO} t t d)|tE | | | Y d} ~ d} ~ ww xY w|%r tW |% t |% z nd}+|+|k s|"|!k r}| | t t d*||"tY |+d+z d,- |dd|
fv r*|#s(t% | ||dg | },||
k r|,}n |dk r|,}n|,}Ct t d.|tY |+d+z d,- |st[ | }-nt] | }-|-rAt t d/ |tE |- g }.|dk rB|$D ]?}(t_ |(d0|-rd1 |- nd }/|. |/ @ta |. }0|0r4t t d2 |0| | t% | ||+||0| ||
ddfv rt|+d0k rnt
d3| |r9t t t
| t# || g c S ||k rnt
d4| |r9t t t
| t# || g c S t | dk r|s|s|r t t d5 |r6t
d6|j1 | | n{|r||r|r|j2 |j2 k s|0t
d7 | | n1|r/t
d8 | | |rDt
d9|3 j1 t | dz
nt
d: |r9t t t
| |S );ae
Given a raw bytes sequence, return the best possibles charset usable to render str objects.
If there is no results, it is a strong indicator that the source is binary/not text.
By default, the process will extract 5 blocs of 512o each to assess the mess and coherence of a given sequence.
And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will.
The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page
but never take it for granted. Can improve the performance.
You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that
purpose.
This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32.
By default the library does not setup any handler other than the NullHandler, if you choose to set the 'explain'
toggle to True it will alter the logger configuration to add a StreamHandler that is suitable for debugging.
Custom logging format and handler can be set manually.
z4Expected object of type bytes or bytearray, got: {0}r zz from_bytes..] " DDD "e,,DDD zacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.c . g | ]}t |d S r- r. r/ s r2 r3 z from_bytes..h r4 r5 z^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.r
z>Trying to detect encoding from a tiny portion of ({}) byte(s).zIUsing lazy str decoding because the payload is quite large, ({}) byte(s).z@Detected declarative mark in sequence. Priority +1 given for %s.zIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.ascii> utf_16utf_32z[Encoding %s wont be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.z2Encoding %s does not provide an IncrementalDecoderg A)encodingz9Code page %s does not fit given bytes sequence at ALL. %sTzW%s is deemed too similar to code page %s and was consider unsuited already. Continuing!zpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes. ignorestrict)errorszaLazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %s g j@z^LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %szc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.d )ndigitsz=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {}g?,z We detected language {} using {}z.Encoding detection: %s is most likely the one.zoEncoding detection: %s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.zONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z7Encoding detection: %s will be used as a fallback matchz:Encoding detection: utf_8 will be used as a fallback matchz:Encoding detection: ascii will be used as a fallback matchz]Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.z=Encoding detection: Unable to determine any suitable charset.)4
isinstance bytearraybytes TypeErrorformattypeloggerlevel
addHandlerexplain_handlersetLevelr lendebug
removeHandlerloggingWARNINGr r logjoinintr r r appendsetr r addr r ModuleNotFoundError ImportErrorstrUnicodeDecodeError LookupErrorr rangemaxdecodeminr sumroundr r
r r r: fingerprintbest)1r r! r" r# r$ r% r& r' previous_logger_levellengthis_too_small_sequenceis_too_large_sequenceprioritized_encodingsspecified_encodingtestedtested_but_hard_failuretested_but_soft_failurefallback_ascii fallback_u8fallback_specifiedresults sig_encoding sig_payload
encoding_ianadecoded_payloadbom_or_sig_availablestrip_sig_or_bomis_multi_byte_decoderesimilar_soft_failure_testencoding_soft_failedr_multi_byte_bonusmax_chunk_gave_upearly_stop_countlazy_str_hard_failure md_chunks md_ratiosi cut_sequencechunkchunk_partial_size_chkjmean_mess_ratiofallback_entrytarget_languages cd_ratioschunk_languagescd_ratios_mergeds1 r2
from_bytesr % sd 8 i)U!3 4 4
B I IY
& /***
^^F
{{ STTT F 1 1 1 OO 1 DW_ E E E |IwUBPR S S TUUU
5 IIl # #
ED|DDD
6 IIl # #
ED|DDD
*u$%%
l
qyyVe^j00% ((
NN-? ? NN.> >
L S S
W ^ ^
.BK y)))t % $$%7888
N
UUF NK G 3I > > L+ $$\222
W
)))+++ $$W---. ? `< `<
M = = M\ 9 9 F " "
=!!! +} < /
4K 5
5
0 0 09M 0 JJm
$:=$I$I ! !#[1 JJD
H
$
)>%)G)G'500 kD k**"3{#3#3c$ii#?@ * #&'500 I"3{#3#3#5#56 * # # # #K0 a--
O!FF
$ * *= 9 9 9 HHHH %*!$; ],@AA
,0)
% JJi $
) ?AAs;/?/?
"
. t+
.O$$v- JJ-
B! , , 11 5 5 % 8 8 A:~
**$QZ %78L#
:(8E(A(A *\ 9
$++!'<J88( , &
w!FF $5 (,%
%
"Q9Q<43G3G),Z)<)<& $
" 5 5 56oMM "1a!eR 0 0 " "'0Q^1C'D / F4D4M4M+6 +EL , 3 3M( 3 S S !8"8!89_LL !E M
U # # # Zy 9 9 : : :} )) A% $555$ 6)9U)B)B
& % *
#d))++&--mH-MMMM%
t!FF (..}===
09 AC NNS^^ + +c i ' '+;?P+P+P # * *= 9 9 9 JJ0 o+Q777
'74F!GGG- H ". }iO" " !$666)7&&"g--%3NN"0K
K /C' 3 3 3
% D 1- @ @ 4] C C JJ8??!3'7#8#8
G # #"
2
2"13>N X)9 : : :TX# # 11111)<< JJ299$m
$
0'7 C C C #%% LL@-
7$$_555 5666!7=#9":;; ; ; ; L ( ( LL1
7$$_555 5666!7=#9":;; ; ; ; ) 7||q . ,> JJa
+ LLI"+
NN- . . . .
+ + 3 4 # 4 +~/III' LLU V V V NN; ' ' ' '
+ LLU V V V NN> * * * V k LLNN # LL1
TUUU / _----... Ns] P&&2QQ BS!!U2AUU*Z
[3[ [7,_$$
`=.A`88`=fpc
R t | ||||||| S )z
Same thing than the function from_bytes but using a file pointer that is already ready.
Will not close the file pointer.
)r read)r r! r" r# r$ r% r& r' s r2 from_fpr s6
r5 pathc t | d 5 }t |||||||| cddd S # 1 swxY w Y dS )z
Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode.
Can raise IOError.
rbN)openr ) r r! r" r# r$ r% r& r' r s r2 from_pathr s
dD
R
s 488c
t t | |||||| }t | }t t | } t | dk r"t d | | }
| dxx d|
j z z
cc< t d t | |d | d 5 } |
|
ddd n # 1 swxY w Y |
S )zi
Take a (text-based) file path and try to create another file next to it, this time using UTF-8.
r z;Unable to normalize "{}", no encoding charset seems to fit.-z{}r+ wbN)r r listr rS IOErrorrL rj r: r r` replacerY writeoutput) r r! r" r# r$ r% r& rw filenametarget_extensionsresultr s r2 normalizer 7 sn
G ~~H Xh / /00
7||q I P P
\\^^FaC&/ 11
CII%%h8I0J0JKKLLd
"
!!!" " " " " " " " " " " " " " "
Ms 9(D--D14D1)r r r NNTF)r r r NNT)1rV os.pathr r typingr r r r osr r_ r` cdr r r
r constantr r r r mdr modelsr r utilsr r r r r r getLoggerrN
StreamHandlerrQ setFormatter FormatterrJ rZ floatboolr r r r r5 r2 r s & & & & & & & & 0 0 0 0 0 0 0 0 0 0 0 0 HHH R Q Q Q Q Q Q Q Q Q Q Q 0 0 0 0 0 0 0 0
/ 0 0''')) GABB " "!%Z ZZ Z Z Z
s) Z s)
Z Z Z Z Z Z Z~ " "!%
s) s)
8 " "!%
s)
s)
: " "!%) )
) ) ) )
s) ) s)
) ) ) ) ) ) ) )s ! ++