Mu!& d dl Z d dlZd dlmZ d dlmZmZ d dlmZm Z m
Z
m Z m Z ddl
mZmZmZmZ ddlmZmZmZmZ ddlmZ dd lmZmZ dd
l m Z m Z m Z m Z m!Z!m"Z"m#Z# e j$ d Z% e j& Z'e'( e j) d d"de*de+de+de,de e
e- de e
e- de.de.defdZ/ d"d e de+de+de,de e
e- de e
e- de.de.defd Z0 d"d d de+de+de,de e
e- de e
e- de.de.defd Z1 d#d d de+de+de,de e
e- de e
e- de.defd!Z2dS )$ N)PathLike)basenamesplitext)AnyBinaryIOListOptionalSet )coherence_ratioencoding_languagesmb_encoding_languagesmerge_coherence_ratios)IANA_SUPPORTEDTOO_BIG_SEQUENCETOO_SMALL_SEQUENCETRACE)
mess_ratio) CharsetMatchCharsetMatches)any_specified_encodingcut_sequence_chunks iana_nameidentify_sig_or_bom
is_cp_similaris_multi_byte_encodingshould_strip_sig_or_bomcharset_normalizerz)%(asctime)s | %(levelname)s | %(message)s 皙?TF sequencessteps
chunk_size threshold cp_isolation cp_exclusionpreemptive_behaviourexplainreturnc t | t t f s/t d t | |rJt j }t t t
t t | } | dk rt
d |rEt t t
|p t j t# t% | dddg d g S |At t d d
| d |D }ng }|At t d d
| d
|D }ng }| ||z k r't t d||| d}| }|dk r| |z |k rt+ | |z }t | t, k }
t | t. k } |
r4t t d | n5| r3t t d | g } |rt1 | nd}
|
6| |
t t d|
t5 }g }g }d}d}d}t# }t7 | \ }}|D| | t t dt | | | d d| vr| d | t8 z D ]}|r||vr
|r||v r||v r| | d}||k }|ot= | }|dv r$|s"t t d| l t? | }n8# t@ tB f$ r$ t t d| Y w xY w | rS|du rOtE |du r| dt+ d n#| t | t+ d | n,tE |du r| n| t | d | }nx# tF tH f$ rd} t | tH s/t t d|tE | | | Y d} ~ d} ~ ww xY wd} |D ]} tK || rd} n| r$t t d || tM |sdnt | | t+ | |z } |o|duot | | k } | r!t t d | t+ t | d z }!tO |!d }!d}"d}#g }$g }% tQ | || |||||| D ]Y}&|$ |& |% tS |&| |%d |k r|"dz
}"|"|!k s|r|du r nZnJ# tF $ r=} t t d!|tE | |!}"d}#Y d} ~ nd} ~ ww xY w|#s| r|s | t+ d" d * |d#$ n\# tF $ rO} t t d%|tE | | | Y d} ~ d} ~ ww xY w|%r tW |% t |% z nd}'|'|k s|"|!k r}| | t t d&||"tY |'d'z d() |dd|
fv r*|#s(t% | ||dg | }(||
k r|(}n |dk r|(}n|(}t t d*|tY |'d'z d() |st[ | })nt] | })|)rAt t d+ |tE |) g }*|dk rB|$D ]?}&t_ |&d,|)rd- |) nd }+|* |+ @ta |* },|,r4t t d. |,| | t% | ||'||,| ||
ddfv rt|'d,k rnt
d/| |r9t t t
| t# || g c S ||k rnt
d0| |r9t t t
| t# || g c S t | dk r|s|s|r t t d1 |r6t
d2|j1 | | n{|r||r|r|j2 |j2 k s|0t
d3 | | n1|r/t
d4 | | |rDt
d5|3 j1 t | dz
nt
d6 |r9t t t
| |S )7ae
Given a raw bytes sequence, return the best possibles charset usable to render str objects.
If there is no results, it is a strong indicator that the source is binary/not text.
By default, the process will extract 5 blocs of 512o each to assess the mess and coherence of a given sequence.
And will give up a particular code page after 20% of measured mess. Those criteria are customizable at will.
The preemptive behavior DOES NOT replace the traditional detection workflow, it prioritize a particular code page
but never take it for granted. Can improve the performance.
You may want to focus your attention to some code page or/and not others, use cp_isolation and cp_exclusion for that
purpose.
This function will strip the SIG in the payload/sequence every time except on UTF-16, UTF-32.
By default the library does not setup any handler other than the NullHandler, if you choose to set the 'explain'
toggle to True it will alter the logger configuration to add a StreamHandler that is suitable for debugging.
Custom logging format and handler can be set manually.
z4Expected object of type bytes or bytearray, got: {0}r zz from_bytes..[ " DDD "e,,DDD zacp_exclusion is set. use this flag for debugging purpose. limited list of encoding excluded : %s.c . g | ]}t |d S r/ r0 r1 s r4 r5 z from_bytes..f r6 r7 z^override steps (%i) and chunk_size (%i) as content does not fit (%i byte(s) given) parameters.r z>Trying to detect encoding from a tiny portion of ({}) byte(s).zIUsing lazy str decoding because the payload is quite large, ({}) byte(s).z@Detected declarative mark in sequence. Priority +1 given for %s.zIDetected a SIG or BOM mark on first %i byte(s). Priority +1 given for %s.ascii> utf_16utf_32z[Encoding %s wont be tested as-is because it require a BOM. Will try some sub-encoder LE/BE.z2Encoding %s does not provide an IncrementalDecoderg A)encodingz9Code page %s does not fit given bytes sequence at ALL. %sTzW%s is deemed too similar to code page %s and was consider unsuited already. Continuing!zpCode page %s is a multi byte encoding table and it appear that at least one character was encoded using n-bytes. zaLazyStr Loading: After MD chunk decode, code page %s does not fit given bytes sequence at ALL. %sg j@strict)errorsz^LazyStr Loading: After final lookup, code page %s does not fit given bytes sequence at ALL. %szc%s was excluded because of initial chaos probing. Gave up %i time(s). Computed mean chaos is %f %%.d )ndigitsz=%s passed initial chaos probing. Mean measured chaos is %f %%z&{} should target any language(s) of {}g?,z We detected language {} using {}z.Encoding detection: %s is most likely the one.zoEncoding detection: %s is most likely the one as we detected a BOM or SIG within the beginning of the sequence.zONothing got out of the detection process. Using ASCII/UTF-8/Specified fallback.z7Encoding detection: %s will be used as a fallback matchz:Encoding detection: utf_8 will be used as a fallback matchz:Encoding detection: ascii will be used as a fallback matchz]Encoding detection: Found %s as plausible (best-candidate) for content. With %i alternatives.z=Encoding detection: Unable to determine any suitable charset.)4
isinstance bytearraybytes TypeErrorformattypeloggerlevel
addHandlerexplain_handlersetLevelr lendebug
removeHandlerloggingWARNINGr r logjoinintr r r appendsetr r addr r ModuleNotFoundError ImportErrorstrUnicodeDecodeError LookupErrorr rangemaxr r decodesumroundr
r r r r< fingerprintbest)-r" r# r$ r% r& r' r( r) previous_logger_levellengthis_too_small_sequenceis_too_large_sequenceprioritized_encodingsspecified_encodingtestedtested_but_hard_failuretested_but_soft_failurefallback_ascii fallback_u8fallback_specifiedresults sig_encoding sig_payload
encoding_ianadecoded_payloadbom_or_sig_availablestrip_sig_or_bomis_multi_byte_decoderesimilar_soft_failure_testencoding_soft_failedr_multi_byte_bonusmax_chunk_gave_upearly_stop_countlazy_str_hard_failure md_chunks md_ratioschunkmean_mess_ratiofallback_entrytarget_languages cd_ratioschunk_languagescd_ratios_mergeds- r4
from_bytesr # s< 8 i)U!3 4 4
B I IY
%+\ /*** i..F
{{ STTT F 1 1 1 OO 1 DW_ E E E |IwUBPR S S TUUU
5 IIl # #
ED|DDD
6 IIl # #
ED|DDD
*u$%%
l
qyyVe^j00% ((
"%i..3E"E"%i..4D"D
L S S
W ^ ^
(* .BK y)))t % $$%7888
N
uuF)+)+-1N*.K15 , . .G 3I > > L+ $$\222
W
)))+++ $$W---. ? E< E<
M = = M\ 9 9 F " "
=!!!)-%1]%B !5 "
:Q ;
;
0 0 09M 0 JJm
*@ *O*O ! !#[1 JJD
H
$
)>%)G)G'500 kD k**"3{#3#3c$ii#?@ * #&'500 I"3{#3#3#5#56 * # # # #K0 a--
O!FF
$ * *= 9 9 9 HHHH +0!$; ],@AA
,0)
% JJi $
) ?AAs;/?/?
"
. t+
.O$$v- JJ-
"%SWWq[!1!1 11 5 5 ! % ! ), $ %
''' E9!=!=>>> R=I--$)$$(999( :-=-F-FE! ) ) ) JJs A
1 $( ! ! ! ! ! ! ) & % *
#d))++&--mH-MMMM%
t!FF (..}===
EN!VY #i..!@!@SV i ' '+;?P+P+P # * *= 9 9 9 JJ0 o+Q777
'74F!GGG- H ". }iO" " !$666)7&&"g--%3NN"0K
K /C' 3 3 3
% D*<]*K*K 4] C C JJ8??!3'7#8#8
G # #"
2
2"13>N X)9 : : :TX# # 11111)<< JJ299$m
$
0'7 C C C #%% LL@-
7$$_555 5666!7=#9":;; ; ; ; L ( ( LL1
7$$_555 5666!7=#9":;; ; ; ; ) 7||q . ,> JJa
+ LLI"+
NN- . . . .
+ + 3 4 # 4 +~/III' LLU V V V NN; ' ' ' '
+ LLU V V V NN> * * * V k LLNN # LL1
TUUU / _----... Ns^ P&&2QQ BS!!U2AUU A1Z22
[9<3[44[9,\00
^ :A^^ fpc
R t | ||||||| S )z
Same thing than the function from_bytes but using a file pointer that is already ready.
Will not close the file pointer.
)r read)r r# r$ r% r& r' r( r) s r4 from_fpr s6
r7 pathz
PathLike[Any]c t | d 5 }t |||||||| cddd S # 1 swxY w Y dS )z
Same thing than the function from_bytes but with one extra step. Opening and reading given file path in binary mode.
Can raise IOError.
rbN)openr ) r r# r$ r% r& r' r( r) r s r4 from_pathr s
dD
R
s 488c
t j dt t | |||||| }t | }t t
| } t | dk r"t d | |
}
| dxx d|
j z z
cc< t d t | |d | d 5 } | |
ddd n # 1 swxY w Y |
S ) zi
Take a (text-based) file path and try to create another file next to it, this time using UTF-8.
z2normalize is deprecated and will be removed in 3.0r z;Unable to normalize "{}", no encoding charset seems to fit.-z{}r- wbN)warningswarnDeprecationWarningr r listr rQ IOErrorrJ rg r< r r^ replacerW writeoutput) r r# r$ r% r& r' r( rt filenametarget_extensionsresultr s r4 normalizer s
M<
G ~~H Xh / /00
7||q I P P
\\^^FaC&/ 11
CII%%h8I0J0JKKLLd
"
!!!" " " " " " " " " " " " " " "
Ms (EE E )r r r! NNTF)r r r! NNT)3rT r osr os.pathr r typingr r r r r
cdr r
r r constantr r r r mdr modelsr r utilsr r r r r r r getLoggerrL
StreamHandlerrO setFormatter FormatterrH rX floatr^ boolr r r r r7 r4 r s & & & & & & & & 5 5 5 5 5 5 5 5 5 5 5 5 5 5 R Q Q Q Q Q Q Q Q Q Q Q 0 0 0 0 0 0 0 0
/ 0 0''')) GABB (,(,!%
49% 49%
H (,(,!%
49% 49%
8 (,(,!%
49%
49%
: (,(,!%. .
. . . .
49% . 49%
. . . . . . . .r7