jR d Z dZddlZddlZddlZddlmZ ddlmZm Z m
Z
ddl m Z m Z ddl
mZ ddlmZmZmZmZmZ dd lmZ ddlZdd
lmZ dd lm Z dd l m Z dd
l m Z m!Z!m"Z" ddl#m$Z$m%Z%m&Z& ddl'm(Z( ddl)m*Z* ddl+m,Z, ddl-m.Z.m/Z/m0Z0 dZ1 ddl2m3Z4 n# e5$ rZ6e6Z1dZ4Y dZ6[6ndZ6[6ww xY we G d d Z7e G d d Z8e G d d Z9 G d de Z:de;d e:fd Z< G d d e Z=e>d k r7 e0 ej? d!ej@ " e= ZA eA dS dS )#z'
This module contains Autotracer class
z ssa.modules N) defaultdict) dataclassfieldasdict)datetime timedelta) fnmatchcase)IteratorOptional
NamedTupleListIterable)urlparse)setup_database)SSAError)Common)iter_domains_dataiter_urls_dataget_url_durations)statusdisabled_usersmisconfiguration_checks)
load_tunables)autotracing_tunables_schema)autotracing_stats_file)is_xray_version_supportedis_kernel_version_supported sentry_initc Z e Zd ZU dZeed< eed< eed< eed< eed< eed< eed< d S )
URLzURL data containeruri avg_duration max_durationtotal_slow_reqs
total_reqs
belongs_tobelongs_to_userN)__name__
__module__ __qualname____doc__str__annotations__int M/opt/cloudlinux/venv/lib64/python3.11/site-packages/ssa/modules/autotracer.pyr r 3 sc HHHOOOOOOr0 r c e Zd ZU dZdZeed< dZeed< dZeed< dZ eed< dZ
eed< dZ eed< dZ eed < dZ
eed
< dZeed < dZeed < dZeed
< dS ) RulesRejectszRejects counters containerr non_wp throttleddensityslowness max_slownessdisablednginx frequency server_limit domain_limit no_domainN)r( r) r* r+ r4 r. r- r5 r6 r7 r8 r9 r: r; r< r= r> r/ r0 r1 r3 r3 ? s $$FCOOOIsGSHcL#HcE3NNNIsL#L#Isr0 r3 c \ e Zd ZU dZeed< dZeed< dZeed< e e
Z e
ed< dS ) Statsz AutoTracer statistics container
rules_versionr urls_processed
urls_selected)default_factoryrejectsN) r( r) r* r+ r, r- rB r. rC r r3 rE r/ r0 r1 r@ r@ O sa ))NCM3 !E, ? ? ?G\?????r0 r@ c ( e Zd ZU dZeed< eed< dS )URLSz"
Representation of an URL
domain_nameuri_pathN)r( r) r* r+ r, r- r/ r0 r1 rG rG X s1 MMMMMr0 rG urlreturnc t | }|j r
d|j nd}|j r |j | nd}|j dd }| d d }t
|| S )z
Split URL into domain_name and uripath including query string
:param url: URL of format protocol://domain/path;parameters?query#fragment
:return: namedtuple URL(domain_name, uripath)
? /zwww.:r )r querypathnetlocreplacesplitrG )rJ fragmentsqsr! _no_www_netloc_no_port_netlocs r1 url_splitrZ ` s
I"+/ 9 Y_ rB%.^
Says if nginx is enabled for a specific user
NF)
skip_nginxrp NginxUserCache
is_enabled)rj r s r1 nginx_is_enabledzAutoTracer.nginx_is_enabled s3 ? A u*955@ @ur0 c t d | j D }| j d| | j |z
S )z>Recalculate limit per server taking into account running tasksc B g | ] }|d dk |d dk | S )r runninguserz
*autotracing*r/ ).0tasks r1
z. sM = = =d !(^y 8 8T!'>),;>< >< ">< >< > >
B' ) ) )$}44r0 full_listc |.t j }| j d| t j t
| j z
fd|D S )zQ
Excludes tasks older than N days from the general list of tasks
NzTask list loaded %s)daysc g | ]S}| d | d ndt k Q|TS )
createtimeNr )getr. timestamp)r r
n_days_agos r1 r z1AutoTracer.excludes_old_tasks.. si x x x.2hh|.D.D.PDHH\***VW[^_i_s_s_u_u[v[vvv vvvr0 )rp rg rc rv r nowr min_retracing_interval)rj r r s @r1 excludes_old_tasksz AutoTracer.excludes_old_tasks s +--I K 3Y ? ? ? \^^i,'. '. '. .
x x x x x x x xr0 current_urlc \ t | }| j D ]}t |d }|j |j k o|j |j k }t |j |j ot |j |j }|s|r%| j d||d dS dS )z
Excludes url from the list if it completely matches the current url
or if domain names match and "*" follows the domain name in the list
rJ z2Skipped: URL %s was traced recently. Matched by %sTF)rZ rg rH rI r rc rv )rj r c task_datat direct_matchwildcard_matchs r1 exclude_thesame_urlsz AutoTracer.exclude_thesame_urls s
k " " I)E*++A =AM9VajAJ>VL (
q} . . J2=aj>?j3J 3J
~
!!H 5!13 3 3 tt
ur0 url_total_reqsdomain_total_reqsc | j rMt j t j || }| j d| || j k S dS )z-Check that URL density passes given thresholdzCalculated density %sT)r6 npamincorrcoefrc rv density_threshold)rj r r url_densitys r1 pass_by_densityzAutoTracer.pass_by_density s] < 8 ' N,=>>@ @K K 5{ C C C !77 7tr0
url_durationsc d| j z
}t j || }| j d|| || j k S )z
The measure of "slowness" for URL is:
at least N% of its requests take more than X seconds.
N% -- self.slow_duration_percentage
X -- self.slow_duration_threshold
d z.Calculated %sth percentile %s for min duration)slow_duration_percentager
percentilerc rv slow_duration_threshold)rj r reversed_percentilereversed_percentile_values r1 pass_by_slowness_percentilez&AutoTracer.pass_by_slowness_percentile s^ "D$A A$&M-2E%G %G! J-3 5 5 5 )D,HHHr0 c t j || j }| j d| j | || j k S )a
The opposite to pass_by_slowness_percentile method.
The measure of "much slowness" for URL is:
at least N% of its requests must take less than X seconds.
N% -- self.max_slow_duration_percentage
X -- self.max_slow_duration_threshold
z.Calculated %sth percentile %s for max duration)r r max_slow_duration_percentagerc rv max_slow_duration_threshold)rj r percentile_values r1 pass_by_max_slowness_percentilez*AutoTracer.pass_by_max_slowness_percentile sV = )-)J L L J ;* , , , 4#CCCr0 url_throttled_reqsc |dS t | t | z dz }| j d| || j k S )zY
Check that percent of throttled requests per URL passes given threshold
NFr z Calculated throttled percent %s)sumrc rv allowed_throttling_percentage)rj r r throttled_percents r1 pass_by_allowed_throttlingz%AutoTracer.pass_by_allowed_throttling sX
%5 !3 4 4S 5H5H H# M ;=NOOO D$FFFr0 wp_statusc 2 | j rt | S dS )zx
Check that URLs of a particular domain should be analyzed.
For now we skip non-wordpress sites
T)only_wpbool)rj r s r1 pass_by_enginezAutoTracer.pass_by_engine s
< # ?? "tr0 c # K t | j D ]5}d}| |j sQ| j d |j }| j j xj |z
c_ | j xj
|z
c_
pt t | j |j
}t | j |j
t | D ]b\ }}|| j v r| j d| | j xj
dz
c_
| | d |d s5| j d | j j xj dz
c_ | |d |j s5| j d | j j xj dz
c_ || }| | s6| j d | j j xj dz
c_ M| | s6| j d
| j j xj dz
c_ |Y| |j
}|6| j d | j j xj dz
c_ |j }t? |tA tC j" | tG |d tG |d tI | |j
|
V d7dS )a
Select all URLs suitable for auto tracing by very basic rules:
- WP site
- suitable throttling
- suitable density
- measure of "slow" URL
ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS
Nz Skipped by engine: non-wordpresszProcessing URL %srm r r z Skipped by throttled percentzSkipped by densityz Skipped by slowness percentilez"Skipped by max slowness percentilez"Skipped by unavailable domain info
url_slow_reqs)r" r$ r% r# r& r' )%r ri r is_a_wordpress_domainrc rv urls_numberrh rE r4 rB dictr rH r rf keysnon_url_fieldsr r r5 r r r6 r r7 r r8 r r> r r r. r meanr max) rj domain_data domain_owner
skipped_countdomain_url_durationsrJ data durationsr s r1 urls_computationally_filteredz(AutoTracer.urls_computationally_filtered$ s -T[99 K 8 K 8K L&&{'HII
!!"DEEE !, 7
"))]:))
))]:))#'(9 [ 4)6 )6 $7 $7 ,DK,7,CTJ^JcJcJeJeEfEf h h :
8 :
8 T$--- !!"5s;;;
))Q.))66 !566=M8NP P K%%&DEEEJ&00A500 ++D1A,B,7,IK K K%%&:;;;J&..!3.. 0 5 77 BB K%%&FGGGJ&//14// ;;IFF K%%&JKKKJ&33q833 '"&"6"6{7N"O"OK"* ))*NOOO
*44944 '2'7 #'*279+=+='>'>*-d?.C*D*D%(.>)?%@%@'*9~~%0%<*6
8 8 8 8 8 8 8i:
8#K 8 K 8r0 stats_collectedc # K | | D ]}|j | j v r;| j d|j | j j xj dz
c_ K| |j r;| j d|j | j j xj
dz
c_
|V dS )z
From selected by computed thresholds URLs take those for which:
- autotracing enabled
- nginx disabled
ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS
z'Skipped: autotracing is disabled for %srm z Skipped: nginx is enabled for %sN) fill_in_disabled_usersr r' r rc rv rh rE r9 r r: )rj r rJ s r1 rC zAutoTracer.urls_selectedz s
##%%%5577 C"d&999 !!"K"%"57 7 7
"++q0++$$S%899
!!"D"%"57 7 7
"((A-((IIII r0 c # K d}t t }| }| j d| | | _ | }|D ]}| |j r| j
j xj dz
c_ 7||k rv||j
| j k r |dz
}||j
xx dz
cc< |V r| j d|j | j | j
j xj dz
c_ | j d|j | | j
j xj dz
c_ dS )a=
Schedule autotracing by sorted list taking into account the limits:
- no same task for 10 days
- limit per server
- limit per domain
ORDER OF RULES MUST NOT BE CHANGED: IT AFFECTS STATISTICS COUNTERS
Return resulting list of URLs scheduled for auto tracing
r zSorted scheduled list %srm z#Skipped URL %s by domain limit (%s)z#Skipped URL %s by server limit (%s)N)r r. urls_sortedrc rv r rg r r r! rh rE r; r& per_domain_limitr= r< )rj general_tasks_countertasks_counter_per_domain sorted_urlsper_server_smart_limitrJ s r1 rx zAutoTracer.urls_scheduled s !" #.s#3#3 &&(( 4kBBB 1133!%!C!C!E!E 5 5C((11
",,1,,$'===+CN;d>SSS)Q.),S^<<<A<<< IIII K%%&K&)g&*&;= = = J&33q8333 !!"G"%'"8: : :
"//14///1 5 5r0 c t t | d d }t |d d S )zd
Sort URLs by total number of requests first
and by average duration second
c | j S N)r" us r1 z(AutoTracer.urls_sorted.. s 1> r0 T)keyreversec | j S r )r% r s r1 r z(AutoTracer.urls_sorted.. s r0 )sortedrf rC )rj first_series r1 r zAutoTracer.urls_sorted s[
T$"4"4"6"6 7 7!9!94I I I k'='= "$ $ $ $r0 c , t | _ dS )z6
Fill internal list of disabled users
N)r )rj s r1 r z!AutoTracer.fill_in_disabled_users s - . .r0 c D | j t t d 5 }t j t | j | ddd dS # 1 swxY w Y dS # t $ r3}| j dt | Y d}~dS d}~ww xY wdS )zI
Save collected statistics for current iteration to file
Nwz*Unable to save iteration stats to file: %s)
rh openr jsondumpr OSErrorrc warningr, )rj
stats_filer~ s r1 r{ z AutoTracer.save_iteration_stats s : !
J0#66 >*IfTZ 0 0*===> > > > > > > > > > > > > > > > > >
J
J
J ##@#a&&J J J J J J J J J
J " !s: A (A A AA AA
B *(BB c x t t 5 }t j | }t |d |d |d t di |d }ddd n # 1 swxY w Y n6# t t j t f$ r t | j }Y nw xY wt | S )z@
Load statistics for latest iteration from file
rA rB rC rE Nr/ ) r r r loadr@ r3 r JSONDecodeErrorKeyErrorrA r )rj r _data stat_datas r1 load_iteration_statsz AutoTracer.load_iteration_stats s 2,--
D *-- !%"8"'(8"9"'"8"."B"By1A"B"B D D
D
D
D
D
D
D
D
D
D
D
D
D
D
D
D -x8 2 2 2 d 011III 2i s5 A7 A A+ A7 +A//A7 2A/3A7 70B*)B*c | }| t t d t t |S )rN r )r disabled_users_quantity)r updater r r r )rj stats_loadeds r1 get_statszAutoTracer.get_stats sb 0022 D88A;$'(8(8$9$9
r0 r )rK N)&r( r) r* r+ r` r staticmethodr r r, r objectr r r r. r rf r r r r r r r r r
r r rC rx r
r r r{ r r
__classcell__)rk s @r1 r\ r\ n s) = = = = = = $ $ $ $ $D :t : : : \: 1S 1Xf-= 1 1 1 \1# $ 5s 5 5 5 5 x xHTN xd x x x x *d +/48
I#
I4
I
I
I
I D7?}DIMD D D D
GXd^
G37
G<@
G
G
G
G 4 T8x} T8 T8 T8 T8n 9= '/~AI# 8*5
*5 *5 *5 *5X$T#Y $ $ $ $/ / / /
J
J
J
J
!d
!
!
!
! 4 r0 r\ __main__zauto_tracer_standalone.log)filenamelevel)Br+ re r ra rq collectionsr dataclassesr r r r r fnmatchr typingr
r r r
r urllib.parser numpyr ssa.dbr ssa.internal.exceptionsr ssa.modules.commonr ssa.modules.storager r r ssa.autotracingr r r ssa.configurationr ssa.configuration.schemesr ssa.internal.constantsr ssa.internal.utilsr r r rt xray.shared_libraryshared_libraryrp ImportErrorr~ r r3 r@ rG r, rZ r\ r( basicConfigDEBUGr r/ r0 r1 r s # # # # # # 0 0 0 0 0 0 0 0 0 0 ( ( ( ( ( ( ( ( A A A A A A A A A A A A A A ! ! ! ! ! ! ! ! ! ! ! ! , , , , , , % % % % % %
L K K K K K K K K K + + + + + + A A A A A A 9 9 9 9 9 9 ******* HHHHHH
@ @ @ @ @ @ @ @ : &3 &4 & & & & G G G G G G G G T zKMMMG!= %m- - - -
AACCCCC s B B&B!!B&