l:9+ d dl mZ d dlZd dlZd dlZd dlZd dlZd dlmZ d dl m
Z
m Z m Z m
Z
mZmZ d dlmZmZ d dlmZ d dlmZmZ d dlmZmZ d d lm Z m Z d d
l m Z m Z d d l!m"Z" d d l#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z) d d
l*m+Z+ d dl,m-Z- d dl.m/Z/ d dl0m1Z2 d dl3m4Z4m5Z5m6Z6 e-e7 Z8 G d de/ Z9 G d d Z: G d d Z;dS ) )absolute_importN)
pid_exists)OptionalDictTextIOUnionListTuple)write_file_via_tempfile
disable_quota)_add_wp_path_info)ClPwddrop_privileges)FeatureStatusEnumextract_suites) acquire_lockget_server_wide_options) userdomainscpusers)"convert_features_dict_to_interface)get_admin_suites_configextract_featuresAWPSuite PremiumSuiteCDNSuite CDNSuitePro)
UserConfig)
setup_logging) WposError)gettext)SCANNING_STATUSLAST_SCANNED_TS
SCAN_CACHEc e Zd ZdZdS )ReportGeneratorErrorzH
Raised when some bad happened during report generating/getting
N)__name__
__module__ __qualname____doc__ py/clwpos/report_generator.pyr% r% 1 s Dr+ r% c e Zd ZdZd
dee fdZd Zd Zd Z d Z
e d ed fd
Z d e
fd Zd efd ZdS )
ScanStatusz3
Type for handly scan status manipulations
Npidc || _ t |t r7t t | d \ | _ | _ d S t |t r$t t | \ | _ | _ d S t t d t | t | d # t $ r. t t d dt | i w xY w)N/z:Unable to parse scan status: %(status)s, type of: %(type)s)statustypemessagecontextz'Unable to parse scan status: %(status)sr2 )
r/
isinstancestrmapintsplitcurrenttotaltupler% _r3
ValueError)selfr2 r/ s r, __init__zScanStatus.__init__< s &#&&
+.sFLL4E4E+F+F( djjjFE**
+.sF+;+;( djjj* Z [ ['*6{{DLL I I &CDD!3v;;/
s A
C 7C ;C 8Dc 8 t | j | j f S N)hashr< r= rA s r, __hash__zScanStatus.__hash__N s T\4:.///r+ c . t |t r | j | j |j k |j fS t |t r%t | | k S t |t rt
| |k S dS )NF)r7 r. r< r= r8 stripr> )rA others r, __eq__zScanStatus.__eq__Q s eZ ( ( (<u}!
Read status file, with shared blocking flock
Nrr/ r< r= z Can't parse scan status json: %s)ospathexistsr! openfcntlflockfilenoLOCK_SHreadLOCK_UNjsonloadsgetr. r: Exceptionloggererror)stfdatar/ es r, r] zScanStatus.reada s
7>>/ * * Ios++
Iq AHHJJ
666vvxx AHHJJ
666
I
I
I
I
I
I
I
II :d++D ((5//C# i $w-8$'1CT BB ! I I ILL!DaHHHHHHHHI!
I
I
I
I
I
I
I
I
I
I
I
I
I
I
I$ s= A:E AD E
E
(EE E
E E #E c P t j | j | j | j d S )z5
Status JSON representation to write
)r< r= r/ )r_ dumpsr< r= r/ rF s r, to_jsonzScanStatus.to_json| s1 z|Z8
r+ c R t | j ot | j S )z<
Return True if scan process is still alive
)boolr/ r rF s r, check_pidzScanStatus.check_pid s! DH~~6*TX"6"66r+ rD )r& r' r( r) r r: rB rG rK rN rQ staticmethodr] r8 rk rm rn r* r+ r, r. r. 8 s HSM $0 0 0 . . . (<( \4 74 7 7 7 7 7 7r+ r. c T e Zd ZdZd Zedefd Zedee fd Z
d dee e de
e ef fdZd e e ddfd
Zdefd Zedee fd Zed e e de e fd
Zede e fd Zedefd Zd dZd dZd dZed Zed Zde
de de f de e eeeeeeeeeef fdZ d!de dedede e
e df fd Z ede dede e
e df fd Z d Z!dS )"ReportGeneratorz3
Class for admin report generation/getting
c t | _ t d | _ di d| _ d | _ t | _ d S )Nr r 1)versionusers)r _clpwdr. r2 resultscan_status_fdrc rF s r, rB zReportGenerator.__init__ sE gg ((
# r+ rR c h t } | r| rdS dS )z
Checks:
1) SCANNING_STATUS file exists
2) SCANNING_STATUS has the scan process pid and it is still alive
TF)r. r] rn )r2 s r, is_scan_runningz ReportGenerator.is_scan_running s9 "" !!
tur+ c 4 t S )z1
Read status file and get status
)r. r] r* r+ r, get_scan_statusz ReportGenerator.get_scan_status s
r+ N user_listc | r?t t d dt | i | }| j d| |r1| || }| j d| t | | j
_ t j
}|r*| j d| d| j
j dS | | t j d d S # t" $ r:}| j d | t j d
Y d }~d S d }~ww xY w) z
Public entry point for the scan procedure
Do some checks, prepare users list, fork process to actually do scan, and return initial scan progress
`user_list` - users to scan; if None - scan all
z$Another scan in progress: %(status)sr2 r4 zFound usernames: %szFiltered usernames: %szScan forked to: %dr )total_users_scannedtotal_users_to_scanz Error during user sites scan: %s N)r{ r% r? r8 r} _ReportGenerator__get_usernamesrc debug"_ReportGenerator__filter_usernameslenr2 r= rU fork_scansysexitrb exception)rA r~ usernamesfprh s r, scanzReportGenerator.scan s ! ! X&q1W/X/X08#d>R>R>T>T:U:U/VX X X
X ((** /;;; C // 9EEI K 6 B B B NN WYY
K 2B 7 7 7+,T[EVWW W
9%%%
%%&H!LLL
s )D: :
E>/E99E>r c t t d | _ t j | j _ | j d| j j |D ]}i }| j xj dz
c_ |
t | D ].\ }}| |g
| /t | 5 t | }ddd n # 1 swxY w Y d | D }| j d |dt' | ii | | j t j t | }| j d| dS ) a
Executes in forked process
Get domain and docroots list for usernames
Count user sites, write to SCAN_CACHE
Create and remove SCANNING_STATUS file
`userdomains` examples:
[{'domain': 'res1.com', 'docroot': '/home/res1/public_html'}],
[{'domain': 'cltest1.com', 'docroot': '/home/cltest1/public_html'},
{'domain': 'mk.cltest1.com', 'docroot': '/home/cltest1/public_html/mk'}]
wzSet status pid to: %dr Nc h | ]Q\ }}| d g D ]5}t j || d 6RS )wp_pathsr1 )ra rU rV joinrstrip).0 _doc_rootvalues_wp_paths r, z(ReportGenerator._scan.. sm BS)V &
:r : : Y1188== r+ rv wp_sites_countzNew scan timestamp: %d)rX r! ry rU getpidr2 r/ rc r r< $_ReportGenerator__update_scan_statusr
setdefaultappendr r
itemsrx updater #_ReportGenerator__write_scan_resultcloseunlink'_ReportGenerator__update_scan_timestamp)rA r username user_data domain_namedoc_rootr _tss r, r zReportGenerator._scan s #?C 8 8 )++ 14;?CCC! HI K 1 $ % % ' ' ')4X)>)>
G
G% X$$Xr2299+FFFF **
9
9 -i 8 8
9
9
9
9
9
9
9
9
9
9
9
9
9
9
9 W`WfWfWhWh H
K ' '+S]];)
""" !!###
/"""**,, 2C88888s C55C9 <C9 c ddddd}t j t s2| dt t
i |S | r8| }| d|j |j
d nt j t sD| t t
| d nk| d t d
t t
| d t j t |S ) z
CLI command to get scan status
Cases:
1) no previous scans
2) scan in progress
3) scan finished (success)
4) scan process crashed
idler N) scan_statusr r last_scan_timer in_progress)r r r )r r rd zFailed to generate report)r
scan_errorr r )rU rV rW r# r r r r{ r} r< r= r! _get_scan_timestampr? r )rA rx r2 s r,
get_statuszReportGenerator.get_status so "#$#' "
w~~j)) MM0#gii..A B B BM ! ! +))++F MM ,'-~'-|
7>>/22
+
+.wyy>>&*&>&>&@&@
#*"#$?"@"@+.wyy>>&*&>&>&@&@ /***
r+ c 2 t t d 5 } | }t | cddd S # 1 swxY w Y n2# t
$ r%}t d| Y d}~nd}~ww xY wdS )z_
Read timestamp from LAST_SCANNED_TS
Returns None in case of any error
rtNz"Can't read last scan timestamp: %s)rX r" r] rI r: rb rc rd )rf trh s r, r z#ReportGenerator._get_scan_timestamp s Cot,,
FFHHNN$$1vv
C C C LL> B B B B B B B B Cts: A% 5A A% A A% A !A% %
B/BBc D t t fd| S )z<
Return usernames from --users cli argument
c | v S rD r* )xto_scans r, z4ReportGenerator.__filter_usernames..1 s Q'\ r+ )listfilter)r r s ` r, __filter_usernamesz"ReportGenerator.__filter_usernames, s'
F11119==>>>r+ c 8 t t S )z*
Get usernames from cpapi
)r r r* r+ r, __get_usernamesz ReportGenerator.__get_usernames3 s
GII r+ c t t j } t t | t d | S )zr
Write current timestamp to LAST_SCANNED_TS
Return timestamp for the later use, if needed
)r: timer r8 r" )tss r, __update_scan_timestampz'ReportGenerator.__update_scan_timestamp: s4
B %@@@ r+ c | | j }| }| d t j |t j t j z | | j |
t j |t j dS )zq
Write current status to SCANNING_STATUS
Data example: "0/10", without any other symbols
r N) ry r[ seekrY rZ LOCK_EXLOCK_NBwriter2 rk flushr^ )rA rf f_nos r, __update_scan_statusz$ReportGenerator.__update_scan_statusD s
xxzz q
D%-%-7888 ##%%&&&
D%-(((((r+ c ` t t j | j t d dS )zJ
Write final scan result to SCANNING_STATUS via temp file
r N)r r_ rj rx r# rF s r, __write_scan_resultz#ReportGenerator.__write_scan_resultQ s(
4; 7 7UKKKKKr+ c
6 i }| t d }| }|s|s| }|S |r|j |d< |j |d< |r| |d< |d }| || \ |d< |d< |d< }}} }
} } }
}t j |t j |t j | t j |
i|d < t j | t j | t j |
t j |i|d
< |S ) a
1. get-report when there is no cache file:
- start generating report
- return total_users_scanned, total_users_to_scan keys
2. get-report when there is no cache file, but scanning
in progress:
- return total_users_scanned, total_users_to_scan keys
3. get-report when cache present, no scanning running
- return data from cache
- no keys total_users_scanned, total_users_to_scan in response
4. get-report when cache present, scanning is running
- return data from cache
- return total_users_scanned, total_users_to_scan keys
T)is_jsonr r r rv total_wordpress_counttotal_users_counttotal_users_count_activetotal_sites_count_active)
_get_from_file_with_lockingr# r} r r< r= r _get_users_reportr namer r r )rA target_usersreport
scanned_cachescanning_statusr users_cachetotal_users_with_active_awp total_users_with_active_premiumtotal_users_with_active_cdn total_users_with_active_cdn_prototal_sites_with_active_awp total_sites_with_active_premiumtotal_sites_with_active_cdn total_sites_with_active_cdn_pros r, ra zReportGenerator.getX s] 88T8RR
..00 _ 99;;DK B,;,CF( ),;,AF( ) '+'?'?'A'AF# $ (0K &&{LAA
w./*++/+/+/+/
: !#B
: "A 2F- .
: !#B
: "A 2F- .
r+ c | |j j t j k }|o t fd|j D S )Nc g | ]m} |j o> |j j t j k nS r* )ra NAMElowerr2 r ALLOWED)r featureuser_features_datas r,
z5ReportGenerator._is_allowed_suite.. sv )3 )3 )3 *<)?)? @R@R@T@T)U)U *e&**7<+=+=+?+?@@GK\Kdd)3 )3 )3r+ )ra r r2 r r anyfeatures)user_suites_datar suiteis_suite_alloweds ` r, _is_allowed_suitez!ReportGenerator._is_allowed_suite sf +//
;;BFWF__ 4C )3 )3 )3 )3#(>)3 )3 )3 %4 %4 4r+ c t | 5 t 5 t | t j }t | t j }t | t j }d d d n # 1 swxY w Y d d d n # 1 swxY w Y |||fS rD ) r r r #wp_paths_with_active_suite_featuresr r r r primary)r wps_with_enabled_awp_features%wps_with_enabled_awp_premium_features wps_with_enabled_cdn_featuress r, _sites_with_active_featuresz+ReportGenerator._sites_with_active_features sY
X
&
& F F
F F,6x,@,@33H4EFF
*4>x4H4H33L4IJJ
2,6x,@,@33H4DEE
* F F F F F F F F F F F F F F F F F F F F F F F F F F F F F F -.SUrrrs5 CBB0$ C0B4 4C7B4 8CC C user_scanned_datar c g }d\ }}d}d}d}d} d}
d} d} d}
t }| D ]\ }}d}d}d}d}|r||vr| | }|s | j d| H t |t
| | }t t
| | }n.# t $ r! | j d i }i }Y nw xY w |
| \ }}}nE# t $ r8}| j dt | g g g }}}Y d}~nd}~ww xY w|dz
}|t |d z
}|
||t r(|r&|dz
}|
t | z
}
t | }|
||t r(|r&|dz
}| t | z
} t | }|r|
||t" r'| dz
} |
t | z
}
t | }nB|
||t$ r&|dz
}| t | z
} t | }| |||d ||||d
|r"t | t | k r n||||||| |
| | |
f S ) z
Returns list of final users data:
example:
[
{"username": "user1", "features": {"object_cache": true}, "wp_sites_count": "1"},
....
]
rs r zCannot get user %s uid)server_wide_optionsz-Unable to get features data from admin configz9Unable to get information for report from user config: %sNr r )r suitesr accelerate_wp_active_sites_count!accelerate_wp_premium_sites_count accelerate_wp_cdn_sites_count!accelerate_wp_cdn_pro_sites_count)r r _uid_by_namerc warningr r r rb r r r8 r: r r r r r r r ) rA r r all_users_infototal_wp_sites_countr r r r r r r r r default_configr scanned_data&wps_per_user_with_enabled_awp_features.wps_per_user_with_enabled_awp_premium_features wps_per_user_with_enabled_cdn!wps_per_user_with_enabled_cdn_prouidr r r r r rh s r, r z!ReportGenerator._get_users_report s
26/ /&'#*+'&'#*+'&'#*+'&'#*+'022&7&=&=&?&? F F "Hl56 2=> :,- )01 -
< <##H--C
##$> t-/TVsVs
##$_adefagaghhhB Wt/T------
" C 5E(F$G$G G %%&68JHUU
\Zw
\+q0++s3P/Q/QQ+9<=Z9[9[6%%&68JLYY
l _D
l/14//37\3]3]]/ADEjAkAk>,
W))*:PRZ[[ W/14//37T3U3UU/478U4V4V1 ! ! $*"./?"@4Z5c1N5V# #
L 1 1S 5H5H H H 46G+-L+-L+-L+-L N Ns* =C(C,+C,0D
E .EE FrV r with_lockc t t j | sdS |r?t | 5 | || cddd S # 1 swxY w Y dS | || S # t
$ r3}t
t d |t | d d}~ww xY w)z3
Reads file with locking if needed
Nz&Can`t read %(file)s. Reason: %(error)s)filerd r4 ) rU rV rW r _read_report_filerb r% r? r8 )rA rV r r rh s r, r z+ReportGenerator._get_from_file_with_locking s8 w~~d## 4 P
=!$'' A A 11$@@A A A A A A A A A A A A A A A A A A --dG<<< P P P&q1Y/Z/Z8$9tCy $9T $9 $9 $9 $9L(D ( ( ( (T # \ ?tCy ?T#Y ? ? ? \? T#Y \ S \ ) ) ) )L L L L? ? ? ?B 4 4 \4 s s \shN #hN d
+hN
tS#sCc3S#E F hN hN hN hNTP P Pd Pt P`efjloqufu`v P P P P + +d +uT3_7M + + + \+ r+ rq )<
__future__r rU r r rY r_ psutilr typingr r r r r r
secureior r clwpos.data_collector_utilsr
clcommon.clpwdr r $clwpos.feature_suites.configurationsr r clwpos.utilsr r clcommon.cpapir r clwpos.optimization_featuresr clwpos.feature_suitesr r r r r r clwpos.user.configr clwpos.logsetupr clwpos.cl_wpos_exceptionsr clwposr r? clwpos.constantsr! r" r# r& rc r% r. rq r* r+ r, r& s ' & & & & &
= = = = = = = = = = = = = = = = ; ; ; ; ; ; ; ; 9 9 9 9 9 9 1 1 1 1 1 1 1 1 R R R R R R R R > > > > > > > > / / / / / / / / K K K K K K * ) ) ) ) ) ) ) ) ) ) ) / / / / / /
x 9 R7 R7 R7 R7 R7 R7 R7 R7jg g g g g g g g g gr+