;a0Z
@ U d Z ddlZddlZddlZddlZddlZddlZddl Z ddl
Z
ddl Z ddl Z ddl
Z
ddlZddlZddlZddlZddlZddlmZmZ ddlmZmZmZ ddlm Z m Z ddl m Z ddl m!Z! ddl"m#Z# dd l$m%Z% dd
l&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/ dd l0m1Z1 dd l2m3Z3 dd
l4m5Z5 ddl6m7Z7 ddl8m9Z9m:Z:m;Z; ddlZ> ddl?m@Z@ ddlAmBZBmCZC ddlDmEZE e#eF ZG e jH d ZIdZJdZKdeLfdZMdZNdZOd ZPd ZQ e jH d ZRd ZSd ZTd!ZUd"ZV e jW ZXe/eYeZe[eLde)eYe'f e+e' f Z\da]e,eC e^d#< da_e,eB e^d$< deLfd%Z`dWd&ZadeCfd'ZbdeBfd(Zc G d) d*ed Ze G d+ d,ed Zfd- Zgd.e
jh d/eZde(fd0Zide\fd1Zjd2eYd3ekddfd4Zl e:efegeU5 d2eYde\fd6 Zmi d7d2eYd8e[fd9Zn e:efegeU5 d2eYd:e[d8e[deLfd; Zoed Zpd?e(fd@Zqd3ekdAeZdeLfdBZr e:efegeUerC d touchr2 loggerdebugr@ rA r6 _persist_ipv6_disabledrH s_ M ""$$$$$ M M M ->???
rA c e Zd ZdZdS )IntegrityErrorzERaised when on disk content does not match hashes in description.jsonN__name__
__module__ __qualname____doc__r@ rA r6 rR rR s OOOOrA rR c e Zd ZdZdS ) UpdateErrora
Raised on other errors during files update.
Possible reasons are:
* server returns non 200 status;
* hash mismatched between downloaded content and description.json;
* urllib errors;
* JSON decoding errors;
* errors while writing to disk.
NrS r@ rA r6 rY rY s rA rY c K t d | | t j t j d|z t z d {V d S )Nz2Files update failed with error: {err}, try: {try_})errtry_r# )rF warningr. asynciosleeprandom randrange_TIMEOUT_MULTIPLICATORexcis r6 _log_failed_updaterf sy
NN<CC! D
-(a003II
J
JJJJJJJJJJrA pathmodec t d 5 t j | t j t j z t j z | }ddd n # 1 swxY w Y t j |d S )zbOpen file at `path` using permission `mode` for writing in binary mode
and return file object.r Nwb)r osr/ O_WRONLYO_CREATO_TRUNCfdopen)rg rh fds r6 _open_with_moderq s
H H
WT2;3bj@$
G
GH H H H H H H H H H H H H H H
9R s ;AA Ac t | | 5 }t j t j |d |d d cd d d S # 1 swxY w Y d S )Ntimeoutfileheaderszutf-8)encoding)
_fetch_urljsonloadio
TextIOWrapperget_content_charset)urlrt responses r6 _fetch_json_syncr s C ) ) )
Xy !),@@II
s AA&&A*-A*r~ rd c t }| rg| rUt d| | | | t d S d S d S )Nz>Network error for %s via IPv6 IP %r, disabling IPv6. Error: %r)rM is_ipv6_enabledlast_ip_was_ipv6rF r] last_ip disable_ipv6rH )r~ rd choosers r6 _disable_ipv6_on_network_errorr s G !W%=%=%?%? ! L OO
! ! ! !rA )on_error max_triesc Z K t j } | dt | | d{V S # t t
j f$ r(}t d || d}~wt j
$ r7}t | | t d | d}~wt $ r7}t | | t d | d}~wt $ r}t d| d| d}~wt j j t" j j f$ r8}t | | t d | | d}~wt( $ r*}t | | t d| d | d}~ww xY w)
zDownload and decode JSON from *url*.
Return decoded JSON. Raise UpdateError:
* HTTP response status code is not 200;
* Unicode or JSON decoding fails;
* on time outs during HTTP request;
* on other HTTP errors.
Nz!json decode error [{}] for url {}request to {} timed outrequest to {} reset%eof error while updating files, url: , err: 8urllib/http error while updating files, url: {}, err: {} Can't fetch
, reason: )r^ get_event_looprun_in_executorr UnicodeDecodeErrorry JSONDecodeErrorrY r. socketrt r ConnectionResetErrorEOFErrorhttpclient
HTTPExceptionurlliberrorURLErrorr2 )r~ rt loopes r6 _fetch_jsonr s ! # #D=))$0@#wOOOOOOOOO 4 5 N N N=DDQLLMMM > A A A&sA...3::3??@@@ = = =&sA.../66s;;<<<
CC C C C C
K %v|'< =
&sA... F M MQ
= = =&sA...;;;;;<<<=sQ ": F*#A33F*2B77
F*2C66
F*D(F* 3E33
F* %F%%F*rv rt c
t j | dt j pdi|d }t || 5 }|j |j fcddd S # 1 swxY w Y dS )z>Perform HEAD http request to *url* with *timeout* & *headers*.Imunify-Server-Id HEAD)rv methodrs N) r requestRequestr
get_server_idrP r/ coderv )r~ rt rv reqrs r6 _perform_http_head_syncr s .
!9!;!;!Ar
! C
sG 4 4 !vqy ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! !s A88A<?A<
current_mtimec K |t u rdS t |d } t t | |d|i d{V \ }}|dk rt d|d| t t 5 t |d }||k r$t
d
||d || ddd n # 1 swxY w Y dS # t j $ r7}t | | t d | d}~wt $ r7}t | | t d | d}~wt j j t& j j f$ rY}t- |d
r|j dk rY d}~dS t | | t d | | d}~ww xY w)zCheck if we need to download description.json file:
- perform HEAD request if local file exists and older return True
otherwise return False
T)usegmtzIf-Modified-Sincer N zUnexpected http code z for z
Last-Modifieda Got code %r, but last modification date %s is earlier than or equal to the date provided in the If-Modified-Since header, the origin server SHOULD generate a 304 (Not Modified) response [rfc7232]. Here's curl cmd:
curl -s -I -w '%%{http_code}' -H 'If-Modified-Since: %s' '%s'r r r i0 Fr )_NEVERr r r rY r Exceptionr timestamprF r] r rt r r. r r r r r r r hasattrr )r~ r rt formatted_mtimer rv
last_mtimer s r6 _need_to_downloadr s t t<< A A A&sA...3::3??@@@ = = =&sA.../66s;;<<< K %v|'< =
1f !&C--55555&sA... F M MQ
sH #C 2ACCC G ,2D
G +2E (G G&3GG T)compressc # K i }|||d< dt j pdi}|r| ddi t j | | } t
j |fi |5 }t 5 }|j
d dk }|rl|sj|j
d d
k rLt d |j
d |j
| |r#| t | n||j d
V ddd n # 1 swxY w Y ddd dS # 1 swxY w Y dS )zs
Fetch *url* as binary file.
If *compress* is true, ungzipping is done automatically
if necessary.
Nrt r r zAccept-Encodinggzipr Content-Encodingz Content-Typezapplication/zipzRequested gzip but got Content-Encoding=%r. Read response as is [identity]. Headers: %s, as curl cmd:
curl -Is -H 'Accept-Encoding: gzip' '%s')fileobj)ru rv )r r updater r r rP r/ r rv getrF rL items
enter_contextr
) r~ rt r
parameters req_headersr r stackgzippeds r6 rx rx O s7 J '
9 &
(@(B(B(HbIK 8 -v6777
.
k
:
:C
9;;
"'"&&'9::fD $$^448III KKJ $$%788 &&((
##HX$>$>$>??? '
%
s7 7E(B>E E(E E(E E((E,/E, dest_filec 0 t j }| }t | || 5 }|d t
x}rL| | | | |d t
x}Lddd n # 1 swxY w Y |d d dk s| |z
} |d dd }
|
Nt |
} | | k r9t j d
| | | z
d
| } | | |k rt d | d | d
| | S )z
Fetch *url* to *dest_file* and return its md5sum.
Raise *urllib.error.ContentTooShortError* if the downloaded file
has unexpected length.
)rt r ru Nrv r r zContent-Lengthz&{got} bytes read, {diff} more expected)gotdiff)messagecontentzcontent fetched from z does not match hash: expected=z, got=)hashlibmd5tellrx r0 _BUFSIZEr writer intr r ContentTooShortErrorr. hexdigestrY )
r~ r rt r md5sumr initial_file_offsetr chunk file_lengthcontent_length_headerexpected_file_length
got_md5sums
r6 _fetch_n_md5sum_urlr | s +--C#..** C8 < < < # ',,X666e # JJu OOE " " " ',,X666e ## # # # # # # # # # # # # # # I " "#5 6 6& @ @ nn&&)<< !) 3 7 78H$ O O ,#&'<#=#= #{22 l77 D K K ' 1K ? L ! 8 J
jF 2 2
4C
4
4
4
4'1
4
4
s A/B55B9<B9re c ( K dt | vS )NzHTTP Error 404)strrc s r6 $_fetch_and_save_should_retry_handlerr s 3s88 ++rA )r r should_retryr r dest_path dest_modec 0 K t || 5 }t t | |||| d{V cddd S # 1 swxY w Y dS # t j $ r7}t | | t
d | d}~wt $ r7}t | | t
d | d}~wt $ r*}t | | t
d| d| d}~wt j j t j j f$ r8}t | | t
d | | d}~wt $ r-}t | | t
d| d | d
| d}~ww xY w) zFetch bytes from `url`, save them to `dest_path`,
and return md5 checksum of downloaded content.
Raise UpdateError:
* HTTP response status code is not 200;
* on time outs during HTTP request;
* on other HTTP errors.
r Nr r r r r r z to r )rq r r r rt r rY r. r r r r r r r r r2 )r~ r rt r r r r r s r6 _fetch_and_saver sT 0 L
Y
2
2 i"#!
> A A A&sA...3::3??@@@ = = =&sA.../66s;;<<<
&sA... CC C C C C
K %v|'< =
&sA... F M MQ
L L L&sA...JJJ)JJqJJKKKLsi A A A AA A A F 2B
F 2C
F%D (F(3E
F((FF_Itemr datac & d | d D S )z,Return a set of _Item for easy manipulation.c F h | ] }t |d |d S )r~ r )r ).0items r6 z_items.. s* I I I4E$u+tH~ . . I I IrA r r@ )r s r6 _itemsr s I I4= I I IIrA c d } || | t j | D ]d\ }}}|D ],} |t j || | -|D ],} |t j || | -edS )zCheck and change file/dir modes recursively.
Starting at dirname, change all inner directory permissions to dir_perm,
file permissions to file_perm
c t j | j dz }||k rmt j | sPt
d| t | t | t j | | d S d S d S # t $ r t
d| Y d S w xY w)Ni zGFixing wrong permission to file/dir %s [%s] expected [%s] (not symlink)z&Failed to change permission to file %s) rk lstatst_moderg islinkrF r] octchmodPermissionErrorr )
file_dir_path
permission current_modes r6 _os_chmodz"check_mode_dirs.._os_chmod s 8M22:UBLz))"'.. 3 3) ;! %%
OO
33333 *))) LL8-
s B
B %B>=B>N)rk walkrg join) dirnamedir_perm file_permr rg dirsfiles directorynames r6 check_mode_dirsr s & Igx WW - - ; ;dE ? ?I Ibgll433X > > > > ; ;D Ibgll4.. : : : : ;; ;rA description_path
files_pathc || j v sJ | j }|}||k rR| r/| d | dd dS |j }||k PdS dS )aP
Try to fix the structure of /var/imunify360/files/ when
NotADirectoryError happens.
It indicates that some part in the path is a file:
/var/imunify360/files/sigs <- is a file
=> open("/var/imunify360/files/sigs/v1/description.json") will fail.
We try to rectify it by deleting the file but up to FILES_DIR.
T)
missing_ok)parentsexist_okN)r parentis_fileunlinkmkdir)r r _dir topmost_dirs r6 _fix_directory_structurer s )1 1 1 1 1 "DK
*
<<>> KK4K ( ( ( dT : : : E{ *
rA c e Zd Z eej Z ee Zi Z i Z
e Z e Z i Z
dZ edez ej ZdDdZd Zd Zdej d d
fd Zdej fd Zd
Ze dddde de de de de de d d
fd Z e d e fd Z!e d e"e fd Z#e de d e fd Z$dEd e fdZ%d e fdZ&d e"e fd Z'e d Z(d e)e fd Z*d Z+e,fd e-fd Z.d e fd!Z/d"e-d e fd#Z0dEd$Z1de2j3 d%e"e4 d d
fd&Z5d'e"e4 d e6e"e4 e"e f fd(Z7e de d e fd) Z8e de d e fd* Z9e:dEd+ Z;e:d,e2j3 d e2j3 fd- Zd e?e@e?e e e z f f fd2ZAd eBe fd3ZCdFd4ZDd5e2j3 d,e2j3 d eEe2j3 fd6ZFd e fd7ZGe:d8ej d9ej d:e"e d d
fd; ZHdZJdFd?ZKdEdFd@ZLe dGdAeEe d d
fdB ZMe de d d
fdC ZNd
S )HIndexz/static )periodTc X || j vrt d| d| j || _ d| _ dg i| _ | } t
| 5 }t j | | _ ddd n # 1 swxY w Y n# t $ rE t d| t t j | t Y nTt t" t j f$ r6}|r#t' d | |d| _ Y d}~nd}~ww xY w|rX| }t- | r5t' d d
| | j s| dS dS ) z
:param bool integrity_check: check if last update
did not break anything (by interrupting it in the middle or
another programmatic error)
:raise IntegrityError:
z*Trying to initiate unregistered file type z. Allowed types Fr Nz Path %s has a file in parentsz cannot read description file {}Tz'some files are missing or corrupted: {}z, )_TYPES
ValueErrortype _is_blank_json_descriptionfile_pathr/ ry rz NotADirectoryErrorr _throttled_log_errorr pathlibPath FILES_DIRFileNotFoundErrorr r rR r. _corrupted_fileslenr r )selftype_integrity_checkrg r5 r bad_filess r6 __init__zIndex.__init__/ s) # # (U ( ( +( (
r]
))++ "d
*q !Yq\\
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*! D D D & &'F M M M $W\$%7%7 C C C C C
" " "
$5<}>)r+ rT r r r# r r$ s r6 __repr__zIndex.__repr__b s\
'
4::<<((
rA r r% Nc t d| j | | | } || j d dS )zjWhether *files_path* dir may be used for this type's file group.
:raises: IntegrityError
zValidating [%s]: %sTr& N)rF rL r _make_file_groupr$ r FileGroups r6 validatezIndex.validatej sT
)49jAAA))
$)T222222rA c 6 G fdd j }|S )zV
Return FileGroup class: Index class with local path == *files_path*.
c 6 e Zd Zededef fd ZdS ))Index._make_file_group..FileGroupr% r% c F |j k sJ t j S z+Return local base path for given file type.)r rk fspath)clsr% r r$ s r6 r z4Index._make_file_group..FileGroup.files_path| s( ))))y,,,rA N)rT rU rV classmethodr r )r r$ s r6 r6 r: { sP
-s
-s
-
-
-
-
-
- [
-
-
-rA r6 )r+ r5 s `` r6 r4 zIndex._make_file_groupv sG
- - - - - - - - - - - rA c . t j | j }t t j t j t t j | j t j
|d |d d S )Ndirru ) r _PERMSr r rk rg normpathr r _PATHSpardir)r$ permss r6 r zIndex.check_mode_dirs sq TY' G Y TY(?KK
%L &M
rA Fall_zip essentialr%
relative_pathr r rH rI c | j | |r| j | || j |< ||d| j |< || j |< dS )a Add a type to known file types.
* relative_path is a relative path to all files for that type.
* dir_perm is permission mask used to create directories.
* file_perm is permission mask used to create files.
* all_zip is a flag which shows whether that type of files can
be downloaded in all.zip archive. all.zip is expected to be on
the server.
* essential is whether the agent can start if there are errors
updating that type.
)rA ru N)r add_ESSENTIAL_TYPESrD rB _ALL_ZIP_SUPPORT)r> r% rJ r r rH rI s r6 add_typezIndex.add_type sj ,
u , $ $U + + + )
5$,i @ @
5&- U###rA c B K t d | j D S )zuWhether essential files exist.
Note: the files may be corrupted (integrity check is not performed).
c 3 D K | ]}t |d j V dS )Fr3 N)r r )r r% s r6 z.Index.essential_files_exist.. sI
eU333= =
rA )allrM r> s r6 essential_files_existzIndex.essential_files_exist s9
-
rA c 4 | j S )z&Return a set of all known files types.)r copyrT s r6 typesz Index.types s z rA c b t j t | j | S r< )rk rg r r rD r> r% s r6 r zIndex.files_path s! w||Isz%'8999rA c
|rJ| j t j j v r2t j | d S t j | | j d S )z9Return local path for description.json for current index.description.json) r r FilesUpdateDISABLEDrk rg r _descriptionfile_path_latestr )r$ latests r6 r zIndex._descriptionfile_path sm di6#5#>>>7<<11335G
w||DOODI 6 68JKKKrA c r d | D d }|S )Nc . g | ]}|d
|d S )r` rA r@ )r xs r6
z6Index._descriptionfile_path_latest.. s% JJJAakJqxJJJrA r ) _get_listvalues)r$ ltss r6 r_ z"Index._descriptionfile_path_latest s5 JJ !1!1!8!8!:!:JJJ1M
rA c R t }t | j D ]}| |j } t |t j t }n%# t $ r |
| Y _w xY w||j k r|
| |S )z9Return a set of file paths that are missing or corrupted.) setr r
localfilepathr~ r r r r r! rL r )r$ r' r rg actuals r6 r" zIndex._corrupted_files s EE 4:&& $ $D%%dh//D
"4h??$
d###
$$
d###s A!! BBc | j | S )z`
usage example:
>> async with Index.locked(WHITELISTS):
...
)_lockrZ s r6 lockedz Index.locked s y rA c D fdt j D S )z(Return iterable over all files in index.c 3 L K | ] } |j V d S r* rj r~ r r r$ s r6 rR z Index.files.. s3 LL""48,,LLLLLLrA )r r r0 s `r6 r z Index.files s' LLLL
9K9KLLLLrA c | j d S )z+Return 'items' field from JSON description.r )r r0 s r6 r z Index.items s z'""rA c t j | d j S # t $ r |cY S w xY w)zBReturn mtime of description file if it exists, otherwise -math.infTr` )rk statr st_mtimer2 )r$ defaults r6 _descriptionfile_mtimez Index._descriptionfile_mtime sO 7455T5BBCCL L NNN s ,/ >>c | }|sdS |t j j z t j k S )z4Return True if last update was too late in the past.T)ry r r] PERIODtime)r$ _desc_mtimes r6 _is_outdatedzIndex._is_outdated s< 1133 4V / 66DDrA rt c K | j pyt | dk pT| o@t | | j | | d{V S )z>Return True if update from server is needed for current index.r N)r r# r" r~ r _descriptionfile_urlr ry )r$ rt s r6 is_update_neededzIndex.is_update_needed s
N
4((**++a/
!!## +--di88//11
rA c t d 5 t j ||| ddd dS # 1 swxY w Y dS # t $ r"}t t | |d}~ww xY w)z)Create local directory for current index.r )rh r N)r rk makedirsr2 rY r )r$ r dir_moder r s r6 _makedirszIndex._makedirs
s - ""
G
G G(XFFFF
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G
G - - - c!ff%%1 , -s2 A 6 A :A :A
A/
A**A/ to_updatec K | | } || j d }|j |j d }|j |j d }|D ]}| |j } t
j | }
t
j |
s| |
|d t |j | |||j d{V dS )zX
Fetch files from *to_update* set, verify hashes, save to *files_path*.
Fr3 rA ru r )r r N) r4 r rB rj r~ rk rg r isdirr r r ) r$ r r rt r6 fgr file_moder filenamer s r6
_update_fileszIndex._update_files s ))
Yty%
8
8
89RW%e,Ibg&v. D''11Hgooh//G7==))
Bw5AAA!#{
rA remote_itemsc t j } fd|D } fd|D }||z
} fd|D }||z
}||fS )zFigure out what should be updated based on current items,
file system state and remote items.
Return tuple of files to fetch and files to delete.
Files to fetch is a set of _Item.
Files to delete is a set of file paths.c D h | ] } |j S r@ rq rr s r6 r z+Index._calculate_changes..5 s) LLLt))$(33LLLrA c D h | ] } |j S r@ rq rr s r6 r z+Index._calculate_changes..6 s) NNN**4844NNNrA c L h | ] } |j v |!S r@ rq )r r r' r$ s r6 r z+Index._calculate_changes..: s>
!!$(++9<<
<<J#>???rA c K t j | | j }t | }| |j dz }| j | j d }| j | j d }| | j }t 5 }| ||d | t j
||d t ||||d d {V } t j |d
5 }
|
| d d d n # 1 swxY w Y t% j | D ]v\ } } }
| D ]5}t% j t$ j | | | 6|
D ]5}t% j t$ j | | | 6w| | | || }nR# t2 t4 t6 t j t j f$ r"}t= t? | |d }~ww xY w| d d d n # 1 swxY w Y t
||tC | | j tD j# j$ vS ) a
Update current type of files using all.zip archive. Directory with
current type of files will be cleared and replaced with all.zip
contents. all.zip is expected to be on the server
Return whether updated.
:param timeout:
:raise UpdateError: if OSError or http error or
integrity check error (got wrong data from
the server)
all.zipru rA Fr T)r )r r Nr )%r r r r r r r r rB r r r callbackr r zipfileZipFile
extractallrk r r rg r r7 _replace_live_with_new_dirr rR r2
BadZipfile LargeZipFilerY r pop_allboolr r] r^ )r$ rt r new_path archive_pathr r all_zip_urlrollback_stack_archiveroot directories filenamesr r old_pathr s r6 _run_update_all_zipzIndex._run_update_all_zipc sU L !;!;<< ++I66 ))(-)*CDD K *62 ;ty)%0'' 22
[[ 0 %N NN8XN > > > # #& !
$
& # A
1 _\377 17&&x0001 1 1 1 1 1 1 1 1 1 1 1 1 1 1
57GH4E4E J J0D+y%0 J J dI!>!>IIII$- J J dH!=!=yIIIIJ
h''' ::8YOO "$
1
1
1 "#a&&))q0
1
" " $ $ $a0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 % 0 %f lh
y 2 ;;;s\ >AJ H*E H E HE B9H
J -I ; II J JJversionforcec K t j | | j }| st d| j d| |dk rY| }d | D d }t d| j | |
d } t |d z
t | k r |st d
| d | j n(# t $ r t d| j d | w xY w|j D ]}| s| rt |d z
t | k rh| |j d
z }| |d | | | d d{V dS t d
|d| j )a Update to the version specified in *version*.
:param version: version to update to
:raise UpdateError: if OSError or http error or
integrity check error (got wrong data from
the server)
zCannot update z , because it is not a symlink: r` c ( g | ]\ }}|d
|S ru r@ )r verprops r6 rd z#Index.update_to.. s5 !Th rA r z%Try to update to latest version %s %sFstrictVERSIONzVersion z is already set for z5, because current version doesn't have VERSION file: liveTtarget_is_directory
is_updatedNz not found in )r r r r
is_symlinkrY re r rF rL resolver
read_textr1 r! r
iterdiris_dirr r
symlink_torename
_run_hooks)r$ r r r versions current_pathrg
new_live_paths r6 update_tozIndex.update_to s L !;!;<< ##%% +999ii)
h ~~''H %-^^%5%5 G
KK7G
!(((66
1<<>>DDFFGG7##$ $ $ "GwGGDIGG ! +'+yyy,,@
$,,.. D??$$
D9 ,7799??AABBgG G %)NN49v3E$F$FM!,,Tt,LLL!((333 //T/:::::::::FFk'''499MNNNs
A#D7 7%E c t j | | j }| si S | d }i }t
d }|j D ]}| r| |k rd}nd}|dz x} rqd } |
}t
| } |dt | d|| < | |k r| }# t $ r t d|| Y w xY w|t
d k r d|| d< |S ) NFr r+ Tr )currentr` rA z Version file %s is not valid: %sr` )r r r r r r r
r
r r? r r r rF r )
r$ r r result max_versionrg r version_filer _vers
r6 re zIndex._get_list s L !;!;<< ##%% I (((66 cll $,,.. D
||~~ -- $y 00 88::
* 4 4 6 6G"7++D#*"' "4yy$ $F4L
k))&* ! LL:$
H
& % %,0F; )
s AD&D>=D>c g }t | d D ]4\ }}|d rdn |d rdnd}| | | 5|S )z/Return list of versions available in the index.c | d S )Nr r@ )rc s r6 z Index.get_list.. s
AaD rA )keyr z
(current)r` z (latest)r )sortedre r append)r$ r r r markers r6 get_listzIndex.get_list
s # NN " " $ $..
0
0MGT
? >[[
MMW.f.. / / / /
rA c t j | | j }| sdS | d }|j D ]}t j
t j j t j
| j t j j z
j }| rf| sR||k rL|t$ j j k r7t* d| j | t/ j |d dS )z~Remove old versions of files.
This is done by removing old directories in the path, that older than 30 days.
NFr z Removing old version of %s: %sTr )r r r r r r r
r r r nowtimezoneutc
fromtimestamprv rw daysr r r] DAYS_TO_KEEPrF rL r r )r$ r r rg days_olds r6 _clean_old_versionszIndex._clean_old_versions s6 L !;!;<< ##%% F (((66 $,,.. 8 8D 00+++IIKK("+/
8 ))
8 L(( 2 ??? & &
9 % & ) "))*555 '//
0BINNN-O0 %
j====
" " $ $ $[- % - % - % - % - % - % - % - % - % - % - % - % - % - % - %^ s8 =A/G
-DG
B FG
F/G
GGc K j }t || d{V } t | \ }}|p|}|s6t
d j dS t j
j }| r| d nd}t | } t 5 }
| j j d d |
t& j | d |r| r|n|} | rAt | | | fd
|D d{V | || d{V t3 | d z j j d 5 } | t7 j | ddd n # 1 swxY w Y | | | }n6# t@ tB f$ r"}
tE tG |
|
d}
~
ww xY w|
$ ddd n # 1 swxY w Y |rE| r1t
d
| t' j |d j tJ j& j' vS )z
Run update, return whether updated.
:raise UpdateError: if OSError or http error or
integrity check error (got wrong data from
the server)
rs Nz updating %s: nothing to update.Fr rA r Tr c 3 L K | ] } |j V d S r* rq rr s r6 rR z$Index._run_update.. sD $ $9=**4844$ $ $ $ $ $rA r\ ru z,Removing old path on file by file update: %s)(r r r r r rF rL _touchr r r r r r r r r rB r r r r _copytreeunionr rq r ry dumpsencoder7 r rR r2 rY r r r r] r^ )r$ rt r~ as_jsonr r need_updater r r r from_pathru r s ` r6 _run_updatezIndex._run_update| s '' 22#C999999999 # 6 6vg G G 9 ,9 KK949 E E E KKMMM5 L !;!;<< 09/C/C/E/E OI U + + +4 ++I66 [[ * %N NN$+di07%
# #
xt
$
%I ):):I
!!
oo OO $ $ $ $AJ$ $ $ $$Xy'$JJ J J J J J J J
1$ 11K *62 = JJtz' 2 2 9 9 ; ;<<< = = = = = = = = = = = = = = =
h''' ::8YOO"G,
1
1
1!#a&&))q0
1
" " $ $ $U* % * % * % * % * % * % * % * % * % * % * % * % * % * % * %Z 8)) 8 KK>
M($ 7 7 7 7 y 2 ;;;s[ C K15)J :I$ J$I( (J+I( ,.JK1K, K KK11K58K5from_dirto_dir
ignored_pathsc ` K fd}t t j | |d|d d{V dS )z7Copy *from_dir* to *to_dir* except for *ignored_paths*.c t t j t sJ t fd|D S )z(Return names that should not be copied.c 3 ` K | ](}t j | v $|V )d S r* )rk rg r )r r r
rg s r6 rR z8Index._copytree..ignore_names.. sJ 7<<d++}<< <<<< rA )
isinstancerk r= r frozenset)rg namesr
s ` r6 ignore_namesz%Index._copytree..ignore_names s_ bioos33 3 33 !
rA T)symlinksignore
dirs_exist_okN)r r copytree)r r r
r s ` r6 r zIndex._copytree su O
rA r~ c t j t | j | j }| j | j }t j | t j | j v s J d
|| t j || }t j | | j | S )z.Return a local file path corresponding to URL.z$url ({}) does not fit file path ({}))
rk rg relpathr _URL_PATH_PREFIXrD r r r r r. r r )r$ r~ url_relpath type_pathrJ s r6 rj zIndex.localfilepath s goo SMM 5
K * L # #w|K'@'@'H H H H 1 8 8i H H
I H H Y??
w||DOODI 6 6
FFFrA c | d }t j | rt j | dS dS # t
$ r2}t t | Y d}~dS d}~ww xY w)z5Update mtime of description.json file so it is fresh.Tru N) r rk rg isfileutimer2 rF r] r )r$ rg r s r6 r z Index._touch s #--T-::Dw~~d##
# # # NN3q66 " " " " " " " " " #s A A
B 'BB c K t | j | j t g D ]}} || | d {V # t t
f$ r&}t d|| Y d }~Hd }~wt $ r&}t d|| Y d }~vd }~ww xY wt
d| j d| z d S )Nzhook %s error: %sz%s files update finished%sz (not updated)) r _HOOKSr r$ rR r rF r r exceptionrL )r$ r hookr s r6 r zIndex._run_hooks s $+di0<.AA ? ?D
?d4,,,,,,,,,,"N3
;
;
; 0$::::::::
?
?
? !4dA>>>>>>>>
? ( I J /
s! ?B$ A11
B$> B B$c $ K t j j }|sy| | d{V s^t d| j t t j j dz | d d{V dS | j
o| j | j }| }|rd}t d| j | t j
| t j j | d{V }|r!t d| j | nG# t j t" f$ r.}t d | j || d
}Y d}~nd}~ww xY w|rd }t d| j | t j
| t j j | d{V }|r!t d| j | nr# t j t" f$ rY}t d | j || | d d{V | j | j v r|Y d}~dS d}~ww xY w| |p| d{V dS ) a Run update for the current `type` of files.
Normally update is performed when either is true:
* index is never been fetched (description.json missing or broken);
* last update was performed longer than configured period of time ago;
* some local files are missing or have wrong content (md5 hash differs
from description.json).
If force is True then update is performed unconditionally.
Raises asyncio.TimeoutError, UpdateError.
Nz(%s was updated less than %s minutes ago.< Fr r zUpdating %s files via %szUpdated %s using %sz%s update error via %s: %sTzfile by file download)r r] TIMEOUTr rF rL r r r{ r r rN r^ wait_forr SOCKET_TIMEOUT TimeoutErrorrY r r rM )r$ r rt rH file_by_filelog_strupdatedr s r6 r z Index.update s $, 4#8#8#A#A A A A A A A KK: F&-344
//U/33 3 3 3 3 3 3 3 F.ET%:49%E"{ $ G KK2DIw G G G
$ ' 0,, *9 ! ! KKK 5ty'JJJ(+6
$
$
$ 0$)Wa $
$
-G KK2DIw G G G
' 0$$V%7%FGG! ! KKK 5ty'JJJ(+6
0$)Wa ooo7777777779 555GFFFFF
oo)9Eo:::::::::::s3 A D2 2E6$E11E6 A H I/AI**I/ only_typec K |rj | |d }| | 4 d{V | | d{V ddd d{V dS # 1 d{V swxY w Y dS |rt d | j D ]i} | |d }| | 4 d{V | | d{V ddd d{V n# 1 d{V swxY w Y jdS t d | j D ]i} | |d }| | 4 d{V | | d{V ddd d{V n# 1 d{V swxY w Y jdS )zkRun update for all registered `types` of files.
Raises asyncio.TimeoutError, UpdateError.
Fr3 NzUpdating essential fileszUpdating all files)rn r rF rL rM r )r> r+ r only_essentialindexr% s r6
update_allzIndex.update_allL ss .C 5999Ezz),,
*
*
*
*
*
*
*
*ll5)))))))))
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
*
. KK2 3 3 3 -
.
.E5999 ::e,, . . . . . . . . ,,u---------. . . . . . . . . . . . . . . . . . . . . . . . . . .
.
.
KK, - - -
.
.E5999 ::e,, . . . . . . . . ,,u---------. . . . . . . . . . . . . . . . . . . . . . . . . . .
.
.s5 A
A&)A&< C**
C4 7C4 E77
F F c F | j | | dS )z:Add a hook for type_ to be called after successful update.N)r rL )r> r% r! s r6 add_hookzIndex.add_hooke s%
5 d#####rA )T)Fr% N)NFF)OrT rU rV r r^ Lockrm ri r rD rB r rM rN r r r rF r r r( r- r1 rk PathLiker7 r4 r r? r r r rO rU r rX r r r_ r" rn r r r r floatry r~ r r r r r r r r r r staticmethodr r r r r r
re r r r r r r r rj r r r r/ r1 r@ rA r6 r r " s
K % %E
[
F
F
F
SUUFsuu 6::QX666v|DD)# )# )# )#V
32;
34
3
3
3
3 2;
. . .. . .
. . .
. . . [.8
D
[
!c#h ! ! ! [! :s :s : : : [:L LS L L L Lc #c( [ Mx} M M M M# # # .4 Ed E E E E
e
- - - -!,36u:
2$ J$ s5z3s8# $$ $ $ $. K K K K K [K B B B B B [B : : : \ : @gl @w| @ @ @ \@N >? % % % %N$s) 8 8 8 84C C18 C ', C C C CJN s
%%i777777777 + .
8)S
s % A" A A"c N K t d d{V S # t j t f$ ri} t d{V r t d| n#t | t j rt | Y d} ~ dS d} ~ ww xY w)z6Update all files. Don't fail if essential files exist.T)r- Nz2Failed to update files [essential files exist]: %s) r r/ r^ r' rY rU rF r r )r[ s r6 !update_all_no_fail_if_files_existr@ s %%T%::::::::: + . ,,.. . . . . . . LLDc
#w 344
!s*
s % B$A B B$r2 )NF)rW r^ r r r http.clientr r{ ry mathrk r r` r r r| r urllib.errorr urllib.request collectionsr r
contextlibr r r email.utilsr r r r
itertoolsr loggingr packaging.versionr
typingr r r r r r r r r urllib.parser defence360agent.contractsr !defence360agent.contracts.licenser "defence360agent.subsys.panels.baser defence360agent.utilsr r r defence360agent.utils.commonr r defence360agent.utils.threadsr #defence360agent.utils.net_transportr! r" hooksr$ rT rF r r> r3 r- r r7 r8 r9 r: r; r r r _MAX_TRIES_FOR_DOWNLOADrb infr r r r5 JSONTyper; __annotations__r< rB rH rM rP RuntimeErrorrR rY rf r4 rq r r r r r r rx r r r r r r r r r< r/ r rU r> r@ r@ rA r6 rZ s
/ / / / / / / / : : : : : : : : : : 9 9 9 9 9 9 9 9 % % % % % %
" ! ! ! ! ! - , , , , , 8 8 8 8 8 8 = = = = = = E E E E E E E E E E 9 9 9 9 9 9 9 9 3 3 3 3 3 3 8 #w|$DEE A :
$ 4
% GL0 1 1 1 ( c5$d38nd3iG H 8< X3
4 ; ; ;%)
H\ " ) ) )Gd G G G GM M M M 6 P P P P P\ P P P
,
K K K "+ S X
h
!
!)
!
!
!
!
!
,8O "=3 "=H "= "= "= "=L *,
!
!
!
!
!
!
!
!
,8O 7 7 "7-27 7 7 7 7t 59 )
)
)
C )
U )
)
)
)
X& & & & &R,I ,# ,$ , , , ,
%5 0L 0L 0L 0L{0L
0L 0L 0L 0L
0Lf
7UH -..J JU J J J J
; ; ;F @I l07 ,F
$ F
$ F
$ F
$ F
$ F
$ F
$ F
$R *
3 ,1
}
rA