rK d Z ddlZddlZddlZddlZddlmZ ddlmZ ddl m
Z
ddlm Z m Z ddlm
Z
ddlmZmZ dd lmZ dd
lmZmZmZmZmZmZmZmZm Z m Z m Z dd l m Z dd l!m"Z"m#Z#m$Z$m%Z%m&Z&m'Z'm(Z(m)Z)m*Z*m+Z+m,Z,m-Z-m.Z.m/Z/m0Z0m1Z1 dd
l2m3Z3 ddl4m5Z5 ddl6m7Z7m8Z8 ddl9m:Z: ddl;mZ> ddl?m@Z@ ddlAmBZBmCZCmDZDmEZEmFZFmGZGmHZH ddlImJZJmKZKmLZL eMZN ejO eP ZQ eF e d d ZR eF e d d ZSeLeJjT jU ZVeLeJjW jU ZXde e7 deeMeMeMf fdZYde e7 deeMeMeMf fdZZde e7 deeMeMeMf fd Z[d Z\d Z]d Z^d Z_d!e`fd"Za G d# d$e Zb G d% d&ece Zd G d' d(e7 Ze G d) d*e7 Zf G d+ d,e7 Zg G d- d.e7 Zh G d/ d0e7 Zi G d1 d2e7 Zj G d3 d4e7 Zk G d5 d6e7 Zl G d7 d8e7 Zm G d9 d:e7 Zn G d; de7 ZpdS )?z,DB tables related to firewall functionality. N) timedelta)Enum)reduce) IPv4Network IPv6Network)starmap)ior
itemgetter)OrderBy) AnyDictIterableIteratorListOptionalSequenceTupleTypeUnionSet)Signal)JOINSQL BooleanFieldCase CharFieldCheck CompositeKey DoesNotExist
FloatFieldForeignKeyField IntegerFieldPrimaryKeyField TextFieldfnprefetchField)
model_to_dict)Reject)Modelinstance) ApplyOrderBy)CHUNK_SIZE_SQL_QUERYsplit_for_chunktimeit)Country)ALLTCPUDP IPNetworkpack_ip_networkunpack_ip_networkis_net)IP IPVersionNumericIPVersionz
0.0.0.0/32 z::/64model
packed_ip_netc b |\ }}}t d||f | j |k z | j |k z S )z
Filters ip addresses/networks contained in ip network net.
:param model: model to apply query
:param packed_ip_net: tuple of integers
:return: peewee expression
(network_address & ?) == ?r netmaskversionr< r= netmaskrB s I/opt/imunify360/venv/lib/python3.11/site-packages/im360/model/firewall.py_filter_ip_net_subnetsrG K sE 'Cw (4+66 =D " =G # % c b |\ }}}t d||f | j |k z | j |k z S )z
Filters ip addresses/networks contained in ip network net.
Does not includes network itself
:param model: model to apply query
:param packed_ip_net: tuple of integers
:return: peewee expression
r? r@ rC s rF _filter_ip_net_subnets_exclusiverJ ] sE 'Cw (4+66 =4 ! =G # %rH c ` |\ }}}t d|f | j |k z | j |k z S )z
Filters ip addresses/networks that includes provided
ip address/network, including network itself
:param model: model to apply query
:param packed_ip_net: tuple of integers
:return: peewee expression
z (? & netmask) == network_addressr@ rC s rF _filter_ip_net_supernetsrL o sC 'Cw .77 =D " =G # %rH c t j | }t | \ }}}|j s.| j | t
| |||f z S | j | t
| |||f z t | |||f z S # t $ r | j | cY S w xY wN) ipaddress
ip_networkr5 hostmaskipcontainsrL rJ
ValueError)r< ip_strip_netrD rE rB s rF _ip_search_conditionrW s %f-- -V 4 4T7 8$$V,,/GT7+0 0
!!&))*53g2FGGH253g:NOOP
) ) )x ((((()s B $CCc v t | \ }}}t | |||f t | |||f z S rN )r5 rL rJ )r< rV rD rE rB s rF _net_search_conditionrY sO (00Cw #
T7# (dG0DEE F FrH c v t fddD r;t d d d }t j | d< nad v r]t j d }t | \ }}} |||d t j | d< S )Nc 3 K | ]}|v V d S rN ).0kkwargss rF z#_add_ip_net_args.. s'
J
J11;
J
J
J
J
J
JrH )network_addressrA rB ra rA rB rR )allr6 r8 ip_net_to_stringadopt_to_ipvX_networkr5 update)r_ rR rD rE rB s ` rF _add_ip_net_argsrf s
J
J
J
J I
J
J
JJJ /
$ %vi'8&:K
*2..t
%fTl
3
3 ,R 0 0T7
# I I
*2..t MrH c t | d t t f r%t | d \ | d< | d< | d< | d= | S )NrR ra rA rB )
isinstancegetr r r5 )argss rF _replace_ip_with_packed_reprrk s\ $((4..; "<==
DJ ' '
" # O O J KrH returnc P | t j k o| t j k S )z Whether expiration time passed.)IPListNEVERtime
expirations rF
is_expiredrs s % C* *CCrH c " e Zd ZdZdZdZdZdZdS )
ActionTypez What to do with matching IPs.dropcaptcha splashscreenignoreN)__name__
__module__ __qualname____doc__DROPCAPTCHA SPLASHSCREENIGNOREr\ rH rF ru ru s* '' DG!L
FFFrH ru c T e Zd ZdZdZeZdZeZdZeZ dZ
e
Z d Z e
defd Zd S )
PurposezIPList's purposes understood by the agent.
An analog of i360.model.firewall.ActionType but for the new
(DEF-17989) server sync case.
whiterv rx rw c | j S rN )valueselfs rF __str__zPurpose.__str__ s
zrH purposec t j t j t j t j t j t j t j t j it | S rN ) r WHITErn r~ BLACKr GRAYr GRAY_SPLASHSCREEN)clsr s rF listnamezPurpose.listname sC
M6< L&, OV[ &":
'
rH N)rz r{ r| r} r r r~ rv r rx r rw r classmethodstrr r\ rH rF r r s~
E E D D!L LGG s [ rH r c $
e Zd ZdZdZdZdZdZdZeeeefZ e
e e e Z
dZdZd ZeZeZd
Z G d d Z ed
Z ed
ed d e g Z dZ e dd Z ed Z e dd Z! e d Z" ed Z# edd Z$ e%d
d
Z& e%d
d Z' e%d Z( e%dd
Z) e d
Z* e d
Z+ e d
Z, ed ededed g Z- G d d Z.e/de0e1 d e1fd Z2e/d e1d e1fd Z3e/d!e0e4 fd" Z5e/d!e0e4 d e6fd# Z7e8d e9e1e6f fd$ Z:e8 dad%e;d&ee8 fd+ Z?e8 dbd%e@e1eAeBf d e ZQe8dcd? ZRe8ed,fd eSe; fd@ ZTe8dA ZUe8dB ZVe8dbdC ZWdD ZXe8dE ZYdF ZZdG Z[dbdHZ\e8d%e;d e0e1 fdI Z]e8dJ Z^e8dK Z_e`dL Zae8 dcd,d,dMd%e@ebjA ebjB f d ezIPList.5 s 3ty{{#3#3 rH r r
country_idr column_namez scope in ('')c D e Zd Zej ZdZ edddd ZdZ dZ
dS ) IPList.Metaiplistra rA rB r ))countryF)r F)rq F)rR FresidentN rz r{ r| r+ dbdatabasedb_tabler primary_keyindexesschemar\ rH rF Metar ^ sB ;"l y)Z
rH r rl c
2 t j j t j t j j t j t j j t j t j j t j dt j i
| t j S )zGiven `action_type` string return corresponding list name.
Return :attr:`GRAY` for an unknown/missing `action_type`.
N) ru r~ r rn r r r r r r ri )r s rF action_type2listnamezIPList.action_type2listnamel s\
O !6< $fk # )6+C #V] &+
#k6;
'
'
(rH r c t j t j j t j t j j t j t j j i| S )z0Return action_type corresponding to iplist name.) rn r ru r~ r r r r r r s rF listname2action_typezIPList.listname2action_typez s?
L*// K+1 $j&=&C
rH
propertiesc x t | | t j nd S )zzGet iplist name corresponding to properties' action_type.
Return GRAY for an unknown/missing action_type
N)rn r ri ACTION_TYPEr s rF get_listname_fromzIPList.get_listname_from s< **%
NN6- . . .
rH c ^ | r | dt j n t j S )zSGet expiration from properties
Return IPList.NEVER if property not definedrr )ri rn ro r s rF get_expiration_fromzIPList.get_expiration_from s+
JNN< 6 6 6
rH c r | | }| j D ]\ }}||k r||fc S J d )z#Return tuple listname and priority.r z can't happen)r IP_LIST_PRIORITIES)r r _listnamepriorityr s rF get_listname_with_priority_fromz&IPList.get_listname_with_priority_from s\ ))*55 "%"8 * * Hh9$$ )))) % . qrH rR srcdestrr full_accessc || j | j fvs
J d t | \ }}}d t t j t j t j
| t j |k t j |k t j
|k D \ } t t j
| t j |k t j |k t j
|k t j | k t |||dd t j
| }
|
t j |k t j |k t j
|k }
|
} |D ]#} | j j | | $| j j || | | S )a Move ip from src lists to dest list,
as `move` used only in UI and CLI we add manual=True
:param ip: ip address
:param src: src lists (WHITE/BLACK/GRAY/GRAY_SPLASHSCREEN)
:param dest: dst list (WHITE/BLACK)
:param expiration: IPs TTL. 0 means permanent
:param full_access: access to all ports
:return int: items moved
z"Move to GRAY list is not supportedc g | ] }|j
S r\ rq )r] recs rF
z IPList.move.. s*
N
rH TF)r rr r manualcaptcha_passedr )r r r5 rn selectr% MAXrr wherer in_ra rA rB deleteexecutere r r sendr ri )
r rR r r rr r rD rE rB max_expirationqrvr s
rF movez IPList.move s ( H !
0
-R 0 0T7
}}RVF,=%>%>??EE##C((&#-$&')
O $ $ "c ) Nd " Ng % /
'))) MM!#
%##C((
)
)
GG "c ) Nd " Ng %
YY[[ 6 6H K $ $X" $ 5 5 5 5 t2 777 rH c t t | j di t | }| j j |j | |S )`
:param kwargs:
:raises: IntegrityError
:return: model instance
r r\ )superrn createrf r r r r r r_ inst __class__s rF r z
IPList.create sU )uVS!!(DD+;F+C+CDD t} 666 rH Nr c |t d t t j | }| | | j |k }t |t r$| t j |k }nTt | \ }}}| t j |k t j |k t j
|k }| }|r&|D ]#} | j j | | $|S )zQDelete ip from lists if exists
Return number of deleted records.
Nzlistname should not be Noner )rT rn r r r r r rh r rR r5 ra rA rB r r r r )
r rR r r r rD rE ver rows_deletedlsts
rF delete_from_listzIPList.delete_from_list s :;; ; MMOO ! !&/"5"5h"?"? @ @
f,--A b# R((AA ,R 0 0NCs&#-$&#% A
yy{{ 5
5
5 #(((4444rH c t t j | }|r$|D ]!}| j j | "|S rN ) rn r r r r r r r r )r listnames num_deletedr s rF clean_listszIPList.clean_lists sy
MMOO ! !&/"5"5i"@"@ A A I I K K 3 %
3
3 #((2222rH c t j t | z
}| j | j k | | z }| j | j k | | z }| j | j k | j dk z | | z }|
||z |z }|S )z
Removes obsoleted graylist/splashscreen+blacklist[manual=False] IPs.
:param num_days: expired more than num_days ago
:return: int rows deleted
)daysF) rp r
total_secondsr r rs r r r r r r )r num_days
expiration_tsgraylist_ip_is_expiredgraysplash_ip_is_expiredblacklist_ip_is_expiredr s rF cleanup_expired_from_bglistz"IPList.cleanup_expired_from_bglist s iX&>&>&>&L&L&N&NN
"%,#(":cnn ?
?
"
LC 1 1 NN= ) )$*
\SY
&zU"
$nn]++
,
JJLL
U&*+)*
WYY rH c T | }|r|| j |k z }|rHt | \ }}}|t j |k t j |k z t j |k z z }| | }|S rN )
rs r r5 rn ra rA rB r r r )r r rR clausesrD rE rB r s rF delete_expiredzIPList.delete_expired7 s .."" 0 s|x/ /G
!0!4!4 Cw '3.>T)+>W,.
G **,,$$W--5577rH c " | j |fddi|S )N
skip_order_byT) _fetch_queryr r filter_argss rF fetch_as_unionzIPList.fetch_as_unionK s" s MMMMMMrH c v t |t sJ | fd|D }t t j t j t j t j t j t j t j
t j t j t j
t j t j t j t j t% j t j j d t0 t2 j t j t0 j k t j | t } |s | t j } || | } || ! | } | | j |k } |rEtE | }
|
r tF |
fntH |f\ } }
| | |
} |r#| t0 j% |k } |2| t j
& | } | S )Nc 4 g | ]}|j j fv|S r\ )r r )r] lnr s rF r z'IPList._fetch_query.._ s8 ch(= >>> >>>rH scopeon)'rh listrn r rR r rr
imported_fromctimedeepcommentr r r auto_whitelistedra rA rB r% ifnullr SCOPE_LOCALaliasjoinr0 r
LEFT_OUTERidr r rs order_bygroup_byhavingr7 rY rW coderS ) r r r r by_ipby_country_code
by_commentr r r rD search_conditions ` rF r zIPList._fetch_queryO s: )T***** ! # I
MM !$
"'& &,88>>wGG
" T'4?'*0LT
N
N
U6?&&y11
2
2
UF%%'''
(
()
, &
69%%A
8$$A A
f,--A 4--C 3&,,*E2
" c
((c2233A 9 788A !//
;;<. s, u/@I/M/M/M/M/MrH c ( g | ]}|j d k
|S r1 r2 r3 s rF r z IPList.fetch.. s, u/@I/M/M/M/M/MrH .)excluder )r )r r, r- r& desc
list_priorityr, get_nodesr splitr r( ri r0 r )r r r, r- r r. r r
purpose_order others_orderordersr4 nodesnoderowsrowentrys rF fetchz IPList.fetch s
C Y 6 6+ 6 6 A A #+ M #+ L F '
z4C%%''' **,,1133
&
G
G$.*0055 " G GDMM"E$))+++FFFFG
F#A C!#~>>>Eyy##
N#0 1L1L1L#M#Mi KK rH c v |s
J d t | j di || S # t $ r |cY S w xY w)z;Return matching row's field value or `default` if not foundz provides kwargs to find by themNr\ )getattrri r )r fieldr r_ s rF get_fieldzIPList.get_field sc 88888v 737,,V,,e44 4 NNN s ) 88c | | j | j | | j |k z | j dk z }|>| j |k }|s || j z }| | }| | | j |k }|S )N ) r rR rr r rs r r is_nullrB )r r r rB r r* s rF fetch_non_expired_queryz IPList.fetch_non_expired_query s JJsvs~ . . 4 4nn
3<8#; <" M
"_ 3F
4#/11333A w.//ArH c # K | ||| } | E d {V d S # t $ r Y d S w xY wrN )rL dictsiterator RuntimeError)r r r rB r s rF fetch_non_expiredzIPList.fetch_non_expired sv ' '+w G G wwyy))++ + + + + + + + + + FF s ,A
AAc # K | t j t j t j }| t j |k t z }| | | j |k }| D ]'}t |d |d |d V (dS )zh
Fetch listname the most efficient (though experiementally found)
way possible.
Nra rA rB ) r rn ra rA rB r r rs r rN r6 )r r r r rB s rF fetch_ipnetwork_listzIPList.fetch_ipnetwork_list s
JJv-v~v~ N N
GGV_0V5F5F5H5H4HH I I
f,--A7799 C#%&II
rH c f t | j S # t $ r Y d S w xY w)Nr )rn ri r r )r rR s rF get_listnamezIPList.get_listname s@ :::$$- - DD s "
00c 0 | j dk | j dz z S )*The result has to be passed to .where(...)r Nrq r s rF is_expirablezIPList.is_expirable s !# 4)?'@@@rH c |t j k r| S | | j t |pt j k z S )rW )rn ro rY rr r rp )r r s rF rs zIPList.is_expired" sY FL ( (##%% % !! Nc-">49;; ? ? ?
rH c b |t j k o | j t j k p
| j |k S )z@Whether the ip record lives longer than given *expiration* time.)rn ro rr r rr s rF lives_longerzIPList.lives_longer+ s. V\)
Ov| + Kt/K
rH c |t j k oJ|rG| dt j t j k p| d |k ndS )zZWhether the properties for ip lives longer than given *expiration*
time.
rr T)rn ro ri )r r rr s rF lives_longer_propzIPList.lives_longer_prop1 sX V\)
JNN< 6 6&, F
9~~l++j8
rH c b | j t j k o|t j k p
| j |k S )z>Whether the ip record lives less than given *expiration* time.)rr rn ro r\ s rF
lives_lesszIPList.lives_less> s. &,.
&, & F$/J*F
rH c | j }t ||| }|j |j k o |j |j k o|j |j k S )z(Analog of 3.7+ self.ip_network.subnet_of)rP r6 rB ra broadcast_address)r rD rE rB abs rF subnet_ofzIPList.subnet_ofD sQ O c4 1 1
I "
;!Q%66
;#q'::
rH c d |
J d |
J d t | j pd| | _ t | j pd| | _ | j |k }|| _ ||| _ | | | j j || | j | j dS )zT
Update blocking properties
:return tuple: real expiration
Nz 'expiration' must not be Nonez'deep' must not be Noner ) force_insertr )rr r
) maxrr r
r r saver r r )r rr r
r r primary_key_changeds rF update_propertieszIPList.update_propertiesN s %%'F%%% !: do 2J?? Q-- "mx 7
DK
2 333 !!(t!444 /I
rH c t | \ }}}| | j | d | | j |k | j |k | j |k
t d
d }|D ]
}|j c S dS )zReturn the name of highest priority list that contains the *ip*.
Return None if *ip* not in any list or record expired.r r; N)r5 r r r9 r r rs ra rA rB r r r8 r- )r rR rD rE rB r rs rF effective_listzIPList.effective_listk s
-R 0 0T7 JJ !!##))*55
U !!!#s* t# w& Xc*oo**,,
-
-
U1XX
A: trH c | j j 5 | ||| cddd S # 1 swxY w Y dS )z.Update ip lists on CaptchaDosAlert atomically.N)_metar atomic*_blacklist_graylisted_on_captcha_dos_alert)r rR rr r s rF )blacklist_graylisted_on_captcha_dos_alertz0IPList.blacklist_graylisted_on_captcha_dos_alert s
Y
&
&
(
( AAJ s AA
Ac t \ }}}g }t d }|D ]H} | j t j k r;| t
j rt dd c S | j t j k rB| t
j r | j rt dd c S | j t j k rH| | s|
| s t d| j d| c S | j k r6| | s!|
| j | j f J|D ] \ }
} | |
| g } | sJ !t j | |k t j |k t j |k f}
t! t j j |
}|d |D z
}t! j |
|r&t2 d d
|D | t j ||d t2 d
t9 t j ft9 | ifd|D S )a Update ip lists on CaptchaDosAlert.
Spec [1]:
if search(ip, "WHITE"): # should not really happen
return
existing_black_supernets = search(ip, "BLACK")
if any(n.expiration >= expiration for n in existing_black_supernets):
# should not really happen
return
existing = search_exactly(ip, "BLACK")
if existing.manual:
# Do nothing if already added manually
return
else:
# exact match with less expiration, remove it
# and replace with new expiration later
remove(ip, "BLACK")
# it can really exist only in GRAY list
for listname in ["GRAY", "GRAY_SPLASHSCREEN", "IGNORE"]:
existing = search_exactly(ip, listname)
if existing and existing.expiration <= expiration:
remove(ip, listname)
add(ip, "BLACK", expiration)
[1]: https://gerrit.cloudlinux.com/#/c/61260/14/src/handbook/message_processing/local_captcha_dos.py
:param ip: attackers ip
:param expiration: when record will expired
:return: Union[Dict, Exception]
r; rq z!Don't blacklist whitelisted ips [z] on CaptchaDosAlertz*Don't blacklist manually blacklisted ips [z. is already blacklisted for long enough time: z >= c 4 g | ]}|j t j fS r\ )rR rn r )r] rn s rF r zEIPList._blacklist_graylisted_on_captcha_dos_alert.. s! FFF!v}-FFFrH zRemoved %s from %s listsc g | ]\ }}|S r\ r\ )r] _Ls rF r zEIPList._blacklist_graylisted_on_captcha_dos_alert.. s 0K0K0Ktq!0K0K0KrH F)rR r rr r r zPut %s on the BLACK listc g | ] \ }}|f
S r\ r\ )r] rx r rR s rF r zEIPList._blacklist_graylisted_on_captcha_dos_alert.. s! GGGKAx"h GGGrH ) blocklist unblocklist) r5 rn find_closest_ip_netsr r r] rp r) r r ra rr rP r& rR r
IgnoreListra bin_andrA rB r r r r loggerinfor dict)r rR rr r networkrE rB r| supernetsrD ip_r r is_subnetignore_subnetss ` rF rs z1IPList._blacklist_graylisted_on_captcha_dos_alert s[ L "1!4!4 w //q/AA $ ; $ ;C|v|++0@0@0M0M+ vrr
,,$$TY[[11 -J - vrr
|v|++ ,, ,47NN:4N4N ,
"
~##C,<, F F H H FF~FFFF !9-55777 KK*B0K0K{0K0K0K
\!
.333 V\"DJ$?$?$? HGGG;GGG
rH c B t | j | j | j S rN r6 ra rA rB r s rF rP zIPList.ip_network s" $,
rH )rr r c , t | \ }}}| t | |||f } | | | } |-| | j | } | | j } | | | j
|k } || | } t | S )a
Returns all supernets containing given network (*ip*)
that are not expired by *expiration* time.
:param ip: ip network to lookup
:param listname: list of listnames
:param limit: number of supernets to return
:param expiration: seconds since the epoch or None
None means "use the current time"
:return: list of matching supernets, ordered by netmask
(desc: from smallest to largest)
)
r5 r r rL rs r r r rA r8 r r- r
)
r rR r r- rr r rD rE rB r s
rF r} zIPList.find_closest_ip_nets s , -R 0 0T7 JJLL $S3g*> ? ?
GGS^^J/// 0 0 ((2233A
JJs{'')) * *
f,--A AAwwrH c J t d fd j D S )Nc v g | ]5\ }}s j |k n t d j j d |k |f6S )")r r r )r] r r r force_no_prefixs rF r z(IPList.list_priority..4 sj
'Hh +JCLH,, < !9 < < <==I
rH )r r )r r s ``rF r9 zIPList.list_priority0 sI
+.*@
rH networksc # K t | t t j t j t j t j dk t j t k z t j dk t j t k z z t
| z }|.| t j t j k }n2| t j
| }| t t j t j k t j t j t j k z t j t j k z t j t j k t j t j k z t j t j k z t j t j k z z |dn!| t j k z }t) | D ]
}t- | V dS )zYield networks which has lasting supernets with higher priority in
db.
If listname isn't provided ignore priorities (for unblock).
Implemented to solve performance issue: DEF-15123
r$ NTr ) _TempIPListfillrn r ra rA rB r IPV4_HOST_MASKIPV6_HOST_MASKrs r r r r r rr ro r9 r settuplesr6 )r r r r r r s rF filter_ips_has_supernetsz IPList.filter_ips_has_supernets? s """ MM '
%.A%&.N*JKNa'FNn,LMO !!-000
2
6<788AA++I6677A
FF ;#66 /77GG -.
>K$77 9 $ .&, > % 0K4J J L ", < > #- =?$ !( D ++--1EE'
8 188:: . .G#W- - - - - . .rH c # p K t | | t j t j t j t t j z
t t j t j k t j t j k z t j t j k z t t j k z t j
t j k t j
t j
k z t j
t j k z z t j
t j k z }t | D ]
}t# | V dS )zYield exact match of ips which already recorded in db with the same
priority and later expiration (so we don't need to add them)
Implemented to solve performance issue: DEF-15123
r N)r r r ra rA rB r rn rs r r r9 r rr ro r r r6 )r r" r r s rF find_ips_with_later_expirationz%IPList.find_ips_with_later_expiration| sa JJ+##
UV&&(((fm^<
=
=
T^{':: -1LLN ~)<<> ++--1EEG $ .&, > % 0K4J J L ", < >
#- = ?
6 188:: . .G#W- - - - - . .rH c
# K t | | t j t j t j t j t j t j k t j t j k t j t j k z t j t j k z z t t j t j k t j t j t j k z t j t j k z t
t j k z t j t j
k t j t j k z t j t j
k z z }| D ] \ }}}}}t ||| ||fV dS )aF Yield
- subnet (include self) with less priority
(listname and less expiration),
- listname of the subnet
- should_unblock which is True if subnet is exact blocked
network with same listname
Implemented to solve performance issue: DEF-15123
r N)r r r rn ra rA rB r r r r9 r rr ro r r6 )r r" r ra rA rB r should_unblocks rF "find_ip_subnets_with_less_priorityz)IPList.find_ip_subnets_with_less_priority s JJ " N N O % 8 (FN:"2f6LLN ~)<<>
$ ;#66*22;3FGG"23
>[%88 : ''))[-AA
C *fl:!, 0FFH #- = ?
N XXZZ
( (
# ' (
(
(
(
( ( (rH )r rr c * t | \ }}}| | j | j |k | j |k z | j |k z }| | | }|-| | j | }| | j
}| | | j |k }t | S )z
Returns all lists containing given network (*ip*)
:param ip: ip network to lookup
:param listname: list of listnames
:return: names of list
)
r5 r r r ra rA rB rs r r r8 r r
) r rR r r rr rD rE rB r s rF
find_listszIPList.find_lists s -R 0 0T7 JJs| $ $ * *
C
'{d"
${g%
'
GGS^^J/// 0 0 ((2233A
JJs|((** + +
f,--AAwwrH c P t j |i t | S rN r ri rk r queryr_ r s rF ri z
IPList.get ' uww{EJ%A&%I%IJJJrH c P t j di t | S Nr\ r
get_or_createrk r r_ r s rF r zIPList.get_or_create ) $uww$LL'CF'K'KLLLrH c P t j di t | S r r
create_or_getrk r s rF r zIPList.create_or_get r rH )include_itselfr
expired_byc t | \ }} | | j | j | j | j |rt | || f nt | || f }|)| | }n8|t j k r(| | | }|-| | j | }| | | j
|k } fd| D S )ar Return ip_network objects containing all
*ip* entries expired by *expired_by* from lists *listname*
which are members of net *ip* including itself if *include_itself*
:param ip: network to lookup members for
:param listname: list name
:param include_itself: whether to include ip itself as a subnet
[default: False]
:param expired_by: expiry date as "seconds since epoch"
return entries expired by given *expired_by* timestamp
- IPList.NEVER :: return all entries (regardless expiration)
- None :: return non-expired entries
Nc B g | ]\ }}}}t || ||f S r\ )r6 )r] rD rE r erB s rF r z+IPList.find_net_members..3 sD
&T8Q sD'
2
2Ha @
rH )r5 r ra rA r rr r rG rJ rs rn ro r r r )
r rR r r r r rD rE r rB s
@rF find_net_memberszIPList.find_net_members
sN 0 -R 0 0T7 JJ clCN
%
M "3dG(< = = =1#T77KLL
)))**AA
6<
'
'z2233A ((2233A
f,--A
*+((**
rH c T | j | }| j |d S )zQReturn list of iplists with less priority than list from given
propertiesNIP_LISTSindexr r r s rF #lists_with_less_or_equal_prioritiesz*IPList.lists_with_less_or_equal_priorities8 s* ""8,,|EFF##rH c Z | j | }| j d|dz S )zRReturn list of iplists with greater priority than list from given
listnameNr; r r s rF &lists_with_greater_or_equal_prioritiesz-IPList.lists_with_greater_or_equal_prioritiesA s. ""8,,|KeaiK((rH c , t | \ }}}| | j |k | j |k | j |k | j |k }| | | j |k }| S rN ) r5 r r r ra rA rB r r )r rR r r r rE rB r s rF removez
IPList.removeK s !0!4!4 w
"" LH $ 7 * K4 K7 "
KK
f 455E}} rH to_blockc t dt | | j D ]L}| |||| j z d Md S )Nr REPLACE)rangelen
BATCH_SIZE insert_many on_conflictr )r r idxs rF
block_manyzIPList.block_manyX sq CMM3>:: C OOHS3 +?%? @ A A M M
giiii rH c t t | t d d j j j } fd j j j D } j | t j t t j t j k t j
t j
k z t j t j k z t j t j k z }g }| D ]-\ }}}} | t% ||| | f . | | |S )zj
Remove *ips* that are not manual from the [iplist] table.
Return ips to unblock.
z({})z, c 3 8 K | ]}t | V d S rN )rF )r] rG r s rF r` z%IPList.remove_many..l s>
$)GC
rH r )r r r formatr rq r field_namesr r rn r rB ra rA r r r& r6 r r r )
r r" primary_key_sqlr to_remove
to_unblockra rA rB r s
` rF remove_manyzIPList.remove_many` s MM$))CI$9$EFF G G
-0Y-B-N
CJ $
UFM>
"
"
T^{':: -1LLN ~)<<> +*>>@
;D;K;K;M;M 7OWgx %owHH
?..y99::BBDDDrH )r FrN NN)NNNNNNF)F)vrz r{ r| r} r r r r r r r
enumeratereversedr r r% r IPv4 VERSION_IP4IPv6 VERSION_IP6r r r rR r r r r ro r" rr r r r
r r r r r r r ra rA rB r r staticmethodr r r r r r r r r r r r4 r boolr r r r r r r r r r r r! r+ frozensetr r r' rD rH rL rQ r rS rU rY rs r] r_ ra rf rl ro rt rs propertyrP rO r} r9 r
r r r r r ri r r r r r r r r r
__classcell__r s @rF rn rn s ;; K
E
E D+ud$56H ii(:(: ; ;<< KK
F KK J &
By
U/66uzz(7K7KLLMMN H
E J
I4(((M L
33
E
> ?" s)" " " " ["H
[
%) :. :.y$sCx.01:. D>:. :. :. [:.x &.y$sCx.01&. &. &. [&.P 8(y$sCx.018( 8( 8( [8(t
s) [: K K K K [K M M M M [M M M M M [M #+
+
+
+
+
s)+
+
eIsC'( )+
+
+
[+
Z $$ # $ $ $ [$ )) # ) ) ) [)
S
[
$t* [ $5D 12$
eIsN# $$ $ $ [$ $ $ $ $rH rn c e Zd Z ed Z ed Z ed Z ed Z ed Z edd Z
G d d Z e d Z
e d Ze d
eeef fd Zd S )
r Fr r Tr c < e Zd Zej ZdZ edddd ZdS )_TempIPList.Meta
tmp_iplistra rA rB r N) rz r{ r| r+ r r r r r r\ rH rF r r s4 ; "l y)Z
rH r c D | j j d d S )Na
CREATE TEMPORARY TABLE IF NOT EXISTS tmp_iplist
(
network_address INT, netmask INT, version INT,
listname TXT VARCHAR(255) NOT NULL CHECK
(listname in ('WHITE','BLACK','GRAY','GRAY_SPLASHSCREEN')),
priority INT,
expiration INT,
PRIMARY KEY (network_address, netmask, version, listname))
)rq r execute_sqlrX s rF _createz_TempIPList._create s. &&
rH c R | d S rN )r r rX s rF _clearz_TempIPList._clear s"
rH r" c $ t |d r| n|}t dt 5 | | d |D }d d d n # 1 swxY w Y |rt dt 5 d}t
dt | | D ]4}| ||||z 5 d d d d S # 1 swxY w Y d S d S )Nitemszprepare tmp_iplist to fillc g | ]g\ }}t t g d g t | t | t | hS ))ra rA rB r r rr )r zipr5 rn r r )r] rR props rF r z$_TempIPList.fill.. s $ B# ,R 0 0 # C CD I I # 6 6t < < rH zfill tmp_iplist r )
hasattrr r/ r r r r r r r )r r" data
batch_sizer s rF r z_TempIPList.fill s $S' 2 2;ciikkk
0&
9
9 KKMMM JJLLL $ !$% D 0 L)622
L
L
CIIz:: L LCOODsZ/?)?$@AAIIKKKKL
L
L
L
L
L
L
L
L
L
L
L
L
L
L
L
L
L
L L Ls$ 5A==BB ADD
DN)rz r{ r| r" ra rA rB r r r rr r r r r r r
r r r\ rH rF r r s "l...Ol&&&Gl&&&Gye$$$H|'''H J
[
[ LuT8^, L L L [ L L LrH r c e Zd ZdZd\ ZZ edd Z edd Z G d d Z
e d
Z e d Z
d S )
LastSynclistzFUsed to track how up-to-date are lists synced from Correlation server.)rR hashTr r F)r r c $ e Zd Zej ZdZdZdS )LastSynclist.Meta
last_synclistr N)rz r{ r| r+ r r r r r\ rH rF r r s ;"rH r c | t j | j |k d S )N) timestamp)re rp r namer )r r s rF update_timestampz LastSynclist.update_timestamp sC
TY[[
))//D0@AAIIKKKKKrH c H | |ddi \ }}|j S )Nr r )r defaults)r r )r r objrx s rF
get_timestampzLastSynclist.get_timestamp s+ "" Q7G"HHQ} rH N)rz r{ r| r} r8 HASHr r r r r r r r r\ rH rF r r s PPHB
a000I 9%T 2 2 2D
L L [L [ rH r c e Zd ZdZ G d d Z e Z ed Ze d Z
e d Z dS ) WhitelistedCrawlerz^
Crawlers for which local alerts must not add IP
to the :attr:`IPList.GRAY` list.
c e Zd Zej ZdZdS )WhitelistedCrawler.Metawhitelisted_crawlersNrz r{ r| r+ r r r r\ rH rF r r s ;)rH r Fr c t j 5 | | }|D ] }t
|| d d d d S # 1 swxY w Y d S )N) description)
crawler_iddomain)r+ r rr insertr WhitelistedCrawlerDomainr )r r domains inserted_idds rF addzWhitelistedCrawler.add s
[
!
! ***==EEGGK
(// *1 0
s A
A77A;>A;c | | j | | }t
}t
|| }g }| d }|D ]6}|j |j d |j D d} |
| 7||fS )NT) clear_limitc g | ] }|j
S r\ )r )r] r s rF r z,WhitelistedCrawler.fetch.. s >>>AH>>>rH )r r r
) r r r r- r, r r& r r r
r& )
r r- r, crawlers_query
domains_querycrawlers_with_domains_queryr' max_countcrawleritems
rF rD zWhitelistedCrawler.fetch s
JJLL ! !#/ 2 2 8 8 ? ? F Fv N N 17799
&.~}&M&M#"((T(:: 2 G j & 2>>go>>> D
MM$ & rH N) rz r{ r| r} r r# r r$ r r r
rD r\ rH rF r r s
* * * * * * * *
B)'''K [ ! ! [! ! !rH r c v e Zd ZdZ G d d Z e Z eeddd Z e
d Z d S )
r zBDomain names used to check if IP is a :class:`WhitelistedCrawler`.c e Zd Zej ZdZdS ) WhitelistedCrawlerDomain.Metawhitelisted_crawler_domainsNr r\ rH rF r r s ;0rH r FCASCADEr
r on_delete related_namer N) rz r{ r| r} r r# r r! r r r$ r r\ rH rF r r s LL1 1 1 1 1 1 1 1
B o
G YE
"
"
"FFFrH r c e Zd ZdZdZdZ ed Z ed ed ee g Z
e dd Z G d
d Z
ed ed
ededefd ZdS )RemoteProxyGroupz9Groups multiple remote proxies together with common data.r
imunify360Fr zsource in ('{}', '{}')r Tr c $ e Zd Zej ZdZdZdS )RemoteProxyGroup.Metaremote_proxy_group))r sourceTN)rz r{ r| r+ r r r r r\ rH rF r r# 8 s ;'/rH r r r& enabledrl c t || }|j |k rdS ||_ | dS )zSet group's enabled status.
Group is identified by name and source. Returns True if enabled
status has changed, False otherwise (it was a noop).r% FT)r ri r' rj )r r r& r' r s rF set_enabledz RemoteProxyGroup.set_enabled= sG !$$$v$>> =G # #5
trH N)rz r{ r| r} MANUAL
IMUNIFY360r r r r r& r r' r r r r r) r\ rH rF r r $ s CC F J 9% D Y
E*11&*EE F F
F lt444G0 0 0 0 0 0 0 0
s
C
$
4
[
rH r c
e Zd ZdZ eed Z ed Z G d d Z e
de e de e de e
d ee fd
Ze
d e d e d
ee fd Ze
d e d
ee fd ZdS ) RemoteProxyz!Remote Proxy networks in a group.Fr c e Zd Zej ZdZdS )RemoteProxy.Meta remote_proxyNr r\ rH rF r r/ R s ;!rH r by_group by_sourcer' rl c | t j t j t j | j t }|#| t j |k }|#| t j |k }|#| t j |k }t | t j
S )zReturns a list of remote proxy networks as dicts.
Results are optionally filtered by group name, source, and enabled
status.) r r r& r r' r r r r
r rN )r r1 r2 r' r s rF r
zRemoteProxy.listV s
JJ # ! $ K
$
(-9::A (/9<==A (0G;<>>rH r r& r c t || \ }}|D ]R}t j t j | }t
||j }| SdS )z>Adds networks to a list of remote proxy in group name, source.r% )r group_idN) r r r8 rc rO rP r- r rj )r r r& r r rx rD proxys rF add_manyzRemoteProxy.add_manyo sw $11tF1KKq C%i&:3&?&?@@C eh???E JJLLLL rH c d |D }g }t t t j |z t j |k }t | D ]0}| |j | 1t t t t j t
t j t j dk }t | D ]}| |S )zrDeletes networks from remote proxy lists.
Only networks coming from groups with given source are deleted.c Z g | ](}t j t j | )S r\ )r8 rc rO rP )r] rD s rF r z/RemoteProxy.delete_networks.. s;
?BB 4S 9 9 : :
rH r )r- r r r r r r& r
r& delete_instancer r r r r% COUNTr )r r& r r r r6 r s rF delete_networkszRemoteProxy.delete_networksy s=
FN
T"
#
#
U;&(2
3
3
U#*f4
5
5
!WW $ $E NN5= ) ) ) ! ! # # # #
# #$4 5 5
T+t
/
/
X&
'
'
VBH[^,,1
2
2
!WW $ $E ! ! # # # #rH N)rz r{ r| r} r! r r r$ r r r r r r r r r
r7 r< r\ rH rF r- r- K s* ++ O ,5 9 9 9EiU###G" " " " " " " " ?3-? C=? $ ?
d ? ? ? [?0 C S [ S DI [ rH r- c N e Zd ZdZ ed Z ed Z ed Z ed Z G d d Z
e fd Z e fd Z
e fd Ze fd Ze d
ed ee fd Ze d
ee fd Z xZS )r~ z
IP addresses from this list are not blocked by firewall. However,
they still can be placed to other lists by either server or local
events or by user request.
Fr c > e Zd Zej ZdZ eddd ZdZ dS )IgnoreList.Meta ignore_listra rA rB r N)
rz r{ r| r+ r r r r r r r\ rH rF r r? s3 ; "l#4iKK rH r c P t j di t | S )r r\ r r rf r s rF r zIgnoreList.create s* uww~99 0 8 8999rH c P t j di t | S r r r s rF r zIgnoreList.create_or_get r rH c P t j |i t | S rN r r s rF ri zIgnoreList.get r rH c P t j di t | S r r r s rF r zIgnoreList.get_or_create r rH supernetrl c # , K t | \ }}}| | j | |k | j |k | j |k }|D ]$}t |j |j |j V %d S rN )r5 r r ra r rA rB r6 )r rF addressrE rB r rB s rF subnetszIgnoreList.subnets s !0!:!: w JJLL
(
(
.
.7 : K4 K7 "
C##S[#+
rH to_deletec t | }|D ]k}t | \ }}}| | j |k | j |k | j |k ld S rN )r r5 r r ra rA rB r )r rJ uniquer rH rE rB s rF r zIgnoreList.remove s Y D%4T%:%: "GT7 JJLL #w. t# w&
giiii
rH )rz r{ r| r} r rR r" ra rA rB r r r r ri r r4 r rI r r r r s @rF r~ r~ s
B"l...Ol&&&Gl&&&G : : : : [: M M M M [M K K K K [K M M M M [M
y
Xi-@
[
tI [ rH r~ c e Zd ZdZ ed Z ed ede de de
d g Z ed Z G d d
Z
e dd Zed
Zedd Zd S ) BlockedPortz]Port blocking configuration.
Effective when `FIREWALL.port_blocking_mode == ALLOW`.
Fr z proto in ('z', 'r r Tc ( e Zd Zej ZdZdZdZdS )BlockedPort.Meta blocked_portr )))portprotoTN rz r{ r| r+ r r r r r r\ rH rF r rP s& ;!
rH r Nc B |M| | j | t j | z }|(| t t | }|#| t
j |k }|S rN )r r rS
IgnoredByPortrW r0 r )r r r r r s rF _add_filter_argsz BlockedPort._add_filter_args s ! $$Z00 '00<<= A ,]EBBCCA & 788ArH c ^ | | j t t
j t t
j t j t j k } | j |fi |}|
S )Nr ) r r distinctr rV r r r0 r rW r )r r r s rF r! zBlockedPort.fetch_count s
JJsv
XZZ
T-
1
1
T!)WZ7
!C 2 2k 2 2wwyyrH r r c
| | j | j | j | j t
j t
j d t
t j
t t j
t
j t j k } | j
|fi |}| | j | j t
j }d }g }t j | | D ]+\ }}d |D } | |d< | | ,||||z S )N
ip_commentr c fddD S )Nc " i | ] }|| S r\ r\ )r] keyrB s rF
z8BlockedPort.fetch..group_key..- s, "%SX rH )r rR rS r r\ rB s `rF group_keyz$BlockedPort.fetch..group_key, s. )K
rH )r^ c @ g | ]}|d
|d |d d S )rR Nr[ )rR r r\ )r] rR s rF r z%BlockedPort.fetch..3 s= d8' $xB|,<=='''rH r" )r r rR rS r rV rR r r r r r0 r rW r itertoolsgroupbyrN r& )
r r- r, r r ra r'
port_protor" ignored_ipss
rF rD zBlockedPort.fetch sP
JJ %++L99
T-
1
1
T!)WZ7
"
!C 2 2k 2 2
JJsxM,< = =
( 0 J J J & &OJ K
!,Ju MM* % % % %fv~-..rH )NNN)r r )rz r{ r| r} r" rR r r r2 r3 r1 rS r r r rW r! rD r\ rH rF rN rN s sN AA %95 J
BiT"""G"l...Ol&&&Gl&&&GiT|<<. se
R
$'r'9#:#: rz?? ; . 1 ,4
rH c F | t | j | j | j fS rN r r` s rF r z$IPListRecord.fetch.. s' %+S[#+ rH )r r, r- map)r r r r, r- r s rF rD zIPListRecord.fetch s We , , A A
rH c T t j di t | }|S r rB r s rF r zIPListRecord.create s, uww~99 0 8 899 rH )rz rN r )rz r{ r| r} r" ra rA rB rx r r r9 IPListIDr r4 r~ r r r r! rD r r r s @rF rt rt s 99"l...Ol&&&Gl&&&G %(((I CH 9 9! 9.6 9 ) 9 9 9 [ 9D
h
3
[
[ [
[
2 [ rH rt c e Zd ZdZ ed Z ed Z G d d Ze de
de e de e
fd Ze de
d
e defd Zed ede fd
ZdS )r z6DB table that stores "purposes" for given iplist_id`s.Fr c @ e Zd Zej ZdZ edd ZdZ dZ
dS )IPListPurpose.Metar r rx ))r1 Fry Nr r\ rH rF r r s5 ;""l9k:: * rH r r{ purposesrl c t t d | | j t | j t j k t j t | k | j
t t t | z S )z=Yield all distinct iplist_id for *ip_version* and *purposes*.r r )r r
r rx rY r rt r rB r: r r r
r r )r r{ r s rF fetch_iplist_idsz IPListPurpose.fetch_iplist_ids s
qMM IIbl # #
XZZ
T,BLL4J$JT
L
L
U %)9*)EE*..c#x&8&8!9!9::; VXX
rH r c H | | j t | j t j k t j t | k | j |k z S )z9How many iplists are there for *ip_version* and *purpose*r )
r rx rY r rt r rB r: r r )r r{ r s rF r! zIPListPurpose.fetch_count sx
IIbl # #
XZZ
T,BLL4J$JT
L
L
U %)9*)EE:(* UWW
rH r c t j t j t j t j t j t j t j t j i| S )z,Return purpose corresponding to iplist name.) rn r r r r~ r r r r r s rF listname2purposez IPListPurpose.listname2purpose( s>
L'- L', K $g&:
rH N)rz r{ r| r} r r r" rx r r r9 r r r r r r! r r r r\ rH rF r r s @@iU###G %(((I
!
-5g->
(
[
I
C
[
3 7 \ rH r )qr} rO rc loggingrp datetimer enumr functoolsr r r r operatorr r
"defence360agent.rpc_tools.validater typingr r
r r r r r r r r r blinkerr peeweer r r r r r r r r r! r" r# r$ r% r& r' playhouse.shortcutsr( "defence360agent.contracts.messagesr) defence360agent.modelr* r+ $defence360agent.model.simplificationr, defence360agent.utilsr- r. r/ im360.model.countryr0 im360.utils.netr1 r2 r3 r4 r5 r6 r7 defence360agent.utils.validater8 r9 r: r r getLoggerrz r r r V4r r V6r rG rJ rL rW rY rf rk r rs ru r r rn r r r r r r- r~ rN rV rt r r\ rH rF r s 2 2 . . . . . . . . $ $ $ $ $ $ $ $ 6 6 6 6 6 6 $ . - - - - - 5 5 5 5 5 5 1 1 1 1 1 1 1 1 = = = = = = O O O O O O O O O O ' ' ' ' ' ' K J J J J J J J J J 8 $ $ \!:!:;;A> W!5!566q9 $ $ ;',S#s]'; $ ;',S#s]'; $ ;',S#s]'; $ &F F F " Dd D D D D
c4 DZ Z Z Z ZU Z Z Zz$DL DL DL DL DL% DL DL DLN 5 0'! '! '! '! '! '! '! '!T# # # # #u # # #&$ $ $ $ $u $ $ $NK K K K K% K K K\A A A A A A A AH^/ ^/ ^/ ^/ ^/% ^/ ^/ ^/BB. B. B. B. B.E B. B. B.Ju u u u u5 u u up5 5 5 5 5E 5 5 5 5 5rH