7(C d Z ddlZddlZddlZddlZddlZddlZddlmZm Z m
Z
G d d Z G d d Z G d d
Z
G d d Zdddej ej fd
ZddZ ddZdS )zDisk Cache Recipes
N )ENOVAL args_to_key full_namec , e Zd ZdZddZd Zd Zd ZdS )AverageraL Recipe for calculating a running average.
Sometimes known as "online statistics," the running average maintains the
total and count. The average can then be calculated at any time.
Assumes the key will not be evicted. Set the eviction policy to 'none' on
the cache to guarantee the key is not evicted.
>>> import diskcache
>>> cache = diskcache.FanoutCache()
>>> ave = Averager(cache, 'latency')
>>> ave.add(0.080)
>>> ave.add(0.120)
>>> ave.get()
0.1
>>> ave.add(0.160)
>>> ave.pop()
0.12
>>> print(ave.get())
None
Nc > || _ || _ || _ || _ d S N_cache_key_expire_tagselfcachekeyexpiretags q/builddir/build/BUILD/imunify360-venv-2.6.2/opt/imunify360/venv/lib/python3.11/site-packages/diskcache/recipes.py__init__zAverager.__init__& " c ( | j d 5 | j | j d \ }}||z
}|dz
}| j | j ||f| j | j ddd dS # 1 swxY w Y dS )zAdd `value` to average.Tretryg r defaultr r r N)r transactgetr
setr r )r valuetotalcounts r addz Averager.add, s
[
!
!
!
-
- ;??49h?GGLE5 UNE QJE KOO |I
s A BB B c f | j | j dd \ }}|dk rdn||z S )z:Get current average or return `None` if count equals zero.r Tr r r N)r r" r
r r% r& s r r" z Averager.get9 7 {ty($OO u zzttuu}4r c f | j | j dd \ }}|dk rdn||z S )z&Return current average and delete key.r Tr) r N)r popr
r* s r r- z Averager.pop> r+ r NN)__name__
__module__ __qualname____doc__r r' r" r- r r r r s_ . 5 5 5
5 5 5 5 5r r c 8 e Zd ZdZd dZd Zd Zd Zd Zd Z dS )
Lockan Recipe for cross-process and cross-thread lock.
Assumes the key will not be evicted. Set the eviction policy to 'none' on
the cache to guarantee the key is not evicted.
>>> import diskcache
>>> cache = diskcache.Cache()
>>> lock = Lock(cache, 'report-123')
>>> lock.acquire()
>>> lock.release()
>>> with lock:
... pass
Nc > || _ || _ || _ || _ d S r
r r s r r z
Lock.__init__T r r c | j | j d| j | j d }|rdS t j d G)z'Acquire lock using spin-lock algorithm.TNr r r MbP?)r r' r
r r timesleep)r addeds r acquirez Lock.acquireZ sZ
KOO |I $ E
Ju
r c H | j | j d dS )z Release lock by deleting key.Tr N)r deleter
r s r releasez Lock.releaseh s% 49D11111r c | j | j v S )z$Return true if the lock is acquired.)r
r r@ s r lockedz Lock.lockedl s yDK''r c . | d S r
r= r@ s r __enter__zLock.__enter__p r c . | d S r
rA r exc_infos r __exit__z
Lock.__exit__s rG r r. )
r/ r0 r1 r2 r r= rA rC rF rL r3 r r r5 r5 D s}
2 2 2( ( ( r r5 c 2 e Zd ZdZddZd Zd Zd Zd ZdS ) RLocka) Recipe for cross-process and cross-thread re-entrant lock.
Assumes the key will not be evicted. Set the eviction policy to 'none' on
the cache to guarantee the key is not evicted.
>>> import diskcache
>>> cache = diskcache.Cache()
>>> rlock = RLock(cache, 'user-123')
>>> rlock.acquire()
>>> rlock.acquire()
>>> rlock.release()
>>> with rlock:
... pass
>>> rlock.release()
>>> rlock.release()
Traceback (most recent call last):
...
AssertionError: cannot release un-acquired lock
Nc > || _ || _ || _ || _ d S r
r r s r r zRLock.__init__ r r c t j }t j }d || } | j d 5 | j | j d \ }}||k s|dk r@| j | j ||dz f| j
| j d d d d S d d d n # 1 swxY w Y t j
d
) z=Acquire lock by incrementing count using spin-lock algorithm.{}-{}Tr Nr r r r r Nr9 )osgetpid threading get_identformatr r! r" r
r# r r r: r; )r pidtidpid_tidr$ r& s r r= z
RLock.acquire sP ikk!##..c** %%D%11
#{ty) L L ue##uzzKOO %!), #| I $
'1
Ju s A$CC Cc t j }t j }d || }| j d 5 | j | j d \ }}||k o|dk }|s
J d | j | j ||dz
f| j
| j d
d
d
d
S # 1 swxY w Y d
S ) z#Release lock by decrementing count.rQ Tr rR r r z cannot release un-acquired lockr r N) rS rT rU rV rW r r! r" r
r# r r )r rX rY rZ r$ r& is_owneds r rA z
RLock.release s' ikk!##..c**
[
!
!
!
-
- ;??49i?HHLE5 %'5EAIH > > > > >8 KOO "|I
s A/CCCc . | d S r
rE r@ s r rF zRLock.__enter__ rG r c . | d S r
rI rJ s r rL zRLock.__exit__ rG r r. r/ r0 r1 r2 r r= rA rF rL r3 r r rN rN w sn * & " r rN c 2 e Zd ZdZd dZd Zd Zd Zd ZdS )
BoundedSemaphorea` Recipe for cross-process and cross-thread bounded semaphore.
Assumes the key will not be evicted. Set the eviction policy to 'none' on
the cache to guarantee the key is not evicted.
>>> import diskcache
>>> cache = diskcache.Cache()
>>> semaphore = BoundedSemaphore(cache, 'max-cons', value=2)
>>> semaphore.acquire()
>>> semaphore.acquire()
>>> semaphore.release()
>>> with semaphore:
... pass
>>> semaphore.release()
>>> semaphore.release()
Traceback (most recent call last):
...
AssertionError: cannot release un-acquired semaphore
r Nc L || _ || _ || _ || _ || _ d S r
)r r
_valuer r )r r r r$ r r s r r zBoundedSemaphore.__init__ s) r c j | j d 5 | j | j | j }|dk r>| j | j |dz
| j | j ddd dS ddd n # 1 swxY w Y t j d ) zBAcquire semaphore by decrementing value using spin-lock algorithm.Tr r r r r Nr9 )
r r! r" r
rc r# r r r: r; r r$ s r r= zBoundedSemaphore.acquire s %%D%11
4;GG199KOO #| I $
Ju s A BB Bc H | j d 5 | j | j | j }| j |k s
J d |dz
}| j | j || j | j ddd dS # 1 swxY w Y dS )z(Release semaphore by incrementing value.Tr r z$cannot release un-acquired semaphorer r N)r r! r" r
rc r# r r re s r rA zBoundedSemaphore.release s
[
!
!
!
-
- KOODIt{OCCE;&&&(N&&& QJE KOO |I
s A.BB Bc . | d S r
rE r@ s r rF zBoundedSemaphore.__enter__ rG r c . | d S r
rI rJ s r rL zBoundedSemaphore.__exit__ rG r )r NNr_ r3 r r ra ra sn *
r ra c . fd}|S )a( Decorator to throttle calls to function.
Assumes keys will not be evicted. Set the eviction policy to 'none' on the
cache to guarantee the keys are not evicted.
>>> import diskcache, time
>>> cache = diskcache.Cache()
>>> count = 0
>>> @throttle(cache, 2, 1) # 2 calls per 1 second
... def increment():
... global count
... count += 1
>>> start = time.time()
>>> while (time.time() - start) <= 2:
... increment()
>>> count in (6, 7) # 6 or 7 calls depending on CPU load
True
c t z t n } |f d t j
fd }|S )NTr8 c d 5
\ }}
}|||z
z z
}d}|k r
|dz
f n+|dk r
||dz
f nd|z
z }d d d n # 1 swxY w Y |r | nn | i |S )NTr r r )r! r" r# )argskwargslasttallynowdelayr r& r funcr rate
sleep_func time_funcs r wrapperz,throttle..decorator..wrapper" sG
^^$^// 3 3"'))C..KD%#)++CcDjD 00E E u}} #UQY'7@@@@ ! #UQY'7@@@@!"Ud 2 3 3 3 3 3 3 3 3 3 3 3 3 3 3 3 Ju%%%%#
& 4((( (s A>B##B'*B')floatr r# functoolswraps)
rr rp rv r rs r r& r namesecondsrt r ru s
` @@r decoratorzthrottle..decorator s uW~~%!% iooo4ikk
#U|F4 HHH ) ) ) ) ) ) ) ) ) ) )
), r r3 ) r r& r{ rz r r ru rt r| s ```````` r throttler} sN < > r c " fd}|S )a Barrier to calling decorated function.
Supports different kinds of locks: Lock, RLock, BoundedSemaphore.
Assumes keys will not be evicted. Set the eviction policy to 'none' on the
cache to guarantee the keys are not evicted.
>>> import diskcache, time
>>> cache = diskcache.Cache()
>>> @barrier(cache, Lock)
... def work(num):
... print('worker started')
... time.sleep(1)
... print('worker finished')
>>> import multiprocessing.pool
>>> pool = multiprocessing.pool.ThreadPool(2)
>>> _ = pool.map(work, range(2))
worker started
worker finished
worker started
worker finished
>>> pool.terminate()
c t n} | t j fd }|S )Nr c L 5 | i |cd d d S # 1 swxY w Y d S r
r3 )rl rm rr locks r rv z+barrier..decorator..wrapper\ s
-
-tT,V,,
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-s )r rx ry ) rr r rv r r r lock_factoryrz r s ` @r r| zbarrier..decoratorX sj !% iooo4|E3v3??? - - - - -
- r r3 )r r rz r r r| s ````` r barrierr > s< 4 r Fr3 c * fd}|S )a/ Memoizing cache decorator with cache stampede protection.
Cache stampedes are a type of system overload that can occur when parallel
computing systems using memoization come under heavy load. This behaviour
is sometimes also called dog-piling, cache miss storm, cache choking, or
the thundering herd problem.
The memoization decorator implements cache stampede protection through
early recomputation. Early recomputation of function results will occur
probabilistically before expiration in a background thread of
execution. Early probabilistic recomputation is based on research by
Vattani, A.; Chierichetti, F.; Lowenstein, K. (2015), Optimal Probabilistic
Cache Stampede Prevention, VLDB, pp. 886-897, ISSN 2150-8097
If name is set to None (default), the callable name will be determined
automatically.
If typed is set to True, function arguments of different types will be
cached separately. For example, f(3) and f(3.0) will be treated as distinct
calls with distinct results.
The original underlying function is accessible through the `__wrapped__`
attribute. This is useful for introspection, for bypassing the cache, or
for rewrapping the function with a different cache.
>>> from diskcache import Cache
>>> cache = Cache()
>>> @memoize_stampede(cache, expire=1)
... def fib(number):
... if number == 0:
... return 0
... elif number == 1:
... return 1
... else:
... return fib(number - 1) + fib(number - 2)
>>> print(fib(100))
354224848179261915075
An additional `__cache_key__` attribute can be used to generate the cache
key used for the given arguments.
>>> key = fib.__cache_key__(100)
>>> del cache[key]
Remember to call memoize when decorating a callable. If you forget, then a
TypeError will occur.
:param cache: cache to store callable arguments and return values
:param float expire: seconds until arguments expire
:param str name: name given for callable (default None, automatic)
:param bool typed: cache different types separately (default False)
:param str tag: text to associate with arguments (default None)
:param set ignore: positional or keyword args to ignore (default ())
:return: callable decorator
c t fn f fdt j
fd fd}|_ S )z/Decorator created by memoize call for callable.Nc n t j } | i |}t j |z
}||fS )z:Time execution of `func` and return result and time delta.)r: )rl rm startresultdeltarr s r timerz2memoize_stampede..decorator..timer s< IKKET4*6**FIKK%'E5= r c 0 j i t dd \ }}|t ur|\ }}t j }||z
}|
z t j t
j z |k r|S t fz } |d|d } | r; fd}
t j |
} d| _
| |S i } |d |d S ) z:Wrapper for callable to cache arguments and return values.T)r expire_timer N)r r c 5 i } | d d d d d S # 1 swxY w Y d S )NTr8 )r# )pairrl r r r rm r r s r recomputezGmemoize_stampede..decorator..wrapper..recompute s " #(5$#9 #9D !II # $'-$'&* & s #488)targetr8 r )
__cache_key__r" r r: mathlograndomr' rU Threaddaemonr r# )rl rm r r r r rp ttl
thread_key thread_addedr threadr betar r r r rv s `` @r rv z4memoize_stampede..decorator..wrapper sw (''888C % !* ! ! D+ 6!! $
ikk!C'FTMDHV]__$=$==DD!M !F9_
$yy ) # ' -Y ? ? ?F$(FMLLNNN
5$)&))D IIc4CtI D D D7Nr c * t | | S )z,Make key for cache given function arguments.)r )rl rm baseignoretypeds r r z:memoize_stampede..decorator..__cache_key__ s tT65&AA Ar )r rx ry r ) rr r r r rv r r r r rz r r s ` @@@r r| z#memoize_stampede..decorator s %)\ $!!w ! ! ! ! !
1 1 1 1 1 1 1 1 1
1 f B B B B B B B !. r r3 )r r rz r r r r r| s ``````` r memoize_stampeder f sU xD D D D D D D D D D DL r )NNN)NFNr r3 )r2 rx r rS r rU r: corer r r r r5 rN ra r; r} r r r3 r r r s
0 0 0 0 0 0 0 0 0 035 35 35 35 35 35 35 35l0 0 0 0 0 0 0 0fD D D D D D D DN= = = = = = = =H
iz= = = =@% % % %R EGB B B B B Br