B
ÿ¥§›”^ä™ã @ s0 d dgZ ddlZddlZddlZddlZddlZddlZddlZddlm Z ddlm
Z
m Z dZ dZ
dZe ¡ Zdd „ Zd
d „ ZG d d
„ d
eƒZG dd„ dƒZdd„ ZG dd„ deƒZd$dd„Zdd„ ZG dd „ d eƒZG dd „ d eƒZ e Z G d d „ d e ƒZ G d d „ d eƒZ G d!d"„ d"e ƒZ G d#d„ deƒZ!dS )%ÚPoolÚ
ThreadPoolé Né )Úutil)Ú get_contextÚ TimeoutErroré c C s t t| Ž ƒS )N)ÚlistÚmap)Úargs© r ú9/opt/alt/python37/lib64/python3.7/multiprocessing/pool.pyÚmapstar+ s r c C s t t | d | d ¡ƒS )Nr r )r Ú itertoolsÚstarmap)r r r r
Ú starmapstar. s r c @ s e Zd Zdd„ Zdd„ ZdS )ÚRemoteTracebackc C s
|| _ d S )N)Útb)Úselfr r r r
Ú__init__6 s zRemoteTraceback.__init__c C s | j S )N)r )r r r r
Ú__str__8 s zRemoteTraceback.__str__N)Ú__name__Ú
__module__Ú __qualname__r r r r r r
r 5 s r c @ s e Zd Zdd„ Zdd„ ZdS )ÚExceptionWithTracebackc C s0 t t|ƒ||¡}d |¡}|| _d| | _d S )NÚ z
"""
%s""")Ú tracebackÚformat_exceptionÚtypeÚjoinÚexcr )r r r r r r
r < s
z ExceptionWithTraceback.__init__c C s t | j| jffS )N)Ú rebuild_excr r )r r r r
Ú
__reduce__A s z!ExceptionWithTraceback.__reduce__N)r r r r r" r r r r
r ; s r c C s t |ƒ| _| S )N)r Ú __cause__)r r r r r
r! D s
r! c s0 e Zd ZdZ‡ fdd„Zdd„ Zdd„ Z‡ ZS )ÚMaybeEncodingErrorzVWraps possible unpickleable errors, so they can be
safely sent through the socket.c s. t |ƒ| _t |ƒ| _tt| ƒ | j| j¡ d S )N)Úreprr ÚvalueÚsuperr$ r )r r r& )Ú __class__r r
r P s
zMaybeEncodingError.__init__c C s d| j | jf S )Nz(Error sending result: '%s'. Reason: '%s')r& r )r r r r
r U s zMaybeEncodingError.__str__c C s d| j j| f S )Nz<%s: %s>)r( r )r r r r
Ú__repr__Y s zMaybeEncodingError.__repr__)r r r Ú__doc__r r r) Ú
__classcell__r r )r( r
r$ L s r$ r Fc
C sÌ |d k r(t |tƒr|dks(td |¡ƒ‚|j}| j}t| dƒrR| j ¡ |j ¡ |d k rb||Ž d}xP|d ks‚|r¸||k r¸y
|ƒ } W n$ t
t fk
r° t
d¡ P Y nX | d krÆt
d¡ P | \}
} } }
}yd| |
|Žf}W nH tk
r. } z(|r| tk rt||jƒ}d|f}W d d }~X Y nX y||
| |fƒ W nR tk
r” } z2t||d ƒ}t
d | ¡ ||
| d|ffƒ W d d }~X Y nX d } }
} } }
}|d7 }qjW t
d
| ¡ d S ) Nr zMaxtasks {!r} is not validÚ_writerr z)worker got EOFError or OSError -- exitingz worker got sentinel -- exitingTFz0Possible encoding error while sending result: %sz worker exiting after %d tasks)Ú
isinstanceÚintÚAssertionErrorÚformatÚputÚgetÚhasattrr, ÚcloseÚ_readerÚEOFErrorÚOSErrorr ÚdebugÚ ExceptionÚ_helper_reraises_exceptionr Ú
__traceback__r$ )ÚinqueueÚoutqueueÚ initializerÚinitargsZmaxtasksZwrap_exceptionr1 r2 Z completedÚtaskÚjobÚiÚfuncr ÚkwdsÚresultÚeÚwrappedr r r
Úworker] sJ
$ rH c C s | ‚dS )z@Pickle-able helper function for use by _guarded_task_generation.Nr )Zexr r r
r: Š s r: c @ s, e Zd ZdZdZdd„ Zd>dd„Zd d
„ Zd d „ Zd
d„ Z dd„ Z
di fdd„Z d?dd„Z d@dd„Z
dAdd„Zdd„ ZdBd d „ZdCd d „Zdi ddfd d!„ZdDd"d#„ZdEd$d%„Zed&d'„ ƒZed(d)„ ƒZed*d+„ ƒZed,d-„ ƒZd.d/„ Zd0d1„ Zd2d3„ Z d4d5„ Z ed6d7„ ƒZ e d8d9„ ƒZ d:d;„ Z!d r? ZmaxtasksperchildÚcontextr r r
r › sT
z
Pool.__init__c C sZ d}xPt tt| jƒƒƒD ]:}| j| }|jdk rt d| ¡ | ¡ d}| j|= qW |S )zCleanup after any worker processes which have exited due to reaching
their specified lifetime. Returns True if any workers were cleaned up.
FNzcleaning up worker %dT)ÚreversedÚrangeÚlenr] Úexitcoder r8 r )r ZcleanedrB rH r r r
Ú_join_exited_workersÔ s
zPool._join_exited_workersc C s| xvt | jt| jƒ ƒD ]^}| jt| j| j| j| j | j
| j fd}| j |¡ |j
dd¡|_
d|_| ¡ t d¡ qW dS )z€Bring the number of pool processes up to the specified number,
for use after reaping workers which have exited.
)rK r rJ Z
PoolWorkerTz added workerN)rq r\ rr r] rJ rH rl rf rU rV rT Ú_wrap_exceptionÚappendÚnameÚreplacerb rc r r8 )r rB Úwr r r
r^ ã s zPool._repopulate_poolc C s | ¡ r| ¡ dS )zEClean up any exited workers and start replacements for them.
N)rt r^ )r r r r
Ú_maintain_poolô s zPool._maintain_poolc C s4 | j ¡ | _| j ¡ | _| jjj| _| jjj| _ d S )N)
rI rO rl rf r, Úsendre r5 Úrecvri )r r r r
rM ú s zPool._setup_queuesc C s | |||¡ ¡ S )zT
Equivalent of `func(*args, **kwds)`.
Pool must be running.
)Ú apply_asyncr2 )r rC r rD r r r
Úapply s z
Pool.applyc C s | ||t|¡ ¡ S )zx
Apply `func` to each element in `iterable`, collecting the results
in a list that is returned.
)Ú
_map_asyncr r2 )r rC ÚiterableÚ chunksizer r r
r
s zPool.mapc C s | ||t|¡ ¡ S )zÌ
Like `map()` method but the elements of the `iterable` are expected to
be iterables as well and will be unpacked as arguments. Hence
`func` and (a, b) becomes func(a, b).
)r r r2 )r rC r€ r r r r
r s z Pool.starmapc C s | ||t|||¡S )z=
Asynchronous version of `starmap()` method.
)r r )r rC r€ r ÚcallbackÚerror_callbackr r r
Ú
starmap_async s zPool.starmap_asyncc
c sn y0d}x&t |ƒD ]\}}||||fi fV qW W n8 tk
rh } z||d t|fi fV W dd}~X Y nX dS )zšProvides a generator of tasks for imap and imap_unordered with
appropriate handling for iterables which throw exceptions during
iteration.éÿÿÿÿr N)Ú enumerater9 r: )r Z
result_jobrC r€ rB ÚxrF r r r
Ú_guarded_task_generation s z Pool._guarded_task_generationr c C s¤ | j tkrtdƒ‚|dkrFt| jƒ}| j | |j||¡|j f¡ |S |dk r\td
|¡ƒ‚t |||¡}t| jƒ}| j | |jt
|¡|j f¡ dd„ |D ƒS dS )zP
Equivalent of `map()` -- can be MUCH slower than `Pool.map()`.
zPool not runningr z Chunksize must be 1+, not {0:n}c s s | ]}|D ]
}|V q
qd S )Nr )Ú.0ÚchunkÚitemr r r
ú E s z Pool.imap..N)rS rR rY Ú IMapIteratorrQ rP r1 rˆ Ú_jobÚ _set_lengthr0 r Ú
_get_tasksr )r rC r€ r rE Ú task_batchesr r r
Úimap) s(
z Pool.imapc C s¤ | j tkrtdƒ‚|dkrFt| jƒ}| j | |j||¡|j f¡ |S |dk r\td
|¡ƒ‚t |||¡}t| jƒ}| j | |jt
|¡|j f¡ dd„ |D ƒS dS )zL
Like `imap()` method but ordering of results is arbitrary.
zPool not runningr z Chunksize must be 1+, not {0!r}c s s | ]}|D ]
}|V q
qd S )Nr )r‰ rŠ r‹ r r r
rŒ b s z&Pool.imap_unordered..N)rS rR rY ÚIMapUnorderedIteratorrQ rP r1 rˆ rŽ r r0 r r r )r rC r€ r rE r‘ r r r
Úimap_unorderedG s&
zPool.imap_unorderedc C sB | j tkrtdƒ‚t| j||ƒ}| j |jd|||fgdf¡ |S )z;
Asynchronous version of `apply()` method.
zPool not runningr N)rS rR rY Ú ApplyResultrQ rP r1 rŽ )r rC r rD r‚ rƒ rE r r r
r} d s
zPool.apply_asyncc C s | ||t|||¡S )z9
Asynchronous version of `map()` method.
)r r )r rC r€ r r‚ rƒ r r r
Ú map_asynco s zPool.map_asyncc
C sª | j tkrtdƒ‚t|dƒs$t|ƒ}|dkrTtt|ƒt| jƒd ƒ\}}|rT|d7 }t|ƒdkrdd}t |||¡}t
| j |t|ƒ||d} | j
| | j||¡df¡ | S )zY
Helper function to implement map, starmap and their async counterparts.
zPool not runningÚ__len__Né r r )rƒ )rS rR rY r3 r Údivmodrr r] r r Ú MapResultrQ rP r1 rˆ rŽ )
r rC r€ Zmapperr r‚ rƒ Zextrar‘ rE r r r
r w s&
zPool._map_asyncc C sT t ¡ }x0|jtks$| jr8|jtkr8| ¡ t d¡ q
W | j
d ¡ t d¡ d S )Ngš™™™™™¹?zworker handler exiting)
r_ Úcurrent_threadrS rR rQ Ú TERMINATErz ÚtimeÚsleeprP r1 r r8 )ÚpoolÚthreadr r r
r` • s zPool._handle_workersc
C sj t ¡ }xöt| jd ƒD ]Ü\}}d }zÀx¸|D ]„}|jr@t d¡ P y ||ƒ W q* tk
r¬ } zB|d d… \}
} y||
| d| f¡ W n t k
rš Y nX W d d } ~ X Y q*X q*W |rÜt d¡ |rÌ|d nd} || d ƒ wP W d d } }}
X qW t d¡ y:t d¡ |
d ¡ t d ¡ x|D ]} |d ƒ q&W W n t k
rZ t d
¡ Y nX t d ¡ d S ) Nz'task handler found thread._state != RUNr Fzdoing set_length()r r… ztask handler got sentinelz/task handler sending sentinel to result handlerz(task handler sending sentinel to workersz/task handler got OSError when sending sentinelsztask handler exiting) r_ r› Úiterr2 rS r r8 r9 Ú_setÚKeyErrorr1 r7 )
Ú taskqueuer1 r= rŸ Úcacher ZtaskseqZ
set_lengthr@ rF rA ÚidxÚpr r r
rd ¢ sB
zPool._handle_tasksc C sÈ t ¡ }xªy
|ƒ }W n" ttfk
r6 t d¡ d S X |jr\|jtksPtdƒ‚t d¡ P |d krpt d¡ P |\}}}y|| ||¡ W n t
k
r¢ Y nX d } }}q
W xœ|rP|jtkrPy
|ƒ }W n" ttfk
rô t d¡ d S X |d kr t d¡ q¶|\}}}y|| ||¡ W n t
k
r@ Y nX d } }}q¶W t | dƒr°t d¡ y,x&t dƒD ]}| j
¡ s†P |ƒ qtW W n ttfk
r® Y nX t d t|ƒ|j¡ d S )
Nz.result handler got EOFError/OSError -- exitingzThread not in TERMINATEz,result handler found thread._state=TERMINATEzresult handler got sentinelz&result handler ignoring extra sentinelr5 z"ensuring that outqueue is not fullé
z7result handler exiting: len(cache)=%s, thread._state=%s)r_ r› r7 r6 r r8 rS rœ r/ r¢ r£ r3 rq r5 Úpollrr )r= r2 r¥ r r@ rA rB Úobjr r r
rh Ð s\
zPool._handle_resultsc c s4 t |ƒ}x&tt ||¡ƒ}|s"d S | |fV q
W d S )N)r¡ Útupler Úislice)rC ÚitÚsizer‡ r r r
r s zPool._get_tasksc C s t dƒ‚d S )Nz:pool objects cannot be passed between processes or pickled)ÚNotImplementedError)r r r r
r" s zPool.__reduce__c C s& t d¡ | jtkr"t| _t| j_d S )Nz closing pool)r r8 rS rR ÚCLOSEra )r r r r
r4 s
z
Pool.closec C s$ t d¡ t| _t| j_| ¡ d S )Nzterminating pool)r r8 rœ rS ra rm )r r r r
Ú terminate s
zPool.terminatec C sn t d¡ | jtkr tdƒ‚n| jttfkr4tdƒ‚| j ¡ | j ¡ | j
¡ x| j D ] }| ¡ qZW d S )Nz joining poolzPool is still runningzIn unknown state) r r8 rS rR rY r° rœ ra r rg rj r] )r r§ r r r
r & s
z Pool.joinc C sD t d¡ | j ¡ x*| ¡ r>| j ¡ r>| j ¡ t d¡ qW d S )Nz7removing tasks from inqueue until task handler finishedr )
r r8 Z_rlockÚacquireÚis_aliver5 r© r| r rž )r< Ú task_handlerr® r r r
Ú_help_stuff_finish2 s
zPool._help_stuff_finishc
C sJ t d¡ t|_t|_t d¡ | ||t|ƒ¡ | ¡ sNt|ƒdkrNtdƒ‚t|_| d ¡ t d¡ t
¡ |k r|| ¡ |r¸t |d dƒr¸t d¡ x |D ]} | j
d krž| ¡ qžW t d¡ t
¡ |k rÖ| ¡ t d ¡ t
¡ |k rô| ¡ |rFt |d dƒrFt d
¡ x0|D ](} | ¡ rt d | j ¡ | ¡ qW d S ) Nzfinalizing poolz&helping task handler/workers to finishr z.Cannot have cache with result_hander not alivezjoining worker handlerr± zterminating workerszjoining task handlerzjoining result handlerzjoining pool workerszcleaning up worker %d)r r8 rœ rS rµ rr r³ r/ r1 r_ r› r r3 rs r± Úpid)
Úclsr¤ r< r= rŸ Zworker_handlerr´ Zresult_handlerr¥ r§ r r r
rk ; s<
zPool._terminate_poolc C s | S )Nr )r r r r
Ú __enter__k s zPool.__enter__c C s | ¡ d S )N)r± )r Úexc_typeZexc_valZexc_tbr r r
Ú__exit__n s z
Pool.__exit__)NNr NN)N)N)NNN)r )r )NNN)NNN)#r r r r* ru rJ r rt r^ rz rM r~ r
r r„ rˆ r’ r” r} r– r Ú staticmethodr` rd rh r r" r4 r± r rµ Ú classmethodrk r¸ rº r r r r
r ’ sF
8
. < 0c @ s@ e Zd Zdd„ Zdd„ Zdd„ Zddd „Zdd
d „Zd d
„ ZdS )r• c C s4 t ¡ | _ttƒ| _|| _|| _|| _| || j< d S )N) r_ ZEventÚ_eventÚnextÚ job_counterrŽ rQ Ú _callbackÚ_error_callback)r r¥ r‚ rƒ r r r
r w s
zApplyResult.__init__c C s
| j ¡ S )N)r½ Zis_set)r r r r
Úready s zApplyResult.readyc C s | ¡ std | ¡ƒ‚| jS )Nz{0!r} not ready)r rY r0 Ú_success)r r r r
Ú
successful‚ s zApplyResult.successfulNc C s | j |¡ d S )N)r½ Úwait)r Útimeoutr r r
rÅ ‡ s zApplyResult.waitc C s, | |¡ | ¡ st‚| jr"| jS | j‚d S )N)rÅ r r rà Ú_value)r rÆ r r r
r2 Š s
zApplyResult.getc C sT |\| _ | _| jr$| j r$| | j¡ | jr<| j s<| | j¡ | j ¡ | j| j= d S )N)rà rÇ rÀ rÁ r½ ÚsetrQ rŽ )r rB rª r r r
r¢ “ s
zApplyResult._set)N)N) r r r r r rÄ rÅ r2 r¢ r r r r
r• u s
r• c @ s e Zd Zdd„ Zdd„ ZdS )rš c C sf t j| |||d d| _d g| | _|| _|dkrLd| _| j ¡ || j= n|| t || ƒ | _d S )N)rƒ Tr )
r• r rà rÇ Ú
_chunksizeÚ _number_leftr½ rÈ rŽ Úbool)r r¥ r Úlengthr‚ rƒ r r r
r ¤ s
zMapResult.__init__c C sº | j d8 _ |\}}|rp| jrp|| j|| j |d | j …< | j dkr¶| jrZ| | j¡ | j| j= | j ¡ nF|s†| jr†d| _|| _| j dkr¶| j r¢| | j¡ | j| j= | j ¡ d S )Nr r F)
rÊ rà rÇ rÉ rÀ rQ rŽ r½ rÈ rÁ )r rB Zsuccess_resultÚsuccessrE r r r
r¢ ± s"
zMapResult._setN)r r r r r¢ r r r r
rš ¢ s
rš c @ s: e Zd Zdd„ Zdd„ Zd dd„ZeZdd „ Zd
d „ ZdS )
r c C sJ t t ¡ ¡| _ttƒ| _|| _t ¡ | _
d| _ d | _ i | _
| || j< d S )Nr )r_ Z ConditionZLockÚ_condr¾ r¿ rŽ rQ Ú collectionsÚdequeÚ_itemsÚ_indexÚ_lengthÚ _unsorted)r r¥ r r r
r Í s
zIMapIterator.__init__c C s | S )Nr )r r r r
Ú__iter__× s zIMapIterator.__iter__Nc C s¨ | j „ y| j ¡ }W nn tk
r„ | j| jkr8td ‚| j |¡ y| j ¡ }W n, tk
r~ | j| jkrttd ‚td ‚Y nX Y nX W d Q R X |\}}|r |S |‚d S )N) rÎ rÑ ÚpopleftÚ
IndexErrorrÒ rÓ Ú
StopIterationrÅ r )r rÆ r‹ rÍ r& r r r
r¾ Ú s" zIMapIterator.nextc C s | j | j|krr| j |¡ | jd7 _x8| j| jkrd| j | j¡}| j |¡ | jd7 _q.W | j ¡ n
|| j|< | j| jkr’| j| j = W d Q R X d S )Nr )
rÎ rÒ rÑ rv rÔ ÚpopÚnotifyrÓ rQ rŽ )r rB rª r r r
r¢ ð s
zIMapIterator._setc C s< | j , || _| j| jkr.| j ¡ | j| j= W d Q R X d S )N)rÎ rÓ rÒ rÚ rQ rŽ )r rÌ r r r
r s
zIMapIterator._set_length)N) r r r r rÕ r¾ Ú__next__r¢ r r r r r
r Ë s
r c @ s e Zd Zdd„ ZdS )r“ c C sP | j @ | j |¡ | jd7 _| j ¡ | j| jkrB| j| j= W d Q R X d S )Nr )rÎ rÑ rv rÒ rÚ rÓ rQ rŽ )r rB rª r r r
r¢
s
zIMapUnorderedIterator._setN)r r r r¢ r r r r
r“ s r“ c @ s: e Zd ZdZedd„ ƒZd dd„Zdd „ Zed
d „ ƒZdS )
r Fc O s ddl m} || |ŽS )Nr )rJ )ZdummyrJ )r rD rJ r r r
rJ s zThreadPool.ProcessNr c C s t | |||¡ d S )N)r r )r rn r> r? r r r
r ! s zThreadPool.__init__c C s, t ¡ | _t ¡ | _| jj| _| jj| _d S )N)rN rO rl rf r1 re r2 ri )r r r r
rM $ s
zThreadPool._setup_queuesc C sN yx| j dd qW W n tjk
r, Y nX xt|ƒD ]}| d ¡ q8W d S )NF)Úblock)r2 rN ZEmptyrq r1 )r< r´ r® rB r r r
rµ * s z ThreadPool._help_stuff_finish)NNr ) r r r ru r» rJ r rM rµ r r r r
r s
)Nr NF)"Ú__all__r_ rN r rÏ rW r r r r r r rR r° rœ Úcountr¿ r r r9 r r r! r$ rH r: Úobjectr r• Z AsyncResultrš r r“ r r r r r
Ú
s@
, f')@