d Z ddlZddlmZmZmZmZ ddlmZm Z m
Z
m Z m Z ddl
mZmZmZmZmZmZmZmZmZmZmZ ddlmZ n # e$ r Y nw xY wddl m Z m Z m Z m Z m!Z!m"Z"m#Z# ddl$m%Z%m&Z&m'Z'm(Z(m)Z) dd l*m+Z+ dd
l,m-Z- dd l.m/Z/ ddl0Z0ddl1Z1ddl2Z2ddl3Z3d Z4d
Z5ddZ6 G d d Z7 G d de Z8 G d de Z9 G d de Z:dS )z3Contains PackIndexFile and PackFile implementations N) BadObjectAmbiguousObjectNameUnsupportedOperation
ParseError)mman LazyMixin unpack_from
bin_to_hexbyte_ord) create_pack_object_headerpack_object_header_infois_equal_canonical_shatype_id_to_type_map write_object stream_copy
chunk_size delta_types OFS_DELTA REF_DELTAmsb_sizePackIndexFile_sha_to_index)OInfoOStream OPackInfo OPackStream ODeltaStreamODeltaPackInfoODeltaPackStream)DecompressMemMapReaderDeltaApplyReader
Sha1Writer
NullStreamFlexibleSha1Writer)pack)crc32) NULL_BYTE)
PackIndexFilePackFile
PackEntityc j | | }t | \ }}}d}d}|t k rZ|} t || }
| dz
} |
dz } |
dz r/t || }
| dz
} | dz
} | dz |
dz z } |
dz /| }| }n |t
k r|dz }||| }n|}||z } |rDt
||d d| }
|| t ||||
fS | t |||||
fS || t ||| fS | t |||| fS )a&
:return: Tuple(abs_data_offset, PackInfo|PackStream)
an object of the correct type according to the type_id of the object.
If as_stream is True, the object will contain a stream, allowing the
data to be read decompressed.
:param data: random accessible data containing all required information
:parma offset: offset in to the data at which the object information is located
:param as_stream: if True, a stream object will be returned that can read
the data, otherwise you receive an info object onlyN F)
use_regionbufferr
r r r r r r r r )cursoroffset as_streamdatatype_id uncomp_sizedata_rela_offsettotal_rela_offset
delta_infoic delta_offsetabs_data_offsetstreams \/builddir/build/BUILD/cloudlinux-venv-1.0.10/venv/lib/python3.11/site-packages/gitdb/pack.pypack_object_atrB L s V $ $ + + - -D-DT-J-J*G[* J ) T!W Q4x $h <a!!A
FA A L (A -!d(;L $h < "
I ,r 1*+<<=
- 00O
]'->-?-?(@%UU "Kf$U$UU U"$4VWkS]_e$f$ff f "Ifg{$K$KK K"N67KQ[$\$\\ \ c d}d}|du}d}|r|} | t }|t | z
}| | } |t | z
} || |rt | | }t | t k rn| } |t | z
} || |rt | | }|||fS )a Copy a stream as read from read function, zip it, and write the result.
Count the number of written bytes and return it
:param base_crc: if not None, the crc will be the base for all compressed data
we consecutively write and generate a crc32 from. If None, no crc will be generated
:return: tuple(no bytes read, no bytes written, crc32) crc might be 0 if base_crc
was falser N)r lencompressr& flush)
readwritezstreambase_crcbrbwwant_crccrcchunk
compresseds
rA write_stream_to_packrR s
B
Bt#H
C Z
c%jj %%e,,
c*oo
j )
C((C u:: # # J#j//B E* %J$$
C=rC c ( e Zd ZdZdZd Zd Zd ZdS ) IndexWriterzUtility to cache index information, allowing to write all information later
in one go to the given stream
**Note:** currently only writes v2 indices_objsc , t | _ d S N)listrU selfs rA __init__zIndexWriter.__init__ s VV
rC c @ | j |||f dS )z&Append one piece of object informationN)rU append)rZ binsharO r4 s rA r] zIndexWriter.append s%
63/00000rC c p | j d t | }|j } |t j |t
dt j t d }| j D ]+}|t |d d xx dz
cc< ,t d D ]6}|| } |t
d| ||dz xx |z
cc< 7 |t
d|d |d d | j D | j D ]$} |t
d|d d
z %t }| j D ]V}|d } | d k r*| | d
t | z dz
} |t
d| d
z W|D ]} |t
d| t | dk sJ || | d }
||
|
S )zWrite the index file using the given write method
:param pack_sha: binary sha over the whole pack that we index
:return: sha1 binary sha over all index file contentsc | d S )Nr )os rA z#IndexWriter.write.. s
ad rC )key>L( r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r r, rC c 3 & K | ] }|d V
dS )r Nra ).0ts rA z$IndexWriter.write.. s& 44A1Q4444444rC i >Qr0 Fas_hex)rU sortr$ rI r( index_v2_signaturer% index_version_defaultrX r rangejoinr] rE sha) rZ pack_sharI
sha_writer sha_writetmplistri r< vofsrv s rA rI zIndexWriter.write s{
NN+++'..
$ -2333 $t] @AABBB z"" , ,A HQqT!W%% & & &! + & & & &s A
A Id4mm $ $ $ AENNNa NNNN $tWS\**+++ #((4444444555 5 5A Id41
!233 4 4 4 4 && 4 4AA$CZ s### 3w<</!3 Id4z!122 3 3 3 3 ' 'C Id4oo & & & & 8}}"""" (nnEn**
c
rC N)__name__
__module__ __qualname____doc__ __slots__r[ r] rI ra rC rA rT rT sR 2 2 I 1 1 16 6 6 6 6rC rT c e Zd ZdZdZdZdZ fdZd Zd Z d Z
d Z d
Z d Z
d Zd
Zd Zd Zd Zd Zd Zd Zd Zd Zd Zd Zd Zd Zd e v rd Z xZ S xZ S ) r( zmA pack index provides offsets into the corresponding pack, allowing to find
locations for offsets faster.i s tOcrl c V t || _ d S rW )superr[
_indexpath)rZ indexpath __class__s rA r[ zPackIndexFile.__init__ s$
#rC c F t j | j d | _ d S rW )r force_map_handle_removal_winr _cursorrY s rA closezPackIndexFile.close s )$/::: rC c |dk r(| j dd | _ d S |dk r(| j dd | _ d S |dk rt j | j | _ t j dk rp| j t j k rDt d| j | j t j fz d S d S | j }|d d | j
k rdpd | _ | j dk r/t d
|d d }|| j k s
J d |z d D ]+}t | |t | d
|| j fz ,| d S )N_packfile_checksumr r zsThe index file at %s is too large to fit into a mapped window (%i > %i). This is a limitation of the implementation rl r, re z Unsupported index version: %i)entryr4 rv rO z_%s_v%i)r mapr r make_cursorr r1 window_size file_sizeAssertionErrorrr _versionr setattrgetattr _initialize)rZ attrmmap
version_idfnames rA _set_cache_zPackIndexFile._set_cache_ s ' ' '&*l&6&6&8&8S&AD # # #
)
)
)&*l&6&6&8&8&>D # # #
Y
+DO<<GGIIDL !!A%%$,*@*@*B*BTEUEWEW*W*W$ &[OT\%;%;%=%=t?O?Q?Q^S &S T T T &%*W*W <##%%D !"1"X)@ @ FQ L1DM}!! (tQ 7 7 :
!T]2224SV`4`222
;
X
XeWT9t}?U3U%V%VWWWW
rC c d t d| j d|dz z dz S )z!:return: tuple(offset, binsha, 0)z>L20s r r r r rZ r< s rA _entry_v1zPackIndexFile._entry_v13 s/ 7DL$4$4$6$6q2v
FFNNrC c j t d| j d|dz z d S )zsee ``_offset_v2``re r r r r r s rA
_offset_v1zPackIndexFile._offset_v17 s/ 4 !1!1!3!3TAF]CCAFFrC c ` d|dz z dz }| j ||dz S )zsee ``_sha_v2``r r r r0 r r rZ r< bases rA _sha_v1zPackIndexFile._sha_v1; s5 q2v"| !!$tby.11rC c dS ) unsupportedr ra r s rA _crc_v1zPackIndexFile._crc_v1@ s qrC c ~ | | | | | | fS )z#:return: tuple(offset, binsha, crc))
_offset_v2_sha_v2_crc_v2r s rA _entry_v2zPackIndexFile._entry_v2G s/ ""DLLOOT\\!__EErC c t d| j | j |dz z d }|dz r. s/ LLU++LLLLLLrC )
r array frombytesr r r r sys byteorderbyteswaptuplert r )rZ as ` rA offsetszPackIndexFile.offsets s =A C A
KK ((**4+>> trC r c " t | | S rW r rZ rv s rA r zPackIndexFile.sha_to_index s -dC88 8rC ) r} r~ r r r rr rs r[ r r r r r r r r r r r r r r r r r r r r globals
__classcell__)r s @rA r( r( s % % # $ $ $ $ $ # # # PO O OG G G2 2 2
F F F 2 2 2
W W WG G G ' ' ' + + +( ( (M M M" 41 1 1f $wwyy00 9 9 9 9 9 9 9 1000rC r( c x e Zd ZdZdZdZdZdZdZd Z d Z
d Z dd Z d Z
d
Zd Zd Zd Zd Zd Zd ZddZdS )r) a A pack is a file written according to the Version 2 for git packs
As we currently use memory maps, it could be assumed that the maximum size of
packs therefore is 32 bit on 32 bit systems. On 64 bit systems, this should be
fine though.
**Note:** at some point, this might be implemented using streams as well, or
streams are an alternate path in the case memory maps cannot be created
for some reason - one clearly doesn't want to read 10GB at once in that
case) _packpathr _sizer iKCAPrl r0 c || _ d S rW r )rZ packpaths rA r[ zPackFile.__init__
s
!rC c F t j | j d | _ d S rW )r r r r rY s rA r zPackFile.close s )$.999 rC c t j | j | _ t d| j d \ }| _ | _ || j k rt d|z d S )N>LLLr zInvalid pack signature: %i) r r r r1 r r r r r pack_signaturer )rZ r r7 s rA r zPackFile._set_cache_ s{ '77BBDD .9 AQAQASASUV-W-W*
d) ) ) 9G CDD D * )rC Tc # K | j }| | j z
}|p| j }t }||k rt ||d \ }}t
|j |j |j t |j j |j k sJ |||j
z
|j z z
}|r|j d |V ||k dS dS )z7Handle the actual iteration of objects within this packTr N)r r footer_sizefirst_object_offsetr# rB r rH rI r r r@ _br pack_offsetcompressed_bytes_readseek) rZ start_offsetr5 r= content_size
cur_offsetnull data_offsetostreams rA
_iter_objectszPackFile._iter_objects s L{{}}t'77 !=T%=
||<''#1!Z#F#F K
dj',
K K K>% 5555 ;)<<@d@d@f@ff fJ
'##A&&&MMM <''''''rC c | j S )z2:return: The amount of objects stored in this pack)r rY s rA r z
PackFile.size:
zrC c | j S )z!:return: the version of this packr rY s rA r zPackFile.version> s
} rC c X | j S )z
:return: read-only data of this pack. It provides random access and usually
is a memory map.
:note: This method is unsafe as it returns a window into a file which might be larger than than the actual window size)r r1 r rY s rA r6 z
PackFile.dataB s$ |&&((,,...rC c | j | j dz
dd S )zE:return: 20 byte sha1 hash on all object sha's contained in this filer0 N)r r1 r r2 rY s rA checksumzPackFile.checksumJ s@ |&&t|'='='?'?"'DEELLNNqqqQQrC c | j S )z :return: path to the packfiler rY s rA r z
PackFile.pathN s
~ rC c t }| j } t ||d d }| | |j t
k r|j |j z
}nnN|S )a
:return: list of pack streams which are required to build the object
at the given offset. The first entry of the list is the object at offset,
the last one is either a full object, or a REF_Delta stream. The latter
type needs its reference object to be locked up in an ODB to form a valid
delta chain.
If the object at offset is no delta, the size of the list is 1.
:param offset: specifies the first byte of the object within this packTr, )rX r rB r] r7 r r r; )rZ r4 r r= r s rA collect_streamszPackFile.collect_streamsU sq ff L $Q55a8G JJw )++ ,w/AA
rC c H t | j |p| j d d S )zRetrieve information about the object at the given file-absolute offset
:param offset: byte offset
:return: OPackInfo instance, the actual type differs depending on the type_id attributeFr, rB r r rZ r4 s rA infoz
PackFile.infor s'
dlF,Nd6NPUVVWXYYrC c H t | j |p| j d d S )zRetrieve an object at the given file-relative offset as stream along with its information
:param offset: byte offset
:return: OPackStream instance, the actual type differs depending on the type_id attributeTr, r r s rA r@ zPackFile.streamy s'
dlF,Nd6NPTUUVWXXrC r c 0 | |d S )a
:return: iterator yielding OPackStream compatible instances, allowing
to access the data in the pack directly.
:param start_offset: offset to the first object to iterate. If 0, iteration
starts at the very first object in the pack.
**Note:** Iterating a pack directly is costly as the datastream has to be decompressed
to determine the bounds between the objectsTr5 r )rZ r s rA stream_iterzPackFile.stream_iter s !!,$!???rC N)Tr )r} r~ r r r r pack_version_defaultr r r[ r r r r r r6 r r r r r@ r ra rC rA r) r) s >I N K" " "
E
E
E 4 / / /R R R :Z Z ZY Y Y @ @ @ @ @ @rC r) c e Zd ZdZdZeZeZd Z d Z
d Z d Z d Z
dd Zd
Zd Zd Zd
Zd Zd ZddZd Zd Zd Zd Zeddej fd Z edej fd Z dS )r* zqCombines the PackIndexFile and the PackFile into one, allowing the
actual objects to be resolved and iterated)_index_pack _offset_mapc t j | \ }}| d|z | _ | d|z | _ dS )zGInitialize ourselves with the path to the respective pack or index filez%s.idxz%s.packN)osr splitext IndexFileClsr! PackFileClsr" )rZ pack_or_index_pathbasenameexts rA r[ zPackEntity.__init__ sT (();<<
#''8(;<< %%i(&:;;
rC c j | j | j d S rW )r! r r" rY s rA r zPackEntity.close s0
rC c t | j }t | j | j j z
}|s
J d d }t | dk r |d |i}nWt | }t | }t | t || }t | }|||d < || _ d S )NzCannot handle empty indicesr, r ) sortedr! r rE r" r6 r iternextzipdictr# )rZ r offsets_sorted last_offset
offset_map iter_offsetsiter_offsets_plus_one consecutives rA r zPackEntity._set_cache_ s
3 3 5 566$*//++,,tz/EE << <<<~
~ ! # #(+[9JJ //L$( $8$8 ! & ' ' ' l,ABBK k**J .9J~b) *%rC c \ | j | }|t | |S )z*:return: index for the given sha, or raise)r! r r )rZ rv r s rA
_sha_to_indexzPackEntity._sha_to_index s- ((-- =C.. rC c # K | j j }| j }t | j D ]} | || || V dS )zPIterate over all objects in our index and yield their OInfo or OStream instencesN)r! rv _objectrt r )rZ r5 _shar= r s rA r zPackEntity._iter_objects sj {,4;++--.. 9 9E'$$u++y%88 8 8 8 8 9 9rC r. c |dk r| | }|| j | }| j | }t | j j | \ }}}|r|t vr;| j
| }t ||j |j
|j
S | | } t j | }
t# ||
j d|
S |t vr t% |t& | | S | | } | d d } t+ | \ }} t+ | | \ }}
| d j t v rt/ |d t% || d j |
S )z:return: OInfo or OStream object providing information about the given sha
:param index: if not -1, its assumed to be the sha's index in the IndexFiler Ni r. z Could not resolve delta object)r; r! rv r4 r
r" r r1 r2 r r@ r typer collect_streams_at_offsetr! newr r r rH r r7 r )rZ rv r5 r r4 r7 r8 r9
packstreamstreamsdstreambufsrc_size target_sizes rA r= zPackEntity._object s 199&&s++E ;+//%((C##E**1HI[IfIfgmInInIuIuIwIw1x1x. . =k)) !Z . .v 6 6
sJOZ_jFWXXX 44V< LLL
44V<Y..!&"3Z@@ M!%!9!9&:K:S:S:U:U!V!V!%!9!9&:K!L!L ~ !Z . .t{/A/A&/I/I J JFNN6****
"Z . .v/@ A AFNN6***! .K//* rC c | | j | | S )ab
As ``PackFile.collect_streams``, but takes a sha instead of an offset.
Additionally, ref_delta streams will be resolved within this pack.
If this is not possible, the stream will be left alone, hence it is adivsed
to check for unresolved ref-deltas and resolve them before attempting to
construct a delta stream.
:param sha: 20 byte sha1 specifying the object whose related streams you want to collect
:return: list of streams, first being the actual object delta, the last being
a possibly unresolved base object.
:raise BadObject:)rA r! r4 r; r s rA r zPackEntity.collect_streams s7 --dk.@.@ASASTWAXAX.Y.YZZZrC Nc |}|s:t |t t f st | }t | }t | }|j }d} d}
|du} |t
dt j t j | | dz
} | rt }
d} |D ]}
| dz
} d}t |
j |
j
}|rt | }nd} || t j | }|
j }t% |j ||| \ }}}||
j
k sJ | r |
|
j || | t | |z z
} | |k r n| |k rt- d|| fz | d }t | d
k sJ || | t | z
} d}| r|
|| }||fS ) a
Create a new pack by putting all objects obtained by the object_iterator
into a pack which is written using the pack_write method.
The respective index is produced as well if index_write is not Non.
:param object_iter: iterator yielding odb output objects
:param pack_write: function to receive strings to write into the pack stream
:param indx_write: if not None, the function writes the index file corresponding
to the pack.
:param object_count: if you can provide the amount of objects in your iteration,
this would be the place to put it. Otherwise we have to pre-iterate and store
all items into a list to get the number, which uses more memory than necessary.
:param zlib_compression: the zlib compression level to use
:return: tuple(pack_sha, index_binsha) binary sha over all the contents of the pack
and over all contents of the index. If index_write was None, index_binsha will be None
**Note:** The destination of the write functions is up to the user. It could
be a socket, or a file for instance
**Note:** writes only undeltified objectsr Nr r r, )rK zKExpected to write %i objects into pack, but received only %i from iteratorsFro r0 )r r rX rE r$ rI r% r) r r rT r r7 r r& rT compressobjr@ rR rH r] r^ r rv )cls object_iter
pack_write index_write object_countzlib_compressionobjs pack_writerpwriter| r wants_index actual_countobjrO hdrrJ r rL rM rw index_shas rA
write_packzPackEntity.write_pack sI . % kE4=99
)K(( t99L )44 "!- tFH 3X5RT`aabbb r " MME C A LC ,CKBBC
Cjj F3KKK &'788GjG.w|VWWZ[[[KBC>>>>
3 SZc222
3s88b= C|++ ,
< ' ' ]amo{`||~ ~
~
??%?008}}""""
8 s8}} ; Hk::I ""rC c < t j dd| \ }t j dd| \ }fd}fd}| ||||| \ } }
t j t j d} t j || t | dfz } t j || t | dfz }
t j || t j ||
| | S )a Create a new on-disk entity comprised of a properly named pack file and a properly named
and corresponding index file. The pack contains all OStream objects contained in object iter.
:param base_dir: directory which is to contain the files
:return: PackEntity instance initialized with the new pack
**Note:** for more information on the other parameters see the write_pack method r% r c . t j | S rW r% rI )r pack_fds rA rc z#PackEntity.create.. s rx 3 3 rC c . t j | S rW r| )r index_fds rA rc z#PackEntity.create.. s 1 5 5 rC z
pack-%s.%sidx) tempfilemkstemprx r% r r ru r
rename)rj rk base_dirrn ro pack_path
index_pathrl rm pack_binsha index_binshafmt
new_pack_pathnew_index_pathr r} s @@rA createzPackEntity.create s & -b&( C C ' /GX F F *3333
5555 $'NN;
KYegw$x$x! \
Xsj6M6Mv5V/VWW
hz+7N7NPU6V0VWW
)]+++
*n---s=!!!rC )r. )F) r} r~ r r r r( r' r) r( r[ r r r; r r= r r@ rN rP r% r r_ ra r rA r classmethodrT Z_BEST_SPEEDrx r ra rC rA r* r* s 2 2I
!LK< < < & & &2 9 9 9'= '= '= '=X( ( (' ' '0 0 0/ / / 5 5 5 5n3 3 3 2 2 2 ! ! !F [ [ [ =A $t7HY# Y# Y# [Y#v 8r sZ
: 9 ??????? D ! ! ! ! ! !
5
1] 1] 1]l% % % %VD D D D D D D DN~9 ~9 ~9 ~9 ~9I ~9 ~9 ~9HQ@ Q@ Q@ Q@ Q@y Q@ Q@ Q@lw" w" w" w" w" w" w" w" w" w"s A A
A