j f d dl Z d dlZd dlZd dlZd dlmZmZ d dlmZmZm Z m
Z
m Z m Z m
Z
mZmZ d dlmZ d dlmZ d dlmZ d dlmZ d dlmZ d Zed
z Zd Z e Z ej d Z G d
de Z d dZ!d Z"d Z#d Z$d Z%d Z&d dZ'd Z(d Z)d Z*ej+ defd Z,dS ) N)datetime timedelta) ColumnBooleanDateTimeIntegerString
create_engineeventfunctext)declarative_base) Inspector)Session)close_all_sessions)
DatabaseErrorz/var/lve/ssa.dbz.old cleanup_old_datac e Zd ZdZdZ eed Z eedd Z eedd Z
eed Z eed Z ee
d Z ee
d Z eed Z eed Z ee
d Z e ed ej d Z e ed ej ej
Zd S )
RequestResultay
Describes processed request stored in database file.
E.g.
{
"timestamp": "1650008727",
"url": "http://mydomain.com/index.php",
"duration": 162077,
"hitting_limits": false,
"throttled_time": 0,
"io_throttled_time": 0,
"wordpress": true
}
Note: created_at, updated_at is saved in local TZ format
scrape_resultT) primary_keyF)indexnullable)r )timezone)server_defaultr )onupdater N)__name__
__module__ __qualname____doc__
__tablename__r r idr domainpath timestampdurationr is_slow_requesthitting_limitsthrottled_timeio_throttled_time wordpressr r now
created_at
updated_at =/opt/cloudlinux/venv/lib64/python3.11/site-packages/ssa/db.pyr r ' sR $M T * * *B
VF$
7
7
7F 6&u 5 5 5Dw///Ivg...H fWu555OVGe444NVGe444N w777w///I $///
RVWWWJ$///($(**U]UYU]U_U_```JJJr1 r ' c t j t t z
}d} t | 5 }| t
d ||d }|j }||z
}|dk r
ddd n ddd n # 1 swxY w Y nt d| t | dS )a
Removes outdated records from database, saving disk space.
Deletions are batched to avoid loading the entire database into
the cgroup's page cache, which causes OOM on large databases.
Runs VACUUM after deletion to reclaim disk space.
)daysr TzrDELETE FROM scrape_result WHERE rowid IN (SELECT rowid FROM scrape_result WHERE created_at < :cutoff LIMIT :batch))cutoffbatchNz Cleanup deleted %d old records) r todayr RETENTION_TIME_DAYS
session_scopeexecuter
rowcountloggerinfovacuum_database)engine
batch_size
n_days_ago
total_deletedsessionresultdeleteds r2 r r M s5 !!I3F$G$G$GGJM
6
"
" g __T\& & %z::< warningclose Exceptionerrorstr)r@ raw_connoriginal_isolation_levelrL db_dires r2 r? r? d s $$&&H ' 7 #' "" !8!899
9VRW % % b NNFVFFF G G G KKGvGG H H H H NN```` a a a 2333x 4555 1 1 1 /s1vv//000000001 $< $< s0 D-E
F)
F *F?F) F F) ) Gc f t | s!t j | d S d S N)
is_db_presentBasemetadata
create_all)r@ s r2 create_db_if_not_existrb s8 )
((((() )r1 c t j t sdS t j | }d | D }t | dk S )NFc g | ]}|S r0 r0 ).0tables r2
z!is_db_present.. s
G
G
Ge
G
G
Gr1 r )rM r% isfilerP r from_engineget_table_nameslen)r@ database_inspectiontabless r2 r^ r^ s^
7>>& ! ! u#/77
G
G!4!D!D!F!F
G
G
GF v;;?r1 c 0 | d d S )NzPRAGMA journal_mode = WAL)r; ) dbapi_con
con_records r2 setup_wal_moderq s
122222r1 c dt | r dfd}t d|d }t j |dt t | |S )Nzfile:z?mode=roc 0 t j d S )NT)uri)sqlite3connect)connection_strings r2 z!_setup_database.. s go&7TBBB r1 sqlite:////F)creatorechorv )rP r
r listenrq rb )readonlyrz r@ rw s @r2 _setup_databaser~ sz ((( ; 0 : : :BBBBG
wU F
LN3336""" Mr1 Fc t | S r] )r~ )r} s r2 setup_databaser s 8 $ $$r1 c t j t r-t t j t t t }| t dt } t | 5 }t | 5 }| t
t || ddd n # 1 swxY w Y ddd n # 1 swxY w Y n# t $ r Y nw xY wt t dz t dz fD ]5}t j | rt j | 6dS )z
Restore database by establish connections to old and new databases,
merge data to new one if possible and delete old one.
ry Nz-walz-shm)rM r% existsrP r replace
OLD_SSA_DBr disposer
r: queryr first!merge_unharmed_data_from_databaser remove)r@
new_engine
old_engine session_old session_newr% s r2 restore_databaser s
w~~f '
6:&&& !!J
NN 9Z 9 9::J
:
&
& H+}Z7P7P HT_ m , , 2 2 4 4 4 -k; G G G H H H H H H H H H H H H H H H H H H H H H H H H H H H H H H
Vf_fvo>
7>>$ IdOOO sZ D D +=C4( D 4C8 8D ;C8 <D ? D DD DD
D$#D$c d}d} | t | | } | }|sdS |D ]}| | nv# t $ ri t | D ]V} | ||z }|s n&| | G# t $ r Y Sw xY wY nw xY w| ||z
})z_
Scrape all unharmed records from malformed database and merge them into new database.
r
TN)
r r offsetlimitallmerger ranger commit) r r r rA r records_to_saverecord pos_in_batchrecord_to_saves r2 r r sN FJ !!-0077??EEjQQ #iikkO"
)
*
*!!&))))
* %j 1 1
%*\\&<2G%H%H%N%N%P%PN) %%n5555$ D * % sH A: A: :C-,CC-CC-
C'$C-&C''C-,C-c t j t r=t j t rt j t t
| 5 }| t d }| }|d d dk cddd S # 1 swxY w Y dS # t $ r Y dS w xY w)z
Try integrity check of database file to see if it is malformed.
If database unable to execute it, will also count as malformed.
zPRAGMA integrity_checkr okNT) rM r% r r rP r r: r; r
fetchallr )r@ dbrE errorss r2 is_malformed_databaser s
w~~j!! bgnnV&<&<
*
6
"
" (bZZ%= > >??F__&&F!9Q<4' ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( ( tts7 #C 2AC: C C C C C
C"!C"returnc # K t | } |V | n# | xY w | dS # | w xY w)zF
Provide a transactional scope around a series of operations.
)bindN)r r rollbackrT )r@ rD s r2 r: r: s
6"""G
s - A AA A4)r3 )F)-rM
contextlibru loggingr r
sqlalchemyr r r r r r
r r r
sqlalchemy.ext.declarativer sqlalchemy.engine.reflectionr sqlalchemy.ormr sqlalchemy.orm.sessionr sqlalchemy.excr rP r r9 r_ getLoggerr= r r r? rb r^ rq r~ r r r r contextmanagerr: r0 r1 r2 r sN
( ( ( ( ( ( ( ( 8 7 7 7 7 7 2 2 2 2 2 2 " " " " " " 5 5 5 5 5 5 ( ( ( ( ( (
f_
- . .#a #a #a #a #aD #a #a #aL .& & &T) ) )
3 3 3
% % % % 8 6
W r1