OwlCyberSecurity - MANAGER
Edit File: subpostgres.pyc
� �mec @ sg d Z d d l Z d d l Z d d l Z d d l Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z m Z d d l m Z d d l m Z d d l m Z d d l m Z e � Z d Z d e f d � � YZ d e! f d � � YZ" d e f d � � YZ# d e f d � � YZ$ d S( s, Run and manage PostgreSQL as a subprocess. i����N( t quote( t which( t ProcessProtocol( t splitSQLString( t Logger( t CachingFilePath( t LineReceiver( t Deferredt succeed( t DBAPIConnector( t postgres( t InternalDataStoreError( t MultiServices. database system is ready to accept connectionst PostgresMonitorc B sV e Z d Z e � Z d d � Z d � Z e Z d � Z d � Z d � Z d � Z RS( sF A monitoring protocol which watches the postgres subprocess. c C sI t � | _ d | j _ | j | j _ | | _ t | _ t � | _ d S( Ns ( R t lineReceivert delimitert lineReceivedt svct Falset isReadyR t completionDeferred( t selfR ( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt __init__9 s c C s? | j d k r d S| j s; t | k r; | j j � q; n d S( N( R t NoneR t _MAGIC_READY_COOKIEt ready( R t line( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyR A s c C s | j j | � d S( N( R t makeConnection( R ( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt connectionMadeJ s c C s= x6 | j d � D]% } | r | j j d d | �q q Wd S( Ns s {msg}t msg( t splitt logt info( R t outR ( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt outReceivedM s c C sM x6 | j d � D]% } | r | j j d d | �q q W| j j | � d S( Ns s {msg}R ( R R t errorR t dataReceived( R t errR ( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt errReceivedS s c C su | j j d d | j j �| j j | � | j j d k rQ | j j d � n | j j d � | j j | � d S( Ns) pg_ctl process ended with status={status}t statusi s* Could not start postgres; see postgres.log( R R t valueR' R t connectionLostR t callbackR R# t errback( R t reason( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt processEndedY s N( t __name__t __module__t __doc__R R R R R R t disconnectingR R" R& R- ( ( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyR 3 s t ErrorOutputc B s e Z d Z RS( sZ The process produced some error output and exited with a non-zero exit code. ( R. R/ R0 ( ( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyR2 i s t CapturingProcessProtocolc B s; e Z d Z d � Z d � Z d � Z d � Z d � Z RS( s� A L{ProcessProtocol} that captures its output and error. @ivar output: a C{list} of all C{str}s received to stderr. @ivar error: a C{list} of all C{str}s received to stderr. c C s( | | _ | | _ g | _ g | _ d S( s� Initialize a L{CapturingProcessProtocol}. @param deferred: the L{Deferred} to fire when the process is complete. @param inputData: a C{str} to feed to the subprocess's stdin. N( t deferredt inputt outputR# ( R R4 t inputData( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyR y s c C s6 | j d k r2 | j j | j � | j j � n d S( s? The process started; feed its input on stdin. N( R5 R t transportt writet closeStdin( R ( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyR � s c C s | j j | � d S( s5 Some output was received on stdout. N( R6 t append( R t data( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyR" � s c C s | j j | � d S( s5 Some output was received on stderr. N( R6 R; ( R R< ( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyR& � s c C s | j j d j | j � � d S( sI The process is over, fire the Deferred with the output. t N( R4 R* t joinR6 ( R t why( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyR- � s ( R. R/ R0 R R R" R& R- ( ( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyR3 p s t PostgresServicec B s� e Z e d d d d d d g d d d g e d d d d d d d � Z e d � � Z d � Z d � Z d d � Z d d d � Z d � Z d � Z d � Z d � Z e Z d � Z d � Z d � Z RS( t subpostgrest clusters postgres.logR= i i t caldavt pg_ctlt initdbc C s t j | � | | _ | | _ | j j d � | _ | | _ t | _ d | _ | | _ | | _ t j j | � | _ | r� t j j | � | _ n d | _ | s� | j d } t j j | � s� t j | � q� n t | � | _ | | _ | rhd | d k r%| d j d � \ | _ | _ n | d d | _ | _ g | D] } | j d � d ^ qC| _ n! | j j | _ d | _ g | _ | | _ | | _ t | s�| n d d � | _ | s�| n d | _ | | _ | | _! | | _" | | _# | | _$ d | _% g | _& d � } | d | � | _' y t j( | j' � j) | _* Wn d | _* n X| d | � | _+ | | _, d | _- d S( s� Initialize a L{PostgresService} pointed at a data store directory. @param dataStoreDirectory: the directory to @type dataStoreDirectory: L{twext.python.filepath.CachingFilePath} @param subServiceFactory: a 1-arg callable that will be called with a 1-arg callable which returns a DB-API cursor. @type subServiceFactory: C{callable} @param spawnedDBUser: the postgres role @type spawnedDBUser: C{str} t workingR= s /psqlsockst :i i i c S su x t | � D] } | SWd d l } d | } | j j | � rY | j | | j � rY | St d j | | � � � d S( Ni����s /usr/lib/postgresql/12/bin/s Unable to locate {} command: {}( R t ost patht isfilet accesst X_OKR t format( t namet cmdt foundRH t fpath( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt locateCommand s 'RD RE N(. R R t subServiceFactoryt dataStoreDirectoryt childt workingDirt resetSchemaR t delayedShutdownR t shutdownDeferredt databaseNamet clusterNameRH RI t abspatht logFilet logDirectoryt existst makedirsR t socketDirt socketNameR t hostt portt listenAddressest txnTimeoutSecondst testModet maxt sharedBufferst maxConnectionst optionst uidt gidt spawnedDBUsert schemat monitort openConnectionst _pgCtlt statt st_inot _pgCtlInodet _initdbt _reactort _postgresPid( R RT RS Ro RW RZ R[ R] R^ Ra Rb Re Rf Ri Rj Rk Rg Rl Rm Rn t pgCtlt initDBt reactort addrRR ( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyR � s^ ( ", c C s2 | j d k r+ d d l m } | | _ n | j S( Ni����( R{ ( Rw R t twisted.internetR{ ( R R{ ( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyR{ ' s c C s t | _ d S( s Call this when starting database initialization code to protect against shutdown. Sets the delayedShutdown flag to True so that if reactor shutdown commences, the shutdown will be delayed until deactivateDelayedShutdown is called. N( t TrueRX ( R ( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt activateDelayedShutdown. s c C s) t | _ | j r% | j j d � n d S( su Call this when database initialization code has completed so that the reactor can shutdown. N( R RX RY R* R ( R ( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt deactivateDelayedShutdown9 s c C s� | d k r | j } n i | d 6} | j j d � rP d j | j � | d <n? d j | j � | d <| j r� d j | d | j � | d <n | j r� | j | d <n+ | j d k r� t j | j � j | d <n | j | d <t j d | � S( Nt databaset /s unix:{}t endpoints tcp:{}s {}:{}t userRf R ( R RZ Rc t startswithRM Rd Rn Rl t pwdt getpwuidt pw_nameRf R t connectorFor( R RZ t kwargs( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt _connectorForB s s <unlabeled>c C s | j | � j | � S( sK Produce a DB-API 2.0 connection pointed at this database. ( R� t connect( R t labelRZ ( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt produceConnectionX s c C s | j r@ y | j d j | j � � Wq@ t j k r< q@ Xn y | j d j | j � � Wn d } n X| j } | j � | j � | d k r� | j � } | j � } x! t | � D] } | j | � q� W| j � | j � n | j d k r| j | j | � j | � n d S( s� Subprocess is ready. Time to initialize the subservice. If the database has not been created and there is a dump file, then the dump file is imported. s drop database {}s' create database {} with encoding 'UTF8'N( RW t executeRM RZ R t DatabaseErrorR Ro t closeR� t cursorR t commitRY RS t setServiceParent( R t createDatabaseConnt createDatabaseCursort sqlToExecutet connectionR� t statement( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyR ^ s4 c C s d S( s� Pause monitoring. This is a testing hook for when (if) we are continuously monitoring output from the 'postgres' process. N( ( R ( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt pauseMonitor� s c C s d S( sD Unpause monitoring. @see: L{pauseMonitor} N( ( R ( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt unpauseMonitor� s c sz � f d � � t � � } g } | j d j t d j � j � � � � � j rw | j d j t � j j � � � n � j r� | j d j t � j � � � n | j d j � j � � | j d j � j � � | j d � | j d � | j � j � � j r]| j d j t � j � � � | j d � | j d � | j d � | j d � n | j d � � j r�| j d � n � j d d j � j � d d d d j | � g } t j d d | �� j j | � j | d � j d � j j d � j d � j �| � _ � � f d � � � � f d � } � � � f d � } � f d � � � j j j | � j | � d! S(" sC Start the database and initialize the subservice. c s� y � j d d � } Wn, t j k rD } t j d d | �� n X| j � } t j d k ro t | j _ n4 t j d k r� | j d � n t d j t � � � | | f S( Ns schema creationR s: Unable to connect to database for schema creation: {error}R# t pg8000t pgdbR� s Unknown Postgres DBM module: {}( R� R R� R R# R� R. R~ t realConnectiont autocommitR� R RM ( R� t eR� ( R ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt createConnection� s s -c listen_addresses={}t ,s -c unix_socket_directories={}s -c port={}s -c shared_buffers={:d}s -c max_connections={:d}s! -c standard_conforming_strings=ons -c unix_socket_permissions=0770s -c log_directory={}s -c log_truncate_on_rotation=ons! -c log_filename=postgresql_%w.logs -c log_rotation_age=1440s -c logging_collector=ons -c log_line_prefix=%ts -c log_statement=allt starts --log={}s --timeout=86400s -ws -ot s% Requesting postgres start via: {args}t argst envRI Rl Rm c sW t j d | � } | d k r9 t | j d � � � _ n � j � � � � j � d S( s Grab the postgres pid from the pgCtl status call in case we need to kill it directly later on in hardStop(). Useful in conjunction with the DataStoreMonitor so we can shut down if DataRoot has been removed/renamed/unmounted. s PID: (\d+)\Di N( t ret searchR t intt groupRx R R� ( t resultt reResult( R� R ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt gotStatus� s c s� t j d d � j �t � _ t � } t | d � } � j j | � j � j d g d � j d � j j d � j d � j �| j � � S( s� We started postgres; we're responsible for stopping it later. Call pgCtl status to get the pid. s {cmd} exitedRO R' R� RI Rl Rm N( R R Rr R~ t shouldStopDatabaseR R3 R R{ t spawnProcessR� RV RI Rl Rm t addCallback( R� t dt statusMonitor( R� R ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt gotReady s c sw t � } t | d � } � j j | � j � j d g d � j d � j j d � j d � j �| j � � j � � S( s� There was an error trying to start postgres. Try to connect because it might already be running. In this case, we won't be the one to stop it. R' R� RI Rl Rm N( R R3 R R{ R� Rr R� RV RI Rl Rm R� t addErrback( t fR� R� ( t giveUpR� R ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt couldNotStart s c s. t j d d | �� j � � j j � d S( sk We can't start postgres or connect to a running instance. Shut down. s3 Can't start or connect to postgres: {failure.value}t failureN( R t criticalR� R{ t stop( R� ( R ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyR� " s N( R R; RM t shell_quoteR> Re Ra RI Rd Ri Rj t extendRk R^ Rg Rr R] R R R{ R� R� RV Rl Rm Rp R R� R� ( R Rp Rk R� R� R� ( ( R� R� R� R s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt startDatabase� sb c s� t j � � � j � � j j � j � } t j j � } � _ | j d | j d � j d � j � � j r� � j j � s� t j d d � j j j d � �� j j � n � j r� � j r� t j � j j � j � j � n t j � j j d � n � j j � s?t j d d � j j j d � �� j j � n � j j � s�t j d d � j j j d � �� j j � n � j r�� j r�t j � j j � j � j � t j � j j � j � j � n | j � s�t j d d | j j d � �t � } � j j t | d � � j � j d d d � j g d | d � j j d � j d � j �� f d � } | j | � n) t j d d | j j d � �� j � d S( Nt PGDATAt PGHOSTt PGUSERs Creating {dir}t dirs utf-8i� s Running initdb for {dir}s -Et UTF8s -UR� RI Rl Rm c sH | j d � d k r: t j | � t d j | � � � n � j � d S( Ns FATAL:i����s* Unable to initialize postgres database: {}( t findR R# R RM R� ( R� ( R ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt doCreate^ s s Cluster already exists at {dir}(! R t startServiceR RT RU R[ RH t environt copyR� t updateRI Rc Rn Ra t isdirR R t decodet createDirectoryRl Rm t chownt chmodRV R R{ R� R3 R Rv R� R� ( R t clusterDirR� t dbInitedR� ( ( R s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyR� 3 sH """" " c s] � j r2 t � } � _ | j � f d � � n t j � � } � f d � } | j | � S( sU Stop all child services, then stop the subprocess, if it's running. c s t j � � S( N( R t stopService( t ignored( R ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt <lambda>v R= c s1 � j r-y t j � j � j } Wn t k r; d } n X� j | k r� t j d � � j r� t j d � y t j � j t j � Wq� t k r� q� Xn t d � St � } � j d d j � j � g } t j d d | �� j j | � j | d � j d � j j d � j d � j �| j Sn d S( Ni����s Postgres control script mismatchs Sending SIGTERM to PostgresR� s --log={}s$ Requesting postgres stop via: {args}R� R� RI Rl Rm ( R� RH Rs Rr Rt t OSErrorRu R R Rx t killt signalt SIGTERMR R R RM R] R{ R� R� RV RI Rl Rm R ( R� t newInodeRp R� ( R ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt superStoppedz s0 ( RX R RY R� R R� ( R R� R� ( ( R s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyR� m s &c C sD | j d k r@ y t j | j t j � Wq@ t k r< q@ Xn d S( s= Stop postgres quickly by sending it SIGQUIT N( Rx R RH R� R� t SIGQUITR� ( R ( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt hardStop� s N( R. R/ R R R t propertyR{ R R� R� R� R R� R� R� R� R� R� R� ( ( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyR@ � s< r , � : 5(% R0 RH R� R� R� t pipesR R� t twisted.python.procutilsR t twisted.internet.protocolR t twext.enterprise.dal.parseschemaR t twext.python.logR t twext.python.filepathR t twisted.protocols.basicR t twisted.internet.deferR R t txdav.base.datastore.dbapiclientR R t txdav.common.icommondatastoreR t twisted.application.serviceR R R R t ExceptionR2 R3 R@ ( ( ( s3 /opt/cpanel-ccs/txdav/base/datastore/subpostgres.pyt <module> s, 61