OwlCyberSecurity - MANAGER
Edit File: util.pyc
� �t�ac @ s* d Z d d d d d d d g Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z d d l m Z m Z m Z m Z d d l m Z m Z d d l m Z m Z d d l m! Z! d d l" m# Z# m$ Z$ m% Z% d d l& m' Z' d d l( m) Z) d d l( m) Z* d d l+ m, Z, d d l- m. Z. d d l- m/ Z/ d d l0 m1 Z1 d d l2 m3 Z3 d d l4 m5 Z5 d d l6 m7 Z8 d d! l9 m: Z: m; Z; m< Z< m= Z= d d" l> m? Z? d d# l@ mA ZA d d$ lB mC ZC d d% lD mE ZE d d& lF mG ZG d d' lH mI ZI d d( lJ mK ZK d d) lL mM ZM d d* lN mO ZO d d+ lP mQ ZQ d d, lR mS ZS d d- lT mU ZU d d. lV mW ZW d d/ lX mY ZY d d0 lZ m[ Z[ d d1 l\ m] Z] d d2 l^ m_ Z_ d d3 l` ma Za mb Zb mc Zc d d4 ld me Ze d d5 lf mg Zg d d6 lh mi Zi d d7 lj mk Zk d d8 ll mm Zm d d9 ln mo Zo d d: lp mq Zq mr Zr d d; ls mt Zt d d< lu mv Zv d d= lw mx Zx d d> ly mz Zz d d? l{ m| Z| d d@ l} m~ Z d dA l� m� Z� d dB l� m� Z� d dC l� m� Z� d dD l� m� Z� m� Z� d dE l� m� Z� d dF l� m� Z� d dG l� m� Z� d dH l� m� Z� d dI l� m� Z� d dJ l� m� Z� d dK l� m� Z� d dL l� m� Z� d dM l� Z� d dM l� Z� d dM l Z d dM l� Z� d dM l� Z� d dM l� Z� y d dN l� m� Z� e� Wn e� k r>e� Z� n Xe, � Z� e� e� dO � Z� dP eC f dQ � � YZ� dR � Z� dS e� f dT � � YZ� dU � Z� dV � Z� dW � Z� dX � Z� dY e� f dZ � � YZ� e� d[ � Z� d\ � Z� d e� f d] � � YZ� e� d^ � Z� d e1 e� f d_ � � YZ� d` � Z� da e� f db � � YZ� dc � Z� dd � Z� de � Z� df � Z� dg � Z� dh � Z� di � Z� dj � Z� d e� f dk � � YZ� dl eA f dm � � YZ� dn e� f do � � YZ� dp eA f dq � � YZ� dr e� f ds � � YZ� dt eE j� f du � � YZ� dv eE j� f dw � � YZ� dx � Z� dM S(y s> Utilities for assembling the service and resource hierarchy. t FakeRequestt getDBPoolt getRootResourcet getSSLPassphraset MemoryLimitServicet AlertPostert preFlightChecksi����( t path( t DirectoryLogWrapperResource( t RootResource( t APNSubscriptionResource( t NotifierFactory( t getAPNTopicFromConfig( t diagnose( t checkDirectory( t WebAdminResource( t WebCalendarResource( t fromfdt AF_UNIXt SOCK_STREAMt socketpair( t Popent PIPE( t ConnectionPoolt ConnectionPoolConnection( t ConnectionPoolClient( t POSTGRES_DIALECTt ORACLE_DIALECTt DatabaseType( t ChainingOpenSSLContextFactory( t CachingFilePath( t Logger( t HTTPDigestCredentialChecker( t! UsernamePasswordCredentialChecker( t Service( t UnauthorizedLogin( t Portal( t reactor( t inlineCallbackst returnValuet Deferredt succeed( t addSystemEventTrigger( t Factory( t Connection( t amp( t which( t UsageError( t doBind( t CacheStoreNotifierFactory( t ConfigurationError( t ControlAPIResource( t, DirectoryAddressBookHomeProvisioningResource( t) DirectoryCalendarHomeProvisioningResource( t QopDigestCredentialFactory( t& DirectoryPrincipalProvisioningResource( t" DirectoryBackedAddressBookResource( t AuthenticationWrapper( t ServerInfoResource( t SimpleResourcet SimpleRedirectResourcet SimpleUnavailableResource( t config( t TimezoneCache( t TimezoneServiceResource( t TimezoneStdServiceResource( t DBAPIConnector( t PostgresService( t DKIMUtilst DomainKeyResource( t buildServersDB( t IScheduleInboxResource( t ConduitResource( t current_sql_schema( t NotAllowedToUpgrade( t DirectoryService( t CachingDirectoryService( t directoryFromConfig( t BasicCredentialFactory( t TLSCredentialsFactoryt TLSCredentials( t auth( t IPrincipalCredentials( t joinURL( t Headers( t Resource( t File( t readPlist( t quoteN( t NegotiateCredentialFactoryc $ C s� t | j � } t | | t d | j j d | j j d | j j d | j j rW | j n d d | j j d | j j d | j j d | j j d | j j d | j j d | j j d | d | d | j d | j j d | j j �S( s� Construct a L{PostgresService} from a given configuration and subservice. @param config: the configuration to derive postgres configuration parameters from. @param subServiceFactory: A factory for the service to start once the L{PostgresService} has been initialized. @param uid: The user-ID to run the PostgreSQL server as. @param gid: The group-ID to run the PostgreSQL server as. @return: a service which can start postgres. @rtype: L{PostgresService} t databaseNamet clusterNamet logFilet logDirectoryt t socketDirt socketNamet listenAddressest txnTimeoutSecondst sharedBufferst maxConnectionst optionst uidt gidt spawnedDBUsert pgCtlt initDB( R t DatabaseRootRC RI t Postgrest DatabaseNamet ClusterNamet LogFilet LogRotationt LogRoott SocketDirectoryt SocketNamet ListenAddressest TxnTimeoutSecondst SharedBufferst MaxConnectionst Optionst SpawnedDBUsert Ctlt Init( R> t subServiceFactoryRf Rg t dbRoot( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt pgServiceFromConfig� s$ t ConnectionWithPeerc B s e Z e Z d � Z d � Z RS( c C s d | j j � t | � f S( Ns <peer: %r %r>( t sockett filenot id( t self( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt getPeer� s c C s d | j j � t | � f S( Ns <host: %r %r>( R� R� R� ( R� ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt getHost� s ( t __name__t __module__t Truet connectedR� R� ( ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR � s c C s[ t | t t � } t j | � t d | � } t | | � } | j | � | j � | j S( s| Create a transaction factory from an inherited file descriptor, such as one created by L{ConnectionDispenser}. t dbtype( R R R t ost closeR R t makeConnectiont startReadingt newTransaction( t dbampfdR� t sktt protocolt transport( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt transactionFactoryFromFD� s t ConnectionDispenserc B s e Z d Z d � Z d � Z RS( sy A L{ConnectionDispenser} can dispense already-connected file descriptors, for use with subprocess spawning. c C s | | _ d S( N( t pool( R� t connectionPool( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt __init__� s c C sN t t t � \ } } t | j � } t | | � } | j | � | j � | S( sp Dispense a socket object, already connected to a server, for a client in a subprocess. ( R R R R R� R R� R� ( R� t ct sR� R� ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt dispense� s ( R� R� t __doc__R� R� ( ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR� � s c C s2 t | | d � } t | | � } | j | � | S( N( t storeFromConfigt NoneRM t setDirectoryService( R> t txnFactoryt storet directory( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt storeFromConfigWithoutDPS� s c C s� t | | d � } t | j � } | j j rI | j t | j j � � n | j j r� t | d | j j d | j j d | j j �} n | j | � | S( Nt expireSecondst lookupsBetweenPurgest negativeCaching( R� R� t DirectoryProxyClientServicet DirectoryRealmNamet Serverst Enabledt setServersDBRF t MaxClientst DirectoryCachingt CachingSecondsRL t LookupsBetweenPurgest NegativeCachingEnabledR� ( R> R� R� R� ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt storeFromConfigWithDPSClient� s c C s2 t | | d � } t | | � } | j | � | S( N( R� R� RM R� ( R> R� R� R� ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt storeFromConfigWithDPSServer� s c C s� i } | j j r1 t | j | j j � | d <n | j r\ | j j j j r\ t � | d <n | j } | d k rz d } n | d k r~| j s� | j r� d j d | � } n d j d | � } | d } d d l m } | | | | t | j � | | j | j | j d | d | j j d | j j d | j j d | j j d | j j d | j d | j j d | j j d | j j � } n= d d l! m } | t | j" � | | | j | j d | �} x | j# � D] } | | _$ q�W| S( s� Produce an L{IDataStore} from the given configuration, transaction factory, and notifier factory. If the transaction factory is C{None}, we will create a filesystem store. Otherwise, a SQL store, using that connection information. t pusht cachei s0 https://{config.ServerHostName}:{config.SSLPort}R> s1 https://{config.ServerHostName}:{config.HTTPPort}s< /calendars/__uids__/%(home)s/dropbox/%(dropbox_id)s/%(name)si����( t CommonDataStoret quotat logLabelst logStatst logStatsLogFilet logSQLt logTransactionWaitst timeoutTransactionst cacheQueriest cachePoolt cacheExpireSecondsN(% t NotificationsR� R t ServerHostNamet CoalesceSecondst EnableResponseCachet Memcachedt Poolst Defaultt ClientEnabledR1 t UserQuotaR� t EnableSSLt BehindTLSProxyt formatt txdav.common.datastore.sqlR� t FilePatht AttachmentsRoott EnableCalDAVt EnableCardDAVt EnableManagedAttachmentst LogDatabaset LabelsInSQLt Statisticst StatisticsLogFilet SQLStatementst TransactionWaitSecondst TransactionTimeoutSecondst QueryCachingt MemcachedPoolt ExpireSecondst txdav.common.datastore.filet DocumentRoott valuesR� ( R> R� t directoryServicet notifierFactoriesR� t urit attachments_urit CommonSQLDataStoreR� t CommonFileDataStoret notifierFactory( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR� � sN t PrincipalCredentialCheckerc B s e Z e f Z e d � � Z RS( c c sM t | � } | j d k r< t d j d | j j � � � n | j j j � so t d j d | j j � � � n y d d l m } Wn t k r� d } n X| r� t | j | � r� t | j | j f � n{ t | j t � r� t | j | j f � nP | j j j | j � Vr+t | j | j f � n t d j d | j j � � � d S( Ns No such user: {user}t users" User not allowed to log in: {user}i����( t NegotiateCredentialss& Incorrect credentials for user: {user}( RR t authnPrincipalR� R# R� t credentialst usernamet recordt isLoginEnabledt twistedcaldav.authkerbR� t ImportErrort isinstanceR' t authzPrincipalRP t verifyCredentials( R� R� R� ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt requestAvatarId< s8 ( R� R� RR t credentialInterfacesR& R� ( ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR� 9 s c A C s� | d\ k r t d � � n | d\ k r0 g } n t � t } t } t } t } t } t } t } t } t } t } t } t } t } t } | j � } | d | � } g } g } t t j � � } | j t | � � | j t | � � | j t � � | j j d � pd } t j d d | �x�| j j � D]�\ } } | j � } d\ } | d r�t j d d | �| d k r�t s�t j d � q5n y> | d } | s�t d d d | j! � } n t d | � } Wq~t" k r�t j d � q5q~Xq�| d k r#t# | d | d | � } q�| d k r>t$ | � } q�| t% j& k r\t% | � } q�| d k rkq�t j' d d | �n | r5| j( | � | j) d t* � r�| j( | � q�q5q5Wt j d d | j+ �| j, r t j d d | �| | d | � } | | _- n | j. rZt j d d | �| | d | � } | | _/ | j0 j1 r�| j2 r�t j d d | �| d! | f d" | d# t3 d$ | j0 j4 d$ � � } t5 j6 r�| j7 � qWt8 d% d& | j7 � qZt9 j: j; | j+ | j0 j4 � } y' t< | � j= � t j d' d( | �WqZt> t? f k rV} | j@ t@ jA k rWt j' d) d( | d* | �qWqZXn | jB rutC | f � } n d\ } t j d+ d | �| | j+ d! | f �}! |! jD d, | d\ k r�| n | � | j, r�|! jD d- | d\ k r�| n | � n | j. rd|! jD d. | d\ k r| n | � | j0 j1 rd| j2 rd|! jD | j0 j4 | d\ k rW| n | � qdn | jE r�t j d/ � tF d! | f d0 tG d1 tF jH � }" |! jD d2 |" � x� | j, d3 d f | j. d4 d f | jI j1 d5 | jI jJ f | jK jL j1 d6 d7 f f D]� \ }# }$ }% |# r�| jM s$| jN r6d8 } | jO }& n d9 } | jP }& |" jD |$ tQ d! | f d0 t* d1 tF jH d | d: |& d( |% � � q�q�Wn xN| jR D]C}' |' j) d; d\ � }( |' j) d( d\ � }) |( s�|) s�|( d< d$ k r�t j' d= d; |( d( |) �q�n |( d> jS d$ � }* |! }+ xO |* d? D]C }, |+ jT |, � }- |- d\ k r^tU � }- |+ jD |, |- � n |- }+ q!W|+ jT |* d? � d\ k r�t j' d@ d; |( d( |) �q�n tV |) � }. |+ jD |* d? |. � t j dA d; |( d( |) �q�Wt j dB � tW jX � | jY r/t j dC d | �| |! � }/ |! jD dD |/ � n | jI j1 r�t j dE d | �| |! � }0 |! jD dF |0 � t5 j6 r�t5 jZ d< |0 j[ � q�t8 d% d& |0 j[ � n | j\ j1 r6 t j dG d | �| |! | dH tG �}1 |! jD | j\ j] | d\ k r�|1 n | � t j dI d | �| |! | � }2 |! jD | j\ j^ |2 � n | jK jL j1 r( t j dJ d | �| |! | � }1 |! jD d6 | d\ k r� |1 n | � t_ j` | � | jK jL ja j1 r( t j dK dL tb �| jK jL ja jc r� | jK jL ja jc n | j! }3 tb |3 | jK jL ja jd | jK jL ja je � }4 |" jD dM |4 � q( n | jf r� t j dN dL | jf �| | jf d! | f �}5 |! jD dO | d\ k rz |5 n | � n | jg r� t j dP � | | jf |! | | d! | f �}6 |! jD dQ |6 � n | jh r t j dR � | |! | | d! | f �}7 |! jD dS |7 � n | ji jj jk }8 |8 j1 rg t j dT d; |8 dU �| |! | � }9 |! jD |8 dU |9 � n | jl r� t j dV d | �| |! � }: |! jD dW |: � n t j dX � i }; | r� x� | D]� \ }) }< }= }> |! jD |) |< |! | |= � � d$ |) }) g |; |) <xt |> D]l } | d k r7 |; |) j( t$ | � � q | d k r | j jm } |; |) j( t# | d | d | � � q q Wt j dY d( |) d |< dZ |> �q� Wn tn |! | | | t jo f d[ |; �}? tp |? | � }@ |! | _q |@ S(] s� Set up directory service and resource hierarchy based on config. Return root resource. Additional resources can be added to the hierarchy by passing a list of tuples containing: path, resource class, __init__ args list, and optional authentication schemes list ("basic", "digest"). If the store is specified, then it has already been constructed, so use it. Otherwise build one with L{storeFromConfig}. s- Internal error, 'newStore' must be specified.s /principals/s utf-8R^ s- Configuring authentication for realm: {realm}t realmR� s Setting up scheme: {scheme}t schemet kerbeross Kerberos support not availablet ServicePrincipalt serviceTypet HTTPt hostnamet principals Could not start Kerberost digestt Algorithmt Qopt basict wikis Unknown scheme: {scheme}t AllowedOverWireUnencrypteds# Setting up document root at: {root}t roots% Setting up calendar collection: {cls}t clss /calendars/s) Setting up address book collection: {cls}s /addressbooks/s( Setting up directory address book: {cls}t principalCollectionst principalDirectoryR� t /t aftert startups Deleted: {path}R s" Could not delete: {path} : {error}t errors Setting up root resource: {cls}t principalst calendarst addressbookss* Setting up .well-known collection resourcet isdirt defaultACLs .well-knownt caldavt carddavt timezonet ischedules /ischedulet httpst httpt portt urli s' Invalid alias: URL: {url} Path: {path}i i����s6 Invalid alias: URL: {url} Path: {path} already existss Added alias {url} -> {path}s Setting up Timezone Caches, Setting up time zone service resource: {cls}t timezoness5 Setting up standard time zone service resource: {cls}t stdtimezoness2 Setting up iSchedule podding inbox resource: {cls}t poddings* Setting up podding conduit resource: {cls}s* Setting up iSchedule inbox resource: {cls}s$ Setting up domainkey resource: {res}t rest domainkeys& Setting up WebCalendar resource: {res}t webcals Setting up WebAdmin resourcet admins Setting up Control API resourcet controls" Setting up APNS resource at /{url}t SubscriptionURLs& Setting up server-info resource: {cls}s server-infos" Configuring authentication wrappers( Overriding {path} with {cls} ({schemes})t schemest overridesN(r R� t RuntimeErrorR0 R R5 RG RH R@ RA R R R4 R8 R R: R7 R3 R� R$ RQ t DavRealmt registerCheckerR! R R� t realmNamet encodet logt infot Authenticationt iteritemst lowerRY R� t ValueErrorR6 RN RO R� R t appendt gett FalseR� R� t calendarCollectionR� t addressBookCollectiont DirectoryAddressBookR� t EnableSearchAddressBookRS t namet _reactort _startedt provisionDirectoryR* R� R t joinR� t removet OSErrort IOErrort errnot ENOENTt MigrationOnlyR= t putChildt EnableWellKnownR; R� t allReadACLt TimezoneServicet URIt Schedulingt iScheduleR� R� t SSLPortt HTTPPortR< t Aliasest splitt getChildRU t FileResourceR? t createt EnableTimezoneServicet callLatert onStartupR� t InboxNamet ConduitNameRD t validConfigurationt DKIMRE t Domaint KeySelectort PublicKeyFilet WebCalendarRoott EnableWebAdmint EnableControlAPIR� t Servicest APNSt EnableServerInfot DigestR9 t IPrincipalR t rootResource(A R> t newStoret resourcest rootResourceClasst calendarResourceClasst iScheduleResourceClasst conduitResourceClasst timezoneServiceResourceClasst timezoneStdServiceResourceClasst webCalendarResourceClasst webAdminResourceClasst addressBookResourceClasst' directoryBackedAddressBookResourceClasst apnSubscriptionResourceClasst serverInfoResourceClasst principalResourceClasst controlResourceClassR� t principalCollectiont wireEncryptedCredentialFactoriest" wireUnencryptedCredentialFactoriest portalR� R� t schemeConfigt credFactoryR� R1 R2 t$ directoryBackedAddressBookCollectiont directoryPatht et unavailableR t wellKnownResourcet enabledt wellknown_namet redirected_toR t aliasR R t urlbitst parentt urlpiecet childt resourcet timezoneServicet timezoneStdServiceR t conduitt domaint dkt webCalendart webAdmint controlAPIt apnConfigt apnResourcet serverInfoR" R t argsR! t authWrappert logWrapper( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR x s" # " % ". ( ( " - % c C s | j d k r t } d } n t } d } d } | j rc t t | j � t | | | j � � } n� | j su d } n� | j s� | j d k r� t | d � j } n t j | j | j � j } t | d t | | | j � d | j �} | j } n t d � � | | f S( s� Inspect configuration to determine what database connection pool to set up. return: (L{ConnectionPool}, transactionFactory) t oraclet numerict pyformatR^ R� Rd s= trying to use DB in slave, but no connection info from parentN( t DBTypeR R R� t DBAMPFDR� t intR t DBFeaturest UseDatabaset SharedConnectionPoolR~ t produceConnectionRB t connectorFort DatabaseConnectiont connectR t MaxDBConnectionsPerPoolt connectionR/ ( R> t dialectt paramstyleR� R� t connectionFactory( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR s* ' c B sh e Z d d d � Z e d � � Z e d � � Z e d � � Z d � Z d � Z d � Z d � Z RS( R c C s^ | | _ | | _ | | _ | | _ i | _ i | _ t � | _ | d k rZ | | _ n d S( N( R` t methodR R� t _resourcesByURLt _urlsByResourceRT t headersR� t _newStoreTransaction( R� R` R� R R� t transaction( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR� H s c c sG | s t | � n | j | | � V\ } } t | j | | � V� d S( N( R' t locateChildt _getChild( R� R� t segmentsR� t remaining( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR� S s c c s[ | j d � } | j d � } | j | j | � V} | rM | j | | � n t | � d S( NR ( t stripRJ R� R` t _rememberResourceR' ( R� R R� R� ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt locateResource[ s c c s� | d k s | d k r% t d � n | j | � } | j d � sP | d 7} n | t | � } | } | j | | g � V} | r� | j | | � n t | � d S( NR ( R� R' t urlForResourcet endswithRX R� R� ( R� R� t childNamet parentURLR t segmentR� ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt locateChildResourced s c C s | | j | <| | j | <| S( N( R� R� ( R� R� R ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR� r s c C s< | | j k r | j | =n | | j k r8 | j | =n d S( N( R� R� ( R� R� R ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt _forgetResourcew s c C sJ | j j | d � } | d k rF d t f d � � Y} | | � � n | S( Nt NoURLForResourceErrorc B s e Z RS( ( R� R� ( ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR� � s ( R� R/ R� R# ( R� R� R R� ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR� } s c O s d S( N( ( R� R� t kwds( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt addResponseFilter� s N( R� R� R� R� R& R� R� R� R� R� R� R� ( ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR F s c C s) t j | � j � } | r" | j S| j S( s� Return the amount of memory in use for the given process. If residentOnly is True, then RSS is returned; if False, then virtual memory is returned. @param pid: process id @type pid: C{int} @param residentOnly: Whether only resident memory should be included @type residentOnly: C{boolean} @return: Memory used by process in bytes @rtype: C{int} ( t psutilt Processt memory_infot rsst vms( t pidt residentOnlyt memoryInfo( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt memoryForPID� s c B s5 e Z d Z d d � Z d � Z d � Z d � Z RS( s� A service which when paired with a DelayedStartupProcessMonitor will periodically examine the memory usage of the monitored processes and stop any which exceed a configured limit. Memcached processes are ignored. c C sb | | _ | | _ | | _ | | _ d | _ | d k rL d d l m } n | | _ t | _ d S( s� @param processMonitor: the DelayedStartupProcessMonitor @param intervalSeconds: how often to check @type intervalSeconds: C{int} @param limitBytes: any monitored process over this limit is stopped @type limitBytes: C{int} @param residentOnly: whether only resident memory should be included @type residentOnly: C{boolean} @param reactor: for testing i����( R% N( t _processMonitort _secondst _bytest _residentOnlyR� t _delayedCallt twisted.internetR% R6 R� t _memoryForPID( R� t processMonitort intervalSecondst limitBytesR� R% ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR� � s c C s5 t t | � j � | j j | j | j � | _ d S( s4 Start scheduling the memory checks N( t superR t startServiceR6 RO R� t checkMemoryR� ( R� ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR� � s c C sN t t | � j � | j d k rJ | j j � rJ | j j � d | _ n d S( s& Stop checking memory N( R� R t stopServiceR� R� t activet cancel( R� ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR� � s c C s0 z x| j j D]� } | j d � r+ q n | j j j | d � } | d k r | j } | j } y | j | | j � } Wn/ t k r� } t j d d | d | �q n X| | j k rt j d d | d | d | j r� d n d d | �| j j | � qq q WWd | j j | j | j � | _ Xd S( s� Stop any processes monitored by our paired processMonitor whose resident memory exceeds our configured limitBytes. Reschedule intervalSeconds in the future. t memcacheds6 Unable to determine memory usage of PID: {pid} ({err})R� t errs7 Killing large process: {name} PID:{pid} {memtype}:{mem}R5 t memtypet Residentt Virtualt memN( R� t processest startswitht protocolsR/ R� R� R� R� R� t ExceptionR( R R� t warnt stopProcessR6 RO R� R� R� ( R� R5 t protot procR� t memoryRy ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR� � s. N( R� R� R� R� R� R� R� R� ( ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR � s c C s� t | j d d t j d t �| j j | j t j � rl t | j d d t j d d | j | j f �n | j j | j t j � r� t | j d d t j d d | j | j f �n | j j | j t j � rt | j d d t j d d | j | j f �n | j j rOt | j d d t j d | j j | j j | j j f �n t | j d d t j d d | j | j f �t | j d d t j d d | j | j f �d S( sM Make sure that various key directories exist (and create if needed) s Server roott accesst waits Data rootRM i� s Document roots Config roots Socket file roots Log roots Run rooti� N( R t ServerRootR� t W_OKR� t DataRootR� t sept UserNamet GroupNameR� t R_OKt ConfigRoott SocketFilesR� t SocketRoott Permissionst Ownert GroupRq t RunRoot( R> ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt checkDirectories� sR t Stepperc B s> e Z d Z d � Z d � Z d � Z d � Z d d � Z RS( s� Manages the sequential, deferred execution of "steps" which are objects implementing these methods: - stepWithResult(result) @param result: the result returned from the previous step @returns: Deferred - stepWithFailure(failure) @param failure: a Failure encapsulating the exception from the previous step @returns: Failure to continue down the errback chain, or a Deferred returning a non-Failure to switch back to the callback chain "Step" objects are added in order by calling addStep(), and when start() is called, the Stepper will call the stepWithResult() of the first step. If stepWithResult() doesn't raise an Exception, the Stepper will call the next step's stepWithResult(). If a stepWithResult() raises an Exception, the Stepper will call the next step's stepWithFailure() -- if it's implemented -- passing it a Failure object. If the stepWithFailure() decides it can handle the Failure and proceed, it can return a non-Failure which is an indicator to the Stepper to call the next step's stepWithResult(). TODO: Create an IStep interface (?) c C s( g | _ d | _ d | _ t | _ d S( N( t stepsR� t failuret resultR0 t running( R� ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR� F s c C s, | j r t d � � n | j j | � | S( s� Adds a step object to the ordered list of steps @param step: the object to add @type step: an object implementing stepWithResult() @return: the Stepper object itself so addStep() calls can be chained s Can't add step after start( R� R# R� R. ( R� t step( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt addStepL s c C s t | � S( N( R) ( R� R� ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt defaultStepWithResultZ s c C sD | j t t t j j f k r@ t j d d | j j d | �n | S( Ns Step failure: {name}R5 R� ( t typeRJ R2 t OpenSSLt SSLt ErrorR( R� t __class__R� ( R� R� R� ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt defaultStepWithFailure] s "c C s� t | _ t � | _ x� | j D]v } t | d � r@ | j } n | j } t | d � rj | j } d } n | j } | f } | j j | | d | �q W| j j | � | j S( sH Begin executing the added steps in sequence. If a step object does not implement a stepWithResult/stepWithFailure method, a default implementation will be used. @param result: an optional value to pass to the first step @return: the Deferred that will fire when steps are done t stepWithResultt stepWithFailuret errbackArgs( ( R� R� R( t deferredR� t hasattrR R� R R t addCallbackst callback( R� R� R� t callBackt errBackR ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt starte s N( R� R� R� R� R� R� R R� R ( ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR� ) s c C sC t j d � t j | � t d t j g d t d t � j � d S( s� Log the shutdown reason and call the shutdown-requesting program. In the case the service is spawned by launchd (or equivalent), if our service decides it needs to shut itself down, because of a misconfiguration, for example, we can't just exit. We may need to go through the system machinery to unload our job, manage reverse proxies, update admin UI, etc. Therefore you can configure the ServiceDisablingProgram plist key to point to a program to run which will stop our service. @param programPath: the full path to a program to call (with no args) @type programPath: C{str} @param reason: a shutdown reason to log @type reason: C{str} s* Shutting down Calendar and Contacts serverR� t stdoutt stderrN( R( R R R> t ServiceDisablingProgramR t communicate( t programPatht reason( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt requestShutdown� s c C s� t | � \ } } | r- t | � \ } } n | rH t | � \ } } n | rc t | � \ } } n | s� | j r� t j j | j � r� t d d t | j | � n t S| GHt j d � n t S( s� Perform checks prior to spawning any processes. Returns True if the checks are ok, False if they don't and we have a ServiceDisablingProgram configured. Otherwise exits. R R i ( t verifyConfigt verifyServerRoott verifyTLSCertificatet verifyAPNSCertificateR R� R t existsR* R R0 t syst exitR� ( R> t successR ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR � s" c C s& | j s | j r t d f St d f S( sD At least one of EnableCalDAV or EnableCardDAV must be True s A protocol is enableds& Neither CalDAV nor CardDAV are enabled( R� R� R� R0 ( R> ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR � s c C sN t j | j � } | t j k r+ t d f S| t j k rD t d f St d f S( s7 Ensure server root is not on a phantom volume s ServerRoot is missings> ServerRoot is supposed to be on a non-boot-volume but it's nots ServerRoot is ok( R t detectPhantomVolumeR� t EXIT_CODE_SERVER_ROOT_MISSINGR0 t EXIT_CODE_PHANTOM_DATA_VOLUMER� ( R> R� ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR � s c C s� t t d � rw | j rj d j | j � } t j j | j � } | rt d j d | d | � } t | f Sq&t d f Sn� | j r t j j | j � s� d j d | j � } t j d d d | j g � t | f St j | j � j } | d k rd j d | j � } t | f S| j } n t d f SyS t | j | j d | j d t d | j d t t j | j � d | j j � �Wnh t k r�} t t d � r�d j d | d t | � � } n d j d | d t | � � } t | f SXt d f S( si If a TLS certificate is configured, make sure it exists, is non empty, and that it's valid. t __SecureTransport__s Keychain: {}sF The configured TLS Keychain Identity ({cert}) cannot be used: {reason}t certR s TLS disableds2 The configured TLS certificate ({cert}) is missingt MissingCertificateAlerti R s0 The configured TLS certificate ({cert}) is emptyt certificateChainFilet passwdCallbackt keychainIdentityt sslmethodt cipherss@ The configured TLS certificate ({cert}) cannot be used: {reason}s TLS enabled( R R� t SSLKeychainIdentityR� t cryptot check_keychain_identityR0 R� t SSLCertificateR� R R R t postAlertt statt st_sizeR t SSLPrivateKeyt SSLAuthorityChainR t getattrR� t SSLMethodt SSLCiphersR� R� t str( R> t certificate_titleR t messaget lengthRy ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR � sT c s� | j j j j r�x�d d f D]�\ } } | j j j | � � j sM q n t j � j � su t d j � j � f St t d � s� t � j � s� t d j � j � f Sn y t | | � � Wn3 t k r� } t j d d g � t t | � f SXyP � j r� f d � } n d } t � j � j d | d t t j d � �Wq t k r�} t t d � r�d j d | d � j d t | � � } n' d j d | d � j d t | � � } t j d d g � t | f SXq Wt d f St d f Sd S( sG If APNS certificates are configured, make sure they're valid. t CalDAVs apns:com.apple.calendart CardDAVs apns:com.apple.contacts" APNS certificate does not exist {}R s APNS certificate expired {}t PushNotificationCertificateAlerti c s � j S( N( t Passphrase( t ignored( t protoConfig( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt <lambda>O R^ R# R% t TLSv1_METHODsD The {proto} APNS Keychain Identity ({cert}) cannot be used: {reason}R� R R s> The {proto} APNS certificate ({cert}) cannot be used: {reason}s APNS enableds APNS disabledN( R7 s apns:com.apple.calendar( R8 s apns:com.apple.contact( R� R[ R\ R� R R t CertificatePathR0 R� R R� t checkCertExpirationR R- R R+ R3 R: R� R t PrivateKeyPathR0 R� R� t KeychainIdentityR� ( R> R� t accountNameRy R# R5 ( ( R< s* /opt/cpanel-ccs/calendarserver/tap/util.pyR / sP c C s� yg t d � d } | d d d d d | g } t d | d t d t � } | j � | j } | d k SWn t k r{ t SXd S( s See if the given certificate is expired. @param certPath: the path of the certificate @type certPath: C{str} @return: True if the cert has not expired (or we can't check because we can't find the openssl command line utility); False otherwise t openssli t x509s -checkendt 0s -noouts -inR� R R N( R. R R R t returncodet IndexErrorR� ( t certPatht opensslToolR� R� t exitStatus( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR@ w s c G s� t j s d St j r� t j j t j � r� t d d t j d t j g d t d t � } | j � \ } } | j r� t j d d t j d | �q� t j d d t j �| j � Sn t j r�t j j t j � r�t t j � �P } d } x@ | j � D]2 } d | k rd } Pq� d | k r� d } Pq� q� WWd QX| d k rYt j d d t j �q�t d t j d j t j t j � | g d t d t � } | j � \ } } | j r�t j d d t j d | �q�| j � Sn d S( NR� t sudos --get-private-key-passphraseR R s+ Could not get passphrase for {key}: {error}t keyR s Obtained passphrase for {key}s -----BEGIN RSA PRIVATE KEY-----t RSAs -----BEGIN DSA PRIVATE KEY-----t DSAs( Could not get private key type for {key}s {}:{}( R> R. R� t SSLCertAdminR� R t isfileR R R RG R( R R) R� t SSLPassPhraseDialogt opent readlinesR� R� RG ( R; R� t outputR t sslPrivKeyt keyTypet line( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR � sT c B s� e Z d Z d Z d Z e d � � Z e d � � Z e d � � Z d d � Z e d � � Z e d d d � � Z e d d d � � Z RS( s Encapsulates the posting of server alerts via a singleton which can be configured differently depending on whether you want to directly spawn the external alert program from this process, or send an AMP request to another process to spawn. Workers should call setupForWorker( ), and then calls to postAlert( ) will send an AMP message to the master. The master should call setupForMaster( ), and then calls to postAlert( ) within the master will spawn the external alert program. t alertc C s | � | _ t | � d S( N( t _alertPostert AMPAlertReceiver( R t controlSocket( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt setupForMaster� s c C s | | � | _ d S( N( RZ ( R R\ ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt setupForWorker� s c C s% | j d k r | � | _ n | j S( N( RZ R� ( R ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt _getAlertPoster� s c C s+ | d k r d | _ n t | � | _ d S( N( R� t sendert AMPAlertSender( R� R\ ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR� � s c C s | j � } t j s d St j j t j � s2 d S| rZ | j | � } | | k rZ d Sn | j d k r� | j | � y< t j | g } | j | � t | d t d t �j � Wq� t k r� } t j d d | d | d | �q� Xn | j j | | � d S( NR R s2 Could not post alert: {alertType} {args} ({error})t alertTypeR� R ( R_ R> t AlertPostingProgramR� R R t secondsSinceLastPostR` R� t recordTimeStampt extendR R R R� R( R t sendAlert( R Rb t ignoreWithinSecondsR� t postert secondst commandLineRy ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR+ � s. c C s� | d k r t j } n | d k r9 t t j � � } n t | � } | j d j | � � } | j � sr d } nG | j � �9 } y" | j � j � } t | � } Wn d } n XWd QX| | S( Ns .{}.timestampi ( R� R> R� R� t timeR� R� R� R RS t readlineR� ( R Rb t timestampsDirectoryt nowt dirFPt childFPt timestampR� RX ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyRd ! s c C st | d k r t j } n | d k r9 t t j � � } n t | � } | j d j | � � } | j t | � � d S( Ns .{}.timestamp( R� R> R� R� Rl R� R� R� t setContentR3 ( R Rb Rn Ro Rp Rq ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyRe 5 s N( R� R� R� t ALERT_ROUTER� RZ t classmethodR] R^ R_ R� R+ Rd Re ( ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR � s 't AMPAlertSendingFactoryc B s e Z d � Z d � Z RS( c C s | | _ d S( N( R` ( R� R` ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR� C s c C s t j � } | | j _ | S( N( R- t AMPR` R� ( R� t addrR� ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt buildProtocolF s ( R� R� R� Ry ( ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyRv A s Ra c B s& e Z d Z d d d � Z d � Z RS( sA Runs in the workers, sends alerts to the master via AMP c C s5 | | _ | d k r1 | j t j t | � � n d S( N( R� R� t addFactoryR Rt Rv ( R� R\ R� ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR� Q s c C s | j j t d | d | �S( NRb R� ( R� t callRemotet PostAlert( R� Rb R� ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyRg V s N( R� R� R� R� R� Rg ( ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyRa L s t AMPAlertReceiverFactoryc B s e Z d � Z RS( c C s t � S( N( t AMPAlertProtocol( R� Rx ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyRy ^ s ( R� R� Ry ( ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR} \ s R[ c B s e Z d Z d � Z RS( s^ Runs in the master, receives alerts from workers, executes the alert posting program c C s) | d k r% | j t j t � � n d S( N( R� Rz R Rt R} ( R� R\ ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR� g s ( R� R� R� R� ( ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR[ b s R| c B sJ e Z d e j � f d e j e j � � f g Z d e j � f g Z RS( Rb R� t status( R� R� R- t Stringt ListOft argumentst response( ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR| o s R~ c B s e Z d Z e j d � � Z RS( sK Defines the AMP protocol for sending alerts from worker to master c C s t j | d | � i d d 6S( s7 The "PostAlert" handler in the master i t OKR ( R R+ ( R� Rb R� ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR+ ~ s ( R� R� R� R| t responderR+ ( ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyR~ y s c C s^ d } d } t j j | � r9 t | � j d | � } n t | t � rZ | j d � } n | S( s� Return the ServerRoot value from the OS X preferences plist. If plist not present, return empty string. @rtype: C{unicode} s* /Library/Server/Preferences/Calendar.plistu R� s utf-8( R� R R RW R/ R� R3 t decode( t defaultPlistPatht serverRoot( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt serverRootLocation� s (� R� t __all__R� R t calendarserver.accesslogR t calendarserver.provision.rootR t calendarserver.push.applepushR t calendarserver.push.notifierR t calendarserver.push.utilR t calendarserver.toolsR t calendarserver.tools.utilR t" calendarserver.webadmin.delegationR t calendarserver.webcal.resourceR R� R R R R t subprocessR R t twext.enterprise.adbapi2R R R t twext.enterprise.ienterpriseR R R t twext.internet.sslR t twext.python.filepathR R� t twext.python.logR t twext.who.checkerR R! t twisted.application.serviceR" t twisted.cred.errorR# t twisted.cred.portalR$ R� R% R6 t twisted.internet.deferR& R' R( R) t twisted.internet.reactorR* t twisted.internet.protocolR+ t twisted.internet.tcpR, t twisted.protocolsR- t twisted.python.procutilsR. t twisted.python.usageR/ t twistedcaldav.bindR0 t twistedcaldav.cacheR1 t twistedcaldav.configR2 t twistedcaldav.controlapiR3 t# twistedcaldav.directory.addressbookR4 t twistedcaldav.directory.calendarR5 t twistedcaldav.directory.digestR6 t! twistedcaldav.directory.principalR7 t( twistedcaldav.directorybackedaddressbookR8 t twistedcaldav.resourceR9 t twistedcaldav.serverinfoR: t twistedcaldav.simpleresourceR; R< R= t twistedcaldav.stdconfigR> t twistedcaldav.timezonesR? t twistedcaldav.timezoneserviceR@ t twistedcaldav.timezonestdserviceRA t txdav.base.datastore.dbapiclientRB t txdav.base.datastore.subpostgresRC t0 txdav.caldav.datastore.scheduling.ischedule.dkimRD RE t8 txdav.caldav.datastore.scheduling.ischedule.localserversRF t4 txdav.caldav.datastore.scheduling.ischedule.resourceRG t' txdav.common.datastore.podding.resourceRH R� RI t* txdav.common.datastore.upgrade.sql.upgradeRJ t txdav.dps.clientRK R� t txdav.who.cacheRL t txdav.who.utilRM t txweb2.auth.basicRN t txweb2.auth.tlsRO RP t txweb2.davRQ t txweb2.dav.authRR t txweb2.dav.utilRS t txweb2.http_headersRT t txweb2.resourceRU t txweb2.staticRV RL t plistlibRW t urllibRX R� R= R� R Rl R� RY R� R� R( R~ R R� t objectR� R� R� R� R� R� R R R R� R� R R� R� R R R R R R R@ R R Rv Ra R} R[ t CommandR| Rw R~ R� ( ( ( s* /opt/cpanel-ccs/calendarserver/tap/util.pyt <module> s� " " ) >? � � 'C N C` ( K H Gm