OwlCyberSecurity - MANAGER

Edit File: MultiServerDeployment.md

---
title: 'Multi-server Deployment'
...

Calendar Server version 3 and later uses a database as the primary data
store, instead of the filesystem store used by previous versions. This
allows for a familiar profile of scalability to multiple servers by
offloading the DB to a dedicated server or cluster, and then adding
front-end Calendar Server hosts as needed. This document highlights the
key elements of a multi-server Calendar Server deployment.

-   Database Connectivity\_: By default, Calendar Server assumes the DB
    is hosted locally on a unix domain socket, so you must add
    configuration to connect to an external DB service over the network.
-   Database Setup and Schema Management\_: When connecting to an
    external DB, the administrator is responsible for applying our
    database schema to initialize the database for use by Calendar
    Server, using the calendarserver\_bootstrap\_database tool.
-   Memcached\_: All Calendar Server hosts need to share access to
    memcached
-   Proxy Database\_: In previous versions, the proxy (delegate)
    database was a separate SQLite or Postgres container, however as of
    6.0 this data is stored in the primary database along with other
    user data, so no special steps are required here.
-   Directory Services\_: All servers should have access to the same
    directory services data that defines users, groups, resources, and
    locations. Calendar Server provides a highly flexible LDAP client to
    leverage existing directory servers, or you can use local XML files.
-   Client Connectivity\_: Use either a load balancer or round robin
    dns, and configure all servers to use the same ServerHostName in
    caldavd.plist
-   Shared Storage for Attachments\_: AttachmentsRoot should point to
    storage shared across all servers, e.g. an NFS mount. Used for file
    attachments to calendar events.
-   General Advice\_: *No one wants advice - only corroboration.* --John
    Steinbeck

Database Connectivity
=====================

First, make sure your Postgres service is connectable over the network ;
by default it may use only unix domain sockets. Accepting connections
over TCP typically involves changes to
[listen\_address](http://www.postgresql.org/docs/current/static/runtime-config-connection.html#GUC-LISTEN-ADDRESSES)
in the main server config file to bind the network socket, and also
edits to
[pg\_hga.conf](http://www.postgresql.org/docs/current/static/auth-pg-hba-conf.html)
to allow connections by IP or netmask. Additional discussion of database
server setup and tuning is beyond the scope of this document.

There are a few configuration parameters in caldavd.plist that control
Calendar Server's behavior with respect to database use. The relevant
caldavd.plist entries and their default values are shown and described
below (as defined in
[stdconfig.py](https://github.com/apple/ccs-calendarserver/blob/master/twistedcaldav/stdconfig.py))

"UseDatabase"  : True, # True: database; False: files

"DBType"       : "",   # 2 possible values: empty, meaning 'spawn postgres
                           # yourself', or 'postgres', meaning 'connect to a
                           # postgres database as specified by the 'DSN'
                           # configuration key.  Will support more values in
                           # the future.

"DSN"          : "",   # Data Source Name.  Used to connect to an external
                           # database if DBType is non-empty.  Format varies
                           # depending on database type.

All of the above are top-level keys in caldavd.plist.

The DSN is a colon separated string defined in PyGreSQL-4.0/pgdb.py and
has the following structure:

dbhost = params[0]
    dbbase = params[1]
    dbuser = params[2]
    dbpasswd = params[3]
    dbopt = params[4]
    dbtty = params[5]

When no hostname is specified, Calendar Server assumes the use of a
local unix domain socket (found in the directory defined by the RunRoot
config key)

Example of a 'remote postgres via TCP' configuration:

<key>DBType</key>
    <string>postgres</string>
    <key>DSN</key>
    <string>hostname:dbname:dbuser:dbpass::</string>

Database Setup and Schema Management
====================================

Whenever DBType is set, Calendar Server is not responsible for the
lifecycle of the database, nor is it responsible for the setup and
schema population - these tasks are now the responsibility of the
administrator. Starting and stopping the database should be accomplished
using native tools (e.g. pg\_ctl). The database should be started before
Calendar Server, and stopped after Calendar Server.

It is critically important that your database server keeps updated
statistics about your database, which allows the database query planner
to select appropriate performance optimizations. Refer to your database
server documentation for details.

Memcached
=========

The default memcached settings are found in
[stdconfig.py](https://github.com/apple/ccs-calendarserver/blob/master/twistedcaldav/stdconfig.py).
By default there is one memcached 'pool' that is automatically managed
by Calendar Server, and memcache is started and stopped along with
Calendar Server. For a multi-server deployment, you should manage the
memcached lifecycle externally, to make it independent of your Calendar
Server hosts. Also, all Calendar Server hosts must be configured to
share this memcached instance. A sample configuration is shown below,
which instructs Calendar Server to connect to the 'Default' pool at
example.com port 11211.

<key>Memcached</key>
    <dict>
      <key>MaxClients</key>
      <integer>5</integer>
      <key>Options</key>
      <array>
        <string>-U</string>
        <string>0</string>
        <string>-m</string>
        <string>6000</string>
      </array>
      <key>Pools</key>
      <dict>
        <key>Default</key>
        <dict>
          <key>ClientEnabled</key>
          <true/>
          <key>ServerEnabled</key>
          <false/>
          <key>BindAddress</key>
          <string>EXAMPLE.COM</string>
          <key>Port</key>
          <integer>11211</integer>
          <key>HandleCacheTypes</key>
          <array>
            <string>Default</string>
          </array>
        </dict>
      </dict>
      <key>memcached</key>
      <string>memcached</string>
    </dict>

In this configuration, the administrator is expected to ensure that
there is a memcached instance running on host EXAMPLE.COM listening on
port 11211 (Default). All calendar servers need to have the same
memcache configuration. Memcache should start first and stop last,
relative to calendar server and postgres. Note also that memcached
should be as close to the Calendar Server hosts as possible to minimize
network latency.

Proxy Database
==============

As of version 6.0, the proxy data is stored in the main database, and no
special steps are required to configure a multi-server deployment to
share proxy data. The rest of the information in this section applies
only to server versions before 6.0.

The Proxy DB (for delegation) is typically stored on disk in an sqlite
DB, which does not allow for concurrent access across multiple hosts. To
address this, create an additional DB in the postgres server, then edit
caldavd.plist to add something like the following, and disable any other
ProxyDB configuration.

<key>ProxyDBService</key>
    <dict>
      <key>type</key>
      <string>twistedcaldav.directory.calendaruserproxy.ProxyPostgreSQLDB</string>
      <key>params</key>
      <dict>
        <key>dbtype</key>
        <string>ProxyDB</string>
        <key>host</key>
        <string>PARADISE-FALLS</string>
        <key>database</key>
        <string>FOSSILS</string>
        <key>user</key>
        <string>MUNTZ</string>
      </dict>
    </dict>

As with the memcache config, all calendar servers should have the same
ProxyDBService config. In the shown example, the server will expect to
access a database called FOSSILS as user MUNTZ on the postgres server
PARADISE-FALLS. Unlike with the primary calendar data store, calendar
server is prepared to initialize the schema of this database at runtime
if it does not exist - so nothing is required beyond creating the empty
db, creating the db user with appropriate access, and applying some
caldavd.plist configuration.

Directory Services
==================

It is critical that all servers use the same directory services data
that defines users (and their passwords), groups, resources, and
locations used by Calendar Server. By default, this data is stored in
local XML files, which is not ideally suited for a multi-server
deployment, although would still work fine if the administrator is
willing to manage the workflow around updating and distributing those
files to all servers.

In addition, Calendar Server provides a very configurable LDAP client
interface for accessing external directory services data. Administrators
familiar with LDAP should need little more than to look at
[twistedcaldav/stdconfig.py](https://github.com/apple/ccs-calendarserver/blob/master/twistedcaldav/stdconfig.py)
for the available options to get started. Calendar Server will perform
standard LDAP bind authentication to authenticate clients.

Open Directory is also available when running on Mac OS X or Mac OS X
Server.

Client Connectivity
===================

Use either a load balancer or round robin dns, and configure all servers
to use the same ServerHostName in caldavd.plist. A load balancer
provides the most optimal distribution of work across available servers,
and greater resiliency incase of individual server failure. Round robin
DNS is simpler, and should work fine - however be aware that DNS caches
may result in a given client 'sticking' to a server for a while. Using
the same ServerHostName everywhere allows for all servers to have the
exact same caldavd.plist, which is strongly recommended for simplicity.

Shared Storage for Attachments
==============================

Set the caldavd.plist key AttachmentsRoot to a filesystem directory that
is shared and writable by all Calendar Server machines, for example an
NFS export. This will be used to store file attachements that users may
attach to calendar events.

General Advice
==============

-   Ensure caldavd.plist is identical on all Calendar Server hosts. This
    is not strictly required, but recommended to keep things as
    predictable as possible. Since you already have shared storage for
    AttachmentsRoot, use that to host the 'conf' directory for all
    servers as well; this way you don't need to push config changes out
    to the servers.
-   Use the various [tools and
    utilities](https://github.com/apple/ccs-calendarserver/tree/master/calendarserver/tools)
    to monitor activity in real time, and also for post-processing
    access logs.
-   Be sure you are getting the most from an individual server before
    you decide you need additional hosts (other than for redundancy). To
    optimize the single-server configuration, play with the
    caldavd.plist keys MultiProcessCount (\# of daemons spawned), and
    MaxRequests (\# of requests a daemon will process concurrently). If
    your Calendar Server isn't above 80% CPU use for sustained periods,
    you most likely don't need more Calendar Server hosts.
-   Ensure that your database's table statistics are updated at a
    reasonable interval. "Reasonable" depends entirely on how quickly
    your data changes in shape and size. In particular, be sure to
    update stats after any bulk changes.
-   Tune the database for performance, using the methodologies
    appropriate for the database you are using. The DB server will need
    to accept up to MultiProcessCount \* MaxRequests connections from
    each Calendar Server, unless MaxDBConnectionsPerPool is set, in
    which case the number is MultiProcessCount \*
    MaxDBConnectionsPerPool per server, plus a handful more for other
    things like the notification sidecar or command line tools.
-   Test Scenario: With a well tuned multi-server deployment of
    identically configured caldav servers behind a load balancer, and a
    separate Postgres server with a fast RAID 0, in a low-latency lab
    environment using simulated iCal client load, it takes 5 or 6 caldav
    servers to saturate the postgres server (which becomes i/o bound at
    a load of about 55,000 simulated users in this test).
-   To eliminate all single points of failure, implement
    high-availability for memcache, the database, the directory service,
    the shared storage for AttachmentsRoot, and the network load
    balancer(s).
-   When using an external directory service such as LDAP or Open
    Directory, overall Calendar Server performance is highly dependent
    on the responsiveness of the directory service.