Add docs for Ceph RGWs with Cephadm

doc/source/configuration/cephadm.rst

@@ -308,6 +308,128 @@ should be used in the Kolla Manila configuration e.g.:
 
   manila_cephfs_filesystem_name: manila-cephfs
 
+RADOS Gateways
+--------------
+
+RGWs are defined with the following:
+
+.. code:: yaml
+
+  cephadm_radosgw_services:
+ - id: myrgw
+   count_per_host: 1
+   spec:
+     rgw_frontend_port: 80
+
+Ceph RGWs require additional configuration for the following features:
+
+  * Support both S3 and Swift APIs.
+
+  * Authenticate user access via Keystone.
+
+  * Allow cross-project and public object access.
+
+The set of commands below configure all of these.
+
+.. code:: yaml
+
+  # Append the following to cephadm_commands_post:
+  - "config set client.rgw rgw_content_length_compat true"
+  - "config set client.rgw rgw_enable_apis 's3, swift, swift_auth, admin'"
+  - "config set client.rgw rgw_enforce_swift_acls true"
+  - "config set client.rgw rgw_keystone_accepted_admin_roles 'admin'"
+  - "config set client.rgw rgw_keystone_accepted_roles 'member, Member, _member_, admin'"
+  - "config set client.rgw rgw_keystone_admin_domain Default"
+  - "config set client.rgw rgw_keystone_admin_password {{ secrets_ceph_rgw_keystone_password }}"
+  - "config set client.rgw rgw_keystone_admin_project service"
+  - "config set client.rgw rgw_keystone_admin_user 'ceph_rgw'"
+  - "config set client.rgw rgw_keystone_api_version '3'"
+  - "config set client.rgw rgw_keystone_token_cache_size '10000'"
+  - "config set client.rgw rgw_keystone_url https://{{ internal_net_vip_address }}:5000"
+  - "config set client.rgw rgw_keystone_verify_ssl false"
+  - "config set client.rgw rgw_max_attr_name_len '1000'"
+  - "config set client.rgw rgw_max_attr_size '1000'"
+  - "config set client.rgw rgw_max_attrs_num_in_req '1000'"
+  - "config set client.rgw rgw_s3_auth_use_keystone true"
+  - "config set client.rgw rgw_swift_account_in_url true"
+  - "config set client.rgw rgw_swift_versioning_enabled true"
+
+As we have configured Ceph to respond to Swift APIs, you will need to tell
+Kolla to account for this when registering Swift endpoints with Keystone. Also,
+when ``rgw_swift_account_in_url`` is set, the equivalent Kolla variable should
+be set in Kolla ``globals.yml`` too:
+
+.. code:: yaml
+
+  ceph_rgw_swift_compatibility: false
+  ceph_rgw_swift_account_in_url: true
+
+``secrets_ceph_rgw_keystone_password`` should be stored in the Kayobe
+``secrets.yml``, and set to the same value as ``ceph_rgw_keystone_password`` in
+the Kolla ``passwords.yml``. As such, you will need to configure Keystone
+before deploying the RADOS gateways. If you are using the Kolla load balancer
+(see :ref:`RGWs-with-hyper-converged-Ceph` for more info), you can specify the
+``haproxy`` and ``loadbalancer`` tags here too.
+
+.. code:: yaml
+
+  kayobe overcloud service deploy -kt ceph-rgw,keystone,haproxy,loadbalancer
+
+
+.. _RGWs-with-hyper-converged-Ceph:
+
+RGWs with hyper-converged Ceph
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you are using a hyper-converged Ceph setup (i.e. your OpenStack controllers
+and Ceph storage nodes share the same hosts), you will need to change the
+``rgw_frontend_port``. This is because port 80 (and 443) will be bound to the
+Kolla-deployed haproxy. You should choose a custom port that does not conflict
+with any OpenStack endpoints (``openstack endpoint list``), such as port 8100.
+
+You may also want to use the Kolla-deployed haproxy to load balance your RGWs.
+This means you will not need to define any Ceph ingress services. Instead, you
+add definitions of your Ceph hosts to Kolla ``globals.yml``:
+
+.. code:: yaml
+
+  ceph_rgw_hosts:
+  - host: controller1
+    ip: <host IP on storage net>
+    port: 8100
+  - host: controller2
+    ip: <host IP on storage net>
+    port: 8100
+  - host: controller3
+    ip: <host IP on storage net>
+    port: 8100
+
+HA with Ingress services
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Ingress services are defined with the following. ``id`` should match name (not
+id) of the RGW service to which ingress will point to. ``spec`` is a service
+specification required by Cephadm to deploy the ingress (haproxy + keepalived
+pair).
+
+.. code:: yaml
+
+  cephadm_ingress_services:
+      - id: rgw.myrgw
+        spec:
+          frontend_port: 443
+          monitor_port: 1967
+          virtual_ip: 10.66.0.1/24
+          ssl_cert: {example_certificate_chain}
+
+When using ingress services, you will need to stop Kolla from configuring your
+RGWs to use the Kolla-deployed haproxy. Set the following in Kolla
+``globals.yml``:
+
+.. code:: yaml
+
+  enable_ceph_rgw_loadbalancer: false
+
 Deployment
 ==========
 
@@ -345,8 +467,14 @@ cephadm.yml playbook to perform post-deployment configuration:
 
    kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/cephadm.yml
 
-The ``cephadm.yml`` playbook imports various other playbooks, which may
-also be run individually to perform specific tasks.
+The ``cephadm.yml`` playbook imports various other playbooks, which may also be
+run individually to perform specific tasks. Note that if you want to deploy
+additional services (such as RGWs or ingress) after an initial deployment, you
+will need to set ``cephadm_bootstrap`` to true. For example:
+
+.. code:: bash
+
+   kayobe playbook run $KAYOBE_CONFIG_PATH/ansible/cephadm.yml -e cephadm_bootstrap=true
 
 Configuration generation
 ------------------------