feat: server-side data system (#304)

This feature branch contains the server-side Data System implementation.

This system allows the server-side SDK to operate within two paradigms:
that of a background connection that streams data from LaunchDarkly, or
a connection to a database mediated by an in-memory caching algorithm.

---------

Co-authored-by: Matthew M. Keeler <[email protected]>
Co-authored-by: Molly <[email protected]>
Co-authored-by: Ryan Lamb <[email protected]>

Author: 4 people

.github/ISSUE_TEMPLATE/server-sdk-redis-source--bug-report.md

@@ -0,0 +1,50 @@
+---
+name: 'C++ Server SDK / Redis Source Bug Report'
+about: Create a report to help us improve
+title: ''
+labels: 'package: sdk/server-redis, bug'
+assignees: ''
+
+---
+
+**Is this a support request?**
+This issue tracker is maintained by LaunchDarkly SDK developers and is intended for feedback on the code in this
+library. If you're not sure whether the problem you are having is specifically related to this library, or to the
+LaunchDarkly service overall, it may be more appropriate to contact the LaunchDarkly support team; they can help to
+investigate the problem and will consult the SDK team if necessary. You can submit a support request by
+going [here](https://support.launchdarkly.com/) and clicking "submit a request", or by emailing
+[email protected].
+
+Note that issues filed on this issue tracker are publicly accessible. Do not provide any private account information on
+your issues. If your problem is specific to your account, you should submit a support request as described above.
+
+**Describe the bug**
+A clear and concise description of what the bug is.
+
+**To reproduce**
+Steps to reproduce the behavior.
+
+**Expected behavior**
+A clear and concise description of what you expected to happen.
+
+**Logs**
+If applicable, add any log output related to your problem.
+To get more logs from the SDK, change the log level using environment variable `LD_LOG_LEVEL`. For example:
+
+  ```
+   LD_LOG_LEVEL=debug ./your-application
+  ```
+
+**SDK version**
+The version of this SDK that you are using.
+
+**Language version, developer tools**
+For instance, C++17 or C11. If you are using a language that requires a separate compiler, such as C, please include the
+name and version of the compiler too.
+
+**OS/platform**
+For instance, Ubuntu 16.04, Windows 10, or Android 4.0.3. If your code is running in a browser, please also include the
+browser type and version.
+
+**Additional context**
+Add any other context about the problem here.

.github/ISSUE_TEMPLATE/server-sdk-redis-source--feature-request.md

@@ -0,0 +1,20 @@
+---
+name: 'C++ Server SDK / Redis Source Feature Request'
+about: Create a report to help us improve
+title: ''
+labels: 'package: sdk/server-redis, feature'
+assignees: ''
+
+---
+
+**Is your feature request related to a problem? Please describe.**
+A clear and concise description of what the problem is. Ex. I would love to see the SDK [...does something new...]
+
+**Describe the solution you'd like**
+A clear and concise description of what you want to happen.
+
+**Describe alternatives you've considered**
+A clear and concise description of any alternative solutions or features you've considered.
+
+**Additional context**
+Add any other context about the feature request here.

.github/workflows/manual-publish-doc.yml

@@ -9,6 +9,7 @@ on:
         options:
           - libs/client-sdk
           - libs/server-sdk
+          - libs/server-sdk-redis-source
 name: Publish Documentation
 jobs:
   build-publish:

.github/workflows/manual-sdk-release-artifacts.yml

@@ -15,6 +15,7 @@ on:
         options:
           - libs/client-sdk:launchdarkly-cpp-client
           - libs/server-sdk:launchdarkly-cpp-server
+          - libs/server-sdk-redis-source:launchdarkly-cpp-server-redis-source
 
 name: Publish SDK Artifacts
 

.github/workflows/server-redis.yml

@@ -0,0 +1,49 @@
+name: libs/server-sdk-redis-source
+
+on:
+  push:
+    branches: [ main ]
+    paths-ignore:
+      - '**.md' #Do not need to run CI for markdown changes.
+  pull_request:
+    branches: [ "main", "feat/**" ]
+    paths-ignore:
+      - '**.md'
+
+jobs:
+  build-test-redis:
+    runs-on: ubuntu-22.04
+    services:
+      redis:
+        image: redis
+        ports:
+          - 6379:6379
+    steps:
+      - uses: actions/checkout@v3
+      - uses: ./.github/actions/ci
+        with:
+          cmake_target: launchdarkly-cpp-server-redis-source
+          simulate_release: true
+  build-redis-mac:
+    runs-on: macos-12
+    steps:
+      - uses: actions/checkout@v3
+      - uses: ./.github/actions/ci
+        with:
+          cmake_target: launchdarkly-cpp-server-redis-source
+          platform_version: 12
+          run_tests: false # TODO: figure out how to run Redis service on Mac
+  build-test-redis-windows:
+    runs-on: windows-2022
+    steps:
+      - uses: actions/checkout@v3
+      - uses: ilammy/msvc-dev-cmd@v1
+      - uses: ./.github/actions/ci
+        env:
+          BOOST_LIBRARY_DIR: 'C:\local\boost_1_81_0\lib64-msvc-14.3'
+          BOOST_LIBRARYDIR: 'C:\local\boost_1_81_0\lib64-msvc-14.3'
+        with:
+          cmake_target: launchdarkly-cpp-server-redis-source
+          platform_version: 2022
+          toolset: msvc
+          run_tests: false # TODO: figure out how to run Redis service on Windows

CMakeLists.txt

@@ -61,6 +61,8 @@ option(LD_DYNAMIC_LINK_OPENSSL
 
 option(LD_BUILD_EXAMPLES "Build hello-world examples." ON)
 
+option(LD_BUILD_REDIS_SUPPORT "Build redis support." OFF)
+
 # If using 'make' as the build system, CMake causes the 'install' target to have a dependency on 'all', meaning
 # it will cause a full build. This disables that, allowing us to build piecemeal instead. This is useful
 # so that we only need to build the client or server for a given release (if only the client or server were affected.)
@@ -142,6 +144,11 @@ add_subdirectory(libs/common)
 add_subdirectory(libs/internal)
 add_subdirectory(libs/server-sent-events)
 
+if (LD_BUILD_REDIS_SUPPORT)
+    message("LaunchDarkly: building server-side redis support")
+    add_subdirectory(libs/server-sdk-redis-source)
+endif ()
+
 # Built as static or shared depending on LD_BUILD_SHARED_LIBS variable.
 # This target "links" in common, internal, and sse as object libraries.
 add_subdirectory(libs/client-sdk)

architecture/server_data_source_arch.md

@@ -115,4 +115,4 @@ classDiagram
       +SetError(StatusCodeType code, std::string message) void
     }
     
-```
+```

architecture/server_data_system_architecture.md

@@ -0,0 +1,95 @@
+# Server Data System Architecture
+
+**Client's usage of the Data System**
+
+```mermaid
+classDiagram
+    direction TB
+    Client --* IDataSystem
+    Client --* DataSourceStatusManager
+    IDataSystem <|-- LazyLoad
+    IDataSystem <|-- BackgroundSync
+    IStore <|-- IDataSystem
+
+```
+
+**Data System - Background Sync**
+
+```mermaid
+
+classDiagram
+    direction TB
+
+    class BackgroundSync {
+    }
+
+    IDataSystem <|-- BackgroundSync
+    BackgroundSync --* IDataSynchronizer
+    IDataDestination <|-- MemoryStore
+    IStore <|-- MemoryStore
+    IDataSynchronizer ..> ChangeNotifier
+    BackgroundSync --* ChangeNotifier
+    ChangeNotifier ..> IDataDestination: writes data updates to..
+    ChangeNotifier ..> IStore: reads necessary data from (to calculate deltas)..
+    BackgroundSync --* MemoryStore
+
+```
+
+In order to satisfy `IDataSystem`, the Background Sync system owns a Memory Store, a Change Notifier,
+and an `IDataSynchronizer`.
+
+Because `IDataSystem` is an `IStore`, Background Sync provides pass-through methods to the Memory Store.
+
+Meanwhile, data updates are received from LaunchDarkly via the `IDataSynchronizer`, and then forwarded into
+the `IDataSynchronizer`'s destination - which is a Change Notifier object owned by Background Sync.
+
+The Change Notifier wants to read existing flag/segment data (`IStore`) to compare
+it to the data it receives from the synchronizer, and then:
+
+- Send out change updates
+- Store the updated data (`IDestination`), while guarding against stale data
+
+Because all flag/segment data is stored in memory, the Memory Store satisfies Change Notifier's `IStore`
+and `IDestination`
+dependencies.
+
+**Data System - Lazy Load**
+
+```mermaid
+
+classDiagram
+    direction TB
+
+    class LazyLoad {
+    }
+
+    IDataSystem <|-- LazyLoad
+    LazyLoad --* IDataReader
+    LazyLoad --* MemoryStore
+    LazyLoad --* ExpirationTracker
+    LazyLoad --* ISerializedDataReader
+    IDataReader <|-- JsonDeserializer: passes back data models
+    JsonDeserializer ..> ISerializedDataReader: reads serialized data
+    ISerializedDataReader <|-- Redis
+    ISerializedDataReader <|-- OtherStores: e.g. DynamoDB, etc.
+
+
+```
+
+In contrast to Background Sync, Lazy Load actively implements `IStore` by fetching data items from
+an `ISerializedDataReader`
+via `IDataReader` and placing them in a Memory Store cache.
+
+Items are cached for a period of time after which they are considered stale and refreshed. This logic is implemented
+by the Expiration Tracker.
+
+Currently, Lazy Load only supports JSON as a storage format.
+
+In practice, the `IDataReader` is therefore a `JsonDeserializer`. Theoretically it could be swapped out for dealing with
+(for example) protobuf.
+
+The `JsonDeserializer` sits in between the Lazy Load system and its `ISerializedDataReader`, intercepting requests.
+
+For example, when it receives a request to get a flag `foo` it forwards that to `ISerializedDataReader`, receives a JSON
+string as a response,
+deserializes that into the SDK's data model, and passes it back to the Lazy Load system.

architecture/server_store_arch.md

@@ -0,0 +1,29 @@
+cmake_minimum_required(VERSION 3.11)
+
+include(FetchContent)
+
+
+FetchContent_Declare(hiredis
+        GIT_REPOSITORY https://github.com/redis/hiredis.git
+        GIT_TAG 60e5075d4ac77424809f855ba3e398df7aacefe8
+        GIT_SHALLOW TRUE
+        SOURCE_DIR _deps/hiredis
+        OVERRIDE_FIND_PACKAGE
+)
+
+
+FetchContent_MakeAvailable(hiredis)
+
+include_directories(${CMAKE_CURRENT_BINARY_DIR}/_deps)
+
+set(REDIS_PLUS_PLUS_BUILD_TEST OFF CACHE BOOL "" FORCE)
+
+# 1.3.7 is the last release that works with FetchContent, due to a problem with CheckSymbolExists
+# when it tries to do feature detection on hiredis.
+FetchContent_Declare(redis-plus-plus
+        GIT_REPOSITORY https://github.com/sewenew/redis-plus-plus.git
+        GIT_TAG 1.3.7
+        GIT_SHALLOW TRUE
+)
+
+FetchContent_MakeAvailable(redis-plus-plus)