Skip to content

Commit 7651552

Browse files
biniona-mongodbbiniona-mongodb
authored
Add FLE + QE Sample App Source Files + Build Scripts + Tests (#1254)
* add sample app build scripts * update source files to match formatted built apps * remove submodule + small tweaks * add newline * tweaks * java remove target dir * update readmes * expand readme * readme * numbering
1 parent 2a4ec69 commit 7651552

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

65 files changed

+10100
-6
lines changed

‎source/includes/sample_apps/csfle/.gitignore

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
build/*/*
2+
.envrc
3+
**/master-key.txt

‎source/includes/sample_apps/csfle/Pipfile

Lines changed: 13 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,13 @@
1+
[[source]]
2+
url = "https://pypi.org/simple"
3+
verify_ssl = true
4+
name = "pypi"
5+
6+
[packages]
7+
pymongo = "*"
8+
pymongocrypt = "*"
9+
10+
[dev-packages]
11+
12+
[requires]
13+
python_version = "3.8"

‎source/includes/sample_apps/csfle/README.md

Lines changed: 84 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,84 @@
1+
# Sample CSFLE Applications
2+
3+
1. Run `cp envrc_template .envrc` and then fill out your `.envrc` file. You can fill out the `.envrc` file by using
4+
the DBX team's test credentials, or by creating accounts
5+
on KMS each KMS provider and filling in the file.
6+
7+
2. Use `direnv` to load the environment variables defined in your `.envrc` file into your environment.
8+
9+
3. Install python packages by running `pip install -r requirements.txt`
10+
11+
> **_NOTE:_** If you must test new QE/FLE fetures in python, run `source python-fle-2 requirements.sh` to build and install `pymongo` and `pymongo-crypt` from source.
12+
13+
4. Install bluehawk by running `npm install -g bluehawk`
14+
15+
5. Run `python build.py` to generate scripts to create a DEK and insert a sample document in all combinations of the following languages and KMS providers:
16+
17+
- Local Key Provider
18+
- AWS KMS
19+
- GCP KMS
20+
- Azure Key Vault
21+
22+
- Dotnet
23+
- Node
24+
- Python
25+
- Java
26+
- Go
27+
28+
Once you build the files, you should see a `build` directory with the following structure:
29+
30+
```
31+
build
32+
├── dotnet
33+
│   ├── aws
34+
│   │   ├── reader
35+
│   │   └── test
36+
│   ├── azure
37+
│   │   ├── reader
38+
│   │   └── test
39+
│   ├── gcp
40+
│   │   ├── reader
41+
│   │   └── test
42+
│   └── local
43+
│   ├── reader
44+
│   └── test
45+
├── go
46+
│   ├── aws
47+
│   │   ├── reader
48+
│   │   └── test
49+
│   ├── azure
50+
│   │   ├── reader
51+
│   │   └── test
52+
│   ├── gcp
53+
│   │   ├── reader
54+
│   │   └── test
55+
│   └── local
56+
│   ├── reader
57+
│   └── test
58+
├── java
59+
│   ├── aws
60+
│   │   ├── reader
61+
│   │   └── test
62+
│   ├── azure
63+
│   │   ├── reader
64+
│   │   └── test
65+
│   ├── gcp
66+
│   │   ├── reader
67+
│   │   └── test
68+
│   └── local
69+
│   ├── reader
70+
│   └── test
71+
...
72+
```
73+
74+
The `reader` directories contains the code to use in docs code snippets, and the `test` directories contains runnable examples for testing.
75+
76+
6. Install the following languages and tools necessary
77+
to run sample applications:
78+
79+
- [JDK](https://dotnet.microsoft.com/en-us/download) and [Maven](https://maven.apache.org/download.cgi)
80+
- [Dotnet SDK](https://dotnet.microsoft.com/en-us/download)
81+
- [Node](https://nodejs.org/en/download/)
82+
- [Go](https://go.dev/dl/)
83+
84+
7. Run `python -m unittest test.py` to test the generated scripts using the KMS credentials in your `.envrc` file.

‎source/includes/sample_apps/csfle/build.py

Lines changed: 92 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,92 @@
1+
import os
2+
import shutil
3+
import click
4+
5+
from const import *
6+
7+
8+
def get_commands(language, get_file_name, state):
9+
"""Get list of commands to build language/state pair"""
10+
file_path = os.path.join(language, get_file_name(language))
11+
output_file = os.path.join(
12+
BUILD_DIR,
13+
language,
14+
*state.split(DIR_SEPERATOR),
15+
os.path.dirname(get_file_name(language)),
16+
)
17+
commands = [
18+
f"{BLUEHAWK} copy {file_path} --state {state} -o {output_file}"
19+
]
20+
if GET_EXTRA_FILES(language):
21+
for l in GET_EXTRA_FILES(language):
22+
next_file_name = os.path.join(language, l)
23+
next_output_file = os.path.join(
24+
BUILD_DIR, language, *state.split(DIR_SEPERATOR), os.path.dirname(l)
25+
)
26+
commands.append(f"{BLUEHAWK} copy {next_file_name} -o {next_output_file}")
27+
return commands
28+
29+
30+
def build_languages(language, build_states):
31+
"""Build sample apps for a language"""
32+
33+
all_commands = []
34+
35+
for f in BUILD_FILES:
36+
for s in build_states:
37+
commands = get_commands(language, f, s)
38+
for c in commands:
39+
all_commands += commands
40+
dedup_commands = list(set(all_commands))
41+
for c in dedup_commands:
42+
os.system(c)
43+
44+
# run formatter
45+
if FILE_MAP[language].get(FORMAT_COMMAND):
46+
print(f"\n\n$$$$ Formatting {language}:\n\n")
47+
os.system(FILE_MAP[language][FORMAT_COMMAND])
48+
49+
50+
def check_bluehawk_installed():
51+
"""Check that bluehawk is installed"""
52+
53+
if shutil.which(BLUEHAWK) == None:
54+
raise ImportError(BLUEHAWK_HELP)
55+
56+
57+
@click.command()
58+
@click.option("--langs", default=None, help="What apps to build")
59+
@click.option(
60+
"--reader",
61+
is_flag=True,
62+
default=False,
63+
help="Whether or not to only build reader directories",
64+
)
65+
def build_apps(langs, reader):
66+
"""Get commmand line arguments and build sample applications"""
67+
68+
check_bluehawk_installed()
69+
70+
if os.path.exists(BUILD_DIR):
71+
shutil.rmtree(BUILD_DIR)
72+
73+
file_map = {}
74+
build_states = []
75+
76+
if not langs:
77+
file_map = FILE_MAP
78+
else:
79+
langs_to_build = [l.strip() for l in langs.split(",")]
80+
file_map = {l: FILE_MAP[l] for l in langs_to_build}
81+
82+
if reader:
83+
build_states = [s for s in BUILD_STATES if TEST not in s]
84+
else:
85+
build_states = BUILD_STATES
86+
87+
for lang in file_map.keys():
88+
build_languages(lang, build_states)
89+
90+
91+
if __name__ == "__main__":
92+
build_apps()

‎source/includes/sample_apps/csfle/build/go/local/reader/insert-encrypted-document.go

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,14 +5,16 @@ import (
55
"encoding/json"
66
"fmt"
77

8-
"io/ioutil"
9-
"log"
10-
118
"go.mongodb.org/mongo-driver/bson"
129
"go.mongodb.org/mongo-driver/mongo"
1310
"go.mongodb.org/mongo-driver/mongo/options"
1411
)
1512

13+
import (
14+
"io/ioutil"
15+
"log"
16+
)
17+
1618
func Insert() error {
1719

1820
// start-key-vault

‎source/includes/sample_apps/csfle/build/go/local/reader/make-data-key.go

Lines changed: 5 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -5,12 +5,14 @@ import (
55
"encoding/base64"
66
"fmt"
77

8+
"go.mongodb.org/mongo-driver/mongo"
9+
"go.mongodb.org/mongo-driver/mongo/options"
10+
)
11+
12+
import (
813
"crypto/rand"
914
"io/ioutil"
1015
"log"
11-
12-
"go.mongodb.org/mongo-driver/mongo"
13-
"go.mongodb.org/mongo-driver/mongo/options"
1416
)
1517

1618
func localMasterKey() []byte {

‎source/includes/sample_apps/csfle/const.py

Lines changed: 102 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,102 @@
1+
import os
2+
3+
BUILD_DIR = "build"
4+
PYTHON = "python"
5+
PYTHON_FLE_2 = "python-fle-2"
6+
JAVA = "java"
7+
CSHARP = "dotnet"
8+
NODE = "node"
9+
NODE_FLE_2 = "node-fle-2"
10+
DEK = "dek"
11+
GO = "go"
12+
INSERT = "insert"
13+
EXTRA_FILES = "extra_files"
14+
FORMAT_COMMAND = "format"
15+
FILE_MAP = {
16+
PYTHON: {
17+
DEK: "make_data_key.py",
18+
INSERT: "insert_encrypted_document.py",
19+
EXTRA_FILES: ["requirements.txt", ".gitignore"],
20+
FORMAT_COMMAND: f"black {os.path.join(BUILD_DIR,'*','*','*','*.py')}",
21+
},
22+
PYTHON_FLE_2: {
23+
DEK: "make_data_key.py",
24+
INSERT: "insert_encrypted_document.py",
25+
EXTRA_FILES: ["requirements.sh", ".gitignore"],
26+
FORMAT_COMMAND: f"black {os.path.join(BUILD_DIR,'*','*','*','*.py')}",
27+
},
28+
JAVA: {
29+
DEK: os.path.join(
30+
"src", "main", "java", "com", "mongodb", "csfle", "makeDataKey.java"
31+
),
32+
INSERT: os.path.join(
33+
"src",
34+
"main",
35+
"java",
36+
"com",
37+
"mongodb",
38+
"csfle",
39+
"insertEncryptedDocument.java",
40+
),
41+
EXTRA_FILES: ["pom.xml", ".gitignore"],
42+
},
43+
CSHARP: {
44+
DEK: "CSFLE/MakeDataKey.cs",
45+
INSERT: "CSFLE/InsertEncryptedDocument.cs",
46+
EXTRA_FILES: ["CSFLE/CSFLE.csproj", "CSFLE/Main.cs", ".gitignore"],
47+
FORMAT_COMMAND: f'find ./{BUILD_DIR} -name "*.csproj" | xargs -n1 dotnet format',
48+
},
49+
NODE: {
50+
DEK: "make_data_key.js",
51+
INSERT: "insert_encrypted_document.js",
52+
EXTRA_FILES: ["package.json", ".gitignore"],
53+
FORMAT_COMMAND: f"prettier --write {BUILD_DIR}",
54+
},
55+
NODE_FLE_2: {
56+
DEK: "make_data_key.js",
57+
INSERT: "insert_encrypted_document.js",
58+
EXTRA_FILES: ["package.json", ".gitignore"],
59+
FORMAT_COMMAND: f"prettier --write {BUILD_DIR}",
60+
},
61+
GO: {
62+
DEK: "make-data-key.go",
63+
INSERT: "insert-encrypted-document.go",
64+
EXTRA_FILES: ["go.mod", "main.go", ".gitignore"],
65+
FORMAT_COMMAND: f"gofmt -l -w -s {BUILD_DIR}",
66+
},
67+
}
68+
69+
DIR_SEPERATOR = "-"
70+
PROD = f"{DIR_SEPERATOR}reader"
71+
TEST = f"{DIR_SEPERATOR}test"
72+
AWS_PROD = f"aws{PROD}"
73+
AWS_TEST = f"aws{TEST}"
74+
GCP_PROD = f"gcp{PROD}"
75+
GCP_TEST = f"gcp{TEST}"
76+
AZURE_PROD = f"azure{PROD}"
77+
AZURE_TEST = f"azure{TEST}"
78+
LOCAL_PROD = f"local{PROD}"
79+
LOCAL_TEST = f"local{TEST}"
80+
BUILD_STATES = [
81+
AWS_PROD,
82+
AWS_TEST,
83+
AZURE_PROD,
84+
AZURE_TEST,
85+
GCP_PROD,
86+
GCP_TEST,
87+
LOCAL_PROD,
88+
LOCAL_TEST,
89+
]
90+
91+
MAKE_KEY = lambda language: FILE_MAP[language][DEK]
92+
MAKE_INSERT = lambda language: FILE_MAP[language][INSERT]
93+
GET_EXTRA_FILES = lambda language: FILE_MAP[language].get(EXTRA_FILES)
94+
BUILD_FILES = (MAKE_KEY, MAKE_INSERT)
95+
96+
BLUEHAWK = "bluehawk"
97+
BLUEHAWK_HELP = f"""Unable to find {BLUEHAWK}. Run
98+
the following command to install {BLUEHAWK}:
99+
100+
npm install -g {BLUEHAWK}
101+
102+
"""

0 commit comments

Comments
 (0)