Skip to content

Commit 7a14a70

Browse files
samodbc-devrevradovan-jorgicerazemk
authored
Add ADaaS documentation. (#64)
* Add ADaaS documentation. * Update fern/versions/public.yml Move the documentation one level up. Co-authored-by: Ben Colborn <[email protected]> * Update fern/versions/public.yml Use hyphens instead of underscores in directory names. Co-authored-by: Ben Colborn <[email protected]> * Update fern/docs/pages/references/adaas/extracting_metadata.mdx Co-authored-by: Ben Colborn <[email protected]> * Update fern/docs/pages/references/adaas/adaas_overview.mdx Co-authored-by: Ben Colborn <[email protected]> * Update fern/docs/pages/references/adaas/extracting_attachments.mdx Co-authored-by: Ben Colborn <[email protected]> * Update fern/docs/pages/references/adaas/adaas_overview.mdx Co-authored-by: Ben Colborn <[email protected]> * Move adaas/ up one level. Use hyphens in file names. Remove duplicate page in index. * Modify titles to sentence case. * Add Keyrings and Import sections to Concepts page. * Add Keyrings and Import sections to Concepts page. * Migrate entities definitions to separate pages. * Lowercase snap-ins. Remove terminology. * Add Extract metadata update. Fix of capitalized terminology. * Add delete phases. Replace Worker with snap-in. * Update fern/docs/pages/adaas/overview.mdx Co-authored-by: radovan-jorgic <[email protected]> * Code review fixes. * Update fern/docs/pages/adaas/overview.mdx Co-authored-by: Ben Colborn <[email protected]> * Update fern/docs/pages/adaas/overview.mdx Co-authored-by: Ben Colborn <[email protected]> * Update fern/docs/pages/adaas/extracting-attachments.mdx Co-authored-by: Ben Colborn <[email protected]> * Update fern/docs/pages/adaas/extracting-attachments.mdx Co-authored-by: Ben Colborn <[email protected]> * Update fern/docs/pages/adaas/delete-phases.mdx Co-authored-by: Ben Colborn <[email protected]> * Update fern/versions/public.yml Co-authored-by: Ben Colborn <[email protected]> * Remove latin abbrevations. * Remove cluttering with EVENT_TYPE. * Fix sentence cases. * Update fern/docs/pages/adaas/getting-started.mdx Co-authored-by: Ben Colborn <[email protected]> * Update fern/docs/pages/adaas/getting-started.mdx Co-authored-by: Ben Colborn <[email protected]> * Update fern/docs/pages/adaas/getting-started.mdx Co-authored-by: Ben Colborn <[email protected]> * First term use in italics, pt. 1. * Modify the order of Getting Started. * Numbered list on extraction phases. * Correct the Extract metadata. Replace with . * Replace Artifacts with artifacts. * Fix Data extraction section. * Fix ATtachments extraction section. * Remove commas from JSON snippet. * Fix a typo. * Image edit. * First uses of terms in italics, pt. 2. * Apply suggestions from code review Co-authored-by: Erazem Kokot <[email protected]> * Code review fixes. * Add chef-cli public repo. --------- Co-authored-by: Ben Colborn <[email protected]> Co-authored-by: radovan-jorgic <[email protected]> Co-authored-by: Erazem Kokot <[email protected]>
1 parent 4044474 commit 7a14a70

16 files changed

+710
-4
lines changed

‎fern/docs/img/adaas/adaas-architecture.mmd

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
---
2+
title: ADaaS architecture high-level overview
3+
---
4+
graph TB
5+
externalSystem[External system]
6+
worker([Worker])
7+
s3interact[s3interact]
8+
airdrop[Airdrop]
9+
s3[(AWS S3)]
10+
11+
externalSystem <-- Get/create users, issues, ... --> worker
12+
13+
worker <-- Exchange artifact upload/download URLs --> s3interact
14+
worker <-- Upload/download artifacts --> s3
15+
worker <-- ADaaS messages and REST API --> airdrop
16+
17+
18+
subgraph DevRev
19+
s3interact <-- Prepare upload/download URL --> s3
20+
airdrop <-- Download/upload artifacts --> s3
21+
end

‎fern/docs/img/adaas/adaas-architecture.svg

Lines changed: 1 addition & 0 deletions
Loading

‎fern/docs/img/adaas/adaas-extraction-phases.mmd

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
---
2+
title: Extraction Phases
3+
---
4+
sequenceDiagram
5+
actor user as UI
6+
participant gw as REST API Gateway
7+
participant ad as Airdrop components
8+
participant ee as External extractor
9+
10+
note over gw,ad: Extract External Sync Units
11+
user ->> gw: Select import connection
12+
gw ->> ad: Start External Sync Unit extraction
13+
ad ->> ee: Start External Sync Unit extraction
14+
ee -->> gw: List of External Sync Units
15+
gw ->> ad: List of External Sync Units
16+
ad ->> gw: List of External Sync Units
17+
gw -->> user: Show available External Sync Units
18+
19+
20+
note over gw,ad: Extract Metadata
21+
ad ->> ee: Start Metadata extraction
22+
ee -->> gw: List of extracted metadata artifacts
23+
gw ->> ad: List of extracted metadata artifacts
24+
25+
note over gw,ad: Extract data
26+
ad ->> ee: Start data extraction
27+
ee -->> gw: List of extracted data artifacts
28+
gw ->> ad: List of extracted data artifacts
29+
30+
note over ad: Transform and import data into DevRev
31+
32+
note over gw,ad: Extract attachments
33+
ad ->> ee: Start attachment extraction
34+
ee -->> gw: List of extracted attachments
35+
gw ->> ad: List of extracted attachments
36+
37+
ad -->> gw: Finished import
38+
gw -->> user: Show finished import and report
39+
40+
note over gw,ad: Delete data
41+
user ->> gw: Delete import
42+
gw ->> ad: Delete import
43+
ad ->> ee: Delete data
44+
ee -->> gw: Finished deleting data
45+
gw ->> ad: Finished deleting data
46+
47+
note over gw,ad: Delete attachments
48+
ad ->> ee: Delete attachments
49+
ee -->> gw: Finished deleting attachments
50+
gw ->> ad: Finished deleting attachments
51+
note over ad: Deletes internal data
52+
ad -->> gw: Finished deleting import
53+
gw -->> user: Import deleted

‎fern/docs/img/adaas/adaas-extraction-phases.svg

Lines changed: 1 addition & 0 deletions
Loading

‎fern/docs/pages/adaas/attachments-extraction.mdx

Lines changed: 64 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,64 @@
1+
For the attachment extraction phase of the import process, the extractor has to upload each attachment to DevRev’s S3
2+
using the `S3Interact` API.
3+
4+
## Triggering event
5+
6+
Airdrop initiates the attachment extraction by starting the snap-in with a message with an event of type `EXTRACTION_ATTACHMENTS_START`.
7+
This is done after the data extraction, transformation and loading into DevRev are completed.
8+
9+
During the attachment extraction phase,
10+
the snap-in extracts attachments from the external system and uploads them as artifacts to DevRev.
11+
12+
The snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_ATTACHMENTS_PROGRESS` together with an optional progress estimate and relevant artifacts
13+
when it extracts some data and the maximum snap-in run time (12 minutes) has been reached.
14+
15+
The snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_ATTACHMENTS_DELAY` and specify a back-off time
16+
when the extraction has been rate-limited by the external system and back-off is required.
17+
18+
In both cases, Airdrop starts the snap-in with a message with an event of type `EXTRACTION_ATTACHMENTS_CONTINUE`.
19+
The restart is immediate in case of `EXTRACTION_ATTACHMENTS_PROGRESS`, or delayed
20+
in case of `EXTRACTION_ATTACHMENTS_DELAY`.
21+
22+
Once the attachment extraction phase is done, the snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_ATTACHMENTS_DONE`.
23+
24+
If attachment extraction fails the snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_ATTACHMENTS_ERROR`.
25+
26+
## Snap-in response
27+
28+
After uploading an attachment or a batch of attachments, the extractor also has to prepare and upload a file specifying
29+
the extracted and uploaded attachments.
30+
31+
It should contain the DevRev IDs of the extracted and uploaded attachments, along with the parent domain object ID
32+
from the external system and the actor ID from the external system.
33+
34+
The uploaded artifact is structured like a normal artifact containing extracted data in JSON Lines (JSONL) format and requires specifying `ssor_attachment` as the item type.
35+
36+
## Examples
37+
38+
Here is an example of an SSOR attachment file:
39+
```json
40+
[
41+
{
42+
"id": {
43+
"external": "don:core:dvrv-us-1:devo/1:artifact/1" // The DON of the uploaded artifact
44+
},
45+
"parent_id": {
46+
"external": "12345" // ID of the parent in the source system
47+
},
48+
"actor_id": {
49+
"external": "123456" // ID of the uploader in the source system
50+
}
51+
},
52+
{
53+
"id": {
54+
"external": "don:core:dvrv-us-1:devo/1:artifact/2" // The DON of the uploaded artifact
55+
},
56+
"parent_id": {
57+
"external": "12344" // ID of the parent in the source system
58+
},
59+
"actor_id": {
60+
"external": "123457" // ID of the uploader in the source system
61+
}
62+
}
63+
]
64+
```

‎fern/docs/pages/adaas/data-attachments-deletion.mdx

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
During the deletion phases, the snap-in may clean up its state or other side effects in the third party systems.
2+
In the most common extraction use cases, there is nothing to do and snap-ins may reply with the completion message.
3+
4+
## Data deletion
5+
6+
### Triggering event
7+
8+
Airdrop initiates the data deletion phase when the import is deleted in the DevRev app.
9+
It is started by sending the worker a message with an event of type `EXTRACTION_DATA_DELETE`.
10+
11+
### Snap-in response
12+
13+
The worker must respond to Airdrop with a message with an event of type `EXTRACTION_DATA_DELETE_DONE` when done or
14+
`EXTRACTION_DATA_DELETE_ERROR` in case of an error.
15+
16+
## Attachments deletion
17+
18+
19+
### Triggering event
20+
21+
Airdrop initiates the attachments deletion phase when an import is deleted from the DevRev app, after the data deletion has completed.
22+
It is started by sending the snap-in a message with an event of type `EXTRACTION_ATTACHMENTS_DELETE`.
23+
24+
### Snap-in response
25+
26+
The snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_ATTACHMENTS_DELETE_DONE` when done,
27+
or `EXTRACTION_ATTACHMENTS_DELETE_ERROR` in case of an error.
28+

0 commit comments

Comments
 (0)