Skip to content

Commit e89cbfb

Browse files
samodsamod
committed
Add delete phases. Replace Worker with snap-in.
1 parent 9db8cd8 commit e89cbfb

11 files changed

+140
-142
lines changed

‎fern/docs/img/adaas/adaas-architecture.mmd

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
---
2+
title: ADaaS architecture high-level overview
3+
---
4+
graph TB
5+
externalSystem[External system]
6+
worker([Worker])
7+
s3interact[s3interact]
8+
airdrop[Airdrop]
9+
s3[(AWS S3)]
10+
11+
externalSystem <-- Get/create users, issues, ... --> worker
12+
13+
worker <-- Exchange artifact upload/download URLs --> s3interact
14+
worker <-- Upload/download artifacts --> s3
15+
worker <-- ADaaS messages and REST API --> airdrop
16+
17+
18+
subgraph DevRev
19+
s3interact <-- Prepare upload/download URL --> s3
20+
airdrop <-- Download/upload artifacts --> s3
21+
end

‎fern/docs/img/adaas/adaas-architecture.svg

Lines changed: 1 addition & 0 deletions
Loading

‎fern/docs/img/adaas/adaas-extraction-phases.mmd

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,53 @@
1+
---
2+
title: Extraction Phases
3+
---
4+
sequenceDiagram
5+
actor user as UI
6+
participant gw as Gateway
7+
participant ad as Airdrop components
8+
participant ee as External extractor
9+
10+
note over gw,ad: Extract External Sync Units
11+
user ->> gw: Select import connection
12+
gw ->> ad: Start External Sync Unit extraction
13+
ad ->> ee: Start External Sync Unit extraction
14+
ee -->> gw: List of External Sync Units
15+
gw ->> ad: List of External Sync Units
16+
ad ->> gw: List of External Sync Units
17+
gw -->> user: Show available External Sync Units
18+
19+
20+
note over gw,ad: Extract Metadata
21+
ad ->> ee: Start Metadata extraction
22+
ee -->> gw: List of extracted metadata artifacts
23+
gw ->> ad: List of extracted metadata artifacts
24+
25+
note over gw,ad: Extract data
26+
ad ->> ee: Start data extraction
27+
ee -->> gw: List of extracted data artifacts
28+
gw ->> ad: List of extracted data artifacts
29+
30+
note over ad: Transform and import data into DevRev
31+
32+
note over gw,ad: Extract attachments
33+
ad ->> ee: Start attachment extraction
34+
ee -->> gw: List of extracted attachments
35+
gw ->> ad: List of extracted attachments
36+
37+
ad -->> gw: Finished import
38+
gw -->> user: Show finished import and report
39+
40+
note over gw,ad: Delete data
41+
user ->> gw: Delete import
42+
gw ->> ad: Delete import
43+
ad ->> ee: Delete data
44+
ee -->> gw: Finished deleting data
45+
gw ->> ad: Finished deleting data
46+
47+
note over gw,ad: Delete attachments
48+
ad ->> ee: Delete attachments
49+
ee -->> gw: Finished deleting attachments
50+
gw ->> ad: Finished deleting attachments
51+
note over ad: Deletes internal data
52+
ad -->> gw: Finished deleting import
53+
gw -->> user: Import deleted

‎fern/docs/img/adaas/adaas-extraction-phases.svg

Lines changed: 1 addition & 0 deletions
Loading

‎fern/docs/pages/adaas/delete-phases.mdx

Lines changed: 28 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
During the delete phases, the snap-in may clean up its state or other side effects in the third party systems.
2+
In the most common extraction use cases, there is nothing to do and snap-ins may reply with the completion message.
3+
4+
## Data deletion
5+
6+
### Triggering event
7+
8+
Airdrop initiates the data delete phase when the sync is deleted from DevRev.
9+
It is started by sending the worker a message with eventType: `EXTRACTION_DATA_DELETE`.
10+
11+
### Snap-in response
12+
13+
The worker must respond to Airdrop with a message with eventType `EXTRACTION_DATA_DELETE_DONE` when done or
14+
`EXTRACTION_DATA_DELETE_ERROR` in case of an error.
15+
16+
## Attachments deletion
17+
18+
19+
### Triggering event
20+
21+
Airdrop initiates the attachments delete phase when the Sync is deleted from DevRev.
22+
It is started by sending the span-in a message with eventType: `EXTRACTION_ATTACHMENTS_DELETE`.
23+
24+
### Snap-in response
25+
26+
The snap-in must respond to Airdrop with a message with eventType `EXTRACTION_ATTACHMENTS_DELETE_DONE` when done,
27+
or `EXTRACTION_ATTACHMENTS_DELETE_ERROR` in case of an error.
28+

‎fern/docs/pages/adaas/extracting-attachments.mdx

Lines changed: 25 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,28 @@
1-
For the attachment extraction phase of the import process,
2-
the extractor has to upload each attachment to DevRev’s S3 using the `S3Interact` API.
1+
For the attachment extraction phase of the import process, the extractor has to upload each attachment to DevRev’s S3 using the `S3Interact` API.
2+
3+
## Triggering event
4+
5+
Airdrop initiates the Attachment Extraction by starting the snap-in with a message with eventType `EXTRACTION_ATTACHMENTS_START`
6+
when transitioning to the Data Extraction phase.
7+
8+
During the Attachment Extraction phase,
9+
the snap-in extracts attachments from the External System and uploads them as Artifacts to DevRev.
10+
11+
The snap-in must respond to Airdrop with a message with eventType `EXTRACTION_ATTACHMENTS_PROGRESS` together with an optional progress estimate and relevant Artifacts
12+
when it extracts some data and the maximum snap-in runtime (12 minutes) has been reached.
13+
14+
The snap-in must respond to Airdrop with a message with eventType `EXTRACTION_ATTACHMENTS_DELAY` and specify a back-off time
15+
when the extraction has been rate-limited by the External System and back-off is required.
16+
17+
In both cases, Airdrop starts the snap-in with a message with eventType `EXTRACTION_ATTACHMENTS_CONTINUE`.
18+
The restart is immediate (in case of `EXTRACTION_ATTACHMENTS_PROGRESS`) or delayed
19+
(in case of `EXTRACTION_ATTACHMENTS_DELAY`).
20+
21+
Once the Attachment Extraction phase is done, the snap-in must respond to Airdrop with a message with eventType `EXTRACTION_ATTACHMENTS_DONE`.
22+
23+
If Attachment Extraction failed, the snap-in must respond to Airdrop with a message with eventType `EXTRACTION_ATTACHMENTS_ERROR`.
24+
25+
## Snap-in response
326

427
After uploading an attachment or a batch of attachments, the extractor also has to prepare and upload a file specifying
528
the extracted and uploaded attachments.

‎fern/docs/pages/adaas/extracting-data.mdx

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,13 @@ so that it can extract only items created and/or updated since this date in the
88

99
## Triggering event
1010

11-
Airdrop initiates data extraction by starting the Worker with a message with eventType `EXTRACTION_DATA_START`
11+
Airdrop initiates data extraction by starting the snap-in with a message with eventType `EXTRACTION_DATA_START`
1212
when transitioning to the data extraction phase.
1313

1414
During the data extraction phase, the snap-in extracts data from an external system,
1515
prepares batches of data and uploads them in the form of Artifacts to DevRev.
1616

17-
The Worker must respond to Airdrop with a message with eventType `EXTRACTION_DATA_PROGRESS`,
17+
The snap-in must respond to Airdrop with a message with eventType `EXTRACTION_DATA_PROGRESS`,
1818
together with an optional progress estimate and relevant artifacts
1919
when it extracts some data and the maximum ADaaS snap-in runtime (12 minutes) has been reached.
2020

‎fern/docs/pages/adaas/extracting-metadata.mdx

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,13 @@ The extraction function of the snap-in must provide a valid metadata file. DevRe
88

99
## Triggering event
1010

11-
Airdrop initiates the Metadata Extraction by starting the Worker with a message with eventType: `EXTRACTION_METADATA_START`.
11+
Airdrop initiates the Metadata Extraction by starting the snap-in with a message with eventType: `EXTRACTION_METADATA_START`.
1212

1313
During the Metadata Extraction phase,
14-
the Worker extracts relevant metadata from an External System
14+
the snap-in extracts relevant metadata from an External System
1515
to prepare and upload the Initial Domain Mapping to DevRev.
1616

17-
The Worker must respond to Airdrop with a message with eventType `EXTRACTION_METADATA_DONE` when done,
17+
The snap-in must respond to Airdrop with a message with eventType `EXTRACTION_METADATA_DONE` when done,
1818
or `EXTRACTION_METADATA_ERROR` in case of an error.
1919

2020
## Snap-in response

0 commit comments

Comments
 (0)