Skip to content

Commit 74009ee

Browse files
authored
chore: Move ADaaS template repo FAQ to public docs (#166)
This commit moves documentation from the airdrop-template-snapin repo to the public docs, as we will archive that repository, and we want all ADaaS documentation to be fully public and stored centrally. https://app.devrev.ai/devrev/works/ISS-154218
1 parent da8c2c7 commit 74009ee

13 files changed

+312
-217
lines changed

fern/docs/pages/adaas/overview.mdx

Lines changed: 0 additions & 66 deletions
This file was deleted.

fern/docs/pages/adaas/attachments-extraction.mdx renamed to fern/docs/pages/airdrop/attachments-extraction.mdx

Lines changed: 25 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,37 +1,44 @@
1-
For the attachment extraction phase of the import process, the extractor has to upload each attachment to DevRev’s S3
2-
using the `S3Interact` API.
1+
For the attachment extraction phase of the import process, the extractor has to upload each
2+
attachment to DevRev's S3 using the `S3Interact` API.
33

44
## Triggering event
55

6-
Airdrop initiates the attachment extraction by starting the snap-in with a message with an event of type `EXTRACTION_ATTACHMENTS_START`.
7-
This is done after the data extraction, transformation and loading into DevRev are completed.
6+
Airdrop initiates the attachment extraction by starting the snap-in with a message with an event of
7+
type `EXTRACTION_ATTACHMENTS_START`.
8+
This is done after the data extraction, transformation, and loading into DevRev are completed.
89

910
During the attachment extraction phase,
1011
the snap-in extracts attachments from the external system and uploads them as artifacts to DevRev.
1112

12-
The snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_ATTACHMENTS_PROGRESS` together with an optional progress estimate and relevant artifacts
13+
The snap-in must respond to Airdrop with a message with an event of type
14+
`EXTRACTION_ATTACHMENTS_PROGRESS` together with an optional progress estimate and relevant artifacts
1315
when it extracts some data and the maximum snap-in run time (12 minutes) has been reached.
1416

15-
The snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_ATTACHMENTS_DELAY` and specify a back-off time
16-
when the extraction has been rate-limited by the external system and back-off is required.
17+
The snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_ATTACHMENTS_DELAY`
18+
and specify a back-off time when the extraction has been rate-limited by the external system and
19+
back-off is required.
1720

18-
In both cases, Airdrop starts the snap-in with a message with an event of type `EXTRACTION_ATTACHMENTS_CONTINUE`.
19-
The restart is immediate in case of `EXTRACTION_ATTACHMENTS_PROGRESS`, or delayed
20-
in case of `EXTRACTION_ATTACHMENTS_DELAY`.
21+
In both cases, Airdrop starts the snap-in with a message with an event of type
22+
`EXTRACTION_ATTACHMENTS_CONTINUE`.
23+
The restart is immediate in case of `EXTRACTION_ATTACHMENTS_PROGRESS`, or delayed in case of
24+
`EXTRACTION_ATTACHMENTS_DELAY`.
2125

22-
Once the attachment extraction phase is done, the snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_ATTACHMENTS_DONE`.
26+
Once the attachment extraction phase is done, the snap-in must respond to Airdrop with a message
27+
with an event of type `EXTRACTION_ATTACHMENTS_DONE`.
2328

24-
If attachment extraction fails the snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_ATTACHMENTS_ERROR`.
29+
If attachment extraction fails the snap-in must respond to Airdrop with a message with an event of
30+
type `EXTRACTION_ATTACHMENTS_ERROR`.
2531

26-
## Snap-in response
32+
## Response from the snap-in
2733

28-
After uploading an attachment or a batch of attachments, the extractor also has to prepare and upload a file specifying
29-
the extracted and uploaded attachments.
34+
After uploading an attachment or a batch of attachments, the extractor also has to prepare and
35+
upload a file specifying the extracted and uploaded attachments.
3036

31-
It should contain the DevRev IDs of the extracted and uploaded attachments, along with the parent domain object ID
32-
from the external system and the actor ID from the external system.
37+
It should contain the DevRev IDs of the extracted and uploaded attachments, along with the parent
38+
domain object ID from the external system and the actor ID from the external system.
3339

34-
The uploaded artifact is structured like a normal artifact containing extracted data in JSON Lines (JSONL) format and requires specifying `ssor_attachment` as the item type.
40+
The uploaded artifact is structured like a normal artifact containing extracted data in JSON Lines
41+
(JSONL) format and requires specifying `ssor_attachment` as the item type.
3542

3643
## Examples
3744

Original file line numberDiff line numberDiff line change
@@ -1,5 +1,6 @@
1-
During the deletion phases, the snap-in may clean up its state or other side effects in the third party systems.
2-
In the most common extraction use cases, there is nothing to do and snap-ins may reply with the completion message.
1+
During the deletion phases, the snap-in may clean up its state or other side effects in the third
2+
party systems. In the most common extraction use cases, there is nothing to do and snap-ins may reply
3+
with the completion message.
34

45
## Data deletion
56

@@ -8,21 +9,21 @@ In the most common extraction use cases, there is nothing to do and snap-ins may
89
Airdrop initiates the data deletion phase when the import is deleted in the DevRev app.
910
It is started by sending the worker a message with an event of type `EXTRACTION_DATA_DELETE`.
1011

11-
### Snap-in response
12+
### Response from the snap-in
1213

13-
The worker must respond to Airdrop with a message with an event of type `EXTRACTION_DATA_DELETE_DONE` when done or
14-
`EXTRACTION_DATA_DELETE_ERROR` in case of an error.
14+
The worker must respond to Airdrop with a message with an event of type `EXTRACTION_DATA_DELETE_DONE`
15+
when done or `EXTRACTION_DATA_DELETE_ERROR` in case of an error.
1516

1617
## Attachments deletion
1718

1819

1920
### Triggering event
2021

21-
Airdrop initiates the attachments deletion phase when an import is deleted from the DevRev app, after the data deletion has completed.
22+
Airdrop initiates the attachments deletion phase when an import is deleted from the DevRev app,
23+
after the data deletion has completed.
2224
It is started by sending the snap-in a message with an event of type `EXTRACTION_ATTACHMENTS_DELETE`.
2325

24-
### Snap-in response
25-
26-
The snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_ATTACHMENTS_DELETE_DONE` when done,
27-
or `EXTRACTION_ATTACHMENTS_DELETE_ERROR` in case of an error.
26+
### Response from the snap-in
2827

28+
The snap-in must respond to Airdrop with a message with an event of type`EXTRACTION_ATTACHMENTS_DELETE_DONE`
29+
when done, or `EXTRACTION_ATTACHMENTS_DELETE_ERROR` in case of an error.
Lines changed: 32 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -1,50 +1,61 @@
11

2-
In the data extraction phase, the extractor is expected to call the external systems APIs
2+
In the data extraction phase, the extractor is expected to call the external system's APIs
33
to retrieve all the items that were updated since the start of the last extraction.
4-
If there was no previous extraction (the current run is an initial import), then all the items should be extracted.
4+
If there was no previous extraction (the current run is an initial import),
5+
then all the items should be extracted.
56

67
The extractor must store at what time it started each extraction in its state,
7-
so that it can extract only items created and/or updated since this date in the next sync run.
8+
so that it can extract only items created or updated since this date in the next sync run.
89

910
## Triggering event
1011

11-
Airdrop initiates data extraction by starting the snap-in with a message with event of type `EXTRACTION_DATA_START`
12-
when transitioning to the data extraction phase.
12+
Airdrop initiates data extraction by starting the snap-in with a message with event of type
13+
`EXTRACTION_DATA_START` when transitioning to the data extraction phase.
1314

1415
During the data extraction phase, the snap-in extracts data from an external system,
1516
prepares batches of data and uploads them in the form of artifacts to DevRev.
1617

1718
The snap-in must respond to Airdrop with a message with event of type `EXTRACTION_DATA_PROGRESS`,
1819
together with an optional progress estimate and relevant artifacts
19-
when it extracts some data and the maximum ADaaS snap-in runtime (12 minutes) has been reached.
20+
when it extracts some data and the maximum Airdrop snap-in runtime (12 minutes) has been reached.
2021

21-
If the extraction has been rate-limited by the external system and back-off is required, the snap-in must respond to
22-
Airdrop with a message with event of type `EXTRACTION_DATA_DELAY` and specifying back-off time with `delay` attribute.
22+
If the extraction has been rate-limited by the external system and back-off is required, the snap-in
23+
must respond to Airdrop with a message with event of type `EXTRACTION_DATA_DELAY` and specifying
24+
back-off time with `delay` attribute.
2325

2426
In both cases, Airdrop starts the snap-in with a message with event of type `EXTRACTION_DATA_CONTINUE`.
25-
The restarting is immediate (in case of `EXTRACTION_DATA_PROGRESS`) or delayed (in case of `EXTRACTION_DATA_DELAY`).
27+
The restarting is immediate (in case of `EXTRACTION_DATA_PROGRESS`) or delayed
28+
(in case of `EXTRACTION_DATA_DELAY`).
2629

27-
Once the data extraction is done, the snap-in must respond to Airdrop with a message with event of type `EXTRACTION_DATA_DONE`.
30+
Once the data extraction is done, the snap-in must respond to Airdrop with a message with event of
31+
type `EXTRACTION_DATA_DONE`.
2832

29-
If data extraction failed in any moment of extraction, the snap-in must respond to Airdrop with a message with event of type `EXTRACTION_DATA_ERROR`.
33+
If data extraction failed in any moment of extraction, the snap-in must respond to Airdrop with a
34+
message with event of type `EXTRACTION_DATA_ERROR`.
3035

31-
## Snap-in response
36+
## Response from the snap-in
3237

33-
During the data extraction phase, the snap-in uploads batches of extracted items (the recommended batch size is 2000 items) formatted in JSONL
34-
(JSON Lines format), gzipped, and submitted as an artifact to S3Interact (with tooling from `@devrev/adaas-sdk`).
38+
During the data extraction phase, the snap-in uploads batches of extracted items (the recommended
39+
batch size is 2000 items) formatted in JSONL (JSON Lines format), gzipped, and submitted as an
40+
artifact to S3Interact (with tooling from `@devrev/adaas-sdk`).
3541

36-
Each artifact is submitted with an `item_type`, defining a separate domain object from the external system and matching the `record_type` in the provided metadata.
42+
Each artifact is submitted with an `item_type`, defining a separate domain object from the
43+
external system and matching the `record_type` in the provided metadata.
3744
Item types defined when uploading extracted data must validate the declarations in the metadata file.
3845

3946
Extracted data must be normalized.
4047

41-
- Null values: All fields without a value should either be omitted or set to null. For example, if an external system provides values such as "", -1 for missing values, those must be set to null.
42-
- Timestamps: Full-precision timestamps should be formatted as RFC3999 (`1972-03-29T22:04:47+01:00`), and dates should be just `2020-12-31`.
48+
- Null values: All fields without a value should either be omitted or set to null.
49+
For example, if an external system provides values such as "", -1 for missing values,
50+
those must be set to null.
51+
- Timestamps: Full-precision timestamps should be formatted as RFC3999 (`1972-03-29T22:04:47+01:00`),
52+
and dates should be just `2020-12-31`.
4353
- References: references must be strings, not numbers or objects.
44-
- Number fields must be valid JSON numbers (not strings)
45-
- Multiselect fields must be provided as an array (not CSV)
54+
- Number fields must be valid JSON numbers (not strings).
55+
- Multiselect fields must be provided as an array (not CSV).
4656

47-
Each line of the file contains an `id` and the optional `created_date` and `modified_date` fields in the beginning of the record.
57+
Each line of the file contains an `id` and the optional `created_date` and `modified_date` fields
58+
in the beginning of the record.
4859
All other fields are contained within the `data` attribute.
4960

5061
```json
@@ -68,4 +79,4 @@ Extracted artifacts can be validated with the `chef-cli` using the following com
6879

6980
```bash
7081
$ chef-cli validate-metadata -m external_domain_metadata.json -r issue < extractor_issues_2.json
71-
```
82+
```

fern/docs/pages/adaas/external-sync-units-extraction.mdx renamed to fern/docs/pages/airdrop/external-sync-units-extraction.mdx

Lines changed: 18 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,37 +1,41 @@
1-
In the external sync unit extraction phase, the extractor is expected to obtain a list of external sync units
2-
that it can extract with the provided credentials and send it to Airdrop in its response.
1+
In the external sync unit extraction phase, the extractor is expected to obtain a list of external
2+
sync units that it can extract with the provided credentials and send it to Airdrop in its response.
33

4-
An _external sync unit_ refers to a single unit in the external system that we are airdropping to DevRev.
5-
In some systems, this is a project; in some it is a repository; in support systems it could be called a brand or an organization.
4+
An _external sync unit_ refers to a single unit in the external system that is being airdropped to DevRev.
5+
In some systems, this is a project; in some it is a repository; in support systems it could be
6+
called a brand or an organization.
67
What a unit of data is called and what it represents depends on the external system's domain model.
78
It usually combines contacts, users, work-like items, and comments into a unit of domain objects.
89

910
Some external systems may offer a single unit in their free plans,
1011
while their enterprise plans may offer their clients to operate many separate units.
1112

12-
The external sync unit ID is the identifier of the sync unit (project, repository or similar) in the external system.
13+
The external sync unit ID is the identifier of the sync unit (project, repository, or similar)
14+
in the external system.
1315
For GitHub, this would be the repository, for example `cli` in `github.com/devrev/cli`.
1416

1517
## Triggering event
1618

1719
External sync unit extraction is executed only during the initial import.
18-
It extracts external sync units available in the external system,
19-
so that the end user can choose which external sync unit should be airdropped during the creation of an **Import** in the DevRev App.
20+
It extracts external sync units available in the external system, so that the end user can choose
21+
which external sync unit should be airdropped during the creation of an **Import** in the DevRev App.
2022

21-
Airdrop initiates the external sync unit extraction phase by starting the worker with a message with an event of type
22-
`EXTRACTION_EXTERNAL_SYNC_UNITS_START`.
23+
Airdrop initiates the external sync unit extraction phase by starting the worker with a message
24+
with an event of type `EXTRACTION_EXTERNAL_SYNC_UNITS_START`.
2325

24-
The snap-in must respond to Airdrop with a message with an event of type `EXTRACTION_EXTERNAL_SYNC_UNITS_DONE`, which contains
25-
a list of external sync units as a payload, or `EXTRACTION_EXTERNAL_SYNC_UNITS_ERROR` in case of an error.
26+
The snap-in must respond to Airdrop with a message with an event of type
27+
`EXTRACTION_EXTERNAL_SYNC_UNITS_DONE`, which contains a list of external sync units as a payload,
28+
or `EXTRACTION_EXTERNAL_SYNC_UNITS_ERROR` in case of an error.
2629

27-
## Snap-in response
30+
## Response from the snap-in
2831

29-
The snap-in provides the list of external sync units in the provided event message `event_data.external_sync_units` containing the following fields:
32+
The snap-in provides the list of external sync units in the provided event message
33+
`event_data.external_sync_units` containing the following fields:
3034
- `id`: The unique identifier in the external system.
3135
- `name`: The human-readable name in the external system.
3236
- `description`: The short description if the external system provides it.
3337
- `item_count`: The number of items (issues, tickets, comments or others) in the external system.
34-
Item count should be provided if it can be obtained in a very lightweight manner, such as by calling an API endpoint.
38+
Item count should be provided if it can be obtained in a lightweight manner, such as by calling an API endpoint.
3539
If there is no such way to get it (for example, if the items would need to be extracted to count them),
3640
then the item count should be `-1` to avoid blocking the import with long-running queries.
3741

0 commit comments

Comments
 (0)