Merge step-by-step with existing

Author: tilenkavcic

fern/docs/pages/airdrop/data-extraction.mdx

@@ -1,4 +1,3 @@
-
 In the data extraction phase, the extractor is expected to call the external system's APIs
 to retrieve all the items that were updated since the start of the last extraction.
 If there was no previous extraction (the current run is an initial import),
@@ -43,12 +42,11 @@ Each artifact is submitted with an `item_type`, defining a separate domain objec
 external system and matching the `record_type` in the provided metadata.
 Item types defined when uploading extracted data must validate the declarations in the metadata file.
 
-Extracted data must be normalized.
-
+Extracted data must be normalized:
 - Null values: All fields without a value should either be omitted or set to null.
 For example, if an external system provides values such as "", -1 for missing values,
 those must be set to null.
-- Timestamps: Full-precision timestamps should be formatted as RFC3999 (`1972-03-29T22:04:47+01:00`),
+- Timestamps: Full-precision timestamps should be formatted as RFC3339 (`1972-03-29T22:04:47+01:00`),
 and dates should be just `2020-12-31`.
 - References: references must be strings, not numbers or objects.
 - Number fields must be valid JSON numbers (not strings).
@@ -74,9 +72,26 @@ All other fields are contained within the `data` attribute.
 }
 ```
 
+## Validating extracted data
 
 Extracted artifacts can be validated with the `chef-cli` using the following command:
 
 ```bash
-chef-cli validate-metadata -m external_domain_metadata.json -r issue < extractor_issues_2.json
+chef-cli validate-data -m external_domain_metadata.json -r issue < extractor_issues_2.json
+```
+
+You can also generate example data to show the format the data has to be normalized to, using:
+
+```bash
+echo '{}' | chef-cli fuzz-extracted -r issue -m external_domain_metadata.json > example_issues.json
 ```
+
+## Deploying and testing the snap-in
+
+Once you have implemented data extraction, you should deploy your snap-in to your test organization and run an import.
+
+To deploy the snap-in, run `make auth` and `make deploy` in the snap-in repository. Then, activate the snap-in by running `devrev snap_in activate`. 
+
+After activation, you can create an import in the DevRev UI, which will initially reach the 'waiting for user input' stage. During this phase, you can verify your data extraction implementation is working correctly.
+
+Relevant documentation can be found under [Snap-in development](/snapin-development/locally-testing-snap-ins).

fern/docs/pages/airdrop/initial-domain-mapping.mdx

@@ -0,0 +1,41 @@
+## Complete the chef-cli initial domain mapping setup
+
+Next, continue with the steps outlined in [chef-cli setup](initial_domain_mapping_setup.md). 
+
+When you are done you should have the chef-cli context set up and have the chef-cli local UI running in your browser.
+
+## Use the local UI to create initial domain mappings
+
+The final artifact of the recipe creation process is the `initial_domain_mapping.json`, which has to be embedded in the extractor.
+
+This mapping, unlike the recipe blueprint of a concrete import, can contain multiple options for each external record type from which the end-user might choose (for example allow 'task' from an external system to map either to issue or ticket in DevRev), and it can contain also mappings that apply to a record type category. When the user runs a new import, and the extractor reports in its metadata record types belonging to this category, that are not directly mapped in the initial domain mappings, the recipe manager will apply the per-category default to them.
+
+After the blueprint of the test import was completed, the 'install in this org' button takes you to the initial domain mapping creation screen, where you can 'merge' the blueprint to the existing initial mappings of the org.
+
+By repeating this process (run a new import, create a different configuration, merge to the initial mappings), you can create an initial mapping that contains multiple options for the user to choose from.
+
+Finally the Export button allows you to retrieve the `initial_domain_mapping.json`.
+
+## Tip: use local metadata in the local UI
+
+You can also provide a local metadata file to the command using the '-m' flag for example: `chef-cli configure-mappings --env prod -m metadata.json`, this enables to use:
+
+- raw jq transformations using an external field as input. (This is an experimental feature)
+
+- filling in example input data for trying out the transformation.
+
+In this case it is not validated that the local file is the same as the one submitted by the snap-in, this has to be ensured by you.
+
+## Test an import with initial mapping using the in-app UI.
+
+Once the initial mappings are prepared and, any new import in the org (with the same snap-in slug and import slug) where they are installed will use them. The end-users can influence the recipe blueprint that gets created for the sync unit trough the mapping screen in the UI, where they can make record-type filtering, mapping, fine grained filtering, low-code field and value mapping, and finally custom field filtering.
+
+Their decisions are constrained by the choices provided in the initial domain mappings. Currently the low-code UI offers limited insight into the mappings and their reasons, and in some cases, mismatches arise when something that worked in chef-cli doesn't offer the right options to the user, or not all fields that should be resolved are solved. To assist debugging such cases, chef-cli provides a command to extract the description of the low-code decisions that are asked in the UI. Please provide this to us when reporting an issue with how the end-user mapping UI behaves.
+
+```bash
+chef-cli low-code --env prod > low_code.json`
+```
+ 
+## Read metadata tips to improve the metadata
+
+See the [metadata tips](tips.md) section for more information on how to improve the metadata.

fern/docs/pages/airdrop/metadata-extraction.mdx

@@ -1,4 +1,3 @@
-
 During the metadata extraction phase, the Airdrop snap-in must provide an
 `external_domain_metadata.json` file on each sync run.
 This file provides a structured way of describing the external system's domain system,
@@ -24,6 +23,33 @@ During the metadata extraction phase, the Airdrop snap-in must provide an
 The transformation can be crafted and finalized further using the `chef-cli` to ensure extracted
 data is mapped consistently to the DevRev domain model.
 
+## Getting started with infer-metadata
+
+The `chef-cli` provides a helpful command to generate initial domain metadata from example data:
+
+```bash
+chef-cli infer-metadata example_data_directory > metadata.json
+```
+
+To get good results with this approach:
+
+1. Collect example data from the external system and place them in a directory. Each file should:
+   - Contain the same type of records, named after their type
+   - Have .json or .jsonl extension, for example `issues.json`
+   - Contain either a single JSON array of objects, or newline-separated objects
+
+2. Run the `infer-metadata` command targeting this directory
+
+3. Inspect the generated metadata, particularly field types and the suggestions the tool generates
+
+For best results:
+- Provide 10-100 examples of each record type (but not more than 1000)
+- Ensure logically distinct fields are separate keys at the top level
+- Use referentially consistent example data if possible
+- Make sure IDs are strings, not numbers
+
+This generated metadata serves as a starting point that will need further refinement.
+
 ## Craft the metadata declaration
 
 Since crafting metadata declaration in the form of an `external_domain_metadata.json` file can be a
@@ -36,7 +62,6 @@ a snap-in run from external system APIs (since they are configurable in the exte
 be changed by the end user at any time, such as mandatory fields or custom fields).
 
 <Steps>
-
   ### Declare the extracted record types
 
     _Record types_ are the types of records that has a well-defined schema you extract from or load
@@ -266,30 +291,126 @@ be changed by the end user at any time, such as mandatory fields or custom field
   existing in that record type, marked 'is_identifier'. For example:
   ```json
   {
-  "record_types": {
-    "users": {
-      "fields": {
-        "email": {
-          "type": "text",
-          "is_identifier":true
+    "record_types": {
+      "users": {
+        "fields": {
+          "email": {
+            "type": "text",
+            "is_identifier":true
+          }
         }
-      }
-    },
-    "comments": {
-      "fields": {
-        "user_email": {
-          "type": "reference",
-          "reference": {
-            "refers_to": {
-              "#record:users": {
-                "by_field": "email"
+      },
+      "comments": {
+        "fields": {
+          "user_email": {
+            "type": "reference",
+            "reference": {
+              "refers_to": {
+                "#record:users": {
+                  "by_field": "email"
+                }
               }
             }
           }
         }
       }
     }
   }
-}
   ```
+
+  ### Define field attributes
+
+  External system fields that shouldn't be mapped in reverse should be marked as `is_read_only`. 
+  Depending on their purpose you can also mark fields as `is_indexed`, `is_identifier`, `is_filterable`, 
+  `is_write_only` etc. By default these will be set to false. You can find the full list of supported 
+  field attributes and their descriptions in the [metadata schema](./external_domain_metadata_schema.json).
+
+  ### Configure state transitions
+
+  If an external record type has some concept of states, between which only certain transitions are 
+  possible (e.g., to move to the 'resolved' status, an issue first has to be 'in_testing'), you 
+  can declare these in the metadata too.
+
+  This will allow creation of a matching 'stage diagram' in DevRev, which enables a much simpler 
+  import and a closer preservation of the external data than mapping to DevRev's builtin stages.
+
+  This is especially important for two-way sync, as setting the transitions correctly ensures that 
+  the transitions a record undergoes in DevRev can be replicated in the external system.
+
+  To declare this in the metadata, make sure the status is represented as an enum field, and then 
+  declare the allowed transitions:
+
+  ```json
+  {
+    "fields": {
+      "status": {
+        "name": "Status",
+        "is_required": true,
+        "type": "enum",
+        "enum": {
+          "values": [
+            {
+              "key": "detected",
+              "name": "Detected"
+            },
+            {
+              "key": "mitigated",
+              "name": "Mitigated"
+            },
+            {
+              "key": "rca_ready",
+              "name": "RCA Ready"
+            },
+            {
+              "key": "archived",
+              "name": "Archived"
+            }
+          ]
+        }
+      }
+    },
+    "stage_diagram": {
+      "controlling_field": "status",
+      "starting_stage": "detected",
+      "all_transitions_allowed": false,
+      "stages": {
+        "detected": {
+          "transitions_to": ["mitigated", "archived", "rca_ready"],
+          "state": "new"
+        },
+        "mitigated": {
+          "transitions_to": ["archived", "detected"],
+          "state": "work_in_progress"
+        },
+        "rca_ready": {
+          "transitions_to": ["archived"],
+          "state": "work_in_progress"
+        },
+        "archived": {
+          "transitions_to": [],
+          "state": "completed"
+        }
+      },
+      "states": {
+        "new": {
+          "name": "New"
+        },
+        "work_in_progress": {
+          "name": "Work in Progress"
+        },
+        "completed": {
+          "name": "Completed",
+          "is_end_state": true
+        }
+      }
+    }
+  }
+  ```
+
+  In the above example:
+  - The status field is the controlling field of the stage diagram
+  - If a status field has no explicit transitions but you still want a stage diagram, set `all_transitions_allowed` to `true`
+  - External systems may categorize statuses (like Jira's status categories), which can be included in the diagram metadata (`states` in the example)
+  - The `starting_stage` defines the initial stage for new object instances
+  - In current metadata format (v0.2.0), the order and human-readable name are taken from the enum values defined on the controlling field
 </Steps>

fern/versions/public.yml

@@ -164,6 +164,10 @@ navigation:
             hidden: false
             slug: data-extraction
             path: ../docs/pages/airdrop/data-extraction.mdx
+          - page: "Initial domain mapping"
+            hidden: false
+            slug: initial-domain-mapping
+            path: ../docs/pages/airdrop/initial-domain-mapping.mdx
           - page: "Attachments extraction"
             hidden: false
             slug: extract-attachments