Skip to content

Dataplane Codegen Quick Start

Jump to bottom
Yuchao Yan edited this page May 13, 2022 · 74 revisions

Getting Started - Generate SDK With Dataplane Codegen

Support

For more questions and general overview of the process, please refer to https://aka.ms/azsdk/dpcodegen

Foreword

We advise use Dataplane-Codegen-Quick-Start-with-tools to generate SDK code which is very convenient. Of course, if you want to know more details about how to generate SDK code with DPG, please go ahead for next part.

Prerequisites

Setup your repo

Project service name and package name

Two key pieces of information for your project are the service_name and package_name.

The service_name is the short name for the Azure service. The service_name should match across all the SDK language repos and should be name of the directory in the specification folder of the azure-rest-api-specs repo that contains the REST API definition file. An example is Service Bus, whose API definitions are in the specifications/servicebus folder of the azure-rest-api-specs repo, and uses the service_name "servicebus". Not every service follows this convention, but it should be the standard unless there are strong reasons to deviate.

In Python, a project's package name is the name used to publish the package in PyPI. For data plane libraries (management plane uses a different convention), the package_name could be just azure-{service_name}. An example is "azure-servicebus".

Some services may need several different packages. For these cases a third component, the module_name, is added to the package_name, as azure-{service_name}-{module_name}. The module_name usually comes from the name of the REST API file itself or one of the directories toward the end of the file path. An example is the Synapse service, with packages azure-synapse, azure-synapse-accesscontrol, azure-synapse-artifacts, etc.

Project folder structure

Before we start, we probably should get to know the project folder for SDK repo.

Normally, the folder structure would be something like:

  • sdk/{service_name}/{package_name}: the PROJECT_ROOT folder
    • azure/{service_name}/{module_name} : folder where generated code is.
    • /swagger: folder of configuration file which is used to generate SDK code with autorest
    • /tests: folder of test files
    • /samples: folder of sample files
    • azure-{service_name}-{module_name}: package name. Usually, package name is same with part of ${PROJECT_ROOT} folder. After release, you can find it in pypi. For example: you can find azure-messaging-webpubsubservice in pypi.
    • there are also some other files (like setup.py, README.md, etc) which are necessary for a complete package.

More details on the structure of Azure SDK repos is available in the Azure SDK common repo.

How to generate SDK code with Dataplane Codegen

We are working on to automatically generate everything right now, but currently we still need some manual work to get a releasable package. Here're the steps of how to get the package.

  1. Create the ${PROJECT_ROOT} folder if it does not already exist

  2. Create a README.md file under ${PROJECT_ROOT}/swagger

    This README.md file will hold the options to be passed to autorest to generate the code. Storing these options in a file in the SDK repo helps make SDK generation reproducible.

    Here's an example of the README.md

    # Azure Purview for Python

> see https://aka.ms/autorest

### Settings

```yaml
input-file: https://raw.githubusercontent.com/Azure/azure-rest-api-specs/main/specification/webpubsub/data-plane/WebPubSub/stable/2021-10-01/webpubsub.json
output-folder: ../azure/messaging/webpubsubservice
namespace: azure.messaging.webpubsubservice
package-name: azure-messaging-webpubsubservice
license-header: MICROSOFT_MIT_NO_VERSION
clear-output-folder: true
no-namespace-folders: true
title: WebPubSubServiceClient
version-tolerant: true
package-version: 1.0.0b1
security: AADToken
security-scopes: https://webpubsub.azure.com/.default
```

    There are some configuration in the template and you could reference autorest/flags to get more info. We need to replace the value of package-name, title, input-file, package-version, security-scopes, output-folder into your own service's package-name, titleetc. Notes:

    • output-folder: it is path where SDK code will be generated. And it shall be relative path based on ${PROJECT_ROOT}/swagger folder. For example, if package-name is azure-messaging-webpubsubservice, replace - with / to compose output-folder like: ../azure/messaging/webpubsubservice

    • security and security-scopes: if authentication is api key, the securityand security-scopes shall be replaced with:

      security: AzureKey
security-header-name: real-service-header-name

      For more details, see authentication

  3. Run autorest to generate the SDK

    Now you can run this command in swagger folder you just created.

autorest --version=3.7.2 --use=@autorest/[email protected] --use=@autorest/[email protected] README.md

  1. Add necessary files under ${PROJECT_ROOT}

    For a complete package, there are some other files you need to add:

    • CHANGELOG.md: introduce the release history and important change for each release.
    • LICENSE: license file
    • README.md: import files which introduce what the service is and how to use this package
    • dev_requirements.txt: claims some package which is needed to run test
    • MANIFEST.in: configuration file to decide which file will be included in package.
    • setup.py: most important files about dependency and supported version for Python.
    • __init__.py: it is used when package the SDK code. Please manually copy init.py under ${PROJECT_ROOT}/{folder_first} and ${PROJECT_ROOT}/{folder_second}

    Here is template files, and you at least need to edit item (like {{pakcage_name}}, {{package_pprint_name}}, etc) in CHANGELOG.md, README.md, setup.py, MANIFEST.in according to specific info of your own service. It is easier to understand how to edit the files with reference existing service: webpubsub.

Notes:

  • folder_first, folder_second: if package-name is azure-messaging-webpubsubservice, folder_first is azure and folder_second is messaging. Because package name shall start with azure, folder_first is always azure.
  • package_pprint_name: The short description for the package.

How to add convenience code

Usually, we don't advise to add convenience code. Instead, we hope you change swagger definition or use directives. But if you have to, please see the following guidance:

How to write tests

You should write tests that ensure all APIs fulfil their contract and algorithms work as specified. This is a requirement of the Azure SDK Guidelines.

See the Dataplane Codegen Quick Start for Test for information on how to write and run tests for the Python SDK.

How to write sample

Samples can make it quicker and easier for customers to use the package. It could be simple which just shows how to use the package; Or it could be a complex scenario example for customers to create their own service quickly.

You should add samples in the samples directory under ${PROJECT_ROOT}. Each sample should have its own folder and contain a Readme.md(example) which introduces simply how to run the sample.

If you want to run sample with dev mode, just run pip install -e .(detail doc) under ${PROJECT_ROOT} instead of pip install azure.myservice.

How to create package

If you want to create a private package, go to ${PROJECT_ROOT} folder and run:

pip install wheel
python setup.py bdist_wheel

Create/Update the ci.yaml

Now, if everything looks good to you, you can submit a PR in azure-sdk-for-python repo with all the changes you made above. Before you do that, you need to add/update the ci.yml file. Depends on whether there's already one in sdk/{servicename} folder.

If there's no such file then you can add the following template.

# DO NOT EDIT THIS FILE
# This file is generated automatically and any changes will be lost.

trigger:
  branches:
    include:
    - main
    - hotfix/*
    - release/*
    - restapi*
  paths:
    include:
    - sdk/{service_name}/

pr:
  branches:
    include:
    - main
    - feature/*
    - hotfix/*
    - release/*
    - restapi*
  paths:
    include:
    - sdk/{service_name}/

extends:
  template: ../../eng/pipelines/templates/stages/archetype-sdk-client.yml
  parameters:
    ServiceDirectory: {service_name}
    Artifacts:
    - name: { package_name }
      safeName: { safeName }

Please change the {service_name}, Artifacts name and safeName into yours.

If there's already a ci.yml file in your project path. then the only thing you need to do is to add the Artifacts name and safeName of yours into that ci.yml.

Usually, Artifacts name is same with package name and remove - in Artifacts name to get safeName. Example: webpubwub, purview

Release

Here is the Release Checklist you should know before release. Once ready, make the PR to https://github.com/Azure/azure-sdk-for-python. After the PR is merged, it is time to release package.

Make PR

DPG can only help you generate SDK code, there is something you need to update manually:

CHANGELOG.md

CHANGELOG can help customers know the change of new version quicky, so you need to update the it according to the change of this new version. It is also necessary to update release date like 1.0.0b1 (2022-02-02)(rough time is fine and no need to be very accurate)

Version Number

You shall update the version number according to package version rule

Test recordings

To check whether the new content works well, it is better to run live test. If the result is successful, there shall be recording files(*.yml) updated. Please commit them together with your code.

Clone this wiki locally