Skip to content

chore: Add publishing section to python plugin template #5

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 26, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
85 changes: 50 additions & 35 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,67 +1,75 @@
# Python Source Plugin Template

This repo contains everything you need to get started with building a new plugin.
To get started all you need to do is change a few names, define some tables, and write an API Client to populate the tables.

[![Mastering CloudQuery: How to build a Source Plugin in Python](https://i.ytimg.com/vi/TSbGHz5Z09M/maxresdefault.jpg)](https://youtu.be/TSbGHz5Z09M "Mastering CloudQuery: How to build a Source Plugin in Python")

## Key files & classes
- plugin/tables/items.py
- Items - A boilerplate table definition
- ItemResolver - A boilerplate table resolver
- plugin/example/client.py
- ExampleClient - A boilerplate API Client
- plugin/client/client.py
- Spec - Defines the CloudQuery Config
- Client (uses: ExampleClient) - Wraps your API Client
- plugin/plugin.py
- ExamplePlugin - The plugin registration / how CloudQuery knows what tables your plugin exposes.


- [plugin/tables/items.py](plugin/tables/items.py)
- `Items` - A boilerplate table definition
- `ItemResolver` - A boilerplate table resolver
- [plugin/example/client.py](plugin/example/client.py)
- `ExampleClient` - A boilerplate API Client
- [plugin/client/client.py]
- `Spec` - Defines the CloudQuery Config
- `Client` (uses: `ExampleClient`) - Wraps your API Client
- [plugin/plugin.py](plugin/plugin.py)
- `ExamplePlugin` - The plugin registration / how CloudQuery knows what tables your plugin exposes.

## Getting started

### Defining your tables

The first thing you need to do is identify the tables you want to create with your plugin.
Conventionally, CloudQuery plugins have a direct relationship between tables and API responses.

For example:
If you had an API endpoint https://api.example.com/items/{num} and for each value of `num` it provided an object
```json
{
"num": {{num}},
"date": "2023-10-12",
"title": "A simple example"
}
```
Then you would design the table class as
```python
class Items(Table):
def __init__(self) -> None:
super().__init__(
name="item",
title="Item",
columns=[
Column("num", pa.uint64(), primary_key=True),
Column("date", pa.date64()),
Column("title", pa.string()),
],
)
...
```
If you had an API endpoint https://api.example.com/items/{num} and for each value of `num` it provided an object

```json
{
"num": {{num}},
"date": "2023-10-12",
"title": "A simple example"
}
```

Then you would design the table class as

```python
class Items(Table):
def __init__(self) -> None:
super().__init__(
name="item",
title="Item",
columns=[
Column("num", pa.uint64(), primary_key=True),
Column("date", pa.date64()),
Column("title", pa.string()),
],
)
...
```

Creating one table for each endpoint that you want to capture.

### API Client

Next you'll need to define how the tables are retrieved, it's recommended to implement this as a generator, as per the example in `plugin/example/client.py`.

### Spec

Having written your API Client you will have, identified the authentication and/or operational variables needed.
Adding these to the CloudQuery config spec can be done by editing the `Spec` `dataclass` using standard python, and adding validation where needed.

### Plugin
Finally, you need to edit the `plugin.py` file to set the plugin name and version, and add the `Tables` to the `get_tables` function.

Finally, you need to edit the `plugin.py` file to set the plugin name and version, and add the `Tables` to the `get_tables` function.

### Test run

To test your plugin you can run it locally.

To automatically manage your virtual environment and install the dependencies listed in the `pyproject.toml` you can use `poetry`.
Expand All @@ -75,6 +83,13 @@ Then to run the plugin `poetry run main serve`, which will launch the plugin man
With that running you can adjust the `TestConfig.yaml` to match your plugin and run `cloudquery sync`.
This should result in the creation of a sqlite database `db.sqlite` where you can validate your tables are as expected.

### Publishing

1. Update the plugin metadata in [plugin/plugin.py](plugin/plugin.py#L13) to match your team and plugin name.
2. Run `python main.py package -m "Initial release" "v0.0.1" .`. `-m` specifies changelog and `v0.0.1` is the version.
3. Run `cloudquery plugin publish -f` to publish the plugin to the CloudQuery registry.

> More about publishing plugins [here](https://docs.cloudquery.io/docs/developers/publishing-an-addon-to-the-hub)

## Links

Expand Down
2 changes: 1 addition & 1 deletion plugin/plugin.py
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
from plugin.client import Client, Spec

PLUGIN_NAME = "example"
PLUGIN_VERSION = "1.0.0" # {x-release-please-version}
PLUGIN_VERSION = "0.0.1"
TEAM_NAME = "cloudquery"
PLUGIN_KIND = "source"

Expand Down