You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: README.md
+50-35Lines changed: 50 additions & 35 deletions
Original file line number
Diff line number
Diff line change
@@ -1,67 +1,75 @@
1
1
# Python Source Plugin Template
2
+
2
3
This repo contains everything you need to get started with building a new plugin.
3
4
To get started all you need to do is change a few names, define some tables, and write an API Client to populate the tables.
4
5
5
6
[](https://youtu.be/TSbGHz5Z09M"Mastering CloudQuery: How to build a Source Plugin in Python")
6
7
7
8
## Key files & classes
8
-
- plugin/tables/items.py
9
-
- Items - A boilerplate table definition
10
-
- ItemResolver - A boilerplate table resolver
11
-
- plugin/example/client.py
12
-
- ExampleClient - A boilerplate API Client
13
-
- plugin/client/client.py
14
-
- Spec - Defines the CloudQuery Config
15
-
- Client (uses: ExampleClient) - Wraps your API Client
16
-
- plugin/plugin.py
17
-
- ExamplePlugin - The plugin registration / how CloudQuery knows what tables your plugin exposes.
-`Client` (uses: `ExampleClient`) - Wraps your API Client
18
+
-[plugin/plugin.py](plugin/plugin.py)
19
+
-`ExamplePlugin` - The plugin registration / how CloudQuery knows what tables your plugin exposes.
20
20
21
21
## Getting started
22
22
23
23
### Defining your tables
24
+
24
25
The first thing you need to do is identify the tables you want to create with your plugin.
25
26
Conventionally, CloudQuery plugins have a direct relationship between tables and API responses.
26
27
27
28
For example:
28
-
If you had an API endpoint https://api.example.com/items/{num} and for each value of `num` it provided an object
29
-
```json
30
-
{
31
-
"num": {{num}},
32
-
"date": "2023-10-12",
33
-
"title": "A simple example"
34
-
}
35
-
```
36
-
Then you would design the table class as
37
-
```python
38
-
classItems(Table):
39
-
def__init__(self) -> None:
40
-
super().__init__(
41
-
name="item",
42
-
title="Item",
43
-
columns=[
44
-
Column("num", pa.uint64(), primary_key=True),
45
-
Column("date", pa.date64()),
46
-
Column("title", pa.string()),
47
-
],
48
-
)
49
-
...
50
-
```
29
+
If you had an API endpoint https://api.example.com/items/{num} and for each value of `num` it provided an object
30
+
31
+
```json
32
+
{
33
+
"num": {{num}},
34
+
"date": "2023-10-12",
35
+
"title": "A simple example"
36
+
}
37
+
```
38
+
39
+
Then you would design the table class as
40
+
41
+
```python
42
+
classItems(Table):
43
+
def__init__(self) -> None:
44
+
super().__init__(
45
+
name="item",
46
+
title="Item",
47
+
columns=[
48
+
Column("num", pa.uint64(), primary_key=True),
49
+
Column("date", pa.date64()),
50
+
Column("title", pa.string()),
51
+
],
52
+
)
53
+
...
54
+
```
51
55
52
56
Creating one table for each endpoint that you want to capture.
53
57
54
58
### API Client
59
+
55
60
Next you'll need to define how the tables are retrieved, it's recommended to implement this as a generator, as per the example in `plugin/example/client.py`.
56
61
57
62
### Spec
63
+
58
64
Having written your API Client you will have, identified the authentication and/or operational variables needed.
59
65
Adding these to the CloudQuery config spec can be done by editing the `Spec``dataclass` using standard python, and adding validation where needed.
60
66
61
67
### Plugin
62
-
Finally, you need to edit the `plugin.py` file to set the plugin name and version, and add the `Tables` to the `get_tables` function.
68
+
69
+
Finally, you need to edit the `plugin.py` file to set the plugin name and version, and add the `Tables` to the `get_tables` function.
63
70
64
71
### Test run
72
+
65
73
To test your plugin you can run it locally.
66
74
67
75
To automatically manage your virtual environment and install the dependencies listed in the `pyproject.toml` you can use `poetry`.
@@ -75,6 +83,13 @@ Then to run the plugin `poetry run main serve`, which will launch the plugin man
75
83
With that running you can adjust the `TestConfig.yaml` to match your plugin and run `cloudquery sync`.
76
84
This should result in the creation of a sqlite database `db.sqlite` where you can validate your tables are as expected.
77
85
86
+
### Publishing
87
+
88
+
1. Update the plugin metadata in [plugin/plugin.py](plugin/plugin.py#L13) to match your team and plugin name.
89
+
2. Run `python main.py package -m "Initial release" "v0.0.1" .`. `-m` specifies changelog and `v0.0.1` is the version.
90
+
3. Run `cloudquery plugin publish -f` to publish the plugin to the CloudQuery registry.
91
+
92
+
> More about publishing plugins [here](https://docs.cloudquery.io/docs/developers/publishing-an-addon-to-the-hub)
0 commit comments