Skip to content

Commit 7d11123

Browse files
Add sequin guide (#1368)
Co-authored-by: James Ritchie <[email protected]>
1 parent 8da495a commit 7d11123

File tree

10 files changed

+477
-0
lines changed

10 files changed

+477
-0
lines changed

docs/guides/frameworks/introduction.mdx

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -10,11 +10,14 @@ import CardNodejs from "/snippets/card-nodejs.mdx";
1010
import CardNextjs from "/snippets/card-nextjs.mdx";
1111
import CardRemix from "/snippets/card-remix.mdx";
1212
import CardSupabase from "/snippets/card-supabase.mdx";
13+
import CardSequin from "/snippets/card-sequin.mdx";
1314

1415
<CardGroup cols={3}>
1516
<CardBun />
1617
<CardNodejs />
1718
<CardNextjs />
1819
<CardRemix />
20+
<CardSequin />
1921
<CardSupabase />
22+
2023
</CardGroup>

docs/guides/frameworks/sequin.mdx

Lines changed: 319 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,319 @@
1+
---
2+
title: "Sequin database triggers"
3+
sidebarTitle: "Sequin"
4+
description: "This guide will show you how to trigger tasks from database changes using Sequin"
5+
icon: "database"
6+
---
7+
8+
[Sequin](https://sequinstream.com) allows you to trigger tasks from database changes. Sequin captures every insert, update, and delete on a table and then ensures a task is triggered for each change.
9+
10+
Often, task runs coincide with database changes. For instance, you might want to use a Trigger.dev task to generate an embedding for each post in your database:
11+
12+
<Frame>
13+
<img src="/images/sequin-intro.svg" alt="Sequin and Trigger.dev Overview" />
14+
</Frame>
15+
16+
In this guide, you'll learn how to use Sequin to trigger Trigger.dev tasks from database changes.
17+
18+
## Prerequisites
19+
20+
You are about to create a [regular Trigger.dev task](/tasks-regular) that you will execute when ever a post is inserted or updated in your database. Sequin will detect all the changes on the `posts` table and then send the payload of the post to an API endpoint that will call `tasks.trigger()` to create the embedding and update the database.
21+
22+
As long as you create an HTTP endpoint that Sequin can deliver webhooks to, you can use any web framework or edge function (e.g. Supabase Edge Functions, Vercel Functions, Cloudflare Workers, etc.) to invoke your Trigger.dev task. In this guide, we'll show you how to setup Trigger.dev tasks using Next.js API Routes.
23+
24+
You'll need the following to follow this guide:
25+
26+
- A Next.js project with [Trigger.dev](https://trigger.dev) installed
27+
<Info>
28+
If you don't have one already, follow [Trigger.dev's Next.js setup guide](/guides/frameworks/nextjs) to setup your project. You can return to this guide when you're ready to write your first Trigger.dev task.
29+
</Info>
30+
- A [Sequin](https://console.sequinstream.com/register) account
31+
- A Postgres database (Sequin works with any Postgres database version 12 and up) with a `posts` table.
32+
33+
## Create a Trigger.dev task
34+
35+
Start by creating a new Trigger.dev task that takes in a Sequin change event as a payload, creates an embedding, and then inserts the embedding into the database:
36+
37+
<Steps titleSize="h3">
38+
<Step title="Create a `create-embedding-for-post` task">
39+
In your `src/trigger/tasks` directory, create a new file called `create-embedding-for-post.ts` and add the following code:
40+
41+
<CodeGroup>
42+
```ts trigger/create-embedding-for-post.ts
43+
import { task } from "@trigger.dev/sdk/v3";
44+
import { OpenAI } from "openai";
45+
import { upsertEmbedding } from "../util";
46+
47+
const openai = new OpenAI({
48+
apiKey: process.env.OPENAI_API_KEY,
49+
});
50+
51+
export const createEmbeddingForPost = task({
52+
id: "create-embedding-for-post",
53+
run: async (payload: {
54+
record: {
55+
id: number;
56+
title: string;
57+
body: string;
58+
author: string;
59+
createdAt: string;
60+
embedding: string | null;
61+
},
62+
metadata: {
63+
table_schema: string,
64+
table_name: string,
65+
consumer: {
66+
id: string;
67+
name: string;
68+
};
69+
};
70+
}) => {
71+
// Create an embedding using the title and body of payload.record
72+
const content = `${payload.record.title}\n\n${payload.record.body}`;
73+
const embedding = (await openai.embeddings.create({
74+
model: "text-embedding-ada-002",
75+
input: content,
76+
})).data[0].embedding;
77+
78+
// Upsert the embedding in the database. See utils.ts for the implementation -> ->
79+
await upsertEmbedding(embedding, payload.record.id);
80+
81+
// Return the updated record
82+
return {
83+
...payload.record,
84+
embedding: JSON.stringify(embedding),
85+
};
86+
}
87+
});
88+
```
89+
90+
```ts utils.ts
91+
import pg from "pg";
92+
93+
export async function upsertEmbedding(embedding: number[], id: number) {
94+
const client = new pg.Client({
95+
connectionString: process.env.DATABASE_URL,
96+
});
97+
await client.connect();
98+
99+
try {
100+
const query = `
101+
INSERT INTO post_embeddings (id, embedding)
102+
VALUES ($2, $1)
103+
ON CONFLICT (id)
104+
DO UPDATE SET embedding = $1
105+
`;
106+
const values = [JSON.stringify(embedding), id];
107+
108+
const result = await client.query(query, values);
109+
console.log(`Updated record in database. Rows affected: ${result.rowCount}`);
110+
111+
return result.rowCount;
112+
} catch (error) {
113+
console.error("Error updating record in database:", error);
114+
throw error;
115+
} finally {
116+
await client.end();
117+
}
118+
}
119+
```
120+
</CodeGroup>
121+
122+
This task takes in a Sequin record event, creates an embedding, and then uppserts the embedding into a `post_embeddings` table.
123+
</Step>
124+
<Step title="Add the task to your Trigger.dev project">
125+
Register the `create-embedding-for-post` task to your Trigger.dev cloud project by running the following command:
126+
127+
```bash
128+
npx trigger.dev@latest dev
129+
```
130+
131+
In the Trigger.dev dashboard, you should now see the `create-embedding-for-post` task:
132+
133+
<Frame>
134+
<img src="/images/sequin-register-task.png" alt="Task added" />
135+
</Frame>
136+
</Step>
137+
</Steps>
138+
139+
<Check>
140+
You've successfully created a Trigger.dev task that will create an embedding for each post in your database. In the next step, you'll create an API endpoint that Sequin can deliver records to.
141+
</Check>
142+
143+
## Setup API route
144+
145+
You'll now create an API endpoint that will receive posts from Sequin and then trigger the `create-embedding-for-post` task.
146+
147+
<Info>
148+
This guide covers how to setup an API endpoint using the Next.js App Router. You can find examples for Next.js Server Actions and Pages Router in the [Trigger.dev documentation](https://trigger.dev/docs/guides/frameworks/nextjs).
149+
</Info>
150+
151+
<Steps titleSize="h3">
152+
<Step title="Create a route handler">
153+
Add a route handler by creating a new `route.ts` file in a `/app/api/create-embedding-for-post` directory:
154+
155+
```ts app/api/create-embedding-for-post/route.ts
156+
import type { createEmbeddingForPost } from "@/trigger/create-embedding-for-post";
157+
import { tasks } from "@trigger.dev/sdk/v3";
158+
import { NextResponse } from "next/server";
159+
160+
export async function POST(req: Request) {
161+
const authHeader = req.headers.get('authorization');
162+
if (!authHeader || authHeader !== `Bearer ${process.env.SEQUIN_WEBHOOK_SECRET}`) {
163+
return NextResponse.json({ error: 'Unauthorized' }, { status: 401 });
164+
}
165+
const payload = await req.json();
166+
const handle = await tasks.trigger<typeof createEmbeddingForPost>(
167+
"create-embedding-for-post",
168+
payload
169+
);
170+
171+
return NextResponse.json(handle);
172+
}
173+
```
174+
175+
This route handler will receive records from Sequin, parse them, and then trigger the `create-embedding-for-post` task.
176+
</Step>
177+
<Step title="Set secret keys">
178+
You'll need to set four secret keys in a `.env.local` file:
179+
180+
```bash
181+
SEQUIN_WEBHOOK_SECRET=your-secret-key
182+
TRIGGER_SECRET_KEY=secret-from-trigger-dev
183+
OPENAI_API_KEY=sk-proj-asdfasdfasdf
184+
DATABASE_URL=postgresql://
185+
```
186+
187+
The `SEQUIN_WEBHOOK_SECRET` ensures that only Sequin can access your APIendpoint.
188+
189+
The `TRIGGER_SECRET_KEY` is used to authenticate requests to Trigger.dev and can be found in the **API keys** tab of the Trigger.dev dashboard.
190+
191+
The `OPENAI_API_KEY` and `DATABASE_URL` are used to create an embedding using OpenAI and connect to your database. Be sure to add these as [environment variables](https://trigger.dev/docs/deploy-environment-variables) in Trigger.dev as well.
192+
</Step>
193+
</Steps>
194+
195+
<Check>
196+
You've successfully created an API endpoint that can receive record payloads from Sequin and trigger a Trigger.dev task. In the next step, you'll setup Sequin to trigger the endpoint.
197+
</Check>
198+
199+
## Create Sequin consumer
200+
201+
You'll now configure Sequin to send every row in your `posts` table to your Trigger.dev task.
202+
203+
<Steps titleSize="h3">
204+
<Step title="Connect Sequin to your database">
205+
1. Login to your Sequin account and click the **Add New Database** button.
206+
2. Enter the connection details for your Postgres database.
207+
<Info>
208+
If you need to connect to a local dev database, flip the **use localhost** switch and follow the instructions to create a tunnel using the [Sequin CLI](/cli).
209+
</Info>
210+
3. Follow the instructions to create a publication and a replication slot by running two SQL commands in your database:
211+
212+
```sql
213+
create publication sequin_pub for all tables;
214+
select pg_create_logical_replication_slot('sequin_slot', 'pgoutput');
215+
```
216+
217+
4. Name your database and click the **Connect Database** button.
218+
219+
Sequin will connect to your database and ensure that it's configured properly.
220+
221+
<Note>
222+
If you need step-by-step connection instructions to connect Sequin to your database, check out our [quickstart guide](/quickstart).
223+
</Note>
224+
</Step>
225+
<Step title="Tunnel to your local endpoint">
226+
Now, create a tunnel to your local endpoint so Sequin can deliver change payloads to your local API:
227+
228+
1. In the Sequin console, open the **HTTP Endpoint** tab and click the **Create HTTP Endpoint** button.
229+
2. Enter a name for your endpoint (i.e. `local_endpoint`) and flip the **Use localhost** switch. Follow the instructions in the Sequin console to [install the Sequin CLI](/cli), then run:
230+
231+
```bash
232+
sequin tunnel --ports=3001:local_endpoint
233+
```
234+
235+
3. Now, click **Add encryption header** and set the key to `Authorization` and the value to `Bearer SEQUIN_WEBHOOK_SECRET`.
236+
4. Click **Create HTTP Endpoint**.
237+
</Step>
238+
<Step title="Create a Push Consumer">
239+
Create a push consumer that will capture posts from your database and deliver them to your local endpoint:
240+
241+
1. Navigate to the **Consumers** tab and click the **Create Consumer** button.
242+
2. Select your `posts` table (i.e `public.posts`).
243+
3. You want to ensure that every post receives an embedding - and that embeddings are updated as posts are updated. To do this, select to process **Rows** and click **Continue**.
244+
<Note>
245+
You can also use **changes** for this particular use case, but **rows** comes with some nice replay and backfill features.
246+
</Note>
247+
4. You'll now set the sort and filter for the consumer. For this guide, we'll sort by `updated_at` and start at the beginning of the table. We won't apply any filters:
248+
<Frame>
249+
<img src="/images/sequin-sort-and-filter.png" alt="Consumer Sort and Filter" />
250+
</Frame>
251+
5. On the next screen, select **Push** to have Sequin send the events to your webhook URL. Click **Continue**.
252+
6. Now, give your consumer a name (i.e. `posts_push_consumer`) and in the **HTTP Endpoint** section select the `local_endpoint` you created above. Add the exact API route you created in the previous step (i.e. `/api/create-embedding-for-post`):
253+
<Frame>
254+
<img src="/images/sequin-consumer-config.png" alt="Consumer Endpoint" />
255+
</Frame>
256+
7. Click the **Create Consumer** button.
257+
</Step>
258+
</Steps>
259+
260+
<Check>
261+
Your Sequin consumer is now created and ready to send events to your API endpoint.
262+
</Check>
263+
264+
## Test end-to-end
265+
266+
<Steps titleSize="h3">
267+
<Step title="Spin up you dev environment">
268+
1. The Next.js app is running: `npm run dev`
269+
2. The Trigger.dev dev server is running `npx trigger.dev@latest dev`
270+
3. The Sequin tunnel is running: `sequin tunnel --ports=3001:local_endpoint`
271+
</Step>
272+
<Step title="Create a new post in your database">
273+
274+
```sql
275+
insert into
276+
posts (title, body, author)
277+
values
278+
(
279+
'The Future of AI',
280+
'An insightful look into how artificial intelligence is shaping the future of technology and society.',
281+
'Alice H Johnson'
282+
);
283+
```
284+
</Step>
285+
<Step title="Trace the change in the Sequin dashboard">
286+
In the Sequin console, navigate to the [**Trace**](https://console.sequinstream.com/trace) tab and confirm that Sequin delivered the event to your local endpoint:
287+
<Frame>
288+
<img src="/images/sequin-trace.png" alt="Trace Event" />
289+
</Frame>
290+
</Step>
291+
292+
<Step title="Confirm the event was received by your endpoint">
293+
In your local terminal, you should see a `200` response in your Next.js app:
294+
295+
```bash
296+
POST /api/create-embedding-for-post 200 in 262ms
297+
```
298+
</Step>
299+
<Step title="Observe the task run in the Trigger.dev dashboard">
300+
Finally, in the Trigger.dev dashboard, navigate to the [**Runs**](https://trigger.dev/runs) tab and confirm that the task run completed successfully:
301+
302+
<Frame>
303+
<img src="/images/sequin-final-run.png" alt="Task run" />
304+
</Frame>
305+
</Step>
306+
</Steps>
307+
308+
<Check>
309+
Every time a post is created or updated, Sequin will deliver the row payload to your API endpoint and Trigger.dev will run the `create-embedding-for-post` task.
310+
</Check>
311+
312+
## Next steps
313+
314+
With Sequin and Trigger.dev, every post in your database will now have an embedding. This is a simple example of how you can trigger long-running tasks on database changes.
315+
316+
From here, add error handling and deploy to production:
317+
318+
- Add [retries](/errors-retrying) to your Trigger.dev task to ensure that any errors are captured and logged.
319+
- Deploy to [production](/guides/frameworks/nextjs#deploying-your-task-to-trigger-dev) and update your Sequin consumer to point to your production database and endpoint.
204 KB
Loading

docs/images/sequin-final-run.png

516 KB
Loading

0 commit comments

Comments
 (0)