|
2 | 2 | // SPDX-License-Identifier: MIT
|
3 | 3 |
|
4 | 4 | // Package queue implements a specialized concurrent queue system for Gitea.
|
5 |
| -// |
| 5 | + |
6 | 6 | // Terminology:
|
7 |
| -// |
8 |
| -// 1. Task: |
9 |
| -// - A task can be a simple value, such as an integer, or a more complex structure that has multiple fields and |
10 |
| -// methods. The aim of a task is to be a unit of work, a set of tasks will be sent to a handler to be processed. |
| 7 | + |
| 8 | +// 1. Item: |
| 9 | +// - An item can be a simple value, such as an integer, or a more complex structure that has multiple fields and |
| 10 | +// methods. Usually a item serves as a unit of work, sets of items will be sent to a handler to be processed. |
| 11 | +// - It's represented as a binary slice on the queue |
| 12 | +// - When an item serve as a unit of work generally it will be referred as a task |
11 | 13 | //
|
12 | 14 | // 2. Batch:
|
13 |
| -// - A collection of tasks that are grouped together for processing. Each worker receives a batch of tasks. |
| 15 | +// - A collection of items that are grouped together for processing. Each worker receives a batch of items. |
14 | 16 | //
|
15 | 17 | // 3. Worker:
|
16 |
| -// - Individual unit of execution designed to process tasks from the queue. It's a goroutine that calls the Handler |
17 |
| -// - Workers will get new tasks through a channel (WorkerPoolQueue is responsible for the distribution) |
18 |
| -// - As workers operate in parallel, the default value of max workers is n/2, where n is the number of logical CPUs |
| 18 | +// - Individual unit of execution designed to process items from the queue. It's a goroutine that calls the Handler. |
| 19 | +// - Workers will get new items through a channel (WorkerPoolQueue is responsible for the distribution). |
| 20 | +// - Workers operate in parallel. The default value of max workers is determined by the setting system. |
19 | 21 | //
|
20 | 22 | // 4. Handler (represented by HandlerFuncT type):
|
21 |
| -// - It's the function responsible to process tasks. Each active worker will call this. |
22 |
| -// - When processing these batches, there might be instances where certain tasks remain unprocessed or "unhandled". |
23 |
| -// In such scenarios, the Handler ensures these unhandled tasks are returned to the base queue after a brief delay. |
24 |
| -// This mechanism is particularly beneficial in cases where the processing entity (like a document indexer) is |
25 |
| -// temporarily unavailable. It ensures that no task is skipped or lost due to transient failures in the processing |
26 |
| -// mechanism. |
| 23 | +// - It's the function responsible for processing items. Each active worker will call it. |
| 24 | +// - When processing batches, there might be instances where certain items remain unprocessed or "unhandled". |
| 25 | +// In such scenarios, the Handler ensures these unhandled items are returned to the base queue after a brief delay. |
| 26 | +// This mechanism is particularly beneficial in cases where the processing entity (like a document indexer) is |
| 27 | +// temporarily unavailable. It ensures that no item is skipped or lost due to transient failures in the processing |
| 28 | +// mechanism. |
27 | 29 | //
|
28 | 30 | // 5. Base queue:
|
29 |
| -// - Represents the underlying storage mechanism for the queue. There are several implementations: |
30 |
| -// - Channel: Uses Go's native channel constructs to manage the queue, suitable for in-memory queuing. |
31 |
| -// - Level, Redis: Especially useful in persistent queues and clusters, where we may have multiple nodes. |
32 |
| -// - Dummy: This is special, it's not a real queue, it's only used as a no-op queue or a testing queue. |
33 |
| -// - They all have the same abstraction, the same interface, and they are tested by the same testing code. |
| 31 | +// - Represents the underlying storage mechanism for the queue. There are several implementations: |
| 32 | +// - Channel: Uses Go's native channel constructs to manage the queue, suitable for in-memory queuing. |
| 33 | +// - LevelDB: Especially useful in persistent queues for single instances. |
| 34 | +// - Redis: Suitable for clusters, where we may have multiple nodes. |
| 35 | +// - Dummy: This is special, it's not a real queue, it's a immediate no-op queue, which is useful for tests. |
| 36 | +// - They all have the same abstraction, the same interface, and they are tested by the same testing code. |
34 | 37 | //
|
35 | 38 | // 6. WorkerPoolQueue:
|
36 |
| -// - It's responsible to glue all together, using the "base queue" to provide "worker pool" functionality. He creates |
37 |
| -// new workers if needed and can flush the queue, running all the tasks synchronously till it finishes. |
| 39 | +// - It's responsible to glue all together, using the "base queue" to provide "worker pool" functionality. It creates |
| 40 | +// new workers if needed and can flush the queue, running all the items synchronously till it finishes. |
38 | 41 | // - Its "Push" function doesn't block forever, it will return an error if the queue is full after the timeout.
|
39 | 42 | //
|
40 | 43 | // 7. Manager:
|
41 | 44 | // - The purpose of it is to serve as a centralized manager for multiple WorkerPoolQueue instances. Whenever we want
|
42 |
| -// to create a new queue, flush, or get a specific queue, we have to use it. |
| 45 | +// to create a new queue, flush, or get a specific queue, we have to use it. |
43 | 46 | //
|
44 | 47 | // A queue can be "simple" or "unique". A unique queue will try to avoid duplicate items.
|
45 | 48 | // Unique queue's "Has" function can be used to check whether an item is already in the queue,
|
46 |
| -// although it's not 100% reliable due to there is no proper transaction support. |
| 49 | +// although it's not 100% reliable due to the lack of proper transaction support. |
47 | 50 | // Simple queue's "Has" function always returns "has=false".
|
48 | 51 | //
|
49 | 52 | // A WorkerPoolQueue is a generic struct; this means it will work with any type but just for that type.
|
50 |
| -// If you want another kind of tasks to run, you would have to call the manager to create a new WorkerPoolQueue for you |
51 |
| -// with a different handler that works with this new type of task. As an example of this: |
| 53 | +// If you want another kind of items to run, you would have to call the manager to create a new WorkerPoolQueue for you |
| 54 | +// with a different handler that works with this new type of item. As an example of this: |
52 | 55 | //
|
53 |
| -// func Init() error { |
54 |
| -// taskQueue = queue.CreateSimpleQueue(graceful.GetManager().ShutdownContext(), "task", handler) |
55 |
| -// ... |
56 |
| -// } |
57 |
| -// func handler(items ...*admin_model.Task) []*admin_model.Task { ... } |
| 56 | +// func Init() error { |
| 57 | +// itemQueue = queue.CreateSimpleQueue(graceful.GetManager().ShutdownContext(), "item", handler) |
| 58 | +// ... |
| 59 | +// } |
| 60 | +// func handler(items ...*admin_model.Item) []*admin_model.Item { ... } |
58 | 61 | //
|
59 |
| -// As you can see, the handler defined the admin_model.Task type for the queue |
| 62 | +// As you can see, the handler defined the admin_model.Item type for the queue |
60 | 63 | package queue
|
61 | 64 |
|
62 | 65 | import "code.gitea.io/gitea/modules/util"
|
|
0 commit comments