Skip to content

Commit 99cdbaa

Browse files
authored
Update queue.go
1 parent faa5074 commit 99cdbaa

File tree

1 file changed

+35
-32
lines changed

1 file changed

+35
-32
lines changed

modules/queue/queue.go

Lines changed: 35 additions & 32 deletions
Original file line numberDiff line numberDiff line change
@@ -2,61 +2,64 @@
22
// SPDX-License-Identifier: MIT
33

44
// Package queue implements a specialized concurrent queue system for Gitea.
5-
//
5+
66
// Terminology:
7-
//
8-
// 1. Task:
9-
// - A task can be a simple value, such as an integer, or a more complex structure that has multiple fields and
10-
// methods. The aim of a task is to be a unit of work, a set of tasks will be sent to a handler to be processed.
7+
8+
// 1. Item:
9+
// - An item can be a simple value, such as an integer, or a more complex structure that has multiple fields and
10+
// methods. Usually a item serves as a unit of work, sets of items will be sent to a handler to be processed.
11+
// - It's represented as a binary slice on the queue
12+
// - When an item serve as a unit of work generally it will be referred as a task
1113
//
1214
// 2. Batch:
13-
// - A collection of tasks that are grouped together for processing. Each worker receives a batch of tasks.
15+
// - A collection of items that are grouped together for processing. Each worker receives a batch of items.
1416
//
1517
// 3. Worker:
16-
// - Individual unit of execution designed to process tasks from the queue. It's a goroutine that calls the Handler
17-
// - Workers will get new tasks through a channel (WorkerPoolQueue is responsible for the distribution)
18-
// - As workers operate in parallel, the default value of max workers is n/2, where n is the number of logical CPUs
18+
// - Individual unit of execution designed to process items from the queue. It's a goroutine that calls the Handler.
19+
// - Workers will get new items through a channel (WorkerPoolQueue is responsible for the distribution).
20+
// - Workers operate in parallel. The default value of max workers is determined by the setting system.
1921
//
2022
// 4. Handler (represented by HandlerFuncT type):
21-
// - It's the function responsible to process tasks. Each active worker will call this.
22-
// - When processing these batches, there might be instances where certain tasks remain unprocessed or "unhandled".
23-
// In such scenarios, the Handler ensures these unhandled tasks are returned to the base queue after a brief delay.
24-
// This mechanism is particularly beneficial in cases where the processing entity (like a document indexer) is
25-
// temporarily unavailable. It ensures that no task is skipped or lost due to transient failures in the processing
26-
// mechanism.
23+
// - It's the function responsible for processing items. Each active worker will call it.
24+
// - When processing batches, there might be instances where certain items remain unprocessed or "unhandled".
25+
// In such scenarios, the Handler ensures these unhandled items are returned to the base queue after a brief delay.
26+
// This mechanism is particularly beneficial in cases where the processing entity (like a document indexer) is
27+
// temporarily unavailable. It ensures that no item is skipped or lost due to transient failures in the processing
28+
// mechanism.
2729
//
2830
// 5. Base queue:
29-
// - Represents the underlying storage mechanism for the queue. There are several implementations:
30-
// - Channel: Uses Go's native channel constructs to manage the queue, suitable for in-memory queuing.
31-
// - Level, Redis: Especially useful in persistent queues and clusters, where we may have multiple nodes.
32-
// - Dummy: This is special, it's not a real queue, it's only used as a no-op queue or a testing queue.
33-
// - They all have the same abstraction, the same interface, and they are tested by the same testing code.
31+
// - Represents the underlying storage mechanism for the queue. There are several implementations:
32+
// - Channel: Uses Go's native channel constructs to manage the queue, suitable for in-memory queuing.
33+
// - LevelDB: Especially useful in persistent queues for single instances.
34+
// - Redis: Suitable for clusters, where we may have multiple nodes.
35+
// - Dummy: This is special, it's not a real queue, it's a immediate no-op queue, which is useful for tests.
36+
// - They all have the same abstraction, the same interface, and they are tested by the same testing code.
3437
//
3538
// 6. WorkerPoolQueue:
36-
// - It's responsible to glue all together, using the "base queue" to provide "worker pool" functionality. He creates
37-
// new workers if needed and can flush the queue, running all the tasks synchronously till it finishes.
39+
// - It's responsible to glue all together, using the "base queue" to provide "worker pool" functionality. It creates
40+
// new workers if needed and can flush the queue, running all the items synchronously till it finishes.
3841
// - Its "Push" function doesn't block forever, it will return an error if the queue is full after the timeout.
3942
//
4043
// 7. Manager:
4144
// - The purpose of it is to serve as a centralized manager for multiple WorkerPoolQueue instances. Whenever we want
42-
// to create a new queue, flush, or get a specific queue, we have to use it.
45+
// to create a new queue, flush, or get a specific queue, we have to use it.
4346
//
4447
// A queue can be "simple" or "unique". A unique queue will try to avoid duplicate items.
4548
// Unique queue's "Has" function can be used to check whether an item is already in the queue,
46-
// although it's not 100% reliable due to there is no proper transaction support.
49+
// although it's not 100% reliable due to the lack of proper transaction support.
4750
// Simple queue's "Has" function always returns "has=false".
4851
//
4952
// A WorkerPoolQueue is a generic struct; this means it will work with any type but just for that type.
50-
// If you want another kind of tasks to run, you would have to call the manager to create a new WorkerPoolQueue for you
51-
// with a different handler that works with this new type of task. As an example of this:
53+
// If you want another kind of items to run, you would have to call the manager to create a new WorkerPoolQueue for you
54+
// with a different handler that works with this new type of item. As an example of this:
5255
//
53-
// func Init() error {
54-
// taskQueue = queue.CreateSimpleQueue(graceful.GetManager().ShutdownContext(), "task", handler)
55-
// ...
56-
// }
57-
// func handler(items ...*admin_model.Task) []*admin_model.Task { ... }
56+
// func Init() error {
57+
// itemQueue = queue.CreateSimpleQueue(graceful.GetManager().ShutdownContext(), "item", handler)
58+
// ...
59+
// }
60+
// func handler(items ...*admin_model.Item) []*admin_model.Item { ... }
5861
//
59-
// As you can see, the handler defined the admin_model.Task type for the queue
62+
// As you can see, the handler defined the admin_model.Item type for the queue
6063
package queue
6164

6265
import "code.gitea.io/gitea/modules/util"

0 commit comments

Comments
 (0)