Spring Boot Redis Stream Consumer with ThreadPoolTaskScheduler Integration

📖 Overview

This Spring Boot application serves as a resilient Redis Stream consumer for processing order payment events (PAYMENT_SUCCESS and PAYMENT_FAILED). It leverages ThreadPoolTaskScheduler to schedule stream polling at fixed intervals, ensuring reliable and scalable event-driven processing.

Unlike traditional Pub/Sub models, this implementation uses Redis Streams with Consumer Groups to guarantee message durability and exactly-once processing semantics. Messages are retained in Redis until they are explicitly acknowledged. Redis interprets the acknowledgment as: this message was correctly processed so it can be evicted from the consumer group.

💡 Why Redis Streams?

Unlike Pub/Sub, Redis Streams offer:

Persistence – Messages are stored in Redis until explicitly acknowledged by a consumer.
Reliability – Ensures that no messages are lost — perfect for critical systems like payments.
Scalability – Built-in support for consumer groups and horizontal scaling.
Replayability – Failed or pending messages can be retried, replayed, or analyzed.

💡 What is a Consumer Group in Redis Streams?

A Consumer Group in Redis Streams is a mechanism for distributing and managing data consumption by multiple consumers in a parallel and coordinated manner. While XREAD (regular) is suitable for a single consumer, a Consumer Group (XREADGROUP) is ideal for multiple consumers processing the stream together. A Consumer Group allows multiple consumers to share the workload of processing messages without duplication. Each message is delivered to only one consumer in the group.

💡 Why Do We Need Consumer Groups?

To enable multiple consumers to collaborate in processing messages.
To track which messages have been read and which are still pending.
To retry processing if a message fails.
To ensure each message is read by only one consumer, unlike Pub/Sub where all consumers receive the same message.

💡 How Consumer Groups Work?

A stream is created (XADD).
A consumer group is created (XGROUP CREATE).
Multiple consumers join the group and start processing messages (XREADGROUP).
Messages are assigned to a consumer within the group (only one consumer gets each message).
Consumers acknowledge (XACK) processed messages, so Redis knows they are handled.

📌 Redis Stream Message ID (RecordId)

Each message published to a Redis stream is assigned a unique RecordId which acts as its primary identifier in the stream. This ID is composed of a timestamp and sequence number, ensuring uniqueness even for multiple messages added in the same millisecond.

💡 What is PEL?

The Pending Entries List (PEL) in Redis Streams acts as a queue of "messages being processed" for each consumer group. Redis tracks the following details for each message in the PEL:

Message ID: The unique identifier of the message.
Consumer: The consumer currently processing the message.
Delivery Count: The number of times the message has been delivered.
Last Delivery Timestamp: The timestamp of the last delivery attempt.

Note: If a consumer reads a message but does not acknowledge it (XACK), the message remains in the PEL. This ensures that unprocessed or failed messages can be retried or reassigned to another consumer.

💡 What is Dead Letter Queue (DLQ)?

A Dead Letter Queue is a separate stream (or queue) where you put messages that failed processing after a certain number of retries — so they don’t keep clogging your main stream.

This allows you to:

Avoid infinite retry loops.
Investigate or alert on problematic messages.
Optionally reprocess them later (manually or scheduled).

🧩 Event-Driven Architecture

This project implements a reliable event-driven architecture using Redis Streams to handle Order Payment creation and processing. Below is a breakdown of the full flow:

[Client]──▶ (HTTP POST /order-payment)──▶ [Spring Boot API] ──▶ [Redis Stream: PAYMENT_SUCCESS or PAYMENT_FAILED]
                                                                                         │
                                                                                         ▼
                                                                            [StreamConsumer - every 5s]
                                                                                         │
                                                                        ┌────────────────┬──────────────────────┐
                                                                        ▼                ▼                      ▼
                                                                [✔ Processed]   [🔁Retry Queue]  [❌ DLQ (failed after max attempts)]

🔄 How It Works

Client Request
A client sends an HTTP POST request to the /order-payment endpoint with the necessary order payment data.
Spring Boot API (Producer)

The API receives the request, processes the initial business logic, and then publishes a message to the appropriate Redis stream:
- PAYMENT_SUCCESS if the payment is successful.
- PAYMENT_FAILED if the payment fails validation or processing.
Each message sent to the stream includes a manually generated RecordId, ensuring consistent tracking and ordering.

Redis Streams

Redis Streams persist these messages until they are acknowledged by a consumer.
This allows for reliable message delivery, replay, and tracking of pending/unprocessed messages.

StreamConsumer (Scheduled Every 5 Seconds)

A scheduled consumer job runs every 5 seconds, using XREADGROUP to read new or pending messages from the stream as part of a consumer group.
It attempts to process each message accordingly:
- ✅Processed Successfully: The consumer handles the message and sends an XACK to acknowledge its completion. The message is then removed from the pending list.
- 🔁Retry Queue: If processing fails temporarily, the message is not acknowledged, allowing it to be retried in the next cycle. If its idle time exceeds a threshold, the consumer can reclaim the message for retry using XCLAIM.
- ❌Dead Letter Queue (DLQ): If the message fails after exceeding the maximum delivery attempts, it is moved to a DLQ stream for manual inspection, alerting, or later analysis.

🚀 Features

Below are the core features that make this solution robust and ready for real-world scenarios:

StreamConsumer processes messages in real-time every 5 seconds using scheduler
Acknowledgment (XACK) after successful processing
Retry mechanism for unacknowledged or failed messages
Dead Letter Queue (DLQ) support for messages that exceed retry limits

⏱ Scheduled Stream Consumers

The application defines the following scheduled tasks, each executed at a fixed rate:

handlePaymentSuccess – Reads and processes new messages from the PAYMENT_SUCCESS stream.
handlePaymentFailed – Reads and processes new messages from the PAYMENT_FAILED stream.
retryPaymentSuccess – Handles unacknowledged messages from the PAYMENT_SUCCESS stream by inspecting the Pending Entries List (PEL) and reprocessing eligible entries.
retryPaymentFailed – Performs retry logic for messages in the PAYMENT_FAILED stream that remain unacknowledged.

📥 Message Consumer Responsibilities

Read Messages – Uses the XREADGROUP command to consume messages from Redis Streams under a specific consumer group. The message is then routed for processing based on its originating stream (success or failure).
retryPendingMessages – Processes entries from the PEL using XPENDING and XCLAIM:

Detects messages that have exceeded the maximum idle time.
Reclaims the message to the current consumer (retry consumer) for retry.
Tracks the delivery count of each message.
If a message exceeds the maximum delivery attempts, it is routed to a Dead Letter Queue (DLQ) to prevent infinite retries and allow for manual intervention or alerting.

🤖 Tech Stack

The technology used in this project are:

Technology	Description
Spring Boot Starter Web	Building RESTful APIs or web applications. Used in this project to handle HTTP requests for creating and managing order payments.
Spring Data Redis (Lettuce)	A high-performance Redis client built on Netty. Enables producing/consuming Redis Streams via Spring abstraction.
RedisTemplate	Abstraction provided by Spring Data Redis for performing Redis operations like `XADD`, `XREADGROUP`, `XACK`, etc.
ThreadPoolTaskScheduler	Used for scheduling background tasks with support for multiple threads.
Lombok	Reduces boilerplate code by auto-generating getters, setters, constructors, and more via annotations.

🧱 Architecture Overview

The project is organized into the following package structure:

📁 redis-stream-consumer/
└── 📂src/
    └── 📂main/
        ├── 📂docker/
        │   └── 📂app/                     # Dockerfile for Spring Boot application (runtime container)
        ├── 📂java/
        │   ├── 📂config/                  # Spring configuration classes
        │   │   ├── 📂redis/               # Redis-specific configuration (e.g., RedisTemplate, Lettuce client setup)
        │   │   └── 📂scheduler/           # Configuration related to task scheduling, such as setting up ThreadPoolTaskScheduler
        │   ├── 📂mapper/                  # Data mappers or converters, mapping between entity and DTOs or other representations
        │   ├── 📂redis/                   # Manages Redis stream message consumption, including reading messages from `PAYMENT_SUCCESS` and `PAYMENT_FAILED` streams using consumer groups.
        │   ├── 📂scheduler/               # Defines scheduled tasks using `ThreadPoolTaskScheduler` to trigger stream reading and retry logic at a fixed rate.
        │   └── 📂service/                 # Encapsulates business logic required to process and act on the consumed payment messages, such as persisting the data, updating order statuses, or notifying external systems.
        │       └── 📂impl/                # Implementation of services
        └── 📂resources/
            └── application.properties     # Application configuration (redis, profiles, etc.)

🛠️ Installation & Setup

Follow these steps to set up and run the project locally:

✅ Prerequisites

Make sure the following tools are installed on your system:

Tool	Description	Required
Java 17+	Java Development Kit (JDK) to run the Spring application	✅
Redis	In-memory data structure store used as a message broker via Streams	✅
Make	Automation tool for tasks like `make run-app`	✅
Docker	To run services like Redis in isolated containers	⚠️ optional

☕ 1. Install Java 17

Ensure Java 17 is installed on your system. You can verify this with:

java --version

If Java is not installed, follow one of the methods below based on your operating system:

🐧 Linux

Using apt (Ubuntu/Debian-based):

sudo apt update
sudo apt install openjdk-17-jdk

🪟 Windows

Use https://adoptium.net to download and install Java 17 (Temurin distribution recommended).
After installation, ensure JAVA_HOME is set correctly and added to the PATH.
You can check this with:

echo $JAVA_HOME

📨 2. Install Redis

Redis doesn't provide official support for Windows, but you can run it via WSL(Windows Subsystem for Linux) or Docker:

🐧 Linux

Using apt (Ubuntu/Debian-based):

sudo apt update
sudo apt install redis

Start Redis:

sudo systemctl start redis

Enable on boot:

sudo systemctl enable redis

Test:

redis-cli ping
# PONG

(Optional) If you want to add a specific user with access to a specific stream channel, you can run the following command in Redis CLI:

redis-cli
ACL SETUSER spring_producer on >P@ssw0rd ~stream:* +xadd +ping
ACL SETUSER spring_consumer on >P@ssw0rd ~stream:* +xread +xreadgroup +xack +ping

Set up Redis user and password in application.properties:

# Redis properties
spring.data.redis.username=spring_producer
spring.data.redis.password=P@ssw0rd

🧰 3. Install `make` (Optional but Recommended)

This project uses a Makefile to streamline common tasks.

Install make if not already available:

🐧 Linux

Install make using APT

sudo apt update
sudo apt install make

You can verify installation with:

make --version

🪟 Windows

If you're using PowerShell:

Install Chocolatey (if not installed):

Set-ExecutionPolicy Bypass -Scope Process -Force; [System.Net.ServicePointManager]::SecurityProtocol = [System.Net.SecurityProtocolType]::Tls12; iex ((New-Object System.Net.WebClient).DownloadString('https://community.chocolatey.org/install.ps1'))

Verify Chocolatey installation:

choco --version

Install make via Chocolatey:

choco install make

After installation, restart your terminal or ensure make is available in your PATH.

🔁 4. Clone the Project

Clone the repository:

git clone https://github.com/yoanesber/Spring-Boot-Redis-Stream-Consumer.git
cd Spring-Boot-Redis-Stream-Consumer

⚙️ 5. Configure Application Properties

Set up your application.properties in src/main/resources:

# Application properties
spring.application.name=redis-stream-consumer
server.port=8083
spring.profiles.active=development

# Redis configuration
spring.data.redis.host=localhost
spring.data.redis.port=6379
spring.data.redis.username=default
spring.data.redis.password=
spring.data.redis.timeout=10
spring.data.redis.connect-timeout=3
spring.data.redis.lettuce.shutdown-timeout=10

# Task Scheduler properties
spring.task.scheduling.pool.size=5
spring.task.scheduling.thread-name-prefix=thread-
spring.task.scheduling.remove-on-cancel-policy=false
spring.task.scheduling.wait-for-tasks-to-complete-on-shutdown=true
spring.task.scheduling.await-termination-seconds=10
spring.task.scheduling.thread-priority=5
spring.task.scheduling.continue-existing-periodic-tasks-after-shutdown-policy=false
spring.task.scheduling.execute-existing-delayed-tasks-after-shutdown-policy=true
spring.task.scheduling.daemon=false

🔐 Notes: Ensure that:
- Redis username, and password are correct.

🚀 6. Running the Application

This section provides step-by-step instructions to run the application either locally or via Docker containers.

Notes:
- All commands are defined in the Makefile.
- To run using make, ensure that make is installed on your system.
- To run the application in containers, make sure Docker is installed and running.

🔧 Run Locally (Non-containerized)

Ensure Redis are running locally, then:

make dev

🐳 Run Using Docker

To build and run all services (Redis, Spring app):

make docker-up

To stop and remove all containers:

make docker-down

Notes:
- Before running the application inside Docker, make sure to update your application.properties
  - Replace localhost with the appropriate container name for services like Redis.
    - For example:
      - Change spring.data.redis.host=localhost to spring.data.redis.host=redis-stream-server
- Make sure the Redis Stream Producer is already running in Docker before starting this consumer service.
  - You can find the producer repository here:
    https://github.com/yoanesber/Spring-Boot-Redis-Stream-Producer

🟢 Application is Running

Now your application is accessible at:

http://localhost:8083

🧪 Testing Scenarios

Below are the descriptions and outcomes of each test scenario along with visual evidence (captured screenshots) to demonstrate the flow and results.

Verify Successful Stream Message Consumption

To confirm that a valid message is consumed, processed, and acknowledged correctly.

Given a new message in the PAYMENT_SUCCESS stream

XADD PAYMENT_SUCCESS 1744045049273-0 "id" "\"1744045049273-0\"" "orderId" "\"ORD123456781\"" "amount" "199.99" "currency" "\"USD\"" "paymentMethod" "\"CREDIT_CARD\"" "paymentStatus" "\"SUCCESS\"" "cardNumber" "\"1234 5678 9012 3456\"" "cardExpiry" "\"31/12\"" "cardCvv" "\"123\"" "paypalEmail" "" "bankAccount" "" "bankName" "" "transactionId" "\"TXN1743950189267\"" "retryCount" "0" "createdAt" "\"2025-04-07T14:36:29.268562700Z\"" "updatedAt" "\"2025-04-07T14:36:29.268562700Z\""

Note: To generate the current timestamp in milliseconds (useful for constructing custom RecordIds manually), you can run the following command in a UNIX-based terminal (Linux/macOS):

date +%s%3N

This outputs a millisecond-precision timestamp like: 1744044544714

When the handlePaymentSuccess scheduler runs

2025-04-07T23:59:25.350+07:00  INFO 21880 --- [redis-stream-consumer] [       thread-4] c.y.r.s.Impl.OrderPaymentServiceImpl     : Processing payment success: {id=1744045049273-0, orderId=ORD123456781, amount=199.99, currency=USD, paymentMethod=CREDIT_CARD, paymentStatus=SUCCESS, cardNumber=1234 5678 9012 3456, cardExpiry=31/12, cardCvv=123, paypalEmail=null, bankAccount=null, bankName=null, transactionId=TXN1743950189267, retryCount=0, createdAt=2025-04-07T14:36:29.268562700Z, updatedAt=2025-04-07T14:36:29.268562700Z}
2025-04-07T23:59:25.585+07:00  INFO 21880 --- [redis-stream-consumer] [       thread-4] c.y.r.redis.MessageConsumer              : Message with ID 1744045049273-0 processed successfully and acknowledged.

Then the message should be read, processed successfully:

127.0.0.1:6379> XREVRANGE PAYMENT_SUCCESS + - COUNT 1
1) 1) "1744045049273-0"
   2)  1) "id"
       2) "\"1744045049273-0\""
       3) "orderId"
       4) "\"ORD123456781\""
       5) "amount"
       6) "199.99"
       7) "currency"
       8) "\"USD\""
       9) "paymentMethod"
      10) "\"CREDIT_CARD\""
      11) "paymentStatus"
      12) "\"SUCCESS\""z
      13) "cardNumber"
      14) "\"1234 5678 9012 3456\""
      15) "cardExpiry"
      16) "\"31/12\""
      17) "cardCvv"
      18) "\"123\""
      19) "paypalEmail"
      20) ""
      21) "bankAccount"
      22) ""
      23) "bankName"
      24) ""
      25) "transactionId"
      26) "\"TXN1743950189267\""
      27) "retryCount"
      28) "0"
      29) "createdAt"
      30) "\"2025-04-07T14:36:29.268562700Z\""
      31) "updatedAt"
      32) "\"2025-04-07T14:36:29.268562700Z\""
127.0.0.1:6379>
127.0.0.1:6379> XREVRANGE PAYMENT_SUCCESS.dlq + - COUNT 1
(empty array)
127.0.0.1:6379>

Conclusion: The PAYMENT_SUCCESS stream message was handled properly, acknowledged successfully, and not moved into the DLQ — indicating that the consumer logic and stream acknowledgment mechanism are functioning as intended.

Simulate Failed Message Processing

To validate that messages that fail during processing are left unacknowledged and stay in the pending list for retry.

Given a message in the PAYMENT_SUCCESS stream but with an invalid paymentMethod value (CC) in the message

To simulate a failure scenario, the OrderPaymentService logic was intentionally modified to reject payments with an invalid payment method:

if (orderPaymentMap.get("paymentMethod").equals("CC")) {
    logger.error("Order payment is not valid: " + orderPaymentMap);
    return false;
}

In this setup, any message containing "paymentMethod": "CC" will be treated as invalid.

XADD PAYMENT_SUCCESS 1744046314461-0 "id" "\"1744046314461-0\"" "orderId" "\"ORD123456781\"" "amount" "199.99" "currency" "\"USD\"" "paymentMethod" "\"CC\"" "paymentStatus" "\"SUCCESS\"" "cardNumber" "\"1234 5678 9012 3456\"" "cardExpiry" "\"31/12\"" "cardCvv" "\"123\"" "paypalEmail" "" "bankAccount" "" "bankName" "" "transactionId" "\"TXN1743950189267\"" "retryCount" "0" "createdAt" "\"2025-04-07T14:36:29.268562700Z\"" "updatedAt" "\"2025-04-07T14:36:29.268562700Z\""

When the handlePaymentFailed scheduler runs

When such a message is published to the Redis stream (e.g., PAYMENT_SUCCESS), the consumer detects the invalid value during processing and logs an error:

2025-04-08T00:18:53.503+07:00  INFO 33024 --- [redis-stream-consumer] [       thread-5] c.y.r.s.Impl.OrderPaymentServiceImpl     : Processing payment success: {id=1744046314461-0, orderId=ORD123456781, amount=199.99, currency=USD, paymentMethod=CC, paymentStatus=SUCCESS, cardNumber=1234 5678 9012 3456, cardExpiry=31/12, cardCvv=123, paypalEmail=null, bankAccount=null, bankName=null, transactionId=TXN1743950189267, retryCount=0, createdAt=2025-04-07T14:36:29.268562700Z, updatedAt=2025-04-07T14:36:29.268562700Z}
2025-04-08T00:18:53.583+07:00 ERROR 33024 --- [redis-stream-consumer] [       thread-5] c.y.r.s.Impl.OrderPaymentServiceImpl     : Order payment is not valid: {id=1744046314461-0, orderId=ORD123456781, amount=199.99, currency=USD, paymentMethod=CC, paymentStatus=SUCCESS, cardNumber=1234 5678 9012 3456, cardExpiry=31/12, cardCvv=123, paypalEmail=null, bankAccount=null, bankName=null, transactionId=TXN1743950189267, retryCount=0, createdAt=2025-04-07T14:36:29.268562700Z, updatedAt=2025-04-07T14:36:29.268562700Z}

Then the message should remain unacknowledged and enter the Pending Entries List (PEL)

Test Retry Mechanism for Pending Messages and Move The Messages to a DLQ stream

To test the ability to reprocess unacknowledged messages after a certain idle time, then the message should be moved to a DLQ stream such as PAYMENT_SUCCESS.dlq

2025-04-08T00:19:55.024+07:00  INFO 33024 --- [redis-stream-consumer] [       thread-2] c.y.r.redis.MessageConsumer              : Retrying (1 attempts) message with ID: 1744046314461-0 after 61 seconds of idle time for consumer: payment-success-consumer-1
2025-04-08T00:19:55.042+07:00  INFO 33024 --- [redis-stream-consumer] [       thread-2] c.y.r.s.Impl.OrderPaymentServiceImpl     : Processing payment success: {id=1744046314461-0, orderId=ORD123456781, amount=199.99, currency=USD, paymentMethod=CC, paymentStatus=SUCCESS, cardNumber=1234 5678 9012 3456, cardExpiry=31/12, cardCvv=123, paypalEmail=null, bankAccount=null, bankName=null, transactionId=TXN1743950189267, retryCount=0, createdAt=2025-04-07T14:36:29.268562700Z, updatedAt=2025-04-07T14:36:29.268562700Z}
2025-04-08T00:19:55.046+07:00 ERROR 33024 --- [redis-stream-consumer] [       thread-2] c.y.r.s.Impl.OrderPaymentServiceImpl     : Order payment is not valid: {id=1744046314461-0, orderId=ORD123456781, amount=199.99, currency=USD, paymentMethod=CC, paymentStatus=SUCCESS, cardNumber=1234 5678 9012 3456, cardExpiry=31/12, cardCvv=123, paypalEmail=null, bankAccount=null, bankName=null, transactionId=TXN1743950189267, retryCount=0, createdAt=2025-04-07T14:36:29.268562700Z, updatedAt=2025-04-07T14:36:29.268562700Z}
2025-04-08T00:19:55.049+07:00 ERROR 33024 --- [redis-stream-consumer] [       thread-2] c.y.r.redis.MessageConsumer              : Failed to process message with ID: 1744046314461-0
2025-04-08T00:20:55.021+07:00  INFO 33024 --- [redis-stream-consumer] [       thread-1] c.y.r.redis.MessageConsumer              : Retrying (2 attempts) message with ID: 1744046314461-0 after 60 seconds of idle time for consumer: payment-success-consumer-1
2025-04-08T00:20:55.028+07:00  INFO 33024 --- [redis-stream-consumer] [       thread-1] c.y.r.s.Impl.OrderPaymentServiceImpl     : Processing payment success: {id=1744046314461-0, orderId=ORD123456781, amount=199.99, currency=USD, paymentMethod=CC, paymentStatus=SUCCESS, cardNumber=1234 5678 9012 3456, cardExpiry=31/12, cardCvv=123, paypalEmail=null, bankAccount=null, bankName=null, transactionId=TXN1743950189267, retryCount=0, createdAt=2025-04-07T14:36:29.268562700Z, updatedAt=2025-04-07T14:36:29.268562700Z}
2025-04-08T00:20:55.030+07:00 ERROR 33024 --- [redis-stream-consumer] [       thread-1] c.y.r.s.Impl.OrderPaymentServiceImpl     : Order payment is not valid: {id=1744046314461-0, orderId=ORD123456781, amount=199.99, currency=USD, paymentMethod=CC, paymentStatus=SUCCESS, cardNumber=1234 5678 9012 3456, cardExpiry=31/12, cardCvv=123, paypalEmail=null, bankAccount=null, bankName=null, transactionId=TXN1743950189267, retryCount=0, createdAt=2025-04-07T14:36:29.268562700Z, updatedAt=2025-04-07T14:36:29.268562700Z}
2025-04-08T00:20:55.030+07:00 ERROR 33024 --- [redis-stream-consumer] [       thread-1] c.y.r.redis.MessageConsumer              : Failed to process message with ID: 1744046314461-0
2025-04-08T00:21:55.002+07:00  INFO 33024 --- [redis-stream-consumer] [       thread-5] c.y.r.redis.MessageConsumer              : Retrying (3 attempts) message with ID: 1744046314461-0 after 60 seconds of idle time for consumer: payment-success-consumer-1
2025-04-08T00:21:55.010+07:00  INFO 33024 --- [redis-stream-consumer] [       thread-5] c.y.r.s.Impl.OrderPaymentServiceImpl     : Processing payment success: {id=1744046314461-0, orderId=ORD123456781, amount=199.99, currency=USD, paymentMethod=CC, paymentStatus=SUCCESS, cardNumber=1234 5678 9012 3456, cardExpiry=31/12, cardCvv=123, paypalEmail=null, bankAccount=null, bankName=null, transactionId=TXN1743950189267, retryCount=0, createdAt=2025-04-07T14:36:29.268562700Z, updatedAt=2025-04-07T14:36:29.268562700Z}
2025-04-08T00:21:55.011+07:00 ERROR 33024 --- [redis-stream-consumer] [       thread-5] c.y.r.s.Impl.OrderPaymentServiceImpl     : Order payment is not valid: {id=1744046314461-0, orderId=ORD123456781, amount=199.99, currency=USD, paymentMethod=CC, paymentStatus=SUCCESS, cardNumber=1234 5678 9012 3456, cardExpiry=31/12, cardCvv=123, paypalEmail=null, bankAccount=null, bankName=null, transactionId=TXN1743950189267, retryCount=0, createdAt=2025-04-07T14:36:29.268562700Z, updatedAt=2025-04-07T14:36:29.268562700Z}
2025-04-08T00:21:55.011+07:00 ERROR 33024 --- [redis-stream-consumer] [       thread-5] c.y.r.redis.MessageConsumer              : Failed to process message with ID: 1744046314461-0
2025-04-08T00:21:55.011+07:00 ERROR 33024 --- [redis-stream-consumer] [       thread-5] c.y.r.redis.MessageConsumer              : Maximum delivery count exceeded for message ID: 1744046314461-0
2025-04-08T00:21:55.025+07:00  INFO 33024 --- [redis-stream-consumer] [       thread-5] c.y.r.redis.MessageConsumer              : Message with ID 1744046314461-0 moved to DLQ.

Conclusion: During testing, a message with RecordID 1744046314461-0 was intentionally sent with an invalid paymentMethod = CC to simulate a failure scenario. The consumer retried processing this message three times, with each attempt occurring after a 60-second idle period. On each retry, the OrderPaymentService detected the invalid payment and logged an error. After reaching the configured maximum retry threshold (3 attempts), the message was not acknowledged and was successfully moved to the Dead Letter Queue (DLQ) stream (PAYMENT_SUCCESS.dlq). This behavior demonstrates the system’s robustness in handling persistent failures by isolating problematic messages without losing them.

You can verify the message in the DLQ using the following command:

127.0.0.1:6379> XREVRANGE PAYMENT_SUCCESS.dlq + - COUNT 1
1) 1) "1744046314461-0"
   2)  1) "id"
       2) "\"1744046314461-0\""
       3) "orderId"
       4) "\"ORD123456781\""
       5) "amount"
       6) "199.99"
       7) "currency"
       8) "\"USD\""
       9) "paymentMethod"
      10) "\"CC\""
      11) "paymentStatus"
      12) "\"SUCCESS\""
      13) "cardNumber"
      14) "\"1234 5678 9012 3456\""
      15) "cardExpiry"
      16) "\"31/12\""
      17) "cardCvv"
      18) "\"123\""
      19) "paypalEmail"
      20) ""
      21) "bankAccount"
      22) ""
      23) "bankName"
      24) ""
      25) "transactionId"
      26) "\"TXN1743950189267\""
      27) "retryCount"
      28) "0"
      29) "createdAt"
      30) "\"2025-04-07T14:36:29.268562700Z\""
      31) "updatedAt"
      32) "\"2025-04-07T14:36:29.268562700Z\""
127.0.0.1:6379>

This confirms the message was safely moved to the DLQ for further inspection or manual handling.

🔗 Related Repositories

For the Redis Stream as Message Producer implementation, check out Order Payment Service with Redis Streams as Reliable Message Producer for PAYMENT_SUCCESS / PAYMENT_FAILED Events.
For the Redis Publisher implementation, check out Spring Boot Redis Publisher with Lettuce.
For the Redis Subscriber implementation, check out Spring Boot Redis Subscriber with Lettuce.

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
src		src
.dockerignore		.dockerignore
.gitattributes		.gitattributes
.gitignore		.gitignore
Makefile		Makefile
README.md		README.md
mvnw		mvnw
mvnw.cmd		mvnw.cmd
pom.xml		pom.xml

yoanesber/Spring-Boot-Redis-Stream-Consumer

Folders and files

Latest commit

History

Repository files navigation

Spring Boot Redis Stream Consumer with ThreadPoolTaskScheduler Integration

📖 Overview

💡 Why Redis Streams?

💡 What is a Consumer Group in Redis Streams?

💡 Why Do We Need Consumer Groups?

💡 How Consumer Groups Work?

📌 Redis Stream Message ID (RecordId)

💡 What is PEL?

💡 What is Dead Letter Queue (DLQ)?

🧩 Event-Driven Architecture

🔄 How It Works

🚀 Features

⏱ Scheduled Stream Consumers

📥 Message Consumer Responsibilities

🤖 Tech Stack

🧱 Architecture Overview

🛠️ Installation & Setup

✅ Prerequisites

☕ 1. Install Java 17

🐧 Linux

🪟 Windows

📨 2. Install Redis

🐧 Linux

🧰 3. Install make (Optional but Recommended)

🐧 Linux

🪟 Windows

🔁 4. Clone the Project

⚙️ 5. Configure Application Properties

🚀 6. Running the Application

🔧 Run Locally (Non-containerized)

🐳 Run Using Docker

🟢 Application is Running

🧪 Testing Scenarios

🔗 Related Repositories

About

Topics

Resources

Uh oh!

Stars

Watchers

Forks

Languages

🧰 3. Install `make` (Optional but Recommended)