Skip to content

Commit 5cadcc3

Browse files
garyrussellartembilan
garyrussell
authored and
artembilan
committed
GH-1130: Repub Recoverer include ex. message size
Resolves #1130 `RepublishingMessageRecoverer` - include the exception message length in the truncation algorithm. Note that the message is included in the stack trace. If the stack trace + exception message would exceed the limit - first truncate the message within the stack trace to 100 bytes -- if the stack trace and original message still exceed the limit -- also truncate the `X_EXCEPTION_MESSAGE` header to 100 bytes and use the remaing space for stack trace If, after truncating the message in the stack trace, there is room remaining the full stack trace as well as the truncated message, re-truncate the `X_EXCEPTION_MESSAGE` header to use the remaining available bytes. examples: message 150 bytes, stack trace 350 bytes, available 300 bytes, stack trace after message truncation 300 bytes - truncate message to 100, trace to 200 message 200 bytes, stack trace 250 bytes, available 300 bytes, stack trace after message truncation 150 bytes - truncate message to 100 bytes, trace to 200 message 200 bytes, stack trace 250 bytes, available 300 bytes, stack trace after message truncation 150 bytes - truncate message to 150 bytes, trace remains at 150 These are for illustration only, the available bytes is generally must larger. **cherry-pick to 2.1.x** * Fix conflicts in the `RepublishMessageRecovererIntegrationTests` for current code base around `ListenerExecutionFailedException` and `MessageProperties`
1 parent ce677bf commit 5cadcc3

File tree

3 files changed

+136
-19
lines changed

3 files changed

+136
-19
lines changed

‎spring-rabbit/src/main/java/org/springframework/amqp/rabbit/retry/RepublishMessageRecoverer.java

Lines changed: 45 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,8 @@
4949
*/
5050
public class RepublishMessageRecoverer implements MessageRecoverer {
5151

52+
private static final int ELIPSIS_LENGTH = 3;
53+
5254
public static final String X_EXCEPTION_STACKTRACE = "x-exception-stacktrace";
5355

5456
public static final String X_EXCEPTION_MESSAGE = "x-exception-message";
@@ -59,6 +61,8 @@ public class RepublishMessageRecoverer implements MessageRecoverer {
5961

6062
public static final int DEFAULT_FRAME_MAX_HEADROOM = 20_000;
6163

64+
private static final int MAX_EXCEPTION_MESSAGE_SIZE_IN_TRACE = 100 - ELIPSIS_LENGTH;
65+
6266
protected final Log logger = LogFactory.getLog(getClass()); // NOSONAR
6367

6468
protected final AmqpTemplate errorTemplate; // NOSONAR
@@ -150,9 +154,15 @@ protected MessageDeliveryMode getDeliveryMode() {
150154
public void recover(Message message, Throwable cause) {
151155
MessageProperties messageProperties = message.getMessageProperties();
152156
Map<String, Object> headers = messageProperties.getHeaders();
153-
String stackTraceAsString = processStackTrace(cause);
157+
String exceptionMessage = cause.getCause() != null ? cause.getCause().getMessage() : cause.getMessage();
158+
String[] processed = processStackTrace(cause, exceptionMessage);
159+
String stackTraceAsString = processed[0];
160+
String truncatedExceptionMessage = processed[1];
161+
if (truncatedExceptionMessage != null) {
162+
exceptionMessage = truncatedExceptionMessage;
163+
}
154164
headers.put(X_EXCEPTION_STACKTRACE, stackTraceAsString);
155-
headers.put(X_EXCEPTION_MESSAGE, cause.getCause() != null ? cause.getCause().getMessage() : cause.getMessage());
165+
headers.put(X_EXCEPTION_MESSAGE, exceptionMessage);
156166
headers.put(X_ORIGINAL_EXCHANGE, messageProperties.getReceivedExchange());
157167
headers.put(X_ORIGINAL_ROUTING_KEY, messageProperties.getReceivedRoutingKey());
158168
Map<? extends String, ? extends Object> additionalHeaders = additionalHeaders(message, cause);
@@ -183,7 +193,7 @@ public void recover(Message message, Throwable cause) {
183193
}
184194
}
185195

186-
private String processStackTrace(Throwable cause) {
196+
private String[] processStackTrace(Throwable cause, String exceptionMessage) {
187197
String stackTraceAsString = getStackTraceAsString(cause);
188198
if (this.maxStackTraceLength < 0) {
189199
int maxStackTraceLen = RabbitUtils
@@ -193,12 +203,39 @@ private String processStackTrace(Throwable cause) {
193203
this.maxStackTraceLength = maxStackTraceLen;
194204
}
195205
}
196-
if (this.maxStackTraceLength > 0 && stackTraceAsString.length() > this.maxStackTraceLength) {
197-
stackTraceAsString = stackTraceAsString.substring(0, this.maxStackTraceLength);
198-
this.logger.warn("Stack trace in republished message header truncated due to frame_max limitations; "
199-
+ "consider increasing frame_max on the broker or reduce the stack trace depth", cause);
206+
boolean truncated = false;
207+
String truncatedExceptionMessage = exceptionMessage.length() <= MAX_EXCEPTION_MESSAGE_SIZE_IN_TRACE
208+
? exceptionMessage
209+
: (exceptionMessage.substring(0, MAX_EXCEPTION_MESSAGE_SIZE_IN_TRACE) + "...");
210+
if (this.maxStackTraceLength > 0) {
211+
if (stackTraceAsString.length() + exceptionMessage.length() > this.maxStackTraceLength) {
212+
if (!exceptionMessage.equals(truncatedExceptionMessage)) {
213+
int start = stackTraceAsString.indexOf(exceptionMessage);
214+
stackTraceAsString = stackTraceAsString.substring(0, start)
215+
+ truncatedExceptionMessage
216+
+ stackTraceAsString.substring(start + exceptionMessage.length());
217+
}
218+
int adjustedStackTraceLen = this.maxStackTraceLength - truncatedExceptionMessage.length();
219+
if (adjustedStackTraceLen > 0) {
220+
if (stackTraceAsString.length() > adjustedStackTraceLen) {
221+
stackTraceAsString = stackTraceAsString.substring(0, adjustedStackTraceLen);
222+
this.logger.warn("Stack trace in republished message header truncated due to frame_max "
223+
+ "limitations; "
224+
+ "consider increasing frame_max on the broker or reduce the stack trace depth", cause);
225+
truncated = true;
226+
}
227+
else if (stackTraceAsString.length() + exceptionMessage.length() > this.maxStackTraceLength) {
228+
this.logger.warn("Exception message in republished message header truncated due to frame_max "
229+
+ "limitations; consider increasing frame_max on the broker or reduce the exception "
230+
+ "message size", cause);
231+
truncatedExceptionMessage = exceptionMessage.substring(0,
232+
this.maxStackTraceLength - stackTraceAsString.length() - ELIPSIS_LENGTH) + "...";
233+
truncated = true;
234+
}
235+
}
236+
}
200237
}
201-
return stackTraceAsString;
238+
return new String[] { stackTraceAsString, truncated ? truncatedExceptionMessage : null };
202239
}
203240

204241
/**

‎spring-rabbit/src/test/java/org/springframework/amqp/rabbit/retry/RepublishMessageRecovererIntegrationTests.java

Lines changed: 81 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -30,11 +30,12 @@
3030
import org.springframework.amqp.rabbit.core.RabbitTemplate;
3131
import org.springframework.amqp.rabbit.junit.RabbitAvailable;
3232
import org.springframework.amqp.rabbit.junit.RabbitAvailableCondition;
33-
34-
import com.rabbitmq.client.LongString;
33+
import org.springframework.amqp.rabbit.listener.exception.ListenerExecutionFailedException;
3534

3635
/**
3736
* @author Gary Russell
37+
* @author Artem Bilan
38+
*
3839
* @since 2.0.5
3940
*
4041
*/
@@ -43,30 +44,99 @@ public class RepublishMessageRecovererIntegrationTests {
4344

4445
public static final String BIG_HEADER_QUEUE = "big.header.queue";
4546

46-
private static final String BIG_EXCEPTION_MESSAGE = new String(new byte[10_000]).replaceAll("\u0000", "x");
47+
private static final String BIG_EXCEPTION_MESSAGE1 = new String(new byte[10_000]).replace("\u0000", "x");
48+
49+
private static final String BIG_EXCEPTION_MESSAGE2 = new String(new byte[10_000]).replace("\u0000", "y");
4750

4851
private int maxHeaderSize;
4952

5053
@Test
5154
public void testBigHeader() {
52-
RabbitTemplate template = new RabbitTemplate(
53-
new CachingConnectionFactory(RabbitAvailableCondition.getBrokerRunning().getConnectionFactory()));
54-
this.maxHeaderSize = RabbitUtils.getMaxFrame(template.getConnectionFactory()) - 20_000;
55+
CachingConnectionFactory ccf = new CachingConnectionFactory(
56+
RabbitAvailableCondition.getBrokerRunning().getConnectionFactory());
57+
RabbitTemplate template = new RabbitTemplate(ccf);
58+
this.maxHeaderSize = RabbitUtils.getMaxFrame(template.getConnectionFactory())
59+
- RepublishMessageRecoverer.DEFAULT_FRAME_MAX_HEADROOM;
60+
assertThat(this.maxHeaderSize).isGreaterThan(0);
61+
RepublishMessageRecoverer recoverer = new RepublishMessageRecoverer(template, "", BIG_HEADER_QUEUE);
62+
recoverer.recover(new Message("foo".getBytes(), new MessageProperties()),
63+
new ListenerExecutionFailedException("Listener failed",
64+
bigCause(new RuntimeException(BIG_EXCEPTION_MESSAGE1)), null));
65+
Message received = template.receive(BIG_HEADER_QUEUE, 10_000);
66+
assertThat(received).isNotNull();
67+
String trace =
68+
received.getMessageProperties()
69+
.getHeaders()
70+
.get(RepublishMessageRecoverer.X_EXCEPTION_STACKTRACE).toString();
71+
assertThat(trace.length()).isEqualTo(this.maxHeaderSize - 100);
72+
String truncatedMessage =
73+
"xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx...";
74+
assertThat(trace).contains(truncatedMessage);
75+
assertThat((String) received.getMessageProperties().getHeaders()
76+
.get(RepublishMessageRecoverer.X_EXCEPTION_MESSAGE))
77+
.isEqualTo(truncatedMessage);
78+
ccf.destroy();
79+
}
80+
81+
@Test
82+
public void testSmallException() {
83+
CachingConnectionFactory ccf = new CachingConnectionFactory(
84+
RabbitAvailableCondition.getBrokerRunning().getConnectionFactory());
85+
RabbitTemplate template = new RabbitTemplate(ccf);
86+
this.maxHeaderSize = RabbitUtils.getMaxFrame(template.getConnectionFactory())
87+
- RepublishMessageRecoverer.DEFAULT_FRAME_MAX_HEADROOM;
88+
assertThat(this.maxHeaderSize).isGreaterThan(0);
89+
RepublishMessageRecoverer recoverer = new RepublishMessageRecoverer(template, "", BIG_HEADER_QUEUE);
90+
ListenerExecutionFailedException cause = new ListenerExecutionFailedException("Listener failed",
91+
new RuntimeException(new String(new byte[200]).replace('\u0000', 'x')), null);
92+
recoverer.recover(new Message("foo".getBytes(), new MessageProperties()),
93+
cause);
94+
Message received = template.receive(BIG_HEADER_QUEUE, 10_000);
95+
assertThat(received).isNotNull();
96+
String trace = received.getMessageProperties().getHeaders()
97+
.get(RepublishMessageRecoverer.X_EXCEPTION_STACKTRACE).toString();
98+
assertThat(trace).isEqualTo(getStackTraceAsString(cause));
99+
ccf.destroy();
100+
}
101+
102+
@Test
103+
public void testBigMessageSmallTrace() {
104+
CachingConnectionFactory ccf = new CachingConnectionFactory(
105+
RabbitAvailableCondition.getBrokerRunning().getConnectionFactory());
106+
RabbitTemplate template = new RabbitTemplate(ccf);
107+
this.maxHeaderSize = RabbitUtils.getMaxFrame(template.getConnectionFactory())
108+
- RepublishMessageRecoverer.DEFAULT_FRAME_MAX_HEADROOM;
55109
assertThat(this.maxHeaderSize).isGreaterThan(0);
56110
RepublishMessageRecoverer recoverer = new RepublishMessageRecoverer(template, "", BIG_HEADER_QUEUE);
111+
ListenerExecutionFailedException cause = new ListenerExecutionFailedException("Listener failed",
112+
new RuntimeException(new String(new byte[this.maxHeaderSize]).replace('\u0000', 'x'),
113+
new IllegalStateException("foo")), null);
57114
recoverer.recover(new Message("foo".getBytes(), new MessageProperties()),
58-
bigCause(new RuntimeException(BIG_EXCEPTION_MESSAGE)));
115+
cause);
59116
Message received = template.receive(BIG_HEADER_QUEUE, 10_000);
60117
assertThat(received).isNotNull();
61-
assertThat(((LongString) received.getMessageProperties().getHeaders()
62-
.get(RepublishMessageRecoverer.X_EXCEPTION_STACKTRACE)).length()).isEqualTo(this.maxHeaderSize);
118+
String trace = received.getMessageProperties().getHeaders()
119+
.get(RepublishMessageRecoverer.X_EXCEPTION_STACKTRACE).toString();
120+
assertThat(trace).contains("Caused by: java.lang.IllegalStateException");
121+
String exceptionMessage =
122+
received.getMessageProperties()
123+
.getHeaders()
124+
.get(RepublishMessageRecoverer.X_EXCEPTION_MESSAGE).toString();
125+
assertThat(trace.length() + exceptionMessage.length()).isEqualTo(this.maxHeaderSize);
126+
assertThat(exceptionMessage).endsWith("...");
127+
ccf.destroy();
63128
}
64129

65130
private Throwable bigCause(Throwable cause) {
66-
if (getStackTraceAsString(cause).length() > this.maxHeaderSize) {
131+
int length = getStackTraceAsString(cause).length();
132+
int wantThisSize = this.maxHeaderSize + RepublishMessageRecoverer.DEFAULT_FRAME_MAX_HEADROOM;
133+
if (length > wantThisSize) {
67134
return cause;
68135
}
69-
return bigCause(new RuntimeException(BIG_EXCEPTION_MESSAGE, cause));
136+
String msg = length + BIG_EXCEPTION_MESSAGE1.length() > wantThisSize
137+
? BIG_EXCEPTION_MESSAGE1
138+
: BIG_EXCEPTION_MESSAGE2;
139+
return bigCause(new RuntimeException(msg, cause));
70140
}
71141

72142
private String getStackTraceAsString(Throwable cause) {

‎src/reference/asciidoc/amqp.adoc

Lines changed: 10 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5644,6 +5644,16 @@ RepublishMessageRecoverer recoverer = new RepublishMessageRecoverer(amqpTemplate
56445644
----
56455645
====
56465646

5647+
Starting with version 2.0.5, the stack trace may be truncated if it is too large; this is because all headers have to fit in a single frame.
5648+
By default, if the stack trace would cause less than 20,000 bytes ('headroom') to be available for other headers, it will be truncated.
5649+
This can be adjusted by setting the recoverer's `frameMaxHeadroom` property, if you need more or less space for other headers.
5650+
Starting with versions 2.1.13, 2.2.3, the exception message is included in this calculation, and the amount of stack trace will be maximized using the following algorithm:
5651+
5652+
* if the stack trace alone would exceed the limit, the exception message header will be truncated to 97 bytes plus `...` and the stack trace is truncated too.
5653+
* if the stack trace is small, the message will be truncated (plus `...`) to fit in the available bytes (but the message within the stack trace itself is truncated to 97 bytes plus `...`).
5654+
5655+
Whenever a truncation of any kind occurs, the original exception will be logged to retain the complete information.
5656+
56475657
Starting with version 2.1, an `ImmediateRequeueMessageRecoverer` is added to throw an `ImmediateRequeueAmqpException`, which notifies a listener container to requeue the current failed message.
56485658

56495659
===== Exception Classification for Spring Retry

0 commit comments

Comments
 (0)