esp32 s3 serial2 read out invalid value when data size is big

[flybeyond]

Board

ESP32S3 Dev Module

Device Description

plain module

Hardware Configuration

GPIO 17 18 as serial2

Version

v2.0.4

IDE Name

Arduino

Operating System

N/A

Flash frequency

40Hz

PSRAM enabled

no

Upload speed

921600

Description

I use esp32 s3 to read data from a cat1 module. when the module send out big data size say 1000, the read out value will be wrong after 200 bytes to 0xff. Use Serial2.setRxBufferSize(2048) will not resolve the issue.
You can see the debug message, full of 0xff which is not correct.

Sketch

none

Debug Message

15:25:02.443 -> d3 00 d0 43 25 a6 6d 5d fc 62 3f 80 24 15 28 06 00 00 00 00 20 20 41 00 67 7e 77 e7 7a 3a 2a 42 12 32 8a 3a 4a 78 e8 ad 25 91 9d 02 aa cc 50 50 2b bb 
15:25:02.443 -> bb 73 00 ab 91 51 a2 b0 09 4d 12 c4 25 a2 4e 07 c5 97 85 ce 17 7c 2c f8 68 e6 75 17 52 a3 d7 3a 3d f4 e4 e9 4b a9 2b 54 2e a1 7f d5 df a4 bf 49 60 89 
15:25:02.443 -> c0 57 82 38 9d 39 2d 32 66 40 3a 81 09 03 64 34 7c 67 d0 d6 20 fb f2 07 a4 38 b4 40 67 ff d1 06 75 a8 ee 71 ea ce cf 9d d6 f6 e4 f1 01 e1 fb c7 67 e4 
15:25:02.443 -> f4 de 80 1d 04 cb ef f1 2b a3 92 39 7f 9c a1 7e 83 f6 fd d7 c7 e6 3b cf 28 ae bc 75 d1 f3 62 7c 25 30 e0 8d c1 02 42 3c fe a9 af f8 ff ff ff ff ff ff 
15:25:02.475 -> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 85 bf e2 b7 c3 70 72 0d ba ac 38 e1 c0 79 b9 81 ea d0 07 fb 83 da d4 bf 77 af be 09 6f 11 e0 6c 4a e1 
15:25:02.475 -> 01 34 12 3e ec e7 fb 73 eb e5 90 4f c3 cd 7d fe c5 f8 82 ef ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff fc 00 00 00 00 2d b6 ea 2b ba 9a ed a6 
15:25:02.475 -> cb 2e c3 08 e3 9a fc b3 be fb f0 d7 5b 6a b6 9a 31 ca eb 40 0f e7 79 0d 0a 0d 0a 4f 4b 0d 0a ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 
15:25:02.500 -> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 
15:25:02.500 -> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 
15:25:02.500 -> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 
15:25:02.500 -> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 
15:25:02.500 -> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 
15:25:02.500 -> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff 
15:25:02.500 -> ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff ff

Other Steps to Reproduce

No response

I have checked existing issues, online documentation and the Troubleshooting Guide

• I confirm I have checked existing issues, online documentation and Troubleshooting guide.

[SuGlider]

@flybeyond - How are you reading the data?

200 bytes is the size for the UART driver to copy UART FIFO data to the internal Arduino Serial ringbuffer.
In order to read, say 1,000 bytes in a single operation, it shall be done using Arduino Stream Class, with a time out, such as Serial.readBytesUntil(character, buffer, length) or Serial.readBytes(buffer, length)

https://www.arduino.cc/reference/en/language/functions/communication/serial/
https://www.arduino.cc/reference/en/language/functions/communication/serial/readbytes/
https://www.arduino.cc/reference/en/language/functions/communication/serial/readbytesuntil/

[flybeyond]

After checking source code, the issue is caused by setRxBufferSize will not change buffer size dynamically. It just changes a variable and the variable will take effect in construct function. Say, if the code looks like Serial.begin(); Serial.setRxBufferSize(1024); it will not work. And Serial.setRxBufferSize(1024); Serial.begin(); It will work. The behavior is not the same as other arduino devices. It's better to change for consistence.

[VojtechBartoska]

maybe related? #6998

[flybeyond]

@SuGlider I tried Serial.readBytes(buffer, length) to read big data, it takes very long time to complete. 699 bytes take about 1.3s.

09:06:21.341 -> data len 699 left len 0
09:06:21.341 -> Start to read bytes ...
09:06:22.060 ->
09:06:22.060 ->
09:06:22.710 -> End to read bytes ...

[flybeyond]

@VojtechBartoska they are not the same issue. In this case, the setRxBufferSize will not take effect after begin(). And in my test, set the rx buffer size to 2k. I found some data should not shown in read function, show up. Some data show in previous readout value. Highly suspect there are some memory corruption. I think this is a critical issue.

[flybeyond]

Here I found a similar issue. Some one said uart reception of long data has issue for years. Is that true？ Any plans to fix it ?
espressif/esp-idf#7478

[SuGlider]

@flybeyond - I used this code and I can read 1024 bytes on UART 0, by sending data using Arduino Serial Monitor:

char uartBuff[1024];

// 128 Chars to send to UART:  SerialMonitor also send \n \r depending on the setup...
//0123456789abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#0123456789abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@

void setup() {
  Serial.setRxBufferSize(1024);
  Serial.begin(115200);

  Serial.println("\nStart reading up to 1024 bytes...");
  size_t alreadyRead = 0;
  while (alreadyRead < sizeof(uartBuff)) {
    size_t s = 0;
    while (Serial.available() && alreadyRead < sizeof(uartBuff)) {
      uartBuff[alreadyRead++] = Serial.read();
      s++;
    }
    if (s) {
      Serial.printf("So far read %d bytes\n", s);
    }
    delay(50);
  }
  
  // dump data read 
  Serial.println("UART has read 1024 bytes");
  for (int i = 0; i < sizeof(uartBuff); i++) {
    if (!(i & 0x3f)) Serial.println();
    Serial.print(uartBuff[i]);
  }
}

void loop() {
  // put your main code here, to run repeatedly:
}

Ouput:

rst:0x1 (POWERON_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
flash read err, 1000
ets_main.c 371 
ets Jun  8 2016 00:22:57

rst:0x10 (RTCWDT_RTC_RESET),boot:0x13 (SPI_FAST_FLASH_BOOT)
configsip: 0, SPIWP:0xee
clk_drv:0x00,q_drv:0x00,d_drv:0x00,cs0_drv:0x00,hd_drv:0x00,wp_drv:0x00
mode:DIO, clock div:1
load:0x3fff0030,len:1344
load:0x40078000,len:13864
load:0x40080400,len:3608
entry 0x400805f0

Start reading up to 1024 bytes...
read 360 bytes
read 664 bytes
UART has read 1024 bytes

0123456789abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#
0123456789abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@
0123456789abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#
0123456789abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@
0123456789abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#
0123456789abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@
0123456789abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#
0123456789abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@
0123456789abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#
0123456789abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@
0123456789abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#
0123456789abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@
0123456789abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#
0123456789abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@
0123456789abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#
0123456789abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@

[SuGlider]

@flybeyond - Another test with an ESP32 sending data and an ESP32-S3 receiving data.

ESP32 sends data using Serial UART 0 port. Sends data no-stop.
ESP32-S3 reads data from Serial 2, GPIO 17, as the issue stated.
Communications goes at 115200 baudrate.
ESP32 GPIO 1 (Serial Tx pin) is connected to ESP32-S3 GPIO 17 (Serial 2 Rx pin) and both boards GND are also connected to each other (common ground).

ESP32 Data Sender Sketch:

// Sends 64 bytes block - non stop - no \n \r
char buff[] = "0123456789abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#0123456789abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@";
void setup() {
  Serial.begin(115200);
}

void loop() {
  Serial.write((const uint8_t *) buff, sizeof(buff) - 1);
  Serial.flush();
}

ESP32-S3 Data Receiver Sketch:

char uartBuff[1024];

void setup() {
  Serial2.setRxBufferSize(2048);
  Serial2.begin(115200, SERIAL_8N1, 17, 18);
  Serial.begin(115200);

  Serial.println("\nStart reading up to 1024 bytes...");
  delay(100);
  Serial2.flush(false); // flush RX & TX
  
  
  size_t alreadyRead = 0;
  while (alreadyRead < sizeof(uartBuff)) {
    size_t s = 0;
    while (Serial2.available() && alreadyRead < sizeof(uartBuff)) {
      uartBuff[alreadyRead++] = Serial2.read();
      s++;
    }
    if (s) {
      Serial.printf("So far read %d bytes\n", s);
    }
  }
  
  // dump data read 
  Serial.println("UART has read 1024 bytes");
  
  for (int i = 0; i < sizeof(uartBuff); i++) {
    if (!(i & 0x3f)) Serial.println();
    Serial.print(uartBuff[i]);
  }
}

void loop() {
  // put your main code here, to run repeatedly:
}

ESP3-S3 Output:

ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x1 (POWERON),boot:0x8 (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fce3808,len:0x43c
load:0x403c9700,len:0xbec
load:0x403cc700,len:0x2a3c
SHA-256 comparison failed:
Calculated: 3611f330726ad4edf64f62982b0568332a5800bc59425cab90be5379db92c108
Expected: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
Attempting to boot anyway...
entry 0x403c98d8

Start reading up to 1024 bytes...
So far read 120 bytes
So far read 120 bytes
So far read 120 bytes
So far read 120 bytes
So far read 120 bytes
So far read 120 bytes
So far read 120 bytes
So far read 120 bytes
So far read 64 bytes
UART has read 1024 bytes

89abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@01234567
89abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#01234567
89abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@01234567
89abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#01234567
89abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@01234567
89abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#01234567
89abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@01234567
89abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#01234567
89abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@01234567
89abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#01234567
89abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@01234567
89abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#01234567
89abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@01234567
89abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#01234567
89abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@01234567
89abcdefghijklmnopqrstuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#01234567

[SuGlider]

If we add a delay(2000) at this point:

  Serial.println("\nStart reading up to 1024 bytes...");
  delay(100);
  Serial2.flush(false); // flush RX & TX

  delay(2000);  // <<<<<===== Here... we force Serial2 to receive all 1024 at once in the "background" of this delay
  
  size_t alreadyRead = 0;

The output is:

ESP-ROM:esp32s3-20210327
Build:Mar 27 2021
rst:0x1 (POWERON),boot:0x8 (SPI_FAST_FLASH_BOOT)
SPIWP:0xee
mode:DIO, clock div:1
load:0x3fce3808,len:0x43c
load:0x403c9700,len:0xbec
load:0x403cc700,len:0x2a3c
SHA-256 comparison failed:
Calculated: 3611f330726ad4edf64f62982b0568332a5800bc59425cab90be5379db92c108
Expected: ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff
Attempting to boot anyway...
entry 0x403c98d8

Start reading up to 1024 bytes...
So far read 1024 bytes
UART has read 1024 bytes

tuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#0123456789abcdefghijklmnopqrs
tuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@0123456789abcdefghijklmnopqrs
tuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#0123456789abcdefghijklmnopqrs
tuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@0123456789abcdefghijklmnopqrs
tuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#0123456789abcdefghijklmnopqrs
tuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@0123456789abcdefghijklmnopqrs
tuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#0123456789abcdefghijklmnopqrs
tuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@0123456789abcdefghijklmnopqrs
tuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#0123456789abcdefghijklmnopqrs
tuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@0123456789abcdefghijklmnopqrs
tuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#0123456789abcdefghijklmnopqrs
tuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@0123456789abcdefghijklmnopqrs
tuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#0123456789abcdefghijklmnopqrs
tuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@0123456789abcdefghijklmnopqrs
tuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$#0123456789abcdefghijklmnopqrs
tuwxyzABCDEFGHIJKLMNOPQRSTUWXYZ!*$@0123456789abcdefghijklmnopqrs

Which means that the Serial 2 Rx buffer has stored at least 1024 bytes at once in the meantime that the sketch was doing something else (delay(2000)).

So, everything works fine....

[VojtechBartoska]

@flybeyond Can this be considered as solved?