NFR Test Results for NGF version 2.0.0 (#3436)

tests/results/dp-perf/2.0.0/2.0.0-oss.md

@@ -0,0 +1,90 @@
+# Results
+
+## Test environment
+
+NGINX Plus: false
+
+NGINX Gateway Fabric:
+
+- Commit: f06b10da4a1754e253a58a31d2eff1bd404e33cd
+- Date: 2025-06-02T15:20:05Z
+- Dirty: false
+
+GKE Cluster:
+
+- Node count: 12
+- k8s version: v1.32.4-gke.1106006
+- vCPUs per node: 16
+- RAM per node: 65851340Ki
+- Max pods per node: 110
+- Zone: us-west1-b
+- Instance Type: n2d-standard-16
+
+## Summary:
+
+- Average latency slightly increased across all routing methods.
+
+## Test1: Running latte path based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.00, 999.97
+Duration      [total, attack, wait]             30.001s, 30s, 855.194µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  632.128µs, 875.548µs, 862.905µs, 983.833µs, 1.031ms, 1.148ms, 6.856ms
+Bytes In      [total, mean]                     4800000, 160.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test2: Running coffee header based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 999.98, 999.95
+Duration      [total, attack, wait]             30.002s, 30.001s, 1.004ms
+Latencies     [min, mean, 50, 90, 95, 99, max]  692.343µs, 908.751µs, 893.265µs, 1.009ms, 1.054ms, 1.176ms, 30.581ms
+Bytes In      [total, mean]                     4830000, 161.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test3: Running coffee query based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.02, 999.99
+Duration      [total, attack, wait]             30s, 29.999s, 841.484µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  695.394µs, 912.502µs, 898.684µs, 1.022ms, 1.068ms, 1.183ms, 10.847ms
+Bytes In      [total, mean]                     5070000, 169.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test4: Running tea GET method based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.01, 999.98
+Duration      [total, attack, wait]             30.001s, 30s, 955.507µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  685.791µs, 902.297µs, 888.939µs, 1.01ms, 1.057ms, 1.166ms, 9.243ms
+Bytes In      [total, mean]                     4740000, 158.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test5: Running tea POST method based routing
+
+```text
+Requests      [total, rate, throughput]         29999, 1000.00, 999.97
+Duration      [total, attack, wait]             30s, 29.999s, 933.986µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  666.013µs, 916.212µs, 887.892µs, 1.011ms, 1.06ms, 1.191ms, 207.34ms
+Bytes In      [total, mean]                     4739842, 158.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:29999  
+Error Set:
+```

tests/results/dp-perf/2.0.0/2.0.0-plus.md

@@ -0,0 +1,90 @@
+# Results
+
+## Test environment
+
+NGINX Plus: true
+
+NGINX Gateway Fabric:
+
+- Commit: f06b10da4a1754e253a58a31d2eff1bd404e33cd
+- Date: 2025-06-02T15:20:05Z
+- Dirty: false
+
+GKE Cluster:
+
+- Node count: 12
+- k8s version: v1.32.4-gke.1106006
+- vCPUs per node: 16
+- RAM per node: 65851340Ki
+- Max pods per node: 110
+- Zone: us-west1-b
+- Instance Type: n2d-standard-16
+
+## Summary:
+
+- Average latency slightly increased across all routing methods.
+
+## Test1: Running latte path based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 999.97, 999.82
+Duration      [total, attack, wait]             30.005s, 30.001s, 4.397ms
+Latencies     [min, mean, 50, 90, 95, 99, max]  674.871µs, 1.095s, 991.559µs, 5.087s, 6.518s, 7.551s, 7.768s
+Bytes In      [total, mean]                     4770000, 159.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test2: Running coffee header based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.04, 1000.00
+Duration      [total, attack, wait]             30s, 29.999s, 1.129ms
+Latencies     [min, mean, 50, 90, 95, 99, max]  658.096µs, 960.704µs, 943.39µs, 1.091ms, 1.144ms, 1.34ms, 4.704ms
+Bytes In      [total, mean]                     4800000, 160.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test3: Running coffee query based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.00, 999.97
+Duration      [total, attack, wait]             30.001s, 30s, 903.354µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  666.726µs, 962.973µs, 938.524µs, 1.094ms, 1.153ms, 1.375ms, 16.612ms
+Bytes In      [total, mean]                     5040000, 168.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test4: Running tea GET method based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.04, 1000.01
+Duration      [total, attack, wait]             30s, 29.999s, 905.444µs
+Latencies     [min, mean, 50, 90, 95, 99, max]  664.584µs, 941.233µs, 924.206µs, 1.079ms, 1.138ms, 1.324ms, 6.779ms
+Bytes In      [total, mean]                     4710000, 157.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```
+
+## Test5: Running tea POST method based routing
+
+```text
+Requests      [total, rate, throughput]         30000, 1000.03, 999.99
+Duration      [total, attack, wait]             30s, 29.999s, 1.099ms
+Latencies     [min, mean, 50, 90, 95, 99, max]  651.328µs, 944.335µs, 927.829µs, 1.081ms, 1.136ms, 1.29ms, 13.497ms
+Bytes In      [total, mean]                     4710000, 157.00
+Bytes Out     [total, mean]                     0, 0.00
+Success       [ratio]                           100.00%
+Status Codes  [code:count]                      200:30000  
+Error Set:
+```

tests/results/reconfig/2.0.0/2.0.0-oss.md

@@ -0,0 +1,150 @@
+# Results
+
+## Test environment
+
+NGINX Plus: false
+
+NGINX Gateway Fabric:
+
+- Commit: f06b10da4a1754e253a58a31d2eff1bd404e33cd
+- Date: 2025-06-02T15:20:05Z
+- Dirty: false
+
+GKE Cluster:
+
+- Node count: 12
+- k8s version: v1.32.4-gke.1106006
+- vCPUs per node: 16
+- RAM per node: 65851340Ki
+- Max pods per node: 110
+- Zone: us-west1-b
+- Instance Type: n2d-standard-16
+
+## Summary:
+
+- Expectations of the test suite have changed a bit due to the new architecture.
+- Time to ready generally increases due to provisioning NGINX.
+- Seeing some jumbled configuration issues, this has been brought to the attention of the NGINX Agent team.
+
+## Test 1: Resources exist before startup - NumResources 30
+
+### Time to Ready
+
+Time To Ready Description: From when NGF starts to when the NGINX configuration is fully configured
+- TimeToReadyTotal: 15s
+
+### Event Batch Processing
+
+- Event Batch Total: 10
+- Event Batch Processing Average Time: 3ms
+- Event Batch Processing distribution:
+	- 500.0ms: 10
+	- 1000.0ms: 10
+	- 5000.0ms: 10
+	- 10000.0ms: 10
+	- 30000.0ms: 10
+	- +Infms: 10
+
+### NGINX Error Logs
+
+## Test 1: Resources exist before startup - NumResources 150
+
+### Time to Ready
+
+Time To Ready Description: From when NGF starts to when the NGINX configuration is fully configured
+- TimeToReadyTotal: 9s
+
+### Event Batch Processing
+
+- Event Batch Total: 10
+- Event Batch Processing Average Time: 6ms
+- Event Batch Processing distribution:
+	- 500.0ms: 10
+	- 1000.0ms: 10
+	- 5000.0ms: 10
+	- 10000.0ms: 10
+	- 30000.0ms: 10
+	- +Infms: 10
+
+### NGINX Error Logs
+
+## Test 2: Start NGF, deploy Gateway, wait until NGINX agent instance connects to NGF, create many resources attached to GW - NumResources 30
+
+### Time to Ready
+
+Time To Ready Description: From when NGINX receives the first configuration created by NGF to when the NGINX configuration is fully configured
+- TimeToReadyTotal: 21s
+
+### Event Batch Processing
+
+- Event Batch Total: 315
+- Event Batch Processing Average Time: 11ms
+- Event Batch Processing distribution:
+	- 500.0ms: 315
+	- 1000.0ms: 315
+	- 5000.0ms: 315
+	- 10000.0ms: 315
+	- 30000.0ms: 315
+	- +Infms: 315
+
+### NGINX Error Logs
+2025/06/02 16:12:16 [emerg] 8#8: unexpected end of file, expecting "}" in /etc/nginx/conf.d/http.conf:666
+2025/06/02 16:12:16 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:1004
+2025/06/02 16:12:17 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:1113
+2025/06/02 16:12:20 [emerg] 8#8: pread() returned only 0 bytes instead of 4086 in /etc/nginx/conf.d/http.conf:715
+
+## Test 2: Start NGF, deploy Gateway, wait until NGINX agent instance connects to NGF, create many resources attached to GW - NumResources 150
+
+### Time to Ready
+
+Time To Ready Description: From when NGINX receives the first configuration created by NGF to when the NGINX configuration is fully configured
+- TimeToReadyTotal: 109s
+
+### Event Batch Processing
+
+- Event Batch Total: 1445
+- Event Batch Processing Average Time: 18ms
+- Event Batch Processing distribution:
+	- 500.0ms: 1445
+	- 1000.0ms: 1445
+	- 5000.0ms: 1445
+	- 10000.0ms: 1445
+	- 30000.0ms: 1445
+	- +Infms: 1445
+
+### NGINX Error Logs
+2025/06/02 16:16:12 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:197
+2025/06/02 16:16:13 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:1004
+2025/06/02 16:16:14 [emerg] 8#8: unknown directive "ts/ssl_keypair_reconfig_cafe-secret.pem" in /etc/nginx/conf.d/http.conf:189
+2025/06/02 16:16:15 [emerg] 8#8: unexpected end of file, expecting "}" in /etc/nginx/conf.d/http.conf:1974
+2025/06/02 16:16:17 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:2639
+2025/06/02 16:16:18 [emerg] 8#8: unexpected end of file, expecting "}" in /etc/nginx/conf.d/http.conf:3282
+2025/06/02 16:16:18 [emerg] 8#8: pread() returned only 0 bytes instead of 4089 in /etc/nginx/conf.d/http.conf:2416
+2025/06/02 16:16:21 [emerg] 8#8: pread() returned only 0 bytes instead of 4069 in /etc/nginx/conf.d/http.conf:193
+2025/06/02 16:16:21 [emerg] 8#8: pread() returned only 0 bytes instead of 4086 in /etc/nginx/conf.d/http.conf:905
+2025/06/02 16:16:25 [emerg] 8#8: invalid number of arguments in "zone" directive in /etc/nginx/conf.d/http.conf:5862
+2025/06/02 16:16:26 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:6949
+2025/06/02 16:16:27 [emerg] 8#8: pread() returned only 0 bytes instead of 4093 in /etc/nginx/conf.d/http.conf:2421
+2025/06/02 16:16:27 [emerg] 8#8: unexpected end of file, expecting "}" in /etc/nginx/conf.d/http.conf:7315
+2025/06/02 16:16:27 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:7494
+2025/06/02 16:16:28 [emerg] 8#8: pread() returned only 0 bytes instead of 4081 in /etc/nginx/conf.d/http.conf:3875
+2025/06/02 16:16:28 [emerg] 8#8: pread() returned only 0 bytes instead of 4095 in /etc/nginx/conf.d/http.conf:4364
+2025/06/02 16:16:28 [emerg] 8#8: pread() returned only 0 bytes instead of 4093 in /etc/nginx/conf.d/http.conf:4847
+2025/06/02 16:16:29 [emerg] 8#8: pread() returned only 0 bytes instead of 4094 in /etc/nginx/conf.d/http.conf:632
+2025/06/02 16:16:29 [emerg] 8#8: unexpected end of file, expecting "}" in /etc/nginx/conf.d/http.conf:8514
+2025/06/02 16:16:29 [emerg] 8#8: pread() returned only 0 bytes instead of 4085 in /etc/nginx/conf.d/http.conf:7409
+2025/06/02 16:16:29 [emerg] 8#8: pread() returned only 0 bytes instead of 4096 in /etc/nginx/conf.d/http.conf:6299
+2025/06/02 16:16:31 [emerg] 8#8: pread() returned only 0 bytes instead of 4084 in /etc/nginx/conf.d/http.conf:6343
+2025/06/02 16:16:33 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:10970
+2025/06/02 16:16:35 [emerg] 8#8: pread() returned only 0 bytes instead of 4095 in /etc/nginx/conf.d/http.conf:4495
+2025/06/02 16:16:35 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:11951
+2025/06/02 16:16:36 [emerg] 8#8: unexpected end of file, expecting "}" in /etc/nginx/conf.d/http.conf:12438
+2025/06/02 16:16:37 [emerg] 8#8: pread() returned only 0 bytes instead of 4095 in /etc/nginx/conf.d/http.conf:8234
+2025/06/02 16:16:39 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:13878
+2025/06/02 16:16:39 [emerg] 8#8: pread() returned only 0 bytes instead of 4076 in /etc/nginx/conf.d/http.conf:2329
+2025/06/02 16:16:41 [emerg] 8#8: unexpected end of file, expecting "}" in /etc/nginx/conf.d/http.conf:15054
+2025/06/02 16:16:41 [emerg] 8#8: pread() returned only 0 bytes instead of 4095 in /etc/nginx/conf.d/http.conf:9862
+2025/06/02 16:16:42 [emerg] 8#8: pread() returned only 0 bytes instead of 4079 in /etc/nginx/conf.d/http.conf:14393
+2025/06/02 16:16:42 [emerg] 8#8: pread() returned only 0 bytes instead of 2780 in /etc/nginx/conf.d/http.conf:15535
+2025/06/02 16:16:43 [emerg] 8#8: pread() returned only 0 bytes instead of 4083 in /etc/nginx/conf.d/http.conf:5118
+2025/06/02 16:16:43 [emerg] 8#8: unexpected end of file, expecting ";" or "}" in /etc/nginx/conf.d/http.conf:16167