Logging every sent and receive onion message

Author: Sharmalm

CONTRIBUTING.md

@@ -88,10 +88,11 @@ be covered by functional tests.
 When refactoring, structure your PR to make it easy to review and don't
 hesitate to split it into multiple small, focused PRs.
 
-The Minimum Supported Rust Version (MSRV) currently is 1.41.1 (enforced by
-our GitHub Actions). Also, the compatibility for LDK object serialization is
-currently ensured back to and including crate version 0.0.99 (see the
-[changelog](CHANGELOG.md)).
+The Minimum Supported Rust Version (MSRV) currently is 1.48.0 (enforced by
+our GitHub Actions). We support reading serialized LDK objects written by any
+version of LDK 0.0.99 and above. We support LDK versions 0.0.113 and above
+reading serialized LDK objects written by modern LDK. Any expected issues with
+upgrades or downgrades should be mentioned in the [changelog](CHANGELOG.md).
 
 Commits should cover both the issue fixed and the solution's rationale. These
 [guidelines](https://chris.beams.io/posts/git-commit/) should be kept in mind.

ci/ci-tests.sh

@@ -68,6 +68,8 @@ if [[ $RUSTC_MINOR_VERSION -gt 67 && "$HOST_PLATFORM" != *windows* ]]; then
 	cargo check --verbose --color always --features esplora-async
 	cargo test --verbose --color always --features esplora-async-https
 	cargo check --verbose --color always --features esplora-async-https
+	cargo test --verbose --color always --features electrum
+	cargo check --verbose --color always --features electrum
 	popd
 fi
 

fuzz/Cargo.toml

@@ -20,8 +20,8 @@ stdin_fuzz = []
 [dependencies]
 lightning = { path = "../lightning", features = ["regex", "hashbrown", "_test_utils"] }
 lightning-rapid-gossip-sync = { path = "../lightning-rapid-gossip-sync" }
-bitcoin = { version = "0.29.0", features = ["secp-lowmemory"] }
-hex = "0.3"
+bitcoin = { version = "0.30.2", features = ["secp-lowmemory"] }
+hex = { package = "hex-conservative", version = "0.1.1", default-features = false }
 hashbrown = "0.8"
 
 afl = { version = "0.12", optional = true }

fuzz/README.md

@@ -1,22 +1,23 @@
 # Fuzzing
 
-Fuzz tests generate a ton of random parameter arguments to the program and then validate that none cause it to crash.
+Fuzz tests generate a ton of random parameter arguments to the program and then validate that none 
+cause it to crash.
 
 ## How does it work?
 
-Typically, Travis CI will run `travis-fuzz.sh` on one of the environments the automated tests are configured for.
-This is the most time-consuming component of the continuous integration workflow, so it is recommended that you detect
-issues locally, and Travis merely acts as a sanity check. Fuzzing is further only effective with
-a lot of CPU time, indicating that if crash scenarios are discovered on Travis with its low
-runtime constraints, the crash is caused relatively easily.
+Typically, CI will run `ci-fuzz.sh` on one of the environments the automated tests are
+configured for. Fuzzing is further only effective with a lot of CPU time, indicating that if crash 
+scenarios are discovered on CI with its low runtime constraints, the crash is caused relatively
+easily.
 
 ## How do I run fuzz tests locally?
 
-You typically won't need to run the entire combination of different fuzzing tools. For local execution, `honggfuzz`
-should be more than sufficient. 
+We support multiple fuzzing engines such as `honggfuzz`, `libFuzzer` and `AFL`. You typically won't 
+need to run the entire suite of different fuzzing tools. For local execution, `honggfuzz`should be 
+more than sufficient. 
 
 ### Setup
-
+#### Honggfuzz
 To install `honggfuzz`, simply run
 
 ```shell
@@ -31,9 +32,18 @@ cargo update -p honggfuzz --precise "0.5.52"
 cargo install --force honggfuzz --version "0.5.52"
 ```
 
+#### cargo-fuzz / libFuzzer
+To install `cargo-fuzz`, simply run
+
+```shell
+cargo update
+cargo install --force cargo-fuzz
+```
+
 ### Execution
 
-To run the Hongg fuzzer, do
+#### Honggfuzz
+To run fuzzing using `honggfuzz`, do
 
 ```shell
 export CPU_COUNT=1 # replace as needed
@@ -46,33 +56,53 @@ cargo hfuzz run $TARGET
 
 (Or, for a prettier output, replace the last line with `cargo --color always hfuzz run $TARGET`.)
 
+#### cargo-fuzz / libFuzzer
+To run fuzzing using `cargo-fuzz / libFuzzer`, run
+
+```shell
+rustup install nightly # Note: libFuzzer requires a nightly version of rust.
+cargo +nightly fuzz run --features "libfuzzer_fuzz" msg_ping_target
+```
+Note: If you encounter a `SIGKILL` during run/build check for OOM in kernel logs and consider 
+increasing RAM size for VM.
+
+If you wish to just generate fuzzing binary executables for `libFuzzer` and not run them:
+```shell 
+cargo +nightly fuzz build --features "libfuzzer_fuzz" msg_ping_target 
+# Generates binary artifact in path ./target/aarch64-unknown-linux-gnu/release/msg_ping_target
+# Exact path depends on your system architecture.
+```
+You can upload the build artifact generated above to `ClusterFuzz` for distributed fuzzing.
+
+### List Fuzzing Targets
 To see a list of available fuzzing targets, run:
 
 ```shell
 ls ./src/bin/
 ```
 
-## A fuzz test failed on Travis, what do I do?
+## A fuzz test failed, what do I do?
 
-You're trying to create a PR, but need to find the underlying cause of that pesky fuzz failure blocking the merge?
+You're trying to create a PR, but need to find the underlying cause of that pesky fuzz failure 
+blocking the merge?
 
 Worry not, for this is easily traced.
 
-If your Travis output log looks like this:
+If your output log looks like this:
 
 ```
 Size:639 (i,b,hw,ed,ip,cmp): 0/0/0/0/0/1, Tot:0/0/0/2036/5/28604
 Seen a crash. Terminating all fuzzing threads
 
 … # a lot of lines in between
 
-<0x0000555555565559> [func:UNKNOWN file: line:0 module:/home/travis/build/rust-bitcoin/rust-lightning/fuzz/hfuzz_target/x86_64-unknown-linux-gnu/release/full_stack_target]
+<0x0000555555565559> [func:UNKNOWN file: line:0 module:./rust-lightning/fuzz/hfuzz_target/x86_64-unknown-linux-gnu/release/full_stack_target]
 <0x0000000000000000> [func:UNKNOWN file: line:0 module:UNKNOWN]
 =====================================================================
 2d3136383734090101010101010101010101010101010101010101010101
 010101010100040101010101010101010101010103010101010100010101
 0069d07c319a4961
-The command "if [ "$(rustup show | grep default | grep stable)" != "" ]; then cd fuzz && cargo test --verbose && ./travis-fuzz.sh; fi" exited with 1.
+The command "if [ "$(rustup show | grep default | grep stable)" != "" ]; then cd fuzz && cargo test --verbose && ./ci-fuzz.sh; fi" exited with 1.
 ```
 
 Note that the penultimate stack trace line ends in `release/full_stack_target]`. That indicates that

fuzz/src/bin/gen_target.sh

@@ -70,3 +70,9 @@ GEN_TEST msg_tx_signatures msg_targets::
 GEN_TEST msg_tx_init_rbf msg_targets::
 GEN_TEST msg_tx_ack_rbf msg_targets::
 GEN_TEST msg_tx_abort msg_targets::
+
+GEN_TEST msg_stfu msg_targets::
+
+GEN_TEST msg_splice msg_targets::
+GEN_TEST msg_splice_ack msg_targets::
+GEN_TEST msg_splice_locked msg_targets::

fuzz/src/bin/msg_splice_ack_target.rs

@@ -0,0 +1,113 @@
+// This file is Copyright its original authors, visible in version control
+// history.
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+// This file is auto-generated by gen_target.sh based on target_template.txt
+// To modify it, modify target_template.txt and run gen_target.sh instead.
+
+#![cfg_attr(feature = "libfuzzer_fuzz", no_main)]
+
+#[cfg(not(fuzzing))]
+compile_error!("Fuzz targets need cfg=fuzzing");
+
+extern crate lightning_fuzz;
+use lightning_fuzz::msg_targets::msg_splice_ack::*;
+
+#[cfg(feature = "afl")]
+#[macro_use] extern crate afl;
+#[cfg(feature = "afl")]
+fn main() {
+	fuzz!(|data| {
+		msg_splice_ack_run(data.as_ptr(), data.len());
+	});
+}
+
+#[cfg(feature = "honggfuzz")]
+#[macro_use] extern crate honggfuzz;
+#[cfg(feature = "honggfuzz")]
+fn main() {
+	loop {
+		fuzz!(|data| {
+			msg_splice_ack_run(data.as_ptr(), data.len());
+		});
+	}
+}
+
+#[cfg(feature = "libfuzzer_fuzz")]
+#[macro_use] extern crate libfuzzer_sys;
+#[cfg(feature = "libfuzzer_fuzz")]
+fuzz_target!(|data: &[u8]| {
+	msg_splice_ack_run(data.as_ptr(), data.len());
+});
+
+#[cfg(feature = "stdin_fuzz")]
+fn main() {
+	use std::io::Read;
+
+	let mut data = Vec::with_capacity(8192);
+	std::io::stdin().read_to_end(&mut data).unwrap();
+	msg_splice_ack_run(data.as_ptr(), data.len());
+}
+
+#[test]
+fn run_test_cases() {
+	use std::fs;
+	use std::io::Read;
+	use lightning_fuzz::utils::test_logger::StringBuffer;
+
+	use std::sync::{atomic, Arc};
+	{
+		let data: Vec<u8> = vec![0];
+		msg_splice_ack_run(data.as_ptr(), data.len());
+	}
+	let mut threads = Vec::new();
+	let threads_running = Arc::new(atomic::AtomicUsize::new(0));
+	if let Ok(tests) = fs::read_dir("test_cases/msg_splice_ack") {
+		for test in tests {
+			let mut data: Vec<u8> = Vec::new();
+			let path = test.unwrap().path();
+			fs::File::open(&path).unwrap().read_to_end(&mut data).unwrap();
+			threads_running.fetch_add(1, atomic::Ordering::AcqRel);
+
+			let thread_count_ref = Arc::clone(&threads_running);
+			let main_thread_ref = std::thread::current();
+			threads.push((path.file_name().unwrap().to_str().unwrap().to_string(),
+				std::thread::spawn(move || {
+					let string_logger = StringBuffer::new();
+
+					let panic_logger = string_logger.clone();
+					let res = if ::std::panic::catch_unwind(move || {
+						msg_splice_ack_test(&data, panic_logger);
+					}).is_err() {
+						Some(string_logger.into_string())
+					} else { None };
+					thread_count_ref.fetch_sub(1, atomic::Ordering::AcqRel);
+					main_thread_ref.unpark();
+					res
+				})
+			));
+			while threads_running.load(atomic::Ordering::Acquire) > 32 {
+				std::thread::park();
+			}
+		}
+	}
+	let mut failed_outputs = Vec::new();
+	for (test, thread) in threads.drain(..) {
+		if let Some(output) = thread.join().unwrap() {
+			println!("\nOutput of {}:\n{}\n", test, output);
+			failed_outputs.push(test);
+		}
+	}
+	if !failed_outputs.is_empty() {
+		println!("Test cases which failed: ");
+		for case in failed_outputs {
+			println!("{}", case);
+		}
+		panic!();
+	}
+}

fuzz/src/bin/msg_splice_locked_target.rs

@@ -0,0 +1,113 @@
+// This file is Copyright its original authors, visible in version control
+// history.
+//
+// This file is licensed under the Apache License, Version 2.0 <LICENSE-APACHE
+// or http://www.apache.org/licenses/LICENSE-2.0> or the MIT license
+// <LICENSE-MIT or http://opensource.org/licenses/MIT>, at your option.
+// You may not use this file except in accordance with one or both of these
+// licenses.
+
+// This file is auto-generated by gen_target.sh based on target_template.txt
+// To modify it, modify target_template.txt and run gen_target.sh instead.
+
+#![cfg_attr(feature = "libfuzzer_fuzz", no_main)]
+
+#[cfg(not(fuzzing))]
+compile_error!("Fuzz targets need cfg=fuzzing");
+
+extern crate lightning_fuzz;
+use lightning_fuzz::msg_targets::msg_splice_locked::*;
+
+#[cfg(feature = "afl")]
+#[macro_use] extern crate afl;
+#[cfg(feature = "afl")]
+fn main() {
+	fuzz!(|data| {
+		msg_splice_locked_run(data.as_ptr(), data.len());
+	});
+}
+
+#[cfg(feature = "honggfuzz")]
+#[macro_use] extern crate honggfuzz;
+#[cfg(feature = "honggfuzz")]
+fn main() {
+	loop {
+		fuzz!(|data| {
+			msg_splice_locked_run(data.as_ptr(), data.len());
+		});
+	}
+}
+
+#[cfg(feature = "libfuzzer_fuzz")]
+#[macro_use] extern crate libfuzzer_sys;
+#[cfg(feature = "libfuzzer_fuzz")]
+fuzz_target!(|data: &[u8]| {
+	msg_splice_locked_run(data.as_ptr(), data.len());
+});
+
+#[cfg(feature = "stdin_fuzz")]
+fn main() {
+	use std::io::Read;
+
+	let mut data = Vec::with_capacity(8192);
+	std::io::stdin().read_to_end(&mut data).unwrap();
+	msg_splice_locked_run(data.as_ptr(), data.len());
+}
+
+#[test]
+fn run_test_cases() {
+	use std::fs;
+	use std::io::Read;
+	use lightning_fuzz::utils::test_logger::StringBuffer;
+
+	use std::sync::{atomic, Arc};
+	{
+		let data: Vec<u8> = vec![0];
+		msg_splice_locked_run(data.as_ptr(), data.len());
+	}
+	let mut threads = Vec::new();
+	let threads_running = Arc::new(atomic::AtomicUsize::new(0));
+	if let Ok(tests) = fs::read_dir("test_cases/msg_splice_locked") {
+		for test in tests {
+			let mut data: Vec<u8> = Vec::new();
+			let path = test.unwrap().path();
+			fs::File::open(&path).unwrap().read_to_end(&mut data).unwrap();
+			threads_running.fetch_add(1, atomic::Ordering::AcqRel);
+
+			let thread_count_ref = Arc::clone(&threads_running);
+			let main_thread_ref = std::thread::current();
+			threads.push((path.file_name().unwrap().to_str().unwrap().to_string(),
+				std::thread::spawn(move || {
+					let string_logger = StringBuffer::new();
+
+					let panic_logger = string_logger.clone();
+					let res = if ::std::panic::catch_unwind(move || {
+						msg_splice_locked_test(&data, panic_logger);
+					}).is_err() {
+						Some(string_logger.into_string())
+					} else { None };
+					thread_count_ref.fetch_sub(1, atomic::Ordering::AcqRel);
+					main_thread_ref.unpark();
+					res
+				})
+			));
+			while threads_running.load(atomic::Ordering::Acquire) > 32 {
+				std::thread::park();
+			}
+		}
+	}
+	let mut failed_outputs = Vec::new();
+	for (test, thread) in threads.drain(..) {
+		if let Some(output) = thread.join().unwrap() {
+			println!("\nOutput of {}:\n{}\n", test, output);
+			failed_outputs.push(test);
+		}
+	}
+	if !failed_outputs.is_empty() {
+		println!("Test cases which failed: ");
+		for case in failed_outputs {
+			println!("{}", case);
+		}
+		panic!();
+	}
+}