Clean the Monero lib for auditing (#577)

* Remove unsafe creation of dalek_ff_group::EdwardsPoint in BP+

* Rename Bulletproofs to Bulletproof, since they are a single Bulletproof

Also bifurcates prove with prove_plus, and adds a few documentation items.

* Make CLSAG signing private

Also adds a bit more documentation and does a bit more tidying.

* Remove the distribution cache

It's a notable bandwidth/performance improvement, yet it's not ready. We need a
dedicated Distribution struct which is managed by the wallet and passed in.
While we can do that now, it's not currently worth the effort.

* Tidy Borromean/MLSAG a tad

* Remove experimental feature from monero-serai

* Move amount_decryption into EncryptedAmount::decrypt

* Various RingCT doc comments

* Begin crate smashing

* Further documentation, start shoring up API boundaries of existing crates

* Document and clean clsag

* Add a dedicated send/recv CLSAG mask struct

Abstracts the types used internally.

Also moves the tests from monero-serai to monero-clsag.

* Smash out monero-bulletproofs

Removes usage of dalek-ff-group/multiexp for curve25519-dalek.

Makes compiling in the generators an optional feature.

Adds a structured batch verifier which should be notably more performant.

Documentation and clean up still necessary.

* Correct no-std builds for monero-clsag and monero-bulletproofs

* Tidy and document monero-bulletproofs

I still don't like the impl of the original Bulletproofs...

* Error if missing documentation

* Smash out MLSAG

* Smash out Borromean

* Tidy up monero-serai as a meta crate

* Smash out RPC, wallet

* Document the RPC

* Improve docs a bit

* Move Protocol to monero-wallet

* Incomplete work on using Option to remove panic cases

* Finish documenting monero-serai

* Remove TODO on reading pseudo_outs for AggregateMlsagBorromean

* Only read transactions with one Input::Gen or all Input::ToKey

Also adds a helper to fetch a transaction's prefix.

* Smash out polyseed

* Smash out seed

* Get the repo to compile again

* Smash out Monero addresses

* Document cargo features

Credit to @hinto-janai for adding such sections to their work on documenting
monero-serai in #568.

* Fix deserializing v2 miner transactions

* Rewrite monero-wallet's send code

I have yet to redo the multisig code and the builder. This should be much
cleaner, albeit slower due to redoing work.

This compiles with clippy --all-features. I have to finish the multisig/builder
for --all-targets to work (and start updating the rest of Serai).

* Add SignableTransaction Read/Write

* Restore Monero multisig TX code

* Correct invalid RPC type def in monero-rpc

* Update monero-wallet tests to compile

Some are _consistently_ failing due to the inputs we attempt to spend being too
young. I'm unsure what's up with that. Most seem to pass _consistently_,
implying it's not a random issue yet some configuration/env aspect.

* Clean and document monero-address

* Sync rest of repo with monero-serai changes

* Represent height/block number as a u32

* Diversify ViewPair/Scanner into ViewPair/GuaranteedViewPair and Scanner/GuaranteedScanner

Also cleans the Scanner impl.

* Remove non-small-order view key bound

Guaranteed addresses are in fact guaranteed even with this due to prefixing key
images causing zeroing the ECDH to not zero the shared key.

* Finish documenting monero-serai

* Correct imports for no-std

* Remove possible panic in monero-serai on systems < 32 bits

This was done by requiring the system's usize can represent a certain number.

* Restore the reserialize chain binary

* fmt, machete, GH CI

* Correct misc TODOs in monero-serai

* Have Monero test runner evaluate an Eventuality for all signed TXs

* Fix a pair of bugs in the decoy tests

Unfortunately, this test is still failing.

* Fix remaining bugs in monero-wallet tests

* Reject torsioned spend keys to ensure we can spend the outputs we scan

* Tidy inlined epee code in the RPC

* Correct the accidental swap of stagenet/testnet address bytes

* Remove unused dep from processor

* Handle Monero fee logic properly in the processor

* Document v2 TX/RCT output relation assumed when scanning

* Adjust how we mine the initial blocks due to some CI test failures

* Fix weight estimation for RctType::ClsagBulletproof TXs

* Again increase the amount of blocks we mine prior to running tests

* Correct the if check about when to mine blocks on start

Finally fixes the lack of decoy candidates failures in CI.

* Run Monero on Debian, even for internal testnets

Change made due to a segfault incurred when locally testing.

https://github.com/monero-project/monero/issues/9141 for the upstream.

* Don't attempt running tests on the verify-chain binary

Adds a minimum XMR fee to the processor and runs fmt.

* Increase minimum Monero fee in processor

I'm truly unsure why this is required right now.

* Distinguish fee from necessary_fee in monero-wallet

If there's no change, the fee is difference of the inputs to the outputs. The
prior code wouldn't check that amount is greater than or equal to the necessary
fee, and returning the would-be change amount as the fee isn't necessarily
helpful.

Now the fee is validated in such cases and the necessary fee is returned,
enabling operating off of that.

* Restore minimum Monero fee from develop

coins/monero/rpc/simple-request/Cargo.toml

@@ -0,0 +1,31 @@
+[package]
+name = "monero-simple-request-rpc"
+version = "0.1.0"
+description = "RPC connection to a Monero daemon via simple-request, built around monero-serai"
+license = "MIT"
+repository = "https://github.com/serai-dex/serai/tree/develop/coins/monero/rpc/simple-request"
+authors = ["Luke Parker <lukeparker5132@gmail.com>"]
+edition = "2021"
+rust-version = "1.79"
+
+[package.metadata.docs.rs]
+all-features = true
+rustdoc-args = ["--cfg", "docsrs"]
+
+[lints]
+workspace = true
+
+[dependencies]
+async-trait = { version = "0.1", default-features = false }
+
+hex = { version = "0.4", default-features = false, features = ["alloc"] }
+digest_auth = { version = "0.3", default-features = false }
+simple-request = { path = "../../../../common/request", version = "0.1", default-features = false, features = ["tls"] }
+tokio = { version = "1", default-features = false }
+
+monero-rpc = { path = "..", default-features = false, features = ["std"] }
+
+[dev-dependencies]
+monero-address = { path = "../../wallet/address", default-features = false, features = ["std"] }
+
+tokio = { version = "1", default-features = false, features = ["macros"] }

coins/monero/rpc/simple-request/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2022-2024 Luke Parker
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

coins/monero/rpc/simple-request/README.md

@@ -0,0 +1,3 @@
+# Monero simple-request RPC
+
+RPC connection to a Monero daemon via simple-request, built around monero-serai.

coins/monero/rpc/simple-request/src/lib.rs

@@ -0,0 +1,290 @@
+#![cfg_attr(docsrs, feature(doc_auto_cfg))]
+#![doc = include_str!("../README.md")]
+#![deny(missing_docs)]
+
+use std::{sync::Arc, io::Read, time::Duration};
+
+use async_trait::async_trait;
+
+use tokio::sync::Mutex;
+
+use digest_auth::{WwwAuthenticateHeader, AuthContext};
+use simple_request::{
+  hyper::{StatusCode, header::HeaderValue, Request},
+  Response, Client,
+};
+
+use monero_rpc::{RpcError, Rpc};
+
+const DEFAULT_TIMEOUT: Duration = Duration::from_secs(30);
+
+#[derive(Clone, Debug)]
+enum Authentication {
+  // If unauthenticated, use a single client
+  Unauthenticated(Client),
+  // If authenticated, use a single client which supports being locked and tracks its nonce
+  // This ensures that if a nonce is requested, another caller doesn't make a request invalidating
+  // it
+  Authenticated {
+    username: String,
+    password: String,
+    #[allow(clippy::type_complexity)]
+    connection: Arc<Mutex<(Option<(WwwAuthenticateHeader, u64)>, Client)>>,
+  },
+}
+
+/// An HTTP(S) transport for the RPC.
+///
+/// Requires tokio.
+#[derive(Clone, Debug)]
+pub struct SimpleRequestRpc {
+  authentication: Authentication,
+  url: String,
+  request_timeout: Duration,
+}
+
+impl SimpleRequestRpc {
+  fn digest_auth_challenge(
+    response: &Response,
+  ) -> Result<Option<(WwwAuthenticateHeader, u64)>, RpcError> {
+    Ok(if let Some(header) = response.headers().get("www-authenticate") {
+      Some((
+        digest_auth::parse(header.to_str().map_err(|_| {
+          RpcError::InvalidNode("www-authenticate header wasn't a string".to_string())
+        })?)
+        .map_err(|_| RpcError::InvalidNode("invalid digest-auth response".to_string()))?,
+        0,
+      ))
+    } else {
+      None
+    })
+  }
+
+  /// Create a new HTTP(S) RPC connection.
+  ///
+  /// A daemon requiring authentication can be used via including the username and password in the
+  /// URL.
+  pub async fn new(url: String) -> Result<SimpleRequestRpc, RpcError> {
+    Self::with_custom_timeout(url, DEFAULT_TIMEOUT).await
+  }
+
+  /// Create a new HTTP(S) RPC connection with a custom timeout.
+  ///
+  /// A daemon requiring authentication can be used via including the username and password in the
+  /// URL.
+  pub async fn with_custom_timeout(
+    mut url: String,
+    request_timeout: Duration,
+  ) -> Result<SimpleRequestRpc, RpcError> {
+    let authentication = if url.contains('@') {
+      // Parse out the username and password
+      let url_clone = url;
+      let split_url = url_clone.split('@').collect::<Vec<_>>();
+      if split_url.len() != 2 {
+        Err(RpcError::ConnectionError("invalid amount of login specifications".to_string()))?;
+      }
+      let mut userpass = split_url[0];
+      url = split_url[1].to_string();
+
+      // If there was additionally a protocol string, restore that to the daemon URL
+      if userpass.contains("://") {
+        let split_userpass = userpass.split("://").collect::<Vec<_>>();
+        if split_userpass.len() != 2 {
+          Err(RpcError::ConnectionError("invalid amount of protocol specifications".to_string()))?;
+        }
+        url = split_userpass[0].to_string() + "://" + &url;
+        userpass = split_userpass[1];
+      }
+
+      let split_userpass = userpass.split(':').collect::<Vec<_>>();
+      if split_userpass.len() > 2 {
+        Err(RpcError::ConnectionError("invalid amount of passwords".to_string()))?;
+      }
+
+      let client = Client::without_connection_pool(&url)
+        .map_err(|_| RpcError::ConnectionError("invalid URL".to_string()))?;
+      // Obtain the initial challenge, which also somewhat validates this connection
+      let challenge = Self::digest_auth_challenge(
+        &client
+          .request(
+            Request::post(url.clone())
+              .body(vec![].into())
+              .map_err(|e| RpcError::ConnectionError(format!("couldn't make request: {e:?}")))?,
+          )
+          .await
+          .map_err(|e| RpcError::ConnectionError(format!("{e:?}")))?,
+      )?;
+      Authentication::Authenticated {
+        username: split_userpass[0].to_string(),
+        password: (*split_userpass.get(1).unwrap_or(&"")).to_string(),
+        connection: Arc::new(Mutex::new((challenge, client))),
+      }
+    } else {
+      Authentication::Unauthenticated(Client::with_connection_pool())
+    };
+
+    Ok(SimpleRequestRpc { authentication, url, request_timeout })
+  }
+}
+
+impl SimpleRequestRpc {
+  async fn inner_post(&self, route: &str, body: Vec<u8>) -> Result<Vec<u8>, RpcError> {
+    let request_fn = |uri| {
+      Request::post(uri)
+        .body(body.clone().into())
+        .map_err(|e| RpcError::ConnectionError(format!("couldn't make request: {e:?}")))
+    };
+
+    async fn body_from_response(response: Response<'_>) -> Result<Vec<u8>, RpcError> {
+      /*
+      let length = usize::try_from(
+        response
+          .headers()
+          .get("content-length")
+          .ok_or(RpcError::InvalidNode("no content-length header"))?
+          .to_str()
+          .map_err(|_| RpcError::InvalidNode("non-ascii content-length value"))?
+          .parse::<u32>()
+          .map_err(|_| RpcError::InvalidNode("non-u32 content-length value"))?,
+      )
+      .unwrap();
+      // Only pre-allocate 1 MB so a malicious node which claims a content-length of 1 GB actually
+      // has to send 1 GB of data to cause a 1 GB allocation
+      let mut res = Vec::with_capacity(length.max(1024 * 1024));
+      let mut body = response.into_body();
+      while res.len() < length {
+        let Some(data) = body.data().await else { break };
+        res.extend(data.map_err(|e| RpcError::ConnectionError(format!("{e:?}")))?.as_ref());
+      }
+      */
+
+      let mut res = Vec::with_capacity(128);
+      response
+        .body()
+        .await
+        .map_err(|e| RpcError::ConnectionError(format!("{e:?}")))?
+        .read_to_end(&mut res)
+        .unwrap();
+      Ok(res)
+    }
+
+    for attempt in 0 .. 2 {
+      return Ok(match &self.authentication {
+        Authentication::Unauthenticated(client) => {
+          body_from_response(
+            client
+              .request(request_fn(self.url.clone() + "/" + route)?)
+              .await
+              .map_err(|e| RpcError::ConnectionError(format!("{e:?}")))?,
+          )
+          .await?
+        }
+        Authentication::Authenticated { username, password, connection } => {
+          let mut connection_lock = connection.lock().await;
+
+          let mut request = request_fn("/".to_string() + route)?;
+
+          // If we don't have an auth challenge, obtain one
+          if connection_lock.0.is_none() {
+            connection_lock.0 = Self::digest_auth_challenge(
+              &connection_lock
+                .1
+                .request(request)
+                .await
+                .map_err(|e| RpcError::ConnectionError(format!("{e:?}")))?,
+            )?;
+            request = request_fn("/".to_string() + route)?;
+          }
+
+          // Insert the challenge response, if we have a challenge
+          if let Some((challenge, cnonce)) = connection_lock.0.as_mut() {
+            // Update the cnonce
+            // Overflow isn't a concern as this is a u64
+            *cnonce += 1;
+
+            let mut context = AuthContext::new_post::<_, _, _, &[u8]>(
+              username,
+              password,
+              "/".to_string() + route,
+              None,
+            );
+            context.set_custom_cnonce(hex::encode(cnonce.to_le_bytes()));
+
+            request.headers_mut().insert(
+              "Authorization",
+              HeaderValue::from_str(
+                &challenge
+                  .respond(&context)
+                  .map_err(|_| {
+                    RpcError::InvalidNode("couldn't respond to digest-auth challenge".to_string())
+                  })?
+                  .to_header_string(),
+              )
+              .unwrap(),
+            );
+          }
+
+          let response = connection_lock
+            .1
+            .request(request)
+            .await
+            .map_err(|e| RpcError::ConnectionError(format!("{e:?}")));
+
+          let (error, is_stale) = match &response {
+            Err(e) => (Some(e.clone()), false),
+            Ok(response) => (
+              None,
+              if response.status() == StatusCode::UNAUTHORIZED {
+                if let Some(header) = response.headers().get("www-authenticate") {
+                  header
+                    .to_str()
+                    .map_err(|_| {
+                      RpcError::InvalidNode("www-authenticate header wasn't a string".to_string())
+                    })?
+                    .contains("stale")
+                } else {
+                  false
+                }
+              } else {
+                false
+              },
+            ),
+          };
+
+          // If the connection entered an error state, drop the cached challenge as challenges are
+          // per-connection
+          // We don't need to create a new connection as simple-request will for us
+          if error.is_some() || is_stale {
+            connection_lock.0 = None;
+            // If we're not already on our second attempt, move to the next loop iteration
+            // (retrying all of this once)
+            if attempt == 0 {
+              continue;
+            }
+            if let Some(e) = error {
+              Err(e)?
+            } else {
+              debug_assert!(is_stale);
+              Err(RpcError::InvalidNode(
+                "node claimed fresh connection had stale authentication".to_string(),
+              ))?
+            }
+          } else {
+            body_from_response(response.unwrap()).await?
+          }
+        }
+      });
+    }
+
+    unreachable!()
+  }
+}
+
+#[async_trait]
+impl Rpc for SimpleRequestRpc {
+  async fn post(&self, route: &str, body: Vec<u8>) -> Result<Vec<u8>, RpcError> {
+    tokio::time::timeout(self.request_timeout, self.inner_post(route, body))
+      .await
+      .map_err(|e| RpcError::ConnectionError(format!("{e:?}")))?
+  }
+}

coins/monero/rpc/simple-request/tests/tests.rs

@@ -0,0 +1,75 @@
+use monero_address::{Network, MoneroAddress};
+
+// monero-rpc doesn't include a transport
+// We can't include the simple-request crate there as then we'd have a cyclical dependency
+// Accordingly, we test monero-rpc here (implicitly testing the simple-request transport)
+use monero_rpc::*;
+use monero_simple_request_rpc::*;
+
+const ADDRESS: &str =
+  "4B33mFPMq6mKi7Eiyd5XuyKRVMGVZz1Rqb9ZTyGApXW5d1aT7UBDZ89ewmnWFkzJ5wPd2SFbn313vCT8a4E2Qf4KQH4pNey";
+
+#[tokio::test]
+async fn test_rpc() {
+  let rpc =
+    SimpleRequestRpc::new("http://serai:seraidex@127.0.0.1:18081".to_string()).await.unwrap();
+
+  {
+    // Test get_height
+    let height = rpc.get_height().await.unwrap();
+    // The height should be the amount of blocks on chain
+    // The number of a block should be its zero-indexed position
+    // Accordingly, there should be no block whose number is the height
+    assert!(rpc.get_block_by_number(height).await.is_err());
+    let block_number = height - 1;
+    // There should be a block just prior
+    let block = rpc.get_block_by_number(block_number).await.unwrap();
+
+    // Also test the block RPC routes are consistent
+    assert_eq!(block.number().unwrap(), block_number);
+    assert_eq!(rpc.get_block(block.hash()).await.unwrap(), block);
+    assert_eq!(rpc.get_block_hash(block_number).await.unwrap(), block.hash());
+
+    // And finally the hardfork version route
+    assert_eq!(rpc.get_hardfork_version().await.unwrap(), block.header.hardfork_version);
+  }
+
+  // Test generate_blocks
+  for amount_of_blocks in [1, 5] {
+    let (blocks, number) = rpc
+      .generate_blocks(
+        &MoneroAddress::from_str(Network::Mainnet, ADDRESS).unwrap(),
+        amount_of_blocks,
+      )
+      .await
+      .unwrap();
+    let height = rpc.get_height().await.unwrap();
+    assert_eq!(number, height - 1);
+
+    let mut actual_blocks = Vec::with_capacity(amount_of_blocks);
+    for i in (height - amount_of_blocks) .. height {
+      actual_blocks.push(rpc.get_block_by_number(i).await.unwrap().hash());
+    }
+    assert_eq!(blocks, actual_blocks);
+  }
+
+  // Test get_output_distribution
+  // It's documented to take two inclusive block numbers
+  {
+    let height = rpc.get_height().await.unwrap();
+
+    rpc.get_output_distribution(0 ..= height).await.unwrap_err();
+    assert_eq!(rpc.get_output_distribution(0 .. height).await.unwrap().len(), height);
+
+    assert_eq!(rpc.get_output_distribution(0 .. (height - 1)).await.unwrap().len(), height - 1);
+    assert_eq!(rpc.get_output_distribution(1 .. height).await.unwrap().len(), height - 1);
+
+    assert_eq!(rpc.get_output_distribution(0 ..= 0).await.unwrap().len(), 1);
+    assert_eq!(rpc.get_output_distribution(0 ..= 1).await.unwrap().len(), 2);
+    assert_eq!(rpc.get_output_distribution(1 ..= 1).await.unwrap().len(), 1);
+
+    rpc.get_output_distribution(0 .. 0).await.unwrap_err();
+    #[allow(clippy::reversed_empty_ranges)]
+    rpc.get_output_distribution(1 .. 0).await.unwrap_err();
+  }
+}