Initial In Instructions pallet and Serai client lib (#233)

* Initial work on an In Inherents pallet

* Add an event for when a batch is executed

* Add a dummy provider for InInstructions

* Add in-instructions to the node

* Add the Serai runtime API to the processor

* Move processor tests around

* Build a subxt Client around Serai

* Successfully get Batch events from Serai

Renamed processor/substrate to processor/serai.

* Much more robust InInstruction pallet

* Implement the workaround from https://github.com/paritytech/subxt/issues/602

* Initial prototype of processor generated InInstructions

* Correct PendingCoins data flow for InInstructions

* Minor lint to in-instructions

* Remove the global Serai connection for a partial re-impl

* Correct ID handling of the processor test

* Workaround the delay in the subscription

* Make an unwrap an if let Some, remove old comments

* Lint the processor toml

* Rebase and update

* Move substrate/in-instructions to substrate/in-instructions/pallet

* Start an in-instructions primitives lib

* Properly update processor to subxt 0.24

Also corrects failures from the rebase.

* in-instructions cargo update

* Implement IsFatalError

* is_inherent -> true

* Rename in-instructions crates and misc cleanup

* Update documentation

* cargo update

* Misc update fixes

* Replace height with block_number

* Update processor src to latest subxt

* Correct pipeline for InInstructions testing

* Remove runtime::AccountId for serai_primitives::NativeAddress

* Rewrite the in-instructions pallet

Complete with respect to the currently written docs.

Drops the custom serializer for just using SCALE.

Makes slight tweaks as relevant.

* Move instructions' InherentDataProvider to a client crate

* Correct doc gen

* Add serde to in-instructions-primitives

* Add in-instructions-primitives to pallet

* Heights -> BlockNumbers

* Get batch pub test loop working

* Update in instructions pallet terminology

Removes the ambiguous Coin for Update.

Removes pending/artificial latency for furture client work.

Also moves to using serai_primitives::Coin.

* Add a BlockNumber primitive

* Belated cargo fmt

* Further document why DifferentBatch isn't fatal

* Correct processor sleeps

* Remove metadata at compile time, add test framework for Serai nodes

* Remove manual RPC client

* Simplify update test

* Improve re-exporting behavior of serai-runtime

It now re-exports all pallets underneath it.

* Add a function to get storage values to the Serai RPC

* Update substrate/ to latest substrate

* Create a dedicated crate for the Serai RPC

* Remove unused dependencies in substrate/

* Remove unused dependencies in coins/

Out of scope for this branch, just minor and path of least resistance.

* Use substrate/serai/client for the Serai RPC lib

It's a bit out of place, since these client folders are intended for the node to
access pallets and so on. This is for end-users to access Serai as a whole.

In that sense, it made more sense as a top level folder, yet that also felt
out of place.

* Move InInstructions test to serai-client for now

* Final cleanup

* Update deny.toml

* Cargo.lock update from merging develop

* Update nightly

Attempt to work around the current CI failure, which is a Rust ICE.

We previously didn't upgrade due to clippy 10134, yet that's been reverted.

* clippy

* clippy

* fmt

* NativeAddress -> SeraiAddress

* Sec fix on non-provided updates and doc fixes

* Add Serai as a Coin

Necessary in order to swap to Serai.

* Add a BlockHash type, used for batch IDs

* Remove origin from InInstruction

Makes InInstructionTarget. Adds RefundableInInstruction with origin.

* Document storage items in in-instructions

* Rename serai/client/tests/serai.rs to updates.rs

It only tested publishing updates and their successful acceptance.

substrate/serai/client/Cargo.toml

@@ -0,0 +1,33 @@
+[package]
+name = "serai-client"
+version = "0.1.0"
+description = "Client library for the Serai network"
+license = "AGPL-3.0-only"
+repository = "https://github.com/serai-dex/serai/tree/develop/client"
+authors = ["Luke Parker <lukeparker5132@gmail.com>"]
+keywords = ["serai"]
+edition = "2021"
+
+[package.metadata.docs.rs]
+all-features = true
+rustdoc-args = ["--cfg", "docsrs"]
+
+[dependencies]
+thiserror = "1"
+
+serde = { version = "1", features = ["derive"] }
+
+scale = { package = "parity-scale-codec", version = "3" }
+scale-value = "0.6"
+subxt = "0.25"
+
+serai-primitives = { path = "../primitives", version = "0.1" }
+in-instructions-primitives = { path = "../../in-instructions/primitives", version = "0.1" }
+serai-runtime = { path = "../../runtime", version = "0.1" }
+
+[dev-dependencies]
+lazy_static = "1"
+
+tokio = "1"
+
+jsonrpsee-server = "0.16"

substrate/serai/client/LICENSE

@@ -0,0 +1,15 @@
+AGPL-3.0-only license
+
+Copyright (c) 2022-2023 Luke Parker
+
+This program is free software: you can redistribute it and/or modify
+it under the terms of the GNU Affero General Public License Version 3 as
+published by the Free Software Foundation.
+
+This program is distributed in the hope that it will be useful,
+but WITHOUT ANY WARRANTY; without even the implied warranty of
+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+GNU Affero General Public License for more details.
+
+You should have received a copy of the GNU Affero General Public License
+along with this program. If not, see <http://www.gnu.org/licenses/>.

substrate/serai/client/src/in_instructions.rs

@@ -0,0 +1,49 @@
+use scale::Decode;
+
+use serai_runtime::{
+  support::traits::PalletInfo as PalletInfoTrait, PalletInfo, in_instructions, InInstructions,
+  Runtime,
+};
+
+pub use in_instructions_primitives as primitives;
+
+use crate::{
+  primitives::{Coin, BlockNumber},
+  Serai, SeraiError,
+};
+
+const PALLET: &str = "InInstructions";
+
+pub type InInstructionsEvent = in_instructions::Event<Runtime>;
+
+impl Serai {
+  pub async fn get_batch_events(
+    &self,
+    block: [u8; 32],
+  ) -> Result<Vec<InInstructionsEvent>, SeraiError> {
+    let mut res = vec![];
+    for event in
+      self.0.events().at(Some(block.into())).await.map_err(|_| SeraiError::RpcError)?.iter()
+    {
+      let event = event.map_err(|_| SeraiError::InvalidRuntime)?;
+      if PalletInfo::index::<InInstructions>().unwrap() == usize::from(event.pallet_index()) {
+        let mut with_variant: &[u8] =
+          &[[event.variant_index()].as_ref(), event.field_bytes()].concat();
+        let event =
+          InInstructionsEvent::decode(&mut with_variant).map_err(|_| SeraiError::InvalidRuntime)?;
+        if matches!(event, InInstructionsEvent::Batch { .. }) {
+          res.push(event);
+        }
+      }
+    }
+    Ok(res)
+  }
+
+  pub async fn get_coin_block_number(
+    &self,
+    coin: Coin,
+    block: [u8; 32],
+  ) -> Result<BlockNumber, SeraiError> {
+    Ok(self.storage(PALLET, "BlockNumbers", Some(coin), block).await?.unwrap_or(BlockNumber(0)))
+  }
+}

substrate/serai/client/src/lib.rs

@@ -0,0 +1,77 @@
+use thiserror::Error;
+
+use serde::Serialize;
+use scale::Decode;
+
+use subxt::{tx::BaseExtrinsicParams, Config as SubxtConfig, OnlineClient};
+
+pub use serai_primitives as primitives;
+use primitives::{Signature, SeraiAddress};
+
+use serai_runtime::{system::Config, Runtime};
+
+pub mod in_instructions;
+
+#[derive(Clone, Copy, PartialEq, Eq, Debug)]
+pub(crate) struct SeraiConfig;
+impl SubxtConfig for SeraiConfig {
+  type BlockNumber = <Runtime as Config>::BlockNumber;
+
+  type Hash = <Runtime as Config>::Hash;
+  type Hashing = <Runtime as Config>::Hashing;
+
+  type Index = <Runtime as Config>::Index;
+  type AccountId = <Runtime as Config>::AccountId;
+  // TODO: Bech32m
+  type Address = SeraiAddress;
+
+  type Header = <Runtime as Config>::Header;
+  type Signature = Signature;
+
+  type ExtrinsicParams = BaseExtrinsicParams<SeraiConfig, ()>;
+}
+
+#[derive(Clone, Error, Debug)]
+pub enum SeraiError {
+  #[error("failed to connect to serai")]
+  RpcError,
+  #[error("serai-client library was intended for a different runtime version")]
+  InvalidRuntime,
+}
+
+#[derive(Clone)]
+pub struct Serai(OnlineClient<SeraiConfig>);
+
+impl Serai {
+  pub async fn new(url: &str) -> Result<Self, SeraiError> {
+    Ok(Serai(OnlineClient::<SeraiConfig>::from_url(url).await.map_err(|_| SeraiError::RpcError)?))
+  }
+
+  async fn storage<K: Serialize, R: Decode>(
+    &self,
+    pallet: &'static str,
+    name: &'static str,
+    key: Option<K>,
+    block: [u8; 32],
+  ) -> Result<Option<R>, SeraiError> {
+    let mut keys = vec![];
+    if let Some(key) = key {
+      keys.push(scale_value::serde::to_value(key).unwrap());
+    }
+
+    let storage = self.0.storage();
+    let address = subxt::dynamic::storage(pallet, name, keys);
+    debug_assert!(storage.validate(&address).is_ok());
+
+    storage
+      .fetch(&address, Some(block.into()))
+      .await
+      .map_err(|_| SeraiError::RpcError)?
+      .map(|res| R::decode(&mut res.encoded()).map_err(|_| SeraiError::InvalidRuntime))
+      .transpose()
+  }
+
+  pub async fn get_latest_block_hash(&self) -> Result<[u8; 32], SeraiError> {
+    Ok(self.0.rpc().finalized_head().await.map_err(|_| SeraiError::RpcError)?.into())
+  }
+}

substrate/serai/client/tests/runner.rs

@@ -0,0 +1,50 @@
+use lazy_static::lazy_static;
+
+use tokio::sync::Mutex;
+
+pub const URL: &str = "ws://127.0.0.1:9944";
+
+lazy_static! {
+  pub static ref SEQUENTIAL: Mutex<()> = Mutex::new(());
+}
+
+#[macro_export]
+macro_rules! serai_test {
+  ($(async fn $name: ident() $body: block)*) => {
+    $(
+      #[tokio::test]
+      async fn $name() {
+        let guard = runner::SEQUENTIAL.lock().await;
+
+        // Spawn a fresh Serai node
+        let mut command = {
+          use core::time::Duration;
+          use std::{path::Path, process::Command};
+
+          let node = {
+            let this_crate = Path::new(env!("CARGO_MANIFEST_DIR"));
+            let top_level = this_crate.join("../../../");
+            top_level.join("target/debug/serai-node")
+          };
+
+          let command = Command::new(node).arg("--dev").spawn().unwrap();
+          while Serai::new(URL).await.is_err() {
+            tokio::time::sleep(Duration::from_secs(1)).await;
+          }
+          command
+        };
+
+        let local = tokio::task::LocalSet::new();
+        local.run_until(async move {
+          if let Err(err) = tokio::task::spawn_local(async move { $body }).await {
+            drop(guard);
+            let _ = command.kill();
+            Err(err).unwrap()
+          } else {
+            command.kill().unwrap();
+          }
+        }).await;
+      }
+    )*
+  }
+}

substrate/serai/client/tests/updates.rs

@@ -0,0 +1,60 @@
+use core::time::Duration;
+
+use tokio::time::sleep;
+
+use serai_runtime::in_instructions::{Batch, Update};
+
+use jsonrpsee_server::RpcModule;
+
+use serai_client::{
+  primitives::{BlockNumber, BlockHash, SeraiAddress, BITCOIN},
+  in_instructions::{primitives::InInstruction, InInstructionsEvent},
+  Serai,
+};
+
+mod runner;
+use runner::URL;
+
+serai_test!(
+  async fn publish_update() {
+    let mut rpc = RpcModule::new(());
+    rpc
+      .register_async_method("processor_coinUpdates", |_, _| async move {
+        let batch = Batch {
+          id: BlockHash([0xaa; 32]),
+          instructions: vec![InInstruction::Transfer(SeraiAddress::from_raw([0xff; 32]))],
+        };
+
+        Ok(vec![Some(Update { block_number: BlockNumber(123), batches: vec![batch] })])
+      })
+      .unwrap();
+
+    let _handle = jsonrpsee_server::ServerBuilder::default()
+      .build("127.0.0.1:5134")
+      .await
+      .unwrap()
+      .start(rpc)
+      .unwrap();
+
+    let serai = Serai::new(URL).await.unwrap();
+    loop {
+      let latest = serai.get_latest_block_hash().await.unwrap();
+      let batches = serai.get_batch_events(latest).await.unwrap();
+      if let Some(batch) = batches.get(0) {
+        match batch {
+          InInstructionsEvent::Batch { coin, id } => {
+            assert_eq!(coin, &BITCOIN);
+            assert_eq!(id, &BlockHash([0xaa; 32]));
+            assert_eq!(
+              serai.get_coin_block_number(BITCOIN, latest).await.unwrap(),
+              BlockNumber(123)
+            );
+            return;
+          }
+          _ => panic!("get_batches returned non-batch"),
+        }
+      }
+      sleep(Duration::from_millis(50)).await;
+    }
+  }
+);

substrate/serai/primitives/Cargo.toml

@@ -15,11 +15,10 @@ rustdoc-args = ["--cfg", "docsrs"]
 scale = { package = "parity-scale-codec", version = "3", default-features = false, features = ["derive"] }
 scale-info = { version = "2", default-features = false, features = ["derive"] }
 
-serde = { version = "1.0", features = ["derive"], optional = true }
+serde = { version = "1", features = ["derive"], optional = true }
 
 sp-core = { git = "https://github.com/serai-dex/substrate", default-features = false }
-sp-std = { git = "https://github.com/serai-dex/substrate", default-features = false }
 
 [features]
-std = ["scale/std", "scale-info/std", "serde", "sp-core/std", "sp-std/std"]
+std = ["scale/std", "scale-info/std", "serde", "sp-core/std"]
 default = ["std"]

substrate/serai/primitives/src/amount.rs

@@ -9,7 +9,7 @@ use serde::{Serialize, Deserialize};
 
 /// The type used for amounts.
 #[derive(
-  Clone, Copy, PartialEq, Eq, PartialOrd, Debug, Encode, Decode, TypeInfo, MaxEncodedLen,
+  Clone, Copy, PartialEq, Eq, PartialOrd, Debug, Encode, Decode, MaxEncodedLen, TypeInfo,
 )]
 #[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
 pub struct Amount(pub u64);

substrate/serai/primitives/src/coins.rs

@@ -4,7 +4,7 @@ use scale_info::TypeInfo;
 use serde::{Serialize, Deserialize};
 
 /// The type used to identify coins.
-#[derive(Clone, Copy, PartialEq, Eq, Debug, Encode, Decode, TypeInfo, MaxEncodedLen)]
+#[derive(Clone, Copy, PartialEq, Eq, Debug, Encode, Decode, MaxEncodedLen, TypeInfo)]
 #[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
 pub struct Coin(pub u32);
 impl From<u32> for Coin {
@@ -13,7 +13,8 @@ impl From<u32> for Coin {
   }
 }
 
-pub const BITCOIN: Coin = Coin(0);
-pub const ETHER: Coin = Coin(1);
-pub const DAI: Coin = Coin(2);
-pub const MONERO: Coin = Coin(3);
+pub const SERAI: Coin = Coin(0);
+pub const BITCOIN: Coin = Coin(1);
+pub const ETHER: Coin = Coin(2);
+pub const DAI: Coin = Coin(3);
+pub const MONERO: Coin = Coin(4);

substrate/serai/primitives/src/lib.rs

@@ -1,7 +1,63 @@
+#![cfg_attr(docsrs, feature(doc_cfg))]
+#![cfg_attr(docsrs, feature(doc_auto_cfg))]
 #![cfg_attr(not(feature = "std"), no_std)]
 
+use scale::{Encode, Decode, MaxEncodedLen};
+use scale_info::TypeInfo;
+#[cfg(feature = "std")]
+use serde::{Serialize, Deserialize};
+
+use sp_core::{
+  H256,
+  sr25519::{Public, Signature as RistrettoSignature},
+};
+
 mod amount;
 pub use amount::*;
 
 mod coins;
 pub use coins::*;
+
+pub type PublicKey = Public;
+pub type SeraiAddress = PublicKey;
+pub type Signature = RistrettoSignature;
+
+/// The type used to identify block numbers.
+// Doesn't re-export tendermint-machine's due to traits.
+#[derive(
+  Clone, Copy, Default, PartialEq, Eq, Hash, Debug, Encode, Decode, MaxEncodedLen, TypeInfo,
+)]
+#[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
+pub struct BlockNumber(pub u32);
+impl From<u32> for BlockNumber {
+  fn from(number: u32) -> BlockNumber {
+    BlockNumber(number)
+  }
+}
+
+/// The type used to identify block hashes.
+// This may not be universally compatible
+// If a block exists with a hash which isn't 32-bytes, it can be hashed into a value with 32-bytes
+// This would require the processor to maintain a mapping of 32-byte IDs to actual hashes, which
+// would be fine
+#[derive(Clone, Copy, PartialEq, Eq, Hash, Debug, Encode, Decode, MaxEncodedLen, TypeInfo)]
+#[cfg_attr(feature = "std", derive(Serialize, Deserialize))]
+pub struct BlockHash(pub [u8; 32]);
+
+impl AsRef<[u8]> for BlockHash {
+  fn as_ref(&self) -> &[u8] {
+    self.0.as_ref()
+  }
+}
+
+impl From<[u8; 32]> for BlockHash {
+  fn from(hash: [u8; 32]) -> BlockHash {
+    BlockHash(hash)
+  }
+}
+
+impl From<H256> for BlockHash {
+  fn from(hash: H256) -> BlockHash {
+    BlockHash(hash.into())
+  }
+}