The `cargo doc` build started to fail with the rolling of `doc_auto_cfg` into
`doc_cfg`, so now we don't build docs for deps (as we can't reasonably update
`generic-array` at this time).
`home` has been patched as we are able to, not as a direct requirement of this
PR.
* Update `build-dependencies` CI action
* Update `develop` to `patch-polkadot-sdk`
Allows us to finally remove the old `serai-dex/substrate` repository _and_
should have CI pass without issue on `develop` again.
The changes made here should be trivial and maintain all prior
behavior/functionality. The most notable are to `chain_spec.rs`, in order to
still use a SCALE-encoded `GenesisConfig` (avoiding `serde_json`).
* CI fixes
* Add `/usr/local/opt/llvm/lib` to paths on macOS hosts
* Attempt to use `LD_LIBRARY_PATH` in macOS GitHub CI
* Use `libp2p 0.56` in `serai-node`
* Correct Windows build dependencies
* Correct `llvm/lib` path on macOS
* Correct how macOS 13 and 14 have different homebrew paths
* Use `sw_vers` instead of `uname` on macOS
Yields the macOS version instead of the kernel's version.
* Replace hard-coded path with the intended env variable to fix macOS 13
* Add `libclang-dev` as dependency to the Debian Dockerfile
* Set the `CODE` storage slot
* Update to a version of substrate without `wasmtimer`
Turns out `wasmtimer` is WASM only. This should restore the node's functioning
on non-WASM environments.
* Restore `clang` as a dependency due to the Debian Dockerfile as we require a C++ compiler
* Move from Debian bookworm to trixie
* Restore `chain_getBlockBin` to the RPC
* Always generate a new key for the P2P network
* Mention every account on-chain before they publish a transaction
`CheckNonce` required accounts have a provider in order to even have their
nonce considered. This shims that by claiming every account has a provider at
the start of a block, if it signs a transaction.
The actual execution could presumably diverge between block building (which
sets the provider before each transaction) and execution (which sets the
providers at the start of the block). It doesn't diverge in our current
configuration and it won't be propagated to `next` (which doesn't use
`CheckNonce`).
Also uses explicit indexes for the `serai_abi::{Call, Event}` `enum`s.
* Adopt `patch-polkadot-sdk` with fixed peering
* Manually insert the authority discovery key into the keystore
I did try pulling in `pallet-authority-discovery` for this, updating
`SessionKeys`, but that was insufficient for whatever reason.
* Update to latest `substrate-wasm-builder`
* Fix timeline for incrementing providers
e1671dd71b incremented the providers for every
single transaction's sender before execution, noting the solution was fragile
but it worked for us at this time. It did not work for us at this time.
The new solution replaces `inc_providers` with direct access to the `Account`
`StorageMap` to increment the providers, achieving the desired goal, _without_
emitting an event (which is ordered, and the disparate order between building
and execution was causing mismatches of the state root).
This solution is also fragile and may also be insufficient. None of this code
exists anymore on `next` however. It just has to work sufficiently for now.
* clippy
* move economic security into it's own pallet
* fix deny
* Update Cargo.toml, .github for the new crates
* Remove unused import
---------
Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
* Clean up Ethereum
* Consistent contract address for deployed contracts
* Flesh out Router a bit
* Add a Deployer for DoS-less deployment
* Implement Router-finding
* Use CREATE2 helper present in ethers
* Move from CREATE2 to CREATE
Bit more streamlined for our use case.
* Document ethereum-serai
* Tidy tests a bit
* Test updateSeraiKey
* Use encodePacked for updateSeraiKey
* Take in the block hash to read state during
* Add a Sandbox contract to the Ethereum integration
* Add retrieval of transfers from Ethereum
* Add inInstruction function to the Router
* Augment our handling of InInstructions events with a check the transfer event also exists
* Have the Deployer error upon failed deployments
* Add --via-ir
* Make get_transaction test-only
We only used it to get transactions to confirm the resolution of Eventualities.
Eventualities need to be modularized. By introducing the dedicated
confirm_completion function, we remove the need for a non-test get_transaction
AND begin this modularization (by no longer explicitly grabbing a transaction
to check with).
* Modularize Eventuality
Almost fully-deprecates the Transaction trait for Completion. Replaces
Transaction ID with Claim.
* Modularize the Scheduler behind a trait
* Add an extremely basic account Scheduler
* Add nonce uses, key rotation to the account scheduler
* Only report the account Scheduler empty after transferring keys
Also ban payments to the branch/change/forward addresses.
* Make fns reliant on state test-only
* Start of an Ethereum integration for the processor
* Add a session to the Router to prevent updateSeraiKey replaying
This would only happen if an old key was rotated to again, which would require
n-of-n collusion (already ridiculous and a valid fault attributable event). It
just clarifies the formal arguments.
* Add a RouterCommand + SignMachine for producing it to coins/ethereum
* Ethereum which compiles
* Have branch/change/forward return an option
Also defines a UtxoNetwork extension trait for MAX_INPUTS.
* Make external_address exclusively a test fn
* Move the "account" scheduler to "smart contract"
* Remove ABI artifact
* Move refund/forward Plan creation into the Processor
We create forward Plans in the scan path, and need to know their exact fees in
the scan path. This requires adding a somewhat wonky shim_forward_plan method
so we can obtain a Plan equivalent to the actual forward Plan for fee reasons,
yet don't expect it to be the actual forward Plan (which may be distinct if
the Plan pulls from the global state, such as with a nonce).
Also properly types a Scheduler addendum such that the SC scheduler isn't
cramming the nonce to use into the N::Output type.
* Flesh out the Ethereum integration more
* Two commits ago, into the **Scheduler, not Processor
* Remove misc TODOs in SC Scheduler
* Add constructor to RouterCommandMachine
* RouterCommand read, pairing with the prior added write
* Further add serialization methods
* Have the Router's key included with the InInstruction
This does not use the key at the time of the event. This uses the key at the
end of the block for the event. Its much simpler than getting the full event
streams for each, checking when they interlace.
This does not read the state. Every block, this makes a request for every
single key update and simply chooses the last one. This allows pruning state,
only keeping the event tree. Ideally, we'd also introduce a cache to reduce the
cost of the filter (small in events yielded, long in blocks searched).
Since Serai doesn't have any forwarding TXs, nor Branches, nor change, all of
our Plans should solely have payments out, and there's no expectation of a Plan
being made under one key broken by it being received by another key.
* Add read/write to InInstruction
* Abstract the ABI for Call/OutInstruction in ethereum-serai
* Fill out signable_transaction for Ethereum
* Move ethereum-serai to alloy
Resolves#331.
* Use the opaque sol macro instead of generated files
* Move the processor over to the now-alloy-based ethereum-serai
* Use the ecrecover provided by alloy
* Have the SC use nonce for rotation, not session (an independent nonce which wasn't synchronized)
* Always use the latest keys for SC scheduled plans
* get_eventuality_completions for Ethereum
* Finish fleshing out the processor Ethereum integration as needed for serai-processor tests
This doesn't not support any actual deployments, not even the ones simulated by
serai-processor-docker-tests.
* Add alloy-simple-request-transport to the GH workflows
* cargo update
* Clarify a few comments and make one check more robust
* Use a string for 27.0 in .github
* Remove optional from no-longer-optional dependencies in processor
* Add alloy to git deny exception
* Fix no longer optional specification in processor's binaries feature
* Use a version of foundry from 2024
* Correct fetching Bitcoin TXs in the processor docker tests
* Update rustls to resolve RUSTSEC warnings
* Use the monthly nightly foundry, not the deleted daily nightly
Moves from concatted Dockerfiles to pseudo-templated Dockerfiles via a dedicated Rust program.
Removes the unmaintained kubernetes, not because we shouldn't have/use it, but because it's unmaintained and needs to be reworked before it's present again.
Replaces the compose with the work in the new orchestrator binary which spawns everything as expected. While this arguably re-invents the wheel, it correctly manages secrets and handles the variadic Dockerfiles.
Also adds an unrelated patch for zstd and simplifies running services a bit by greater utilizing the existing infrastructure.
---
* Delete all Dockerfile fragments, add new orchestator to generate Dockerfiles
Enables greater templating.
Also delete the unmaintained kubernetes folder *for now*. This should be
restored in the future.
* Use Dockerfiles from the orchestator
* Ignore Dockerfiles in the git repo
* Remove CI job to check Dockerfiles are as expected now that they're no longer committed
* Remove old Dockerfiles from repo
* Use Debian for monero-wallet-rpc
* Remove replace_cmds for proper usage of entry-dev
Consolidates ports a bit.
Updates serai-docker-tests from "compose" to "build".
* Only write a new dockerfile if it's distinct
Preserves the updated time metadata.
* Update serai-docker-tests
* Correct the path Dockerfiles are built from
* Correct inclusion of orchestration folder in Docker builds
* Correct debug/release flagging in the cargo command
Apparently, --debug isn't an effective NOP yet an error.
* Correct path used to run the Serai node within a Dockerfile
* Correct path in Monero Dockerfile
* Attempt storing monerod in /usr/bin
* Use sudo to move into /usr/bin in CI
* Correct 18.3.0 to 18.3.1
* Escape * with quotes
* Update deny.toml, ADD orchestration in runtime Dockerfile
* Add --detach to the Monero GH CI
* Diversify dockerfiles by network
* Fixes to network-diversified orchestration
* Bitcoin and Monero testnet scripts
* Permissions and tweaks
* Flatten scripts folders
* Add missing folder specification to Monero Dockerfile
* Have monero-wallet-rpc specify the monerod login
* Have the Docker CMD specify env variables inserted at time of Dockerfile generation
They're overrideable with the global enviornment as for tests. This enables
variable generation in orchestrator and output to productionized Docker files
without creating a life-long file within the Docker container.
* Don't add Dockerfiles into Docker containers now that they have secrets
Solely add the source code for them as needed to satisfy the workspace bounds.
* Download arm64 Monero on arm64
* Ensure constant host architecture when reproducibly building the wasm
Host architecture, for some reason, can effect the generated code despite the
target architecture always being foreign to the host architecture.
* Randomly generate infrastructure keys
* Have orchestrator generate a key, be able to create/start containers
* Ensure bash is used over sh
* Clean dated docs
* Change how quoting occurs
* Standardize to sh
* Have Docker test build the dev Dockerfiles
* Only key_gen once
* cargo update
Adds a patch for zstd and reconciles the breaking nightly change which just
occurred.
* Use a dedicated network for Serai
Also fixes SERAI_HOSTNAME passed to coordinator.
* Support providing a key over the env for the Serai node
* Enable and document running daemons for tests via serai-orchestrator
Has running containers under the dev network port forward the RPC ports.
* Use volumes for bitcoin/monero
* Use bitcoin's run.sh in GH CI
* Only use the volume for testnet (not dev)
* Route validators for any active set through sc-authority-discovery
Additionally adds an RPC route to retrieve their P2P addresses.
* Have the coordinator get peers from substrate
* Have the RPC return one address, not up to 3
Prevents the coordinator from believing it has 3 peers when it has one.
* Add missing feature to serai-client
* Correct network argument in serai-client for p2p_validators call
* Add a test in serai-client to check DHT population with a much quicker failure than the coordinator tests
* Update to latest Substrate
Removes distinguishing BABE/AuthorityDiscovery keys which causes
sc_authority_discovery to populate as desired.
* Update to a properly tagged substrate commit
* Add all dialed to peers to GossipSub
* cargo fmt
* Reduce common code in serai-coordinator-tests with amore involved new_test
* Use a recursive async function to spawn `n` DockerTests with the necessary networking configuration
* Merge UNIQUE_ID and ONE_AT_A_TIME
* Tidy up the new recursive code in tests/coordinator
* Use a Mutex in CONTEXT to let it be set multiple times
* Make complimentary edits to full-stack tests
* Augment coordinator P2p connection logs
* Drop lock acquisitions before recursing
* Better scope lock acquisitions in full-stack, preventing a deadlock
* Ensure OUTER_OPS is reset across the test boundary
* Add cargo deny allowance for dockertest fork
* Move pallet-asset-conversion
* update licensing
* initial integration
* Integrate Currency & Assets types
* integrate liquidity tokens
* fmt
* integrate dex pallet tests
* fmt
* compilation error fixes
* integrate dex benchmarks
* fmt
* cargo clippy
* replace all occurrences of "asset" with "coin"
* add the actual add liq/swap logic to in-instructions
* add client side & tests
* fix deny
* Lint and changes
- Renames InInstruction::AddLiquidity to InInstruction::SwapAndAddLiquidity
- Makes create_pool an internal function
- Makes dex-pallet exclusively create pools against a native coin
- Removes various fees
- Adds new crates to GH workflow
* Fix rebase artifacts
* Correct other rebase artifact
* Correct CI specification for liquidity-tokens
* Correct primitives' test to the standardized pallet account scheme
---------
Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
Resolves#353
Implements code such that:
- 80% of validators (by stake) must be in favor of a signal for the network to
be
- 80% of networks (by stake) must be in favor of a signal for it to be locked
in
- After a signal has been locked in for two weeks, the network halts
The intention is to:
1) Not allow validators to unilaterally declare new consensus rules.
No method of declaring new consensus rules is provided by this pallet. Solely a
way to deprecate the current rules, with a signaled for successor. All nodes
must then individually decide whether or not to download and run a new node
which has new rules, and if so, which rules.
2) Not place blobs on chain.
Even if they'd be reproducible, it's just a lot of data to chuck on the
blockchain.
* initial implementation
* add function to get a balance of an account
* add support for multiple coins
* rename pallet to "coins-pallet"
* replace balances, assets and tokens pallet with coins pallet in runtime
* add total supply info
* update client side for new Coins pallet
* handle fees
* bug fixes
* Update FeeAccount test
* Fmt
* fix pr comments
* remove extraneous Imbalance type
* Minor tweaks
---------
Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
* initial staking pallet
* add staking pallet to runtime
* support session rotation for serai
* optimizations & cleaning
* fix deny
* add serai network to initial networks
* a few tweaks & comments
* fix some pr comments
* Rewrite validator-sets with logarithmic algorithms
Uses the fact the underlying DB is sorted to achieve sorting of potential
validators by stake.
Removes release of deallocated stake for now.
---------
Co-authored-by: Luke Parker <lukeparker5132@gmail.com>
* Design and document a multisig rotation flow
* Make Scanner::eventualities a HashMap so it's per-key
* Don't drop eventualities, always follow through on them
Technical improvements made along the way.
* Start creating an isolate object to manage multisigs, which doesn't require being a signer
Removes key from SubstrateBlock.
* Move Scanner/Scheduler under multisigs
* Move Batch construction into MultisigManager
* Clarify "should" in Multisig Rotation docs
* Add block_number to MultisigManager, as it controls the scanner
* Move sign_plans into MultisigManager
Removes ThresholdKeys from prepare_send.
* Make SubstrateMutable an alias for MultisigManager
* Rewrite Multisig Rotation
The prior scheme had an exploit possible where funds were sent to the old
multisig, then burnt on Serai to send from the new multisig, locking liquidity
for 6 hours. While a fee could be applied to stragglers, to make this attack
unprofitable, the newly described scheme avoids all this.
* Add mini
mini is a miniature version of Serai, emphasizing Serai's nature as a
collection of independent clocks. The intended use is to identify race
conditions and prove protocols are comprehensive regarding when certain clocks
tick.
This uses loom, a prior candidate for evaluating the processor/coordinator as
free of race conditions (#361).
* Use mini to prove a race condition in the current multisig rotation docs, and prove safety of alternatives
Technically, the prior commit had mini prove the race condition.
The docs currently say the activation block of the new multisig is the block
after the next Batch's. If the two next Batches had already entered the
mempool, prior to set_keys being called, the second next Batch would be
expected to contain the new key's data yet fail to as the key wasn't public
when the Batch was actually created.
The naive solution is to create a Batch, publish it, wait until it's included,
and only then scan the next block. This sets a bound of
`Batch publication time < block time`. Optimistically, we can publish a Batch
in 24s while our shortest block time is 2m. Accordingly, we should be fine with
the naive solution which doesn't take advantage of throughput. #333 may
significantly change latency however and require an algorithm whose throughput
exceeds the rate of blocks created.
In order to re-introduce parallelization, enabling throughput, we need to
define a safe range of blocks to scan without Serai ordering the first one.
mini demonstrates safety of scanning n blocks Serai hasn't acknowledged, so
long as the first is scanned before block n+1 is (shifting the n-block window).
The docs will be updated next, to reflect this.
* Fix Multisig Rotation
I believe this is finally good enough to be final.
1) Fixes the race condition present in the prior document, as demonstrated by
mini.
`Batch`s for block `n` and `n+1`, may have been in the mempool when a
multisig's activation block was set to `n`. This would cause a potentially
distinct `Batch` for `n+1`, despite `n+1` already having a signed `Batch`.
2) Tightens when UIs should use the new multisig to prevent eclipse attacks,
and protection against `Batch` publication delays.
3) Removes liquidity fragmentation by tightening flow/handling of latency.
4) Several clarifications and documentation of reasoning.
5) Correction of "prior multisig" to "all prior multisigs" regarding historical
verification, with explanation why.
* Clarify terminology in mini
Synchronizes it from my original thoughts on potential schema to the design
actually created.
* Remove most of processor's README for a reference to docs/processor
This does drop some misc commentary, though none too beneficial. The section on
scanning, deemed sufficiently beneficial, has been moved to a document and
expanded on.
* Update scanner TODOs in line with new docs
* Correct documentation on Bitcoin::Block::time, and Block::time
* Make the scanner in MultisigManager no longer public
* Always send ConfirmKeyPair, regardless of if in-set
* Cargo.lock changes from a prior commit
* Add a policy document on defining a Canonical Chain
I accidentally committed a version of this with a few headers earlier, and this
is a proper version.
* Competent MultisigManager::new
* Update processor's comments
* Add mini to copied files
* Re-organize Scanner per multisig rotation document
* Add RUST_LOG trace targets to e2e tests
* Have the scanner wait once it gets too far ahead
Also bug fixes.
* Add activation blocks to the scanner
* Split received outputs into existing/new in MultisigManager
* Select the proper scheduler
* Schedule multisig activation as detailed in documentation
* Have the Coordinator assert if multiple `Batch`s occur within a block
While the processor used to have ack_up_to_block, enabling skips in the block
acked, support for this was removed while reworking it for multiple multisigs.
It should happen extremely infrequently.
While it would still be beneficial to have, if multiple `Batch`s could occur
within a block (with the complexity here not being worth adding that ban as a
policy), multiple `Batch`s were blocked for DoS reasons.
* Schedule payments to the proper multisig
* Correct >= to <
* Use the new multisig's key for change on schedule
* Don't report External TXs to prior multisig once deprecated
* Forward from the old multisig to the new one at all opportunities
* Move unfulfilled payments in queue from prior to new multisig
* Create MultisigsDb, splitting it out of MainDb
Drops the call to finish_signing from the Signer. While this will cause endless
re-attempts, the Signer will still consider them completed and drop them,
making this an O(n) cost at boot even if we did nothing from here.
The MultisigManager should call finish_signing once the Scanner completes the
Eventuality.
* Don't check Scanner-emitted completions, trust they are completions
Prevents needing to use async code to mark the completion and creates a
fault-free model. The current model, on fault, would cause a lack of marked
completion in the signer.
* Fix a possible panic in the processor
A shorter-chain reorg could cause this assert to trip. It's fixed by
de-duplicating the data, as the assertion checked consistency. Without the
potential for inconsistency, it's unnecessary.
* Document why an existing TODO isn't valid
* Change when we drop payments for being to the change address
The earlier timing prevents creating Plans solely to the branch address,
causing the payments to be dropped, and the TX to become an effective
aggregation TX.
* Extensively document solutions to Eventualities being potentially created after having already scanned their resolutions
* When closing, drop External/Branch outputs which don't cause progress
* Properly decide if Change outputs should be forward or not when closing
This completes all code needed to make the old multisig have a finite lifetime.
* Commentary on forwarding schemes
* Provide a 1 block window, with liquidity fragmentation risks, due to latency
On Bitcoin, this will be 10 minutes for the relevant Batch to be confirmed. On
Monero, 2 minutes. On Ethereum, ~6 minutes.
Also updates the Multisig Rotation document with the new forwarding plan.
* Implement transaction forwarding from old multisig to new multisig
Identifies a fault where Branch outputs which shouldn't be dropped may be, if
another output fulfills their next step. Locking Branch fulfillment down to
only Branch outputs is not done in this commit, but will be in the next.
* Only let Branch outputs fulfill branches
* Update TODOs
* Move the location of handling signer events to avoid a race condition
* Avoid a deadlock by using a RwLock on a single txn instead of two txns
* Move Batch ID out of the Scanner
* Increase from one block of latency on new keys activation to two
For Monero, this offered just two minutes when our latency to publish a Batch
is around a minute already. This does increase the time our liquidity can be
fragmented by up to 20 minutes (Bitcoin), yet it's a stupid attack only
possible once a week (when we rotate). Prioritizing normal users' transactions
not being subject to forwarding is more important here.
Ideally, we'd not do +2 blocks yet plus `time`, such as +10 minutes, making
this agnostic of the underlying network's block scheduling. This is a
complexity not worth it.
* Split MultisigManager::substrate_block into multiple functions
* Further tweaks to substrate_block
* Acquire a lock on all Scanner operations after calling ack_block
Gives time to call register_eventuality and initiate signing.
* Merge sign_plans into substrate_block
Also ensure the Scanner's lock isn't prematurely released.
* Use a HashMap to pass to-be-forwarded instructions, not the DB
* Successfully determine in ClosingExisting
* Move from 2 blocks of latency when rotating to 10 minutes
Superior as noted in 6d07af92ce10cfd74c17eb3400368b0150eb36d7, now trivial to
implement thanks to prior commit.
* Add note justifying measuring time in blocks when rotating
* Implement delaying of outputs received early to the new multisig per specification
* Documentation on why Branch outputs don't have the race condition concerns Change do
Also ensures 6 hours is at least N::CONFIRMATIONS, for sanity purposes.
* Remove TODO re: sanity checking Eventualities
We sanity check the Plan the Eventuality is derived from, and the Eventuality
is handled moments later (in the same file, with a clear call path). There's no
reason to add such APIs to Eventualities for a sanity check given that.
* Add TODO(now) for TODOs which must be done in this branch
Also deprecates a pair of TODOs to TODO2, and accepts the flow of the Signer
having the Eventuality.
* Correct errors in potential/future flow descriptions
* Accept having a single Plan Vec
Per the following code consuming it, there's no benefit to bifurcating it by
key.
* Only issue sign_transaction on boot for the proper signer
* Only set keys when participating in their construction
* Misc progress
Only send SubstrateBlockAck when we have a signer, as it's only used to tell
the Tributary of what Plans are being signed in response to this block.
Only immediately sets substrate_signer if session is 0.
On boot, doesn't panic if we don't have an active key (as we wouldn't if only
joining the next multisig). Continues.
* Correctly detect and set retirement block
Modifies the retirement block from first block meeting requirements to block
CONFIRMATIONS after.
Adds an ack flow to the Scanner's Confirmed event and Block event to accomplish
this, which may deadlock at this time (will be fixed shortly).
Removes an invalid await (after a point declared unsafe to use await) from
MultisigsManager::next_event.
* Remove deadlock in multisig_completed and document alternative
The alternative is simpler, albeit less efficient. There's no reason to adopt
it now, yet perhaps if it benefits modeling?
* Handle the final step of retirement, dropping the old key and setting new to existing
* Remove TODO about emitting a Block on every step
If we emit on NewAsChange, we lose the purpose of the NewAsChange period.
The only concern is if we reach ClosingExisting, and nothing has happened, then
all coins will still be in the old multisig until something finally does. This
isn't a problem worth solving, as it's latency under exceptional dead time.
* Add TODO about potentially not emitting a Block event for the reitrement block
* Restore accidentally deleted CI file
* Pair of slight tweaks
* Add missing if statement
* Disable an assertion when testing
One of the test flows currently abuses the Scanner in a way triggering it.
Adds 17 new crates, which I'm extremely unhappy about. Unfortunately, it's
needed to resolve a security issue (RUSTSEC-2023-0052) and is inevitable.
Closes#355.
Achieves three notable updates.
1) Resolves RUSTSEC-2022-0093 by updating libp2p-identity.
2) Removes 3 old rand crates via updating ed25519-dalek (a dependency of
libp2p-identity).
3) Sets serde_derive to 1.0.171 via updating to time 0.3.26 which pins at up to
1.0.171.
The last one is the most important. The former two are niceties.
serde_derive, since 1.0.171, ships a non-reproducible binary blob in what's a
complete compromise of supply chain security. This is done in order to reduce
compile times, yet also for the maintainer of serde (dtolnay) to leverage
serde's position as the 8th most downloaded crate to attempt to force changes
to the Rust build pipeline.
While dtolnay's contributions to Rust are respectable, being behind syn, quote,
and proc-macro2 (the top three crates by downloads), along with thiserror,
anyhow, async-trait, and more (I believe also being part of the Rust project),
they have unfortunately decided to refuse to listen to the community on this
issue (or even engage with counter-commentary). Given their political agenda
they seem to try to be accomplishing with force, I'd go as far as to call their
actions terroristic (as they're using the threat of the binary blob as
justification for cargo to ship 'proper' support for binary blobs).
This is arguably representative of dtolnay's past work on watt. watt was a wasm
interpreter to execute a pre-compiled proc macro. This would save the compile
time of proc macros, yet sandbox it so a full binary did not have to be run.
Unfortunately, watt (while decreasing compile times) fails to be a valid
solution to supply chain security (without massive ecosystem changes). It never
implemented reproducible builds for its wasm blobs, and a malicious wasm blob
could still fundamentally compromise a project. The only solution for an end
user to achieve a secure pipeline would be to locally build the project,
verifying the blob aligns, yet doing so would negate all advantages of the
blob.
dtolnay also seems to be giving up their role as a FOSS maintainer given that
serde no longer works in several environments. While FOSS maintainers are not
required to never implement breaking changes, the version number is still 1.0.
While FOSS maintainers are not required to follow semver, releasing a very
notable breaking change *without a new version number* in an ecosystem which
*does follow semver*, then refusing to acknowledge bugs as bugs with their work
does meet my personal definition of "not actively maintaining their existing
work". Maintenance would be to fix bugs, not introduce and ignore.
For now, serde > 1.0.171 has been banned. In the future, we may host a fork
without the blobs (yet with the patches). It may be necessary to ban all of
dtolnay's maintained crates, if they continue to force their agenda as such,
yet I hope this may be resolved within the next week or so.
Sources:
https://github.com/serde-rs/serde/issues/2538 - Binary blob discussion
This includes several reports of various workflows being broken.
https://github.com/serde-rs/serde/issues/2538#issuecomment-1682519944
dtolnay commenting that security should be resolved via Rust toolchain edits,
not via their own work being secure. This is why I say they're trying to
leverage serde in a political game.
https://github.com/serde-rs/serde/issues/2526 - Usage via git broken
dtolnay explicitly asks the submitting user if they'd be willing to advocate
for changes to Rust rather than actually fix the issue they created. This is
further political arm wrestling.
https://github.com/serde-rs/serde/issues/2530 - Usage via Bazel broken
https://github.com/serde-rs/serde/issues/2575 - Unverifiable binary blob
https://github.com/dtolnay/watt - dtolnay's prior work on precompilation
* Move monero-serai from std to std-shims, where possible
* no-std fixes
* Make the HttpRpc its own feature, thiserror only on std
* Drop monero-rs's epee for a homegrown one
We only need it for a single function. While I tried jeffro's, it didn't work
out of the box, had three unimplemented!s, and is no where near viable for
no_std.
Fixes#182, though should be further tested.
* no-std monero-serai
* Allow base58-monero via git
* cargo fmt
* Partial move to ff 0.13
It turns out the newly released k256 0.12 isn't on ff 0.13, preventing further
work at this time.
* Update all crates to work on ff 0.13
The provided curves still need to be expanded to fit the new API.
* Finish adding dalek-ff-group ff 0.13 constants
* Correct FieldElement::product definition
Also stops exporting macros.
* Test most new parts of ff 0.13
* Additionally test ff-group-tests with BLS12-381 and the pasta curves
We only tested curves from RustCrypto. Now we test a curve offered by zk-crypto,
the group behind ff/group, and the pasta curves, which is by Zcash (though
Zcash developers are also behind zk-crypto).
* Finish Ed448
Fully specifies all constants, passes all tests in ff-group-tests, and finishes moving to ff-0.13.
* Add RustCrypto/elliptic-curves to allowed git repos
Needed due to k256/p256 incorrectly defining product.
* Finish writing ff 0.13 tests
* Add additional comments to dalek
* Further comments
* Update ethereum-serai to ff 0.13
Updates to polkadot-v0.9.40, with a variety of dependency updates accordingly.
Substrate thankfully now uses k256 0.13, pathing the way for #256. We couldn't
upgrade to polkadot-v0.9.40 without this due to polkadot-v0.9.40 having
fundamental changes to syncing. While we could've updated tendermint, it's not
worth the continued development effort given its inability to work with
multiple validator sets.
Purges sc-tendermint. Keeps tendermint-machine for #163.
Closes#137, #148, #157, #171. #96 and #99 should be re-scoped/clarified. #134
and #159 also should be clarified. #169 is also no longer a priority since
we're only considering temporal deployments of tendermint. #170 also isn't
since we're looking at effectively sharded validator sets, so there should
be no singular large set needing high performance.
