Disaster Recovery and Node Migration
What to do if something goes wrong
In all cases, NodeSet recommends operators keep backup hardware ready. If you have a testnet node running, you can use it for mainnet while you diagnose the issue with your other node.
Diagnosing Issues
If your node is accessible via SSH but is no longer attesting correctly
First, check to ensure Hyperdrive and your operating system are up to date.
Next, you should check the node logs for errors or other signs of unexpected behavior.
If you cannot access your node via SSH
First, check the physical hardware for issues, especially your network connection. If it's still running, manually log into the system and attempt to diagnose the issue that way.
Migrating to a New Node
Whether you're upgrading your node hardware or recovering from a problem, migrating to a new node is easy with Hyperdrive.
If your old node is accessible
The easiest way to migrate to new node hardware is to re-use your old storage drive with new hardware. If there's nothing wrong with your filesystem, you can simply move the old drive over to the new system with no major consequences.
If your filesystem is lost -- due to drive failure, for example -- you should physically destroy the drive before disposal to ensure sophisticated actors cannot recover any private keys from the device. Then, you can begin the wallet recovery process with new hardware using your seed phrase backup.
Wallet Recovery
Wallet recovery is the process for restoring a node from an offline seed phrase backup. Note that wallet recovery does not recover your configuration, however.
To minimize downtime, you should initiate a wallet recovery on new hardware as soon as you realize the old filesystem is not recoverable.
First, install Hyperdrive as usual on your new node. When Hyperdrive asks you to create a wallet, reply n. This will save you the step of creating and validating a new mnemonic which won't be used.
Next, use the hyperdrive wallet recover command to regenerate your node wallet based on your seed phrase backup. From here, you should review your configuration with hyperdrive service config and monitor the node as usual to ensure it is operational.
Recovering the StakeWise Module
After you've recovered your wallet, you can enable the StakeWise module in the Hyperdrive Configuration terminal hyperdrive service config -> Modules menu.
After exiting the configuration terminal, Hyperdrive asks you to verify you don't have any running validators, and a final verification that if you did have validators running, you've waited 15 minutes, because Hyperdrive can't determine if you attested in the last 15 minutes. Because you're about to regenerate the same keys, this will apply to you, and you MUST ensure that the node you're recovering hasn't attested in the last 15 minutes or you may be slashed!
After you've enabled the StakeWise module, you need to regenerate your validator keys. The reason for this is that the new Hyperdrive installation currently has no way of knowing how many keys were generated before. Regenerate the same number of keys with hyperdrive stakewise wallet generate -c X where X is the number of keys to generate. The alias version of this command is hyperdrive sw w g -c X.
Hyperdrive will ask if you want to continue uploading your deposit data. Even though it's already been uploaded, it's safe to do this, as Hyperdrive will detect they've already been uploaded, and report back the status.
Recovering the Constellation Module
After you've recovered your wallet, you can enable the Constellation module in the Hyperdrive Configuration terminal hyperdrive service config -> Modules menu.
After exiting the configuration terminal, Hyperdrive asks you to verify you don't have any running validators, and a final verification that if you did have validators running, you've waited 15 minutes, because Hyperdrive can't determine if you attested in the last 15 minutes. Because you're about to regenerate the same keys, this will apply to you, and you MUST ensure that the node you're recovering hasn't attested in the last 15 minutes or you may be slashed!
After you've enabled the Constellation module, you need to rebuild your validator keys. The Constellation module searches for activated keys from the Constellation wallet derivation path, by default from index 0 to index 500. You can rebuild your validator keys with hyperdrive constellation wallet rebuild. The alias version of this command is hyperdrive cs w b.
Lost Seed Phrase
All node operators should keep an offline backup of your seed phrase in case of emergency. If you lose your seed phrase but still have access to your old node, the private key for your node wallet is located inside your wallet file, which is typically located at ~/.hyperdrive/data/wallet
If you lose your seed phrase and no longer have access your old node's filesystem, you should immediately contact NodeSet for help.
You are at risk of being ejected from NodeSet!
Last updated