Disaster Recovery and Node Migration

What to do if something goes wrong

In all cases, NodeSet recommends operators keep backup hardware ready. If you have a testnet node running, you can use it for mainnet while you diagnose the issue with your other node.

Diagnosing Issues

If your node is accessible via SSH but is no longer attesting correctly

First, check to ensure Hyperdrive and your operating system are up to date.

Next, you should check the node logs for errors or other signs of unexpected behavior.

If you cannot access your node via SSH

First, check the physical hardware for issues, especially your network connection. If it's still running, manually log into the system and attempt to diagnose the issue that way.

Migrating to a New Node

Whether you're upgrading your node hardware or recovering from a problem, migrating to a new node is easy with Hyperdrive.

If your old node is accessible

The easiest way to migrate to new node hardware is to re-use your old storage drive with new hardware. If there's nothing wrong with your filesystem, you can simply move the old drive over to the new system with no major consequences.

If your filesystem is lost -- due to drive failure, for example -- you should physically destroy the drive before disposal to ensure sophisticated actors cannot recover any private keys from the device. Then, you can begin the wallet recovery process with new hardware using your seed phrase backup.

Wallet Recovery

Wallet recovery is the process for restoring a node from an offline seed phrase backup. Note that wallet recovery does not recover your configuration, however.

To minimize downtime, you should initiate a wallet recovery on new hardware as soon as you realize the old filesystem is not recoverable.

First, install Hyperdrive as usual on your new node. When Hyperdrive asks you to create a wallet, reply n. This will save you the step of creating and validating a new mnemonic which won't be used.

Next, use the hyperdrive wallet recover command to regenerate your node wallet based on your seed phrase backup. From here, you should review your configuration with hyperdrive service config and monitor the node as usual to ensure it is operational.

Recovering the StakeWise Module

After you've recovered your wallet, you can enable the StakeWise module in the Hyperdrive Configuration terminal hyperdrive service config -> Modules menu.

After exiting the configuration terminal, Hyperdrive asks you to verify you don't have any running validators, and a final verification that if you did have validators running, you've waited 15 minutes, because Hyperdrive can't determine if you attested in the last 15 minutes. Because you're about to regenerate the same keys, this will apply to you, and you MUST ensure that the node you're recovering hasn't attested in the last 15 minutes or you may be slashed!

NOTE: There is currently no way to remove a validator's deposit data from the NodeSet service once you've uploaded it. The key will be eligible for activation at any time, so this node must remain online at all times to handle activation and validation duties.
If you turn the node off, you may be removed from NodeSet for negligence of duty!

Do you want to continue uploading your deposit data? [y/n]
y

Uploading deposit data to the NodeSet server...
All of your validator keys are already registered (10 in total).
7 are pending activation.
3 have been activated already.

After you've enabled the StakeWise module, you need to recover your validator keys. The reason for this is that the new Hyperdrive installation currently has no way of knowing how many keys were generated before. Recover your keys with:

hyperdrive stakewise wallet recover-keys

This will ask nodeset.io which keys you have registered already, and procedurally generate validator keys until it finds all of them:

The following validator keys have been registered and can be recovered:
Hoodi Dev Vault (0xb163b6d4E1B317F3B4BacE8770d74C1c21C4a131):
	0x94f566037b51b6f6b6a3a676bba6b0b31f83b018327eca47aa2f1baf0b24f5110745a03b11073fddad2277b03ed751ac
	0x810bab09ed301446e04f6d8a28aff39457c8198ac485fb0abb08082d74099791907d2f4b27643276115a5d3fa1102f42
	0xa526d331333773a2b4480121f482ea9bb05d589f4bf594f23e2100bfd36b88660c4e2329a3af36e215bf625cf1f13869
	0x903bee1b9f05c133548f4afae99b7c51cfd1646389a629554a49bf69cb7ce0e4216ee50e3b882a665ca595949fff65aa

NOTE: Key recovering may take a long time. Progress will be printed after checking every 5 keys.
Are you ready to begin key recovery? [y/n]
y

Searching index 0 to 4...
Searching index 5 to 9...
Searching index 10 to 14...
Recovered 0x810bab09ed301446e04f6d8a28aff39457c8198ac485fb0abb08082d74099791907d2f4b27643276115a5d3fa1102f42 (index 11)
Searching index 12 to 16...
Recovered 0x94f566037b51b6f6b6a3a676bba6b0b31f83b018327eca47aa2f1baf0b24f5110745a03b11073fddad2277b03ed751ac (index 12)
Searching index 13 to 17...
Recovered 0xa526d331333773a2b4480121f482ea9bb05d589f4bf594f23e2100bfd36b88660c4e2329a3af36e215bf625cf1f13869 (index 13)
Searching index 14 to 18...
Recovered 0x903bee1b9f05c133548f4afae99b7c51cfd1646389a629554a49bf69cb7ce0e4216ee50e3b882a665ca595949fff65aa (index 14)
Key recovery complete.

Restarting Validator Client to load the recovered keys... done!
Your recovered keys are now loaded.
Your node can now attest for these validators.

Note that the recover-keys command will only recover keys that you used for deposits already. It won't recover any keys that haven't been used yet, so you'll have to generate new keys again with the hyperdrive stakewise wallet generate-keys command. Remember to always keep enough ETH in your node wallet to fund the creation of new validators with your unused keys.

Recovering the Constellation Module

After you've recovered your wallet, you can enable the Constellation module in the Hyperdrive Configuration terminal hyperdrive service config -> Modules menu.

After exiting the configuration terminal, Hyperdrive asks you to verify you don't have any running validators, and a final verification that if you did have validators running, you've waited 15 minutes, because Hyperdrive can't determine if you attested in the last 15 minutes. Because you're about to regenerate the same keys, this will apply to you, and you MUST ensure that the node you're recovering hasn't attested in the last 15 minutes or you may be slashed!

The following keys will be rebuilt:
        Minipool: 0xeDa3C8D065B97609Be536bF54fEC757D11D79CD5
        Validator: 0x992b0131967c3bd4e840075e2189c7a6836c5b858ee964e4fdd05a1d3230d9668259fdb3651b5ef63bad06e17b37b330
        Index: 1814204

Are you ready to rebuild these keys? [y/n]
y

Rebuilding 0x992b0131967c3bd4e840075e2189c7a6836c5b858ee964e4fdd05a1d3230d9668259fdb3651b5ef63bad06e17b37b330... done! (2.228147452s)
1 keys have been rebuilt.

Your Constellation Validator Client must be restarted in order to load the validator keys and resume attesting wth them.
Would you like to restart the Constellation Validator Client now? [y/n]
y

Successfully restarted the Constellation Validator Client. Your rebuilt validator keys are now loaded.

After you've enabled the Constellation module, you need to rebuild your validator keys. The Constellation module searches for activated keys from the Constellation wallet derivation path, by default from index 0 to index 500. You can rebuild your validator keys with hyperdrive constellation wallet rebuild. The alias version of this command is hyperdrive cs w b.

Lost Seed Phrase

All node operators should keep an offline backup of your seed phrase in case of emergency. If you lose your seed phrase but still have access to your old node, the private key for your node wallet is located inside your wallet file, which is typically located at ~/.hyperdrive/data/wallet

If you lose your seed phrase and no longer have access your old node's filesystem, you should immediately contact NodeSet for help.

You are at risk of being ejected from NodeSet!