Common Problems

We'll continue updating this page so you can easily troubleshoot your node.

How does this page work?

This page is designed to help you fix all of your problems on your own!

First of all you need dump recent logs from your terminal.

journalctl -f -u "chainflip-*"

In order to quickly understand your issue, please first check if your log output resembles any of these.

Not Funded

You will see this log before the Chainflip network witnesses that you added funds to your validator. Once you have successfully added funds, it can take a while for the network to register it. Your node might appear Offline until then.

Logs

{
  "tag": "",
  "msg": "Your Chainflip account cFNugEinCy5Zxv44B4o8uU4PSigKmmzdUYabbZbfNjAAZx18a is not funded. WAITING for account to be funded at block: 143222",
  "level": "warn",
  "ts": "2022-11-18T09:45:37.474214013+00:00",
  "component": "StateChainConnector"
}

Solution

Please make sure you followed all the steps outlined in Funding.

If after 30 minutes your engine is still not online, try restart your engine
sudo systemctl restart chainflip-engine.

Incorrectly configured the ETH RPC

This can have a number of possible issues.

Logs

{"tag":"","msg":"Connecting new HTTP web3 client to https://eth****","level":"debug","ts":"2022-11-18T09:25:21.576142234+00:00","component":"Eth-DualRpcClient"}
Error: Failed to create EthDualRpcClient
 
Caused by:
    Inconsistent chain configuration. Terminating.Expected ETH chain id 5, received 1 through WebSocket., Expected ETH chain id 5, received 1 through HTTP.

Solution

You have not selected the correct Ethereum network. In this case, you have selected mainnet. The Chainflip network uses Görli. Please reread #pointing-to-the-ethereum-client.

Logs

{"tag":"","msg":"Connecting new WebSocket web3 client to wss://eth****","level":"debug","ts":"2022-11-18T10:09:02.643481895+00:00","component":"Eth-DualRpcClient"}
Error: Failed to create EthDualRpcClient
 
Caused by:
    0: Failed to create EthWsRpcClient
    1: Error: 'web3::transports::WebSocket::new(&eth_settings.ws_endpoint).await' with type 'core::result::Result<web3::transports::ws::WebSocket, web3::error::Error>' failed at https://github.com/chainflip-io/chainflip-backend/tree/5e37992a14bb14d8d0609687ebe8f72e9402fbc0/engine/src/eth/rpc.rs#L266#C5
    2: code 401

Solution

You have entered your RPC endpoint in incorrectly. Please make sure you copy it over without any mistakes.

Logs

{"tag":"","msg":"Connecting new WebSocket web3 client to wss://SOM****.goerli.ws.rivet.cloud","level":"debug","ts":"2022-11-18T10:14:58.473000904+00:00","component":"Eth-DualRpcClient"}
Error: Failed to create EthDualRpcClient
 
Caused by:
    0: Failed to create EthWsRpcClient
    1: Error: 'web3::transports::WebSocket::new(&eth_settings.ws_endpoint).await' with type 'core::result::Result<web3::transports::ws::WebSocket, web3::error::Error>' failed at https://github.com/chainflip-io/chainflip-backend/tree/5e37992a14bb14d8d0609687ebe8f72e9402fbc0/engine/src/eth/rpc.rs#L266#C5
    2: Ssl(Error { code: ErrorCode(1), cause: Some(Ssl(ErrorStack([Error { code: 337047686, library: "SSL routines", function: "tls_process_server_certificate", reason: "certificate verify failed", file: "../ssl/statem/statem_clnt.c", line: 1913 }]))) }, X509VerifyResult { code: 62, error: "Hostname mismatch" })

Solution

You have copied the chainflip-engine config without changing any of the values! Please re-read the steps in Engine Settings . You need to make that you use the URL from your Ethereum Client. #pointing-to-the-ethereum-client.

Node killed Out Of Memory

Logs

If your node logs contain the following error:

chainflip-node.service: Main process exited, code=killed, status=9/KILL

It means that your node is running OOM and it is getting killed by the OS.

Solution

You need to create a swapfile (8GB), you can follow this guide in order to do so: https://linuxize.com/post/create-a-linux-swap-file/ (opens in a new tab)

Auctions App

Node Offline — Reasons

Likely you haven't completed all the steps. Please start from Generating Keys section and work your way very slowly, making sure you have read everything.

It can be, that you did not successfully submit the chainflip-cli commands. Please run all the commands in Funding and Bidding section.

Turning Authority Online

You need to wait until the next Rotation. Make sure your node is Online with the labels Backup and Bidding and Qualified.

Authority Offline — Reasons

Your engine has stopped submitting Hearbeats. This means either your node, your engine or both have crashed or are misconfigured.

Verify this with:

sudo systemctl status chainflip-node
sudo systemctl status chainflip-engine

Now check the logs:

journalctl -f -u "chainflip-*"

Backup and Bidding but not Qualified — Reasons

You did not successfully submit the chainflip-cli commands. Please run all the commands in Funding and Bidding section.

The commands that already ran successfully will fail with an error resembling the following:

This is expected, continue with the next command until you complete them all, after completing this your node should appear Backup and Bidding and Qualified.

Ethereum Keyfile Error

thread 'main' panicked at 'called Result::unwrap() on an Err value: Failed to decode Ethereum Private Key file at /root/chainflip/ethereum_key_file

This is due to your key likely either having 0x on the front, or having a newline at the end. You can use nano to edit the file. Make sure to include the -L argument, otherwise nano will save the file with a new line at the end! Make sure that you replace /path/to/ethereum_key_file with the correct file path on your system.

nano -L /path/to/ethereum_key_file

Your validator wallet is out of gETH

This is a very common problem if you have joined the network early.
Make sure you have at least 0.1 gETH at all time in your validator's wallet. Otherwise you'll fail to submit transactions to the Ethereum network.

Why Slash? Why low reputation?

Slashoor has paid you a visit, hasn't it?

Overview

If you find yourself being penalized and your reputation points dropping, it is possible that the chainflip-engine is not functioning optimally. This discussion will focus specifically on ETH witnessing issues.

One potential cause is using a free-tier managed service for your RPC endpoint provider (such as Alchemy, Rivet, or Infura). This can lead to rate-limiting problems based on network activity and the stability of your engine. Rate-limiting models differ among services. Some have a fixed daily request limit, while others offer unlimited daily requests but impose a cap on "compute units" per second. These varying models may result in different behaviors under certain conditions.

If you encounter rate-limiting, your engine will be unable to witness ETH events, which can lead to immediate penalties or reductions in reputation points, potentially culminating in slashing. Additionally, failure to properly witness ETH events can cause rotations to take longer and reduce your chances of becoming an Authority member.

Solution

You have multiple options here:

Using Two Different ETH RPC Providers

This is a good method for setting up your RPC configuration. Sign up for two accounts, such as Infura (opens in a new tab), Rivet (opens in a new tab), and Alchemy (opens in a new tab) (choose two) and update your engine configuration to use Infura for the rpc and Rivet for backup_rpc, or vice versa, for each chain. This approach ensures protection against service outages from either provider.

Subscribe for a paid tier at your RPC Endpoint provider

Subscribe to a paid tier at your RPC endpoint provider: Though this option incurs a monthly fee, it requires minimal effort and is recommended for those who want a hassle-free start without setting up their own ETH nodes or worry about rate-limiting.

Run your own ETH node and expose it as an RPC endpoint

This option, which involves connecting directly to the ETH network, is the most reliable solution. However, it demands strong technical skills for setting up and operating ETH nodes, which is beyond the scope of this guide. For more information, visit: https://geth.ethereum.org/docs/getting-started (opens in a new tab)

Updating your config file

After setting up your RPC endpoints as described above, you need to update the config file and restart your engine. You can do this by running:

sudo nano /etc/chainflip/config/Settings.toml

Your config file will look something like this:

Save your changes and exit then restart the engine by running:

sudo systemctl restart chainflip-engine.service

It still doesn't work?

Follow instructions under Submitting an issue.