Managing The Validator

This page describes the regular tasks that you may need to do, as a node runner.

Handling Upgrades

Depending on the type of upgrade being undertaken by the wider network, one of the options below should be used to perform the upgrade. Automatic upgrades via the x/upgrade module and/or Cosmovisor are not yet supported.

The type of upgrade required by the network impacts which options you may select.

Option 1: Upgrade with chain-id change. Without coordination.

This type of upgrade is the most commonly used by Exocore. For example, the testnet was upgraded from exocoretestnet_233-5 to exocoretestnet_233-6 with this option.

  1. Stop the node.

  2. Remove the existing data

    rm -rf $HOMEDIR/config/genesis.json $HOMEDIR/data/*.db $HOMEDIR/data/snapshots $HOMEDIR/data/cs.wal

  3. Overwrite the validator's signature history, if any

    jq -n '{"height": "0", "round": 0, "step": 0}' > $HOMEDIR/data/priv_validator_state.json

  4. Download the newer binary, overwriting the previous one /usr/bin/exocored. Verify that the version of this binary is correct by running exocored version

  5. Store the new CHAIN_ID in the environment with CHAIN_ID=<new-chain-id>

  6. Set the new CHAIN_ID in the configuration with exocored --home $HOMEDIR config chain-id $CHAIN_ID and make other configuration changes, if any

  7. Download the genesis file

    wget -O $HOMEDIR/config/genesis.json https://raw.githubusercontent.com/ExocoreNetwork/testnets/main/genesis/$CHAIN_ID.json

  8. Start the node

Option 2: Upgrade with chain-id change. With coordination.

  1. Initialize a new node with the increased chain-id at a new location (for example exocoretestnet_233-3 becomes exocoretestnet_233-4) and configure it. Make sure to re-use the same validator private key ($HOMEDIR/config/priv_validator_key.json) or provide the mnemonic via --recover during the initialization.

  2. Modify the systemd service to use the provided --halt-height or --halt-time so that all v3 nodes stop at the same time.

  3. Once the network is halted, obtain the new genesis file and add it to the new $HOMEDIR Additionally, upgrade the binary by overwriting the existing one.

  4. Edit the systemd service to point to the new $HOMEDIR and remove the --halt parameters.

  5. Restart the service.

Option 3: No chain-id change. With coordination.

  1. Edit the systemd service with the specified --halt-height or --halt-time.

  2. Once the network is halted, remove the --halt parameters from the systemd file and upgrade the binary.

  3. Restart the service.

Upgrading from Testnet v5 to v6

This upgrade was performed using Option 1. The genesis file retained the operator registrations as well as the associations between the stakers and the operators. However, deposits, delegations and opt-ins to the Exocore-chain-as-AVS were dropped to avoid unnecessary slashing. To that end, validators must redo (almost) all of the steps listed under Becoming a Validator, skipping only the register-operator and associateOperatorWithEVMStaker transactions. In other words, tokens must be acquired, deposited, and delegated again, and, the opt-into-avs transaction must be sent again.

The base denomination changed from aexo to hua as part of this upgrade. This change affects the configuration of minimum-gas-prices as well as the CLI commands specifying the gas price with the option --gas-prices.

Additionally, the oracle price feeder needs a new configuration file $HOMEDIR/config/oracle_env_beaconchain.yaml, which is documented on its page.

Monitoring Uptime

Enable Prometheus metrics in the configuration by editing the configuration files.

$HOMEDIR/config/app.toml
[telemetry]

# Prefixed with keys to separate services.
service-name = ""

# Enabled enables the application telemetry functionality. When enabled,
# an in-memory sink is also enabled by default. Operators may also enabled
# other sinks such as Prometheus.
enabled = true # THE LINE TO EDIT

# Enable prefixing gauge values with hostname.
enable-hostname = false

# Enable adding hostname to labels.
enable-hostname-label = false

# Enable adding service to labels.
enable-service-label = false

# PrometheusRetentionTime, when positive, enables a Prometheus metrics sink.
prometheus-retention-time = 0

# GlobalLabels defines a global set of name/value label tuples applied to all
# metrics emitted using the wrapper functions defined in telemetry package.
#
# Example:
# [["chain_id", "cosmoshub-1"]]
global-labels = [
]
$HOMEDIR/config/config.toml
[instrumentation]

# When true, Prometheus metrics are served under /metrics on
# PrometheusListenAddr.
# Check out the documentation for the list of available metrics.
prometheus = true # THE LINE TO EDIT

Set up a Prometheus instance to scrape the metrics exposed by your node.

scrape_configs:
  - job_name: 'exocored'
    static_configs:
      - targets: ['localhost:26660']

Visualize these with Grafana by using its dashboards. Metrics include:

  • cometbft_consensus_latest_block_height: The latest block height

  • cometbft_consensus_block_syncing: Whether the node is synced or still catching up.

  • cometbft_consensus_block_interval_seconds: Time between blocks, useful for monitoring block production delays.

In addition, system level information can be loaded with prometheus-node-exporter into Grafana to complement the metrics exposed by exocored.

Lastly, third-party monitoring services such as ping.pub may be used when they add support for Exocore.

Useful Commands

(assuming the node port hasn't changed; if it has, it may be supplied with --node)

PreviousFaucetsNextSecurity Best Practices

Last updated