Managing The Validator

This page describes the regular tasks that you may need to do, as a node runner.

Handling Upgrades

Depending on the type of upgrade being undertaken by the wider network, one of the options below should be used to perform the upgrade. Automatic upgrades via the x/upgrade module and/or Cosmovisor are not yet supported.

The type of upgrade required by the network impacts which options you may select.

Option 1: Upgrade with chain-id change. Without coordination.

This type of upgrade is the most commonly used on the Exocore testnet. For example, the testnet was upgraded from exocoretestnet_233-5 to exocoretestnet_233-6 with this option.

Stop the node.
Remove the existing data
rm -rf $HOMEDIR/config/genesis.json $HOMEDIR/data/*.db $HOMEDIR/data/snapshots $HOMEDIR/data/cs.wal
Overwrite the validator's signature history, if any
jq -n '{"height": "0", "round": 0, "step": 0}' > $HOMEDIR/data/priv_validator_state.json
Download the newer binary, overwriting the previous one /usr/bin/exocored. Verify that the version of this binary is correct by running exocored version
Store the new CHAIN_ID in the environment with CHAIN_ID=<new-chain-id>
Set the new CHAIN_ID in the configuration with exocored --home $HOMEDIR config chain-id $CHAIN_ID and make other configuration changes, if any
Download the genesis file
wget -O $HOMEDIR/config/genesis.json https://raw.githubusercontent.com/ExocoreNetwork/testnets/main/genesis/$CHAIN_ID.json
Start the node

Option 2: Upgrade with chain-id change. With coordination.

Initialize a new node with the increased chain-id at a new location (for example exocoretestnet_233-3 becomes exocoretestnet_233-4) and configure it. Make sure to re-use the same validator private key ($HOMEDIR/config/priv_validator_key.json) or provide the mnemonic via --recover during the initialization.
Modify the systemd service to use the provided --halt-height or --halt-time so that all v3 nodes stop at the same time.
Once the network is halted, obtain the new genesis file and add it to the new $HOMEDIR Additionally, upgrade the binary by overwriting the existing one.
Edit the systemd service to point to the new $HOMEDIR and remove the --halt parameters.
Restart the service.

Option 3: No chain-id change. With coordination.

Edit the systemd service with the specified --halt-height or --halt-time.
Once the network is halted, remove the --halt parameters from the systemd file and upgrade the binary.
Restart the service.

Upgrading from Testnet v6 to v7

This upgrade was performed using Option 2. The genesis file retained all of the information in the previous version. Validators can use the following steps to upgrade.

Rename the older binary with mv $(which exocored) $(dirname $(which exocored))/exocored_v1.0.6.
Download the newer binary 1.0.7 as instructed previously.
Create a new home directory $NEW_HOMEDIR for testnet v7 and copy the configuration + private keys from the old directory. You will need rsync installed.

HOMEDIR="$HOME/.exocored" # your previously chosen path
NEW_HOMEDIR="$HOME/.new-exocored" # choose another path
mkdir -p "$NEW_HOMEDIR/data"
rsync -av --exclude "data" "$HOMEDIR/" "$NEW_HOMEDIR/"
OLD_HOMEDIR="$HOMEDIR"
HOMEDIR="$NEW_HOMEDIR"
unset NEW_HOMEDIR
CHAIN_ID="exocoretestnet_233-7"
exocored --home $HOMEDIR config chain-id $CHAIN_ID
jq -n \
  '{"height": "0", "round": 0, "step": 0}' > $HOMEDIR/data/priv_validator_state.json
wget -O $HOMEDIR/config/genesis.json \
  https://raw.githubusercontent.com/ExocoreNetwork/testnets/main/genesis/$CHAIN_ID.json

After the above step, the names are swapped to say OLD_HOMEDIR and HOMEDIR for the old and the new directories, respectively.

Update the price feeder configuration as documented on its page.
Create a new service file with $OLD_HOMEDIR, but don't start it yet.

/etc/systemd/system/old_exocore.service

[Unit]
Description=Exocore v6 node
After=network.target

[Service]
Type=simple
# This is set to HOME=/root on most systems 
# which causes Cosmos SDK to complain
Environment="HOME=/home/exocoreuser"
User=exocoreuser
Group=exocoreuser
# Halts at 4 am UTC on 10 Dec 2024
# If you're not running a validator, you can skip `--oracle`
ExecStart=/usr/bin/exocored_v1.0.6 start --oracle --halt-time=1733803200 --home <OLD_HOMEDIR> # replace with actual value
Restart=no

[Install]
WantedBy=default.target

Create a systemd timer to start the new binary on a schedule.

/etc/systemd/system/exocore.timer

[Unit]
Description=Schedule Start Testnet v7 Exocore node

[Timer]
# The expected time of the first block is 5:20 am UTC.
# We are ready well before that. The logs will indicate
# Genesis time is in the future. Sleeping until then... genTime=2024-12-10T05:20:48Z
OnCalendar=2024-12-10 04:30:00 UTC
Persistent=true

[Install]
WantedBy=timers.target

Modify the exocore.service to point to the new home directory; replacing the path with the actual value.

/etc/systemd/system/exocore.service

7c7
< ExecStart=/home/user/.local/bin/exocored start --home <OLD_HOMEDIR> --oracle
---
> ExecStart=/home/user/.local/bin/exocored start --home <HOMEDIR> --oracle

Set up the services

# Stop running v6 node
systemctl stop exocore
# Reload the files from disk
systemctl daemon-reload
# Start v6 node
systemctl start old_exocore
# Enable timer for v7
systemctl enable --now exocore.timer

Verify that the timer is running via systemctl list-timers --all | grep -i exocore. Your output will look like the line below.

Tue 2024-12-10 04:30:00 UTC 18h -- exocore.timer exocore.service

The oracle price feeder configuration files have had some edits as part of this upgrade, which are documented on its page.

Once your testnet v7 validator is up and running, the older directory $OLD_HOMEDIR and the old service file /etc/systemd/system/old_exocore.service may be deleted.

If you're running the price feeder as an independent binary, please create a similar service for it to be triggered at 05:10 UTC. Note that the name of the timer should match the name of the service exactly for this to work. Additionally, adding a Restart=yes to the service file will be helpful.

/etc/systemd/system/price-feeder.timer

[Unit]
Description=Schedule Start Testnet v7 Price Feeder

[Timer]
# The expected time of the first block is 5:20 am UTC.
# We need to be within less than 30 minutes of that.
# The closer, the better.
OnCalendar=2024-12-10 05:10:00 UTC
Persistent=true

[Install]
WantedBy=timers.target

Enable the timer.

systemctl daemon-reload
systemctl enable --now price-feeder.timer

If you edited the price feeder service file within this step, restart that as well.

systemctl restart price-feeder.service

Monitoring Uptime

Enable Prometheus metrics in the configuration by editing the configuration files.

$HOMEDIR/config/app.toml

[telemetry]

# Prefixed with keys to separate services.
service-name = ""

# Enabled enables the application telemetry functionality. When enabled,
# an in-memory sink is also enabled by default. Operators may also enabled
# other sinks such as Prometheus.
enabled = true # THE LINE TO EDIT

# Enable prefixing gauge values with hostname.
enable-hostname = false

# Enable adding hostname to labels.
enable-hostname-label = false

# Enable adding service to labels.
enable-service-label = false

# PrometheusRetentionTime, when positive, enables a Prometheus metrics sink.
prometheus-retention-time = 0

# GlobalLabels defines a global set of name/value label tuples applied to all
# metrics emitted using the wrapper functions defined in telemetry package.
#
# Example:
# [["chain_id", "cosmoshub-1"]]
global-labels = [
]

$HOMEDIR/config/config.toml

[instrumentation]

# When true, Prometheus metrics are served under /metrics on
# PrometheusListenAddr.
# Check out the documentation for the list of available metrics.
prometheus = true # THE LINE TO EDIT

Set up a Prometheus instance to scrape the metrics exposed by your node.

scrape_configs:
  - job_name: 'exocored'
    static_configs:
      - targets: ['localhost:26660']

Visualize these with Grafana by using its dashboards. Metrics include:

cometbft_consensus_latest_block_height: The latest block height
cometbft_consensus_block_syncing: Whether the node is synced or still catching up.
cometbft_consensus_block_interval_seconds: Time between blocks, useful for monitoring block production delays.

In addition, system level information can be loaded with prometheus-node-exporter into Grafana to complement the metrics exposed by exocored.

Lastly, third-party monitoring services such as ping.pub may be used when they add support for Exocore.

Useful Commands

Stop the node

systemctl stop exocore

Start the node

systemctl start exocore

Restart the node

systemctl restart exocore

Find my exo1... (account) address

exocored --home $HOMEDIR keys show -a $ACCOUNT_KEY_NAME

Get current height

exocored query block | jq .block.header.height

Check sync status

exocored status | jq

Get current validator set

exocored query tendermint-validator-set

Get own validator address

exocored --home $HOMEDIR tendermint show-address

Convert address formats

exocored debug addr <VALUE>

Get bytes32 val pub key

exocored --home $HOMEDIR --output json keys consensus-pubkey-to-bytes | jq -r .bytes32

Get bytes32 val pub key (another method)

exocored debug pubkey $(exocored --home $HOMEDIR tendermint show-validator)

Get JSON val pub key

exocored --home $HOMEDIR tendermint show-validator

Check node logs

journalctl -u exocore -f

Get the number of peers

curl -s http://localhost:26657/net_info

Check transaction status

exocored query tx <hash>

Find my consensus address

exocored --home $HOMEDIR tendermint show-address

Find my P2P ID

exocored --home $HOMEDIR tendermint show-node-id

(assuming the node port hasn't changed; if it has, it may be supplied with --node)

PreviousFaucets NextSecurity Best Practices

Last updated 11 days ago