Fix for https://github.com/huin/goupnp/issues/33 : do not use loopbacK IP addresses

I have a private ethereum node running on aws instance, the time when started the node i set the etherbase/coinbase address to the first account of the node. As i started the node i find that first account is getting the rewards of mining and everything was as expected, but after few hours the miner/etherbase address automatically get changed, as a result of which funds from first account get transferred to miner address as a mining reward. And i unlocked the first account to send the transaction over the node and as all the funds get transferred to newly assigned miner address, all my transactions become pending due to insufficient funds in sender account. Any help will be appreciated. Thanks in advance.

Here are the few details of geth client and command i used :

GO ETHEREUM CLIENT DETAILS : Geth Version: 1.9.12-stable Git Commit: b6f1c8dcc058a936955eb8e5766e2962218924bc Architecture: amd64 Protocol Versions: [65 64 63] Go Version: go1.13.8 Operating System: linux GOPATH= GOROOT=/build/ethereum-YoJS3q/.go

GETH COMMAND : /usr/bin/geth --datadir node01 --networkid 10 --allow-insecure-unlock --targetgaslimit 9000000000000 --nousb --etherbase '0x669980d2EF6b14e15e4e1D32E70F2ccC83EC98c9' --rpc --rpcaddr "0.0.0.0" --rpcapi admin,db,eth,debug,miner,net,shh,txpool,personal,web3 --rpccorsdomain '*'

GENESIS.JSON : { "alloc": { "0x669980d2EF6b14e15e4e1D32E70F2ccC83EC98c9": { "balance": "10000000000000000000000000000000000000" }, "0x61722d9D8F75E97A2F8B4C63Be1C8820D944C940":{ "balance": "1000000000000000000000000000000000000" } }, "coinbase": "0x669980d2EF6b14e15e4e1D32E70F2ccC83EC98c9", "config": { "homesteadBlock": 0, "byzantiumBlock": 0, "constantinopleBlock": 0, "chainId": 10, "eip150Block": 0, "eip155Block": 0, "eip150Hash": "0x0000000000000000000000000000000000000000000000000000000000000000", "eip158Block": 0, "maxCodeSize": 35, "maxCodeSizeChangeBlock" : 0, "isQuorum": true }, "difficulty": "0x0", "extraData": "0x0000000000000000000000000000000000000000000000000000000000000000", "gasLimit": "0xE0000000", "mixhash": "0x00000000000000000000000000000000000000647572616c65787365646c6578", "nonce": "0x0", "parentHash": "0x0000000000000000000000000000000000000000000000000000000000000000", "timestamp": "0x00" }

@rjl493456442 the archive is started with -rpcapi eth,web3,debug - guess this is what you mean? Also debug_traceBlock is working as expected

Logs from restart with --nousb option added.

Error lines gone, but restart still almost 4 mins:

INFO [01-24|20:02:39.803] Starting pprof server                    addr=http://127.0.0.1:6060/debug/pprof
INFO [01-24|20:02:39.804] Starting Geth on Ethereum mainnet... 
INFO [01-24|20:02:39.805] Enabling metrics collection 
INFO [01-24|20:02:39.806] Maximum peer count                       ETH=25 LES=0 total=25
INFO [01-24|20:02:39.807] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
INFO [01-24|20:06:31.322] Set global gas cap                       cap=25000000

--nousb

I'll try this. If anything hopefully it removes those error messages 😄

When you stop geth, do you have a custom timeout? Geth needs time to flush DB, if it's only given 15 or 30 secs to shut down, DB may be in a state that takes more time to on start. I set Geth shutdown timeout to 3 minutes, that seems to work well.

It is managed by systemd, so whatever the default there is.

Here's the log from the last restart:

INFO [01-24|01:15:49.052] Got interrupt, shutting down... 
INFO [01-24|01:15:49.556] HTTP server stopped                      endpoint=127.0.0.1:8545
INFO [01-24|01:15:49.557] IPC endpoint closed                      url=/mnt/ethereum/geth.ipc
INFO [01-24|01:15:49.558] Ethereum protocol stopped 
INFO [01-24|01:15:49.565] Transaction pool stopped 
INFO [01-24|01:15:49.565] Writing clean trie cache to disk         path=/mnt/ethereum/geth/triecache threads=8
INFO [01-24|01:15:50.046] Persisted the clean trie cache           path=/mnt/ethereum/geth/triecache elapsed=480.110ms
INFO [01-24|01:15:50.046] Blockchain stopped 

And journalctl doesn't show anything obviously wrong:

Jan 24 01:15:49 geth-mainnet systemd[1]: Stopping Geth Service...
Jan 24 01:16:20 geth-mainnet systemd[1]: geth.service: Succeeded.
Jan 24 01:16:20 geth-mainnet systemd[1]: Stopped Geth Service.

LevelDB triggers compaction when you open it more aggressively than during normal mode of operation (I guess their rationale was that a startup is a golden opportunity to shuffle some data before coming online). If that's the cause, there's not much we can do, an archive node is gigantic, so database maintenance will inherently be killer on perf.

If you think of a way to test if this is the cause, I'd be happy to try it out.

LevelDB triggers compaction when you open it more aggressively than during normal mode of operation (I guess their rationale was that a startup is a golden opportunity to shuffle some data before coming online). If that's the cause, there's not much we can do, an archive node is gigantic, so database maintenance will inherently be killer on perf.

I followed Somer Esat's guide. It synced from 0 in fast mode and was shut off for a few days after it was synced.

When you stop geth, do you have a custom timeout? Geth needs time to flush DB, if it's only given 15 or 30 secs to shut down, DB may be in a state that takes more time to on start. I set Geth shutdown timeout to 3 minutes, that seems to work well.

Are you using the snapshot? Can you provide the commands to start the Geth?

You have to explicitly expose the debug namespace APIs via the http, by default it's not.

You have to explicitly expose the debug namespace APIs via the http, by default it's not.

On a side node: wondering if we really need the copyright 2 times in there

Out of curiosity, what happens if you add the --nousb option ?

I understand this might not constitute a proper "start" definition, but it's where I'm now seeing a long delay upon start.

Hi @ligi - I'm measuring the time between the log line:

Starting pprof server

and the log line:

Set global gas cap                       cap=25000000

Timestamps above :)

Can you elaborate how you define "start" and how you measure the times (where is the start where is the end of measurement)?

System information

Geth version: geth version

$ geth version
Geth
Version: 1.9.24-stable
Git Commit: cc05b050df5f88e80bb26aaf6d2f339c49c2d702
Git Commit Date: 20201112
Architecture: amd64
Protocol Versions: [65 64 63]
Go Version: go1.15.5
Operating System: linux
GOPATH=
GOROOT=/usr/local/go

OS & Version: Linux (Ubuntu 20.04 LTS)

Expected behaviour

geth should start within a few seconds.

Actual behaviour

geth takes 4+ minutes to start

Steps to reproduce the behaviour

All I need to do is restart geth and I'll see the startup sequence take a significant amount of time

Backtrace

INFO [01-24|00:37:23.691] Starting pprof server                    addr=http://127.0.0.1:6060/debug/pprof
INFO [01-24|00:37:23.691] Starting Geth on Ethereum mainnet... 
INFO [01-24|00:37:23.691] Enabling metrics collection 
INFO [01-24|00:37:23.693] Maximum peer count                       ETH=25 LES=0 total=25
INFO [01-24|00:37:23.693] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
ERROR[01-24|00:41:31.845] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[01-24|00:41:31.845] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[01-24|00:41:31.845] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[01-24|00:41:32.021] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[01-24|00:41:32.021] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[01-24|00:41:32.021] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
INFO [01-24|00:41:32.022] Set global gas cap                       cap=25000000

Additional info

Starting geth with the following flags

--datadir /mnt/ethereum --syncmode full --gcmode archive --http --http.api "debug,personal,eth,web3" --rpc.txfeecap 2 --miner.gastarget 1000000 --cache 2048 --gpo.percentile 30 --maxpeers 25 --metrics --pprof --pprof.addr=127.0.0.1 --allow-insecure-unlock

Node history: 2020-06-17: Fresh sync on v1.9.14-stable-6d74d1e5 2020-07-11: Upgraded to v1.9.16-stable-ea3b00ad. No delay on restart. 2020-08-18: Restarted. 1.5 minute delay on restart. 2020-08-21: Upgraded to v1.9.19-stable-3e064192. No delay on restart. 2020-09-23: Upgraded to v1.9.21-stable-0287d548. No delay on restart. 2020-09-24: Restarted. 4+ minute delay on restart. 2020-11-05: Restarted. 4+ minute delay on restart. 2020-11-12: Upgraded to v1.9.24-stable-cc05b050. 4+ minute delay on restart.

Logs

2020-07-11 restart logs:

INFO [07-11|14:33:21.520] Starting pprof server                    addr=http://127.0.0.1:6060/debug/pprof
INFO [07-11|14:33:21.520] Starting Geth on Ethereum mainnet... 
INFO [07-11|14:33:21.520] Enabling metrics collection 
INFO [07-11|14:33:21.523] Maximum peer count                       ETH=25 LES=0 total=25
INFO [07-11|14:33:21.524] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
ERROR[07-11|14:33:21.526] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[07-11|14:33:21.527] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[07-11|14:33:21.527] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[07-11|14:33:21.527] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[07-11|14:33:21.527] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[07-11|14:33:21.527] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
INFO [07-11|14:33:21.527] Set global gas cap                       cap=25000000

2020-08-18 restart logs:

INFO [08-18|16:47:27.891] Starting pprof server                    addr=http://127.0.0.1:6060/debug/pprof
INFO [08-18|16:47:27.891] Starting Geth on Ethereum mainnet... 
INFO [08-18|16:47:27.891] Enabling metrics collection 
INFO [08-18|16:47:27.896] Maximum peer count                       ETH=25 LES=0 total=25
INFO [08-18|16:47:27.896] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
ERROR[08-18|16:47:27.899] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[08-18|16:47:27.900] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[08-18|16:47:27.900] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[08-18|16:47:27.900] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[08-18|16:47:27.900] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[08-18|16:47:27.900] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
INFO [08-18|16:47:27.901] Set global gas cap                       cap=25000000

2020-08-21 restart logs:

INFO [08-21|14:22:18.023] Starting pprof server                    addr=http://127.0.0.1:6060/debug/pprof
INFO [08-21|14:22:18.023] Starting Geth on Ethereum mainnet... 
INFO [08-21|14:22:18.024] Enabling metrics collection 
INFO [08-21|14:22:18.024] Maximum peer count                       ETH=25 LES=0 total=25
INFO [08-21|14:22:18.024] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
ERROR[08-21|14:22:18.024] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[08-21|14:22:18.025] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[08-21|14:22:18.025] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[08-21|14:22:18.025] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[08-21|14:22:18.025] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[08-21|14:22:18.025] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
INFO [08-21|14:22:18.025] Set global gas cap                       cap=25000000

2020-09-23 restart logs:

INFO [09-23|19:40:33.433] Starting pprof server                    addr=http://127.0.0.1:6060/debug/pprof
INFO [09-23|19:40:33.433] Starting Geth on Ethereum mainnet... 
INFO [09-23|19:40:33.433] Enabling metrics collection 
INFO [09-23|19:40:33.435] Maximum peer count                       ETH=25 LES=0 total=25
INFO [09-23|19:40:33.435] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
ERROR[09-23|19:40:33.435] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[09-23|19:40:33.435] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[09-23|19:40:33.435] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[09-23|19:40:33.435] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[09-23|19:40:33.435] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[09-23|19:40:33.435] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
INFO [09-23|19:40:33.435] Set global gas cap                       cap=25000000

2020-09-24 restart logs:

INFO [09-24|16:12:48.629] Starting pprof server                    addr=http://127.0.0.1:6060/debug/pprof
INFO [09-24|16:12:48.629] Starting Geth on Ethereum mainnet... 
INFO [09-24|16:12:48.629] Enabling metrics collection 
INFO [09-24|16:12:48.631] Maximum peer count                       ETH=25 LES=0 total=25
INFO [09-24|16:12:48.631] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
ERROR[09-24|16:17:15.528] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[09-24|16:17:15.529] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[09-24|16:17:15.529] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[09-24|16:17:15.701] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[09-24|16:17:15.701] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[09-24|16:17:15.701] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
INFO [09-24|16:17:15.702] Set global gas cap                       cap=25000000

2020-11-05 restart logs:

INFO [11-05|02:24:42.315] Starting pprof server                    addr=http://127.0.0.1:6060/debug/pprof
INFO [11-05|02:24:42.315] Starting Geth on Ethereum mainnet... 
INFO [11-05|02:24:42.315] Enabling metrics collection 
INFO [11-05|02:24:42.317] Maximum peer count                       ETH=25 LES=0 total=25
INFO [11-05|02:24:42.317] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
ERROR[11-05|02:29:07.009] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[11-05|02:29:07.009] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[11-05|02:29:07.009] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[11-05|02:29:07.173] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[11-05|02:29:07.173] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[11-05|02:29:07.173] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
INFO [11-05|02:29:07.174] Set global gas cap                       cap=25000000

2020-11-12 restart logs:

INFO [11-12|23:43:33.095] Starting pprof server                    addr=http://127.0.0.1:6060/debug/pprof
INFO [11-12|23:43:33.095] Starting Geth on Ethereum mainnet... 
INFO [11-12|23:43:33.096] Enabling metrics collection 
INFO [11-12|23:43:33.101] Maximum peer count                       ETH=25 LES=0 total=25
INFO [11-12|23:43:33.101] Smartcard socket not found, disabling    err="stat /run/pcscd/pcscd.comm: no such file or directory"
ERROR[11-12|23:47:59.142] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[11-12|23:47:59.143] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[11-12|23:47:59.143] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=1 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[11-12|23:47:59.305] Failed to enumerate USB devices          hub=ledger vendor=11415 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[11-12|23:47:59.305] Failed to enumerate USB devices          hub=trezor vendor=21324 failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
ERROR[11-12|23:47:59.305] Failed to enumerate USB devices          hub=trezor vendor=4617  failcount=2 err="failed to initialize libusb: libusb: unknown error [code -99]"
INFO [11-12|23:47:59.305] Set global gas cap                       cap=25000000

According to the JSON RPC spec eth_signTransaction uses a gas default. However, when trying to sign a transaction with the gas field missing, geth reports { code: -32000, message: "gas not specified" }.

This does not happen when e.g. using the same transaction with the eth_sendTransaction procedure.

I am not sure if this is intended behavior or a bug. This behavior was observed using geth in --dev mode.

I think this is a great idea - then we could also mark this one as "default"

not yet sure about adding the line describing the behavior as it breaks with the rest of the help layout - but for sure command should be marked optional

Also consider my suggestion of adding a command "node" to actually start the Ethereum node. I filed this report after staring on the help screen for like 10 minutes and wondering "how to I actually start the node"

the archive node is currently running:

archive@archive:~$ ./geth version
Geth
Version: 1.9.25-stable
Git Commit: e7872729012a4871397307b12cc3f4772ffcbec6
Git Commit Date: 20201211
Architecture: amd64
Protocol Versions: [65 64 63]
Go Version: go1.15.6
Operating System: linux
GOPATH=/home/archive/go
GOROOT=go

The code for TraceChain is still present in the codebase, but somehow it's not callable from the rpc interface nor the console. debug_traceBlock* work as expected.

System information

Geth version: 1.9.26-unstable && whatever runs on the archive node @ligi OS & Version: Linux Commit hash : master

Expected behaviour

Returning a list of trace results.

Actual behaviour

$ curl -X POST -H "Content-Type: application/json" --data '{"method": "debug_traceChain", "params": ["0x1", "0x2"], "id": 1}' http://example.com | json_pp
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   186  100   121  100    65   1163    625 --:--:-- --:--:-- --:--:--  1788
{
   "error" : {
      "code" : -32601,
      "message" : "the method debug_traceChain does not exist/is not available"
   },
   "id" : 1,
   "jsonrpc" : "2.0"
}

	fetcher *fetcher.BlockFetcher       // The underlying fetcher which takes care of block header retrieval.

	ns      *nodestate.NodeStateMachine // The global peerset of light client which is shared by all components

This PR removes the old serverPeerSet and uses NodeStateMachine to realize the peer set logic of clients, similarly to what has already been done with clientPeerSet. This change makes the code more consistent by removing multiple custom peer maps used by different mechanisms. It also allows further client side feature improvements (mostly related to incentivization). The NodeStateMachine is also extended with another ForEach iterator that iterates on nodes having a certain field set.

good point actually - fixing via 22220 not yet sure about adding the line describing the behavior as it breaks with the rest of the help layout - but for sure command should be marked optional