Symptoms
Observe the logs in /var/log/daemon.log
Nov
8 01:13:27 localhost.localdomain emerg logger[6270]: Re-starting restjavad
Java core generated in folder /var/core.
Impact
Restjavad will be restarted.
Conditions
1. Provision ASM
2. Huge number of requests to restjavad
3. cpu is hitting 100%
Workaround
More heap memory can reduce cpu consuming operations, fewer GC cycles, less frequent minor GCs, overall less overhead for memory management
can add for less cpu usage.
Please increase the value of provision.extramb and provision.restjavad.extramb by 200MB at a time ( 400, 600, 800 ...) till the issue resolves. Since changing the value of provision.extramb is service affecting you may want to start with a higher value so there is more room to experiment to find a good value for restjavad heap size. Note 500MB is equivalent to large management provisioning and 200MB is the same as medium management provisioning.
NB provision.extramb value doesn't sync between peers (by design) and must be changed on each peer, one at a time, and is service affecting when changed on active. On ASM provisioned system it can take approximately 15 minutes for system to reprovision.
tmsh modify sys db provision.extramb value 200 ( 400, 600, 800 ...)
tmsh modify sys db provision.restjavad.extramb value 600 (800, 1000, 1200 ...
bigstart restart restjavad
Increase timeout
# tmsh modify sys db icrd.timeout value 300
# tmsh modify sys db restjavad.timeout value 300
# tmsh modify sys db restnoded.timeout value 300
bigstart restart restjavad restnoded
