Dell - Defect ID: 246098
Avamar and Embedded Service Enabler (ESE): Kernel error events

Symptoms

The Avamar server is on version 19.10 or later which supports Dell Connectivity with Embedded Service Enabler (ESE). Kernel error events reported by Avamar may look similar to the following output. Event Code = 1 Event Date/Time = 10/28-06:42:15.00000 Event Type = ERROR Event Severity = PROCESS Event Summary = kernel error: Oct 28 06:42:08 avamarlab ese_launcher[24557]: 24557 2024-10-28 06:42:08,250 App Poller ERROR DellESE.ese.remote_support.app_polling.app_polling_interface_connection LN: 67 Interface details not found Software Source = DPN:Unknown Hardware Source = avamarlab Administrator Server = avamarlab Administrator Server version = 19.10.0-166_HF338862 Avamar Server = avamarlab Avamar Server systemid = 1589815007@00:50:56:8E:1E:BD Avamar Server productId = null Avamar Server version = 19.10.0-166 Backup & Recovery Manager Url = avamarlab Data Domain System = ddlab Data Domain System OS version = 7.7.5.11-1046187 Data Domain System serial number = AUDVSDBC4XDDRW Data Domain System = ddlab Data Domain System OS version = 7.7.5.1-1040473 Data Domain System serial number = AUDVSDBC4XDDRW Event Data = Entry key = date Entry value = 2024/10/28 Entry key = code Entry value = 0001 Entry key = time Entry value = 05:42:15.09665 UTC Entry key = thread Entry value = log-messages:109 Entry key = type Entry value = ERROR Entry key = message Entry value = kernel error: Oct 28 06:42:08 avamarlab ese_launcher[24557]: 24557 2024-10-28 06:42:08,250 App Poller ERROR DellESE.ese.remote_support.app_polling.app_polling_interface_connection LN: 67 Interface details not found Entry key = nodeid Entry value = 0.0 Entry key = requestor Entry value = <requestor domain="/" product="MCS" role="Administrator" user="root" />

Cause

Embedded Service Enabler (ESE) is a product-independent python software which can be configured on Avamar to send telemetry information to Dell Technologies. ESE can send telemetry data with direct connections from the Avamar server to the Dell backend or by using Secure Connect Gateway (SCG). Avamar configures ESE in systemd with the "ese_launcher" service, which by default logs to the following file. /var/log/messages The configuration which sends the logs there is rsyslog. /etc/rsyslog.conf Because of the following line in the rsyslog configuration, it logs mostly everything except mail and news in systemd to the messages file location. *.*;mail.none;news.none -/var/log/messages Whenever there is an error in the ESE service, it logs to the messages location and the server log scanner configuration marks it as a kernel error. Marking the ESE errors as kernel errors is not accurate, as ESE is not part of the kernel and has its own logging at the following location. /opt/ese/var/log/*

Resolution

Since the ESE service has its own logging implementation, troubleshooting the ESE service should happen from the ESE log folder, not the messages files. Stop the logging of the ESE service to the messages file and false kernel error reporting for the service by changing the rsyslog configuration.Take a backup of the rsyslog configuration file. cp -p /etc/rsyslog.conf /etc/x-rsyslog.conf As the root user, run the following command: grep -q "if \$programname == 'ese_launcher' then stop" /etc/rsyslog.conf || sed -i '1i if \$programname == '\''ese_launcher'\'' then stop' /etc/rsyslog.conf Restart the rsyslog service. systemctl restart rsyslog

Support Cases