OPERATIONAL DEFECT DATABASE
...
...
Connection events not received on FMC or remote connectors, SFDC down/waiting: - SFDC is down/waiting: root@firepower:/etc/sf# pmtool status | egrep -i ' - [^running]' SFDataCorrelator (normal) - Down HostInput_Daemon (normal) - Waiting sybase_arbiter (system,gui) - Waiting ... - The Following is seen in the logs or any version of this error message that relates to Flow:Maximum or "Failed to process DB configuration" SFDataCorrelator:config [ERROR] Error loading value for Event:Maximum SFDataCorrelator:config [ERROR] Error ReconfigureDBConfiguration SFDataCorrelator:SFDataCorrelator [ERROR] Failed to process DB configuration - The above indicates an issue with being able to read the /etc/sf/SFDataCorrelator.conf. However, the file will very likely be in tact and have no issues. The issue here is that the database where the file configuration is read to has been change somehow and this is why the SFDC process will not start. - Normal dce_config database: root@firepower:/etc/sf# OmniQuery.pl -db sdb -e "select * from dce_config;" +-------------------------+---------+ | name | data | +-------------------------+---------+ | SFD:NetflowDuplicate | 1 | | SFD:RNADuplicate | 1 | | FlowChunk:AliasExternal | 1 | | Flow:Maximum | 1000000 | | FlowChunk:Maximum | 2000000 | | Event:Maximum | 1000000 | | RUAEvent:Maximum | 1000000 | | RUAHistory:Maximum | 1000000 | | Event:DCEMaximum | 1000000 | | FlowPrioritized:Maximum | 1000000 | +-------------------------+---------+ 10 rows in set (0.003005 seconds) - Abnormal dce_config database: root@firepower:/etc/sf# OmniQuery.pl -db sdb -e "select * from dce_config;" +-------------------------+---------+ | name | data | +-------------------------+---------+ | SFD:NetflowDuplicate | 1 | | SFD:RNADuplicate | 1 | | FlowChunk:AliasExternal | 1 | +-------------------------+---------+ 3 rows in set (0.003005 seconds) Abnormal is anything that isn't 10 rows, as there are 10 fields in the conf file. The error seen will correlate with the first missing field in this case "Flow:Maximum" but it "could" be different. - Note the signature in the logs (/var/log/messages): Nov 18 14:50:47 firepower SF-IMS[10354]: [10354] SFDataCorrelator:config [ERROR] Error loading value for Flow:Maximum from dce_config table <--- this may look different depending on your version. Nov 18 14:50:47 firepower SF-IMS[10354]: [10354] SFDataCorrelator:config [ERROR] Error ReconfigureDBConfiguration Nov 18 14:50:47 firepower SF-IMS[10354]: [10354] SFDataCorrelator:SFDataCorrelator [ERROR] Failed to process DB configuration
SFDC down/waiting, no connection events
Re-save the database configuration. This will write to the dce_config database. NOTE: values held here are system defaults. If this configuration is saved with lower numbers than previously listed the database size will be shrunk and therefore eliminating any events that were greater than the saved size. Due to this, ensure that the configuration saved meets with your eventing needs. - System > Configuration > Database > Save - Restart SFDC, on FMC as root: pmtool restartbyid SFDataCorrelator - Check pmtool status: root@firepower:/etc/sf# pmtool status | egrep -i ' - [^running]' HostInput_Daemon (normal) - Waiting sybase_arbiter (system,gui) - Waiting fireamp (normal) - Waiting sfipmid (normal) - Waiting snmpd (normal) - Waiting vjdbc (normal) - Waiting SSEConnector (system) - Waiting TelemetryApp (system) - Waiting CSDApp (system) - Waiting dockerd (system) - Waiting EventHandler (normal) - Waiting datadog-agent (normal) - Waiting datadog-process-agent (normal) - Waiting datadog-system-probe (normal) - Waiting osquery (normal) - Waiting - Validate the Database change: root@firepower:/etc/sf# OmniQuery.pl -db sdb -e "select * from dce_config;" +-------------------------+---------+ | name | data | +-------------------------+---------+ | SFD:NetflowDuplicate | 1 | | SFD:RNADuplicate | 1 | | FlowChunk:AliasExternal | 1 | | Flow:Maximum | 1000000 | | FlowChunk:Maximum | 2000000 | | Event:Maximum | 1000000 | | RUAEvent:Maximum | 1000000 | | RUAHistory:Maximum | 1000000 | | Event:DCEMaximum | 1000000 | | FlowPrioritized:Maximum | 1000000 | +-------------------------+---------+ 10 rows in set (0.003005 seconds)
We do not know how the system enters this state. However, the system needs to be able to perform a monitor check of the table and report to the user that the table has changed. Then promt them to check their settings in the "Database" settings of their configuration file.
Cisco Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
