Symptoms

Windows Security logs indicate that avtar.exe is accessing every user profile on a client. For active user profiles, the entries look like: Log Name: Security Source: Microsoft-Windows-Security-Auditing Date: 5/27/2017 4:00:07 PM Event ID: 4648 Task Category: Logon Level: Information Keywords: Audit Success User: N/A Computer: CNCSD1C.corp.emc.com Description: A logon was attempted using explicit credentials. Subject: Security ID: SYSTEM Account Name: CNCSD1C$ Account Domain: CORP Logon ID: 0x3e7 Logon GUID: {00000000-0000-0000-0000-000000000000} Account Whose Credentials Were Used: Account Name: testuser Account Domain: CORP Logon GUID: {1d662ff0-b57a-9c60-620c-b7f5c70ad1df} Target Server: Target Server Name: localhost Additional Information: localhost Process Information: Process ID: 0x1544 Process Name: C:\Program Files\avs\bin\avtar.exe ----- Log Name: Security Source: Microsoft-Windows-Security-Auditing Date: 5/27/2017 4:00:07 PM Event ID: 4624 Task Category: Logon Level: Information Keywords: Audit Success User: N/A Computer: CNCSD1C.corp.emc.com Description: An account was successfully logged on. Subject: Security ID: SYSTEM Account Name: CNCSD1C$ Account Domain: CORP Logon ID: 0x3e7 Logon Type: 3 New Logon: Security ID: CORP\testuser Account Name: testuser Account Domain: CORP Logon ID: 0x8150fc1 Logon GUID: {cac983ee-8bf7-3789-896f-c9be1e852ead} Process Information: Process ID: 0x1334 Process Name: C:\Program Files\avs\bin\avtar.exe For expired user profiles, it looks like: Log Name: Security Source: Microsoft-Windows-Security-Auditing Date: 5/27/2017 12:51:58 PM Event ID: 4625 Task Category: Logon Level: Information Keywords: Audit Failure User: N/A Computer: CNCSD1C.corp.emc.com Description: An account failed to log on. Subject: Security ID: SYSTEM Account Name: W8001DB03$ Account Domain: INTERNAL Logon ID: 0x3e7 Logon Type: 3 Account For Which Logon Failed: Security ID: NULL SID Account Name: Account Domain: Failure Information: Failure Reason: The specified user account has expired. Status: 0xc0000193 Sub Status: 0xc0000193 Process Information: Caller Process ID: 0xe7c Caller Process Name: C:\Program Files\avs\bin\avtar.exe For disabled user profiles, it looks like: Log Name: Security Source: Microsoft-Windows-Security-Auditing Date: 5/27/2017 12:51:58 PM Event ID: 4625 Task Category: Logon Level: Information Keywords: Audit Failure User: N/A Computer: CNCSD1C.corp.emc.com Description: An account failed to log on. Subject: Security ID: SYSTEM Account Name: W8001DB03$ Account Domain: INTERNAL Logon ID: 0x3e7 Logon Type: 3 Account For Which Logon Failed: Security ID: NULL SID Account Name: Account Domain: Failure Information: Failure Reason: Account currently disabled. Status: 0xc000006e Sub Status: 0xc0000072 Process Information: Caller Process ID: 0xe7c Caller Process Name: C:\Program Files\avs\bin\avtar.exe Entries such as the following can also be seen: Log Name: Security Source: Microsoft-Windows-Security-Auditing Date: 5/27/2017 12:51:58 PM Event ID: 4625 Task Category: Logon Level: Information Keywords: Audit Failure User: N/A Computer: CNCSD1C.corp.emc.com Description: An account failed to log on. Subject: Security ID: Account Name: testuser Account Domain: CORP Logon ID: 0x3e7 Logon Type: 3 Account For Which Logon Failed: Security ID: NULL SID Account Name: Account Domain: Failure Information: Failure Reason: Error occured during Logon. Status: 0xc000018b Sub Status: 0x0 Process Information: Caller Process ID: 0x1544 Caller Process Name: C:\Program Files\avs\bin\avtar.exe The following is a list of common statuses that may be encountered: Status Code Description 0XC000005E There are currently no logon servers available to service the logon request. 0xC0000064 User logon with misspelled or bad user account 0xC000006A User logon with misspelled or bad password 0XC000006D This is either due to a bad username or authentication information 0XC000006E Unknown user name or bad password. 0xC000006F User logon outside authorized hours 0xC0000070 User logon from unauthorized workstation 0xC0000071 User logon with expired password 0xC0000072 User logon to account disabled by administrator 0XC00000DC Indicates the Sam Server was in the wrong state to perform the desired operation. 0XC0000133 Clocks between DC and other computer too far out of sync 0XC000015B The user has not been granted the requested logon type (aka logon right) at this machine 0XC000018C The logon request failed because the trust relationship between the primary domain and the trusted domain failed. 0XC0000192 An attempt was made to logon, but the Netlogon service was not started. 0xC0000193 User logon with expired account 0XC0000224 User is required to change password at next logon 0XC0000225 Evidently a bug in Windows and not a risk 0xC0000234 User logon with account locked 0XC00002EE Failure Reason: An Error occurred during Logon 0XC0000413 Logon Failure: The machine you are logging onto is protected by an authentication firewall. The specified account is not allowed to authenticate to the machine. For a full list, see Error ode ntstatus.h (External Link) These entries are found in the Security Log for every user profile on the client machine every time the backup runs.

Cause

At the end of each backup, the Plugin spawned avtar process gathers information for every user profile on the client. In the avtar log, the following line can be found (notice, the number varies depending on the number of profiles): avtar Info : Reading 14 user profiles avtar Info : Done reading user profiles This gathering of profiles happens at the end of every avtar session on a Windows machine. It happens not only at the end of a Windows File System backup (avtar), but also every time a different plug-in like avexvss (Exchange), avsql (SQL), avvss (VSS) spawns an avtar.exe process. If a Windows VSS backup spawns three avtar processes to backup various volumes, the profiles are gathered three times and adds to the overhead times.Although user profile gathering is supposed to be quick process, in some rare instance like orphaned security identifier (SID) entries it takes a long time impacting Avamar performance. Example of such logged entry: 2017-05-25 04:34:18 avtar Info : Reading 37 user profiles Followed over two hours later by: 2017-05-25 06:50:34 avtar Info : Done reading user profiles Profile gathering at the end of the backup can even fail when invoking "AuthzInitializeContextFromSid": 2023-10-13 09:51:21 avtar Warning : AuthzInitializeContextFromSid failed: 2 More details about the use of this API in profile gathering is located at: https://learn.microsoft.com/en-us/troubleshoot/sql/reporting-services/call-authzinitializecontextfromsid-api-fails In such cases, some SIDs had the corresponding username entries missing and avtar stuck or failed processing these orphaned SIDs. This can happen when deleting user accounts but not deleting the corresponding user home directory. This profile gathering is turned on by default, but is only used for desktop or laptop (DTLT) restores. For each user profile, avtar obtains all groups the user belongs to in order to determine whether the user is a local administrator. This information is used to determine which files the logged in user can see and restore using the DTLT web interface.

Resolution

Although these security entries can be safely ignored, profile gathering can be disabled on Windows Server clients. It should not be disabled on desktops or laptops if the DTLT web interface is being used. To disable User Profile gathering, add the following avtar flag in the avtar.cmd file on the client or the associated Dataset. --x05=65536 The disabling of profile gathering can be handled in two ways. For a single client: Create a text file in C:\Program Files\avs\var called avtar.cmdIn the avtar.cmd file, add the following flag: --x05=65536 This affects all backups on the client, since avtar uses it every time it is started. For multiple clients using a Dataset: In the dataset, go to the 'Options' tabSelect the appropriate Plug-In type from the dropdown listClick the 'More' button. For Windows File System backups: Under 'Enter Attribute: Enter x05Under 'Enter Attribute Value', enter 65536Then click the + button For all other Windows plug-ins: Under 'Enter Attribute:' Enter [avtar]x05Under 'Enter Attribute Value', enter 65536Then click the + button This must be done for each plug-in type that is part of the dataset and for every dataset that is assigned to a group that the client is a member of.

Support Cases