Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
Version Identified Workspace ONE UEM 1908 and above Symptoms: A large number of files, ranging 50k or higher, are found in the operating system's Machine Keys folder (typically C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys). These files may consume excessive disk space on the application server which can adversely affect operation of services or applications hosted on the server.This can affect application servers hosting the following Workspace ONE UEM components: Web ConsoleDevice Services (includes Catalog and Self-Service Portal)API
Microsoft introduced support for asymmetric encryption and digital certificates in Windows NT 4.0. Workspace ONE UEM leverages Microsoft's Cryptography API (CAPI) for such processes. CAPI is susceptible to leaving machine keys on disk forever.
If the key accumulation is significant, it can consume excessive space on the C:\ drive of the server, resulting in degraded performance or hamper start-up/operation of services. Depending on the component hosted on the affected server(s), this may result in issues such as the inability to load the administration console, enroll devices, perform REST API operations, etc.
Depending on the current version of your on-premise Workspace ONE UEM environment, additional action may be required as noted below. Update: Workspace ONE UEM 2204 and above no longer leave behind machine key files on disk. However, you will need to keep using the below Clean-MachineKeys tool until the number of key files deleted falls below 100. (Other products on the server may orphan private key files.).
Clean-MachineKeys - standalone tool VMware has developed a standalone tool - Clean-MachineKeys, to help clear these files in a safe manner. The tool is available for download here for On-premise customers. Please review the following details and contact VMware Global Support to receive assistance for deploying the tool (or if you are unable to access it).Clean-MachineKeys scans each folder passed to it as a command parameter, deleting the private key files contained within. It is possible that such files are in active use. For example, those referenced by a well-known name (e.g. used by IIS, MSMQ, or .NET Framework), by a certificate imported into the certificate stores, or by a running process that has created a key pair or digital certificate in memory. Therefore, the tool excludes from deletion key files which: Have a well-known nameAre found in the certificate stores (which the tool scans at startup)Were created recently While the tool performs reasonable checks to ensure the files being deleted are no longer in use, there is still an element of risk given the nature of machine keys. Please ensure that you follow these guidelines to mitigate risk: The tool can only be executed on Windows Server 2012R2 and aboveTake VM snapshots of the servers before running the tool. This will all you to safely rollback changes performed by the tool in the event that an issue arisesReboot the server BEFORE running the tool. If a server reboot is not feasible, at the least, restart IIS and all Workspace ONE UEM services NOTE: The tool may have environment-specific difficulties scanning the certificate stores to find keys that it needs to exclude, in which case it will report the errors and stop. If this happens, save the entire tool output into a text file and contact VMware Global Support for assistance. Do not ignore the errors; if you allow the tool to continue, it could delete keys that are critical to the system operationThe tool can be executed through Powershell and takes certain parameters: -createdBefore - Defines retention period and takes values in ##_hours_ago or ##_days_ago formats. Only files created before the specified time are considered for deletion-m - Files are moved rather than deleted, thus allowing a simpler means to restore them if any issues are observedFile path to the keys folder exhibiting this issue. For example: C:\ProgramData\Microsoft\Crypto\RSA\MachineKeysC:\ProgramData\Microsoft\Crypto\RSA\S-1-5-18C:\Windows\ServiceProfiles\LocalService\AppData\Roaming\Microsoft\Crypto\RSA\S-1-5-19C:\Windows\ServiceProfiles\NetworkService\AppData\Roaming\Microsoft\Crypto\RSA\S-1-5-20C:\Windows\ServiceProfiles\NetworkService\AppData\Roaming\Microsoft\SystemCertificates\Request\Certificates NOTE: The relevant file path to check will also depend on which service account is being used for your application pools or services. e.g. C:\Windows\ServiceProfiles\{UserAccountBeingUsed}\AppData\Roaming\Microsoft\SystemCertificates\Request\Certificates Executing the Clean-MachineKeys tool While the tool offers flexibility with how it can be executed, it is recommended that customers take the following approach while adhering to the guidelines stipulated above. Extract the Clean-MachineKeys folder to a convenient location on the affected server First-time execution on a given server 1. Before execution of the tool, reboot the server (or restart IIS and WS1 UEM services on the server)2. Open Windows Explorer and navigate to the MachineKeys folder where files have accumulated to ensure the logged-in user has access to that file path3. Launch a Powershell window with Administrator privileges (Run As Administrator)4. Change the working directory in Powershell to the Clean-MachineKeys folder5. Use the -m option, and move files created more than 180 days ago .\Clean-MachineKeys -m -createdBefore 180_days_ago C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys 6. A warning message is presented (screenshot below) - type Y to continue 7. The tool will attempt to read local certificate stores and present an error message if it cannot access any of them. Should this occur, please capture the entire output and contact VMware Support for further assistance. If not, proceed to Step 8. The output can be captured by re-executing the tool and piping the output to a file .\Clean-MachineKeys -m -createdBefore 180_days_ago C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys | Tee-Object -file .\ServerName_MachineKeyLog.txt You may have to press Y for the execution to complete. The log file can be found in the Clean-MachineKeys folderVMware Support will help analyze the error and provide steps to address or avoid itNOTE: All certificate store validations are performed by the tool before it begins moving/deleting any files. As a result, the error validation/exclusion exercise is not expected to be needed more than once for a given server. 8. When the tool has executed successfully, it will provide an overview of the number of files that were moved vs skipped. Upon successful execution of the tool, validate if any issues are observed, copy the moved files (located in MachineKeysFilepath_saved folder) back to the original location or restore the VM to its snapshot. If no issues are observed, proceed to the next section Subsequent execution on a given server 1. Before execution of the tool, reboot the server (or restart IIS and WS1 UEM services on the server)2. Open Windows Explorer and navigate to the MachineKeys folder where files have accumulated to ensure the logged-in user has access to that file path3. Launch a Powershell window with Administrator privileges (Run As Administrator)4. Change the working directory in Powershell to the Clean-MachineKeys folder5. Delete files created more than 7 days ago. Write output to file .\Clean-MachineKeys -createdBefore 7_days_ago C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys 4. Accept any prompts presented by pressing YNOTE: Please ensure the steps noted above are carried out manually before setting up periodic execution. Periodic execution Scheduling for periodic execution of the tool is akin to setting up recurring execution of any Powershell script on a Windows server. There are many ways to accomplish this result. The following serves as a basic example for setting up periodic execution of Clean-MachineKeys with a 7-day retention period.1. Copy and save the following as <<FileName>>.ps1 in the same folder where the Clean-MachineKeys tool is present. In this example, it is named Clear7daysAgo.ps1 <<FolderLocationofClean-MachineKeys.exe>>\Clean-MachineKeys -q -y -createdBefore 7_days_ago C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys 2. Copy and save the following as CreateTaskSchedule.ps1 on server $taskAction = New-ScheduledTaskAction ` -Execute '"C:\Program Files\PowerShell\7\pwsh.exe"' ` ##Powershell install path on server -Argument 'Clear7DaysAgo.ps1' ` ##FileName from previous step -WorkingDirectory 'E:\Clean-MachineKeys1.5.1\' ##Folder where Clear7daysAgo.ps1 is located $taskTrigger = New-ScheduledTaskTrigger -Weekly -DaysOfWeek Monday -At 5AM ##Schedule to run every Monday at 5AM Server Time $taskName = "CleanMachineKeysWeekly" $taskDescription = "Run Clean-MachineKeys ever Monday at 5 AM with retention period of 7 days" Register-ScheduledTask ` -TaskName $taskName ` -Action $taskAction ` -Trigger $taskTrigger ` -Description $taskDescription 3. Run CreateTaskSchedule.ps1 as Administrator to create the Scheduled Task. You can then launch Task Scheduler on the server and validate the presence of a Task named CleanMachineKeysWeekly4. Remember to manually carry out or schedule a server reboot or a restart of all Workspace ONE UEM services and IIS, before the tool is executed.NOTE: To suppress prompts and the need for manual intervention during scheduled execution, the following parameters can be used: -q - Quiet mode - the tool will not report progress during execution-y - Suppress confirmation prompts (this bypasses warning messages) Additional Information: The tool is built to run alongside Workspace ONE UEM and does not require any services to be stopped during executionThe tool can be executed on multiple servers in parallelThe processing speed of the tool is subject to environmental factors such as server resources and the number of files accumulated in the target folderPlease review the related article https://kb.vmware.com/s/article/82294. Periodic execution may be required if your environment is running any of the fix versions listed therein. In other words, deploying the patches to address CRSVC-18259, will have to be supplemented by running the Clean-MachineKeys tool periodically thereafter The frequency of execution will vary by the environment but we recommend that the tool is not run more often than once-a-week Troubleshooting Before execution of the tool, reboot the server (or restart IIS and WS1 UEM services on the server). If this is not performed, you may see the following exception in UEM server logs: System.Security.Cryptography.CryptographicException: Keyset does not exist If these errors are seen after executing the tool, perform the server reboot or restart of IIS and WS1 UEM services to resolve it.