Dear colleagues,

On Wednesday, February 26, 2025, over 200 computers were locked out due to BitLocker encryption and remained unusable until they were recovered through the IT Help Desk. This occurred as part of our transition from Trellix to BitLocker, which began on February 25.

The good news is that all affected users were successfully recovered without data loss. We sincerely appreciate the assistance of our IT partners across campus, who played a crucial role in helping with the recovery process. However, this incident significantly disrupted operations for many customers. While the actual recovery process took only a few minutes per device, the high volume of calls led to extended wait times for assistance. On behalf of UNLV IT, I deeply apologize for this disruption. 

Major disruptions like this are unacceptable. In the spirit of transparency and accountability, I want to share how we arrived at this situation, what happened, and how we plan to improve moving forward. 

How we got here
On July 1, 2024, we launched Software Center, granting UNLV IT-managed device users the ability to install common software without contacting IT support. Since its launch, Software Center has saved an estimated 1,500+ staff hours by eliminating the need to request installations for common software like Adobe Acrobat, Office 2021, and SPSS. This tool is part of Microsoft Configuration Manager, an integrated endpoint management solution that has allowed us to consolidate multiple management tools into a single environment. 

One of the features we began leveraging in Configuration Manager is device encryption. Previously, we relied on Trellix, which required additional support, cost, and complexity. Since Configuration Manager provides BitLocker—a widely used encryption solution—we saw an opportunity to consolidate. Starting in June 2024, new and rebuilt devices were encrypted using BitLocker instead of Trellix. By July, we expanded encryption to previously unencrypted devices. By January 2025, we had successfully encrypted over 1,075 devices with BitLocker with minimal disruption. Throughout this process, very few devices in customer use required recovery. 

What happened during the incident
Given BitLocker’s success, the next logical step was transitioning devices from Trellix to BitLocker. We initiated this process on February 25, 2025. 

By the morning of February 26, we noticed a surge in support requests from users encountering the BitLocker recovery screen. In response: 

• We immediately expanded access to generate recovery keys, enabling additional IT staff and campus partners to assist users.
• Simultaneously, our technicians visited customers in person to investigate. The initial assessment pointed to a Secure Boot issue that led devices to detect an unauthorized change.
• Further investigation suggested that residual Trellix components left after the switchover process contributed to the problem.
• To mitigate further impact, we deployed a fix via Software Center before noon on February 26, preventing additional devices from being locked out upon reboot.
• By February 27, the number of lockout incidents had significantly decreased, with fewer than 30 new recovery requests. 

During our post-incident review, we identified another contributing factor: outdated firmware and driver updates on many affected devices. This was critical because encryption relies on proper communication between Windows and the device’s Trusted Platform Module (TPM). TPMs provide a secure cryptographic environment for encryption and key storage, making firmware and driver updates essential for system stability. 

What we are doing to improve
While our goal was to complete this transition with minimal user disruption, it is now clear that we attempted to move too many devices at once. Despite our success encrypting over 1,075 devices, this transition should have been phased in gradually. 

Since launching Software Center, we have successfully completed major upgrade campaigns with minimal impact. These included: 

• Upgrading Adobe Acrobat a common package, and providing continuous updates (first for UNLV)
• Upgrading Office 2016/2019 to Office 2021
• Upgrading web browsers and Zoom to a minimum baseline version 

A common success factor was allowing a three- to four-week window for users to complete updates, naturally spreading the deployment load. 

Going forward, we are refining this approach by adopting a ring-based deployment strategy for high-impact deployments. This method starts with a small subset of devices (10-20% of the fleet), then gradually expands the rollout as initial deployments prove successful. Randomizing device assignments to these rings will ensure we test updates across a diverse range of hardware configurations.

Our first implementation of this strategy begins on Tuesday, March 4, 2025, with firmware and driver updates for all Dell devices. We will start with 10% of devices, then expand deployment over several weeks while closely monitoring the rollout to prevent similar incidents in the future.

In conclusion, I want to reiterate that large disruptions like these are unacceptable. I personally apologize to all those affected and commit to making improvements. We are excited to be working to streamline our systems, reduce technical debt and complexity, and reduce costs; though this cannot come at the expense of large scale disruptive incidents. I appreciate the patience of the campus community and the dedication of those who assisted in resolving this incident. I am committed to delivering more positive, proactive, and better solutions for our campus community.

Sincerely,
Bob Soulliere
Assistant Vice President for Digital Services and Solutions