7/24/2025 - Server Critical Failure - TrueNAS Upgrade VM Loss - DT <6hrs - Service Restored

I'm currently dealing with a critical server failure caused by a TrueNAS SCALE upgrade that resulted in complete loss of the main virtual machine hosting all media streaming and automation services. The system experienced VM destruction during routine maintenance, requiring emergency recovery procedures.

Issue Summary

• Date: July 24, 2025

• Duration: <4 hours (emergency recovery)

• Affected Services: All services (Plex, media automation, downloads)

• Root Cause: TrueNAS upgrade destroyed VM configuration and access

• Status: Enterprise media server fully restored

Reported outage at 8:45am EST Server up: 2:10pm EST

Issue identified: TrueNAS SCALE upgrade to 25.04.1 replaced working virtualization system with experimental version that provided no method to access or recover existing VMs. The 6.59TB enterprise media server became completely inaccessible despite successful storage preservation.

The new virtualization interface showed "No instances" with no buttons or options to create, import, or recover the production VM containing months of optimization work and enterprise-grade configuration.

Resolution being Implemented:

Emergency snapshot recovery using TrueNAS automatic backup system to restore complete VM from recent snapshot, followed by boot environment rollback to stable version with working virtualization tools. Additional filesystem repair resolved secondary corruption issues during recovery.