7/22/2025 - Server Critical Failure - Filesystem Corruption Recovery - DT <1hrs - Server Restored!

I'm currently dealing with a critical server failure caused by filesystem corruption on the main virtual machine that provides all media streaming and automation services. The system experienced corruption during a recent performance optimization that required emergency recovery procedures.

Issue Summary

• Date: July 16, 2025

• Duration: <1 hours (ongoing recovery)

• Affected Services: All services (Plex, media automation, downloads)

• Root Cause: EXT4 filesystem corruption during VM resource reallocation

• Status: Emergency filesystem repair in progress Reported outage at 12:15pm EST Server up 12:58pm EST Issue identified: VM CPU allocation increased from 24 to 44 cores without proportional memory scaling for containers. This created a resource imbalance where containers had significantly more processing power but were constrained by original memory limits. The increased CPU capacity caused containers to process workloads more aggressively, leading to memory pressure and over 1.2 million OOM (Out of Memory) events. The OOM-killer's repeated process termination during active I/O operations resulted in filesystem corruption and system instability.

  Resolution Implemented:

  Applied enterprise-grade resource allocation strategy with proportional CPU and memory limits across all containers.