Operating System Upgrade Strategy

Keep two LVM partitions on each machine.  When upgrading, always upgrade the secondary partition, leaving the primary partition in a working state.  The secondary partition then becomes the actively used one.