Veeam agent error with asynchronous read operation failed

We use Veeam Agent v5 (and v4, same error) for our backups and found we have some Windows 2019 physical servers using a Volume level backup that have recently failed in their Operating System backup with the following error.

The text of the error gives a little bit more to work with.

31/03/2021 10:00:43 AM :: Error: The device is not ready. Asynchronous read operation failed Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: The device is not ready. Asynchronous read operation failed Unable to retrieve next block transmission command. Number of already processed blocks: [0]. Failed to download disk '1140af5b-d567-46db-9b74-11c85ca1fb25'.  

So we logged a job with Veeam and their answer was to log a job with Microsoft as it was a server error. Time to have a deeper look before we went down that path.

This was a bit strange, the jobs were happy until a round of Windows Updates done some time in February. Looking at the volume picker below showed what the job was trying to back up.

If we removed the Recovery volume from the selected volumes, there is an error about an incomplete backup but the job runs successfully with the remaining two volumes. Including the Recovery volume into the backup causes it to fail again. A backup of only the Recovery volume also had the same failure.

A hunt of the Veeam forums showed that a few others had the same issue.

Hunting around with PowerShell provided this from the Recovery partition

(Get-Partition).Where({$_.type -eq "Recovery"}) | FormatList


UniqueId             : {00000000-0000-0000-0000-100000000000}600508B1001C480977D3EE56F92B41FB

AccessPaths          : {\\?\Volume{62480ffe-130d-4cfd-a21f-e44b531d5a34}\}

DiskNumber           : 4

DiskPath             : \\?\scsi#disk&ven_hpe&prod_logical_volume#5&25daa67a&0&000100#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}

DriveLetter          :

Guid                 : {62480ffe-130d-4cfd-a21f-e44b531d5a34}

IsActive             : False

IsBoot               : False

IsHidden             : False

IsOffline            : True

IsReadOnly           : False

IsShadowCopy         : False

IsDAX                : False

IsSystem             : False

NoDefaultDriveLetter : True

Offset               : 1048576

OperationalStatus    : Offline

PartitionNumber      : 1

Size                 : 499 MB

Type                 : Recovery

Operational Status is Offline, how about we look at that. So with

(Get-Partition).Where({$_.type -eq "Recovery"}) | Set-Partition -IsOffline $False

We set the Recovery partition to being online, run a backup job with the Recovery volume included and it works. Huzzah!

Then a reboot to test and we found that the Offline setting returns to $True after a reboot and the Veeam Agent job would fail. Setting a PowerShell script with the above line in it and making it run as a Scheduled Task on Startup ensured the Recovery volume was online. Not sure if that is best practice, but it makes the job run and our backup engineer happy.