Certificate Connector for Microsoft Intune - Could not create SSL/TLS secure channel

We have the Certificate Connector for Microsoft Intune (v6.2301.1.0) installed on a Server 2019 box and were hitting errors in the CertificateConnector Operational event logs for the PKI Revoke Service and the PKI Create Service looking like this:

Error 5001 - HealthMessageUploadFailedAttempt

Pki Revoke Service:
Failed to upload health messages. Requeuing messages
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---

Error 3003 - RevokeDownloadFailure

Pki Revoke Service:
Failed to download Revocation requests.
ActivityId: xxxx
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---

Error 2 - Exception

Pki Create Service:
Microsoft.Intune.Connectors.Pki.AgentRenewalProcessor.Process threw an exception.
System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.Net.WebException: The request was aborted: Could not create SSL/TLS secure channel.
   at System.Net.HttpWebRequest.EndGetResponse(IAsyncResult asyncResult)
   at System.Net.Http.HttpClientHandler.GetResponseCallback(IAsyncResult ar)
   --- End of inner exception stack trace ---

After a bit of time with our GPO's to see what might have been the cause and Wireshark'ing packets, it came down to a setting enabling the TLSv1.3 protocol in the registry in one GPO. While it's not supported in this version of the OS, but the following was set as a catch for one day when it might be:

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.3\Client]

"Enabled"=dword:00000001

"DisabledByDefault"=dword:00000000

[HKEY_LOCAL_MACHINE\SYSTEM\ControlSet002\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.3\Server]

"Enabled"=dword:00000000

"DisabledByDefault"=dword:00000001

Delete those 4 lines from either the GPO or the registry and the PKI Revoke service will it start downloading and uploading correctly. I can only assume that the app was written to try TLSv1.3 if it is mentioned in the registry and it will fail (at this time). Perhaps one day when TLSv1.3 is supported in Server 2019, it might work properly. Hope this helps someone else out there.

Veeam agent error with asynchronous read operation failed

We use Veeam Agent v5 (and v4, same error) for our backups and found we have some Windows 2019 physical servers using a Volume level backup that have recently failed in their Operating System backup with the following error.

The text of the error gives a little bit more to work with.

31/03/2021 10:00:43 AM :: Error: The device is not ready. Asynchronous read operation failed Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: The device is not ready. Asynchronous read operation failed Unable to retrieve next block transmission command. Number of already processed blocks: [0]. Failed to download disk '1140af5b-d567-46db-9b74-11c85ca1fb25'.  

So we logged a job with Veeam and their answer was to log a job with Microsoft as it was a server error. Time to have a deeper look before we went down that path.

This was a bit strange, the jobs were happy until a round of Windows Updates done some time in February. Looking at the volume picker below showed what the job was trying to back up.

If we removed the Recovery volume from the selected volumes, there is an error about an incomplete backup but the job runs successfully with the remaining two volumes. Including the Recovery volume into the backup causes it to fail again. A backup of only the Recovery volume also had the same failure.

A hunt of the Veeam forums showed that a few others had the same issue.

Hunting around with PowerShell provided this from the Recovery partition

(Get-Partition).Where({$_.type -eq "Recovery"}) | FormatList


UniqueId             : {00000000-0000-0000-0000-100000000000}600508B1001C480977D3EE56F92B41FB

AccessPaths          : {\\?\Volume{62480ffe-130d-4cfd-a21f-e44b531d5a34}\}

DiskNumber           : 4

DiskPath             : \\?\scsi#disk&ven_hpe&prod_logical_volume#5&25daa67a&0&000100#{53f56307-b6bf-11d0-94f2-00a0c91efb8b}

DriveLetter          :

Guid                 : {62480ffe-130d-4cfd-a21f-e44b531d5a34}

IsActive             : False

IsBoot               : False

IsHidden             : False

IsOffline            : True

IsReadOnly           : False

IsShadowCopy         : False

IsDAX                : False

IsSystem             : False

NoDefaultDriveLetter : True

Offset               : 1048576

OperationalStatus    : Offline

PartitionNumber      : 1

Size                 : 499 MB

Type                 : Recovery

Operational Status is Offline, how about we look at that. So with

(Get-Partition).Where({$_.type -eq "Recovery"}) | Set-Partition -IsOffline $False

We set the Recovery partition to being online, run a backup job with the Recovery volume included and it works. Huzzah!

Then a reboot to test and we found that the Offline setting returns to $True after a reboot and the Veeam Agent job would fail. Setting a PowerShell script with the above line in it and making it run as a Scheduled Task on Startup ensured the Recovery volume was online. Not sure if that is best practice, but it makes the job run and our backup engineer happy.

Set-AzStorageBlobContent and Illegal characters in path

Occasionally I upload some content to an Azure Blob storage account for long term archival from a Windows 2016 server using a script (upload-azure.ps1) and the AZ PowerShell modules, specifically Set-AzStorageBlobContent.
When I last ran this 4 months ago, back in Jan 2020, it went fine. I got the error below a month or so ago and just used the web interface to upload instead. Today, when I have lots of content to upload it was time to get the script working. This is what was getting:

Set-AzStorageBlobContent : Failed to open file C:\Upload\Al.7z: Illegal characters in path..
At C:\Upload\upload-azure.ps1:95 char:13
+             Set-AzStorageBlobContent -File $FilePath -Container $cont ...
+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : CloseError: (:) [Set-AzStorageBlobContent], TransferException
    + FullyQualifiedErrorId : TransferException,Microsoft.WindowsAzure.Commands.Storage.Blob.SetAzureBlobContentCommand

The relevant line 95 from the script is:

Set-AzStorageBlobContent -File $FilePath -Container $containerName -Context $storageAccount.Context -Blob $file.Name -Metadata $Metadata

I updated the Az.Storage PowerShell modules (see the fun and games here) to v1.14.0 and tried again. Same issue. However, it turns out there has been some path handling changes in .Net Framework that turned up on my server of recent. This article https://docs.microsoft.com/en-gb/archive/blogs/jeremykuhne/new-net-path-handling-sneak-peek hints to it. Checking the basics of path handling with this below points to something more in .Net than in the Az commands:

PS C:\Users\Al> [System.IO.Path]::GetFullPath("\\?\c:\pagefile.sys")
Exception calling "GetFullPath" with "1" argument(s): "Illegal characters in path."
At line:1 char:1
+ [System.IO.Path]::GetFullPath("\\?\c:\pagefile.sys")
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : NotSpecified: (:) [], MethodInvocationException
    + FullyQualifiedErrorId : ArgumentException

Then reading this again, https://github.com/Azure/azure-powershell/issues/8473, it now has a solution on it that easy to apply - thanks fimcle!

$registryPath = "HKLM:\SOFTWARE\Microsoft.NETFramework\AppContext"
New-Item -Path $registryPath
New-ItemProperty -Path $registryPath -Name "Switch.System.IO.UseLegacyPathHandling" -Value "false"

Run that and open a new PowerShell window and hazaah, I can now upload content to my Blob storage again.

Error "PowerShell Gallery is currently unavailable"

I was trying to update some Az modules on a Windows 2016 server (PowerShell version 5.1.14393.3471 for what it's worth) today to connect to some Azure storage and thought I'd run the normal Update-Module to keep things nice and fresh. However, this time it was failing with:

PS C:\Users\Al> Update-Module
Get-PSGalleryApiAvailability : PowerShell Gallery is currently unavailable.  Please try again later.
At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:2038 char:13
+             Get-PSGalleryApiAvailability -Repository (Get-SourceName  ...
+             ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : InvalidOperation: (:) [Get-PSGalleryApiAvailability], InvalidOperationException
    + FullyQualifiedErrorId : PowerShellGalleryUnavailable,Get-PSGalleryApiAvailability

So running it with a debug gets me this (with some cruft removed):

PS C:\Users\Al>Update-Module -Debug
DEBUG: 00:00:47.3393273 Source 'https://www.powershellgallery.com/api/v2' is not one of the 'NuGet' provider.
...SNIP....
VERBOSE: An error occurred while sending the request.
VERBOSE: Retry downloading 'https://www.powershellgallery.com/api/v2' for '2' more times
VERBOSE: An error occurred while sending the request.
VERBOSE: Retry downloading 'https://www.powershellgallery.com/api/v2' for '1' more times
VERBOSE: An error occurred while sending the request.
VERBOSE: Retry downloading 'https://www.powershellgallery.com/api/v2' for '0' more times
WARNING: Unable to resolve package source 'https://www.powershellgallery.com/api/v2'.
...SNIP...

Interesting. Lets just try check the repository is good first:

PS C:\Users\Al> Get-PSRepository
WARNING: MSG:UnableToDownload «https://go.microsoft.com/fwlink/?LinkID=627338&clcid=0x409» «»
WARNING: Unable to download the list of available providers. Check your internet connection.

Name                      InstallationPolicy   SourceLocation
----                      ------------------   --------------
PSGallery                 Untrusted            https://www.powershellgallery.com/api/v2

OK. Well it looks good but that warning is funny. Looking at the firewall logs it showed successful connections to www.powershellgallery.com so it wasn't access out to the internet. Lets try and install the module again.

PS C:\Users\Al> Install-Module -Name Az
WARNING: Unable to resolve package source 'https://www.powershellgallery.com/api/v2'.
PackageManagement\Install-Package : No match was found for the specified search criteria and module name 'Az'. Try
Get-PSRepository to see all available registered module repositories.
At C:\Program Files\WindowsPowerShell\Modules\PowerShellGet\1.0.0.1\PSModule.psm1:1772 char:21
+ ...          $null = PackageManagement\Install-Package @PSBoundParameters
+                      ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Microsoft.Power....InstallPackage:InstallPackage) [Install-Package], Ex
   ception
    + FullyQualifiedErrorId : NoMatchFoundForCriteria,Microsoft.PowerShell.PackageManagement.Cmdlets.InstallPackage

Bother. At this point, having found three different error messages and being able to download the URL's in the warnings using IE, I looked into some Wireshark dumps and noticed that while the IE was using TLS1.2, this PowerShell was not and failing at the server side to connect properly. Time to enforce TLS1.2 in the PowerShell session with this command:

[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

Rerun the Update-Module and it works. I'm not sure what I might have done to stop it working, perhaps a GPO or some such to set PowerShell to default to TLS1.1 or 1.0 or perhaps PowerShellGallery turned off TLS1.0 and 1.1 support (smart move). You can Google that one to find a better place to put that line so it sticks for each session.

Online Archive missing in Outlook after Office ProPlus upgrade

We recently upgraded a bunch of users from Office 2013 / 2016 Professional Plus to Office 2016 ProPlus using our O365 E3 licences. A bunch of users, both on-prem and in O365, could no longer see their Online Archive in their Outlook. The online archive was gone, kaput, missing, absent, you get the idea.

Interestingly, it did show up in their Outlook Online / on-prem webmail however.

This pinpointed it to probably being their Office 2016 KMS licence key from the previous install. A check of the current licences a user had looked like:

"c:\windows\system32\cscript.exe "C:\Program Files (x86)\Microsoft Office\Office16\ospp.vbs" /dstatus"

---Processing--------------------------
---------------------------------------
PRODUCT ID: xxxxx-xxxxx-xxxxx-xxxxx
SKU ID: d450596f-894d-49e0-966a-fd39ed4c4c64
LICENSE NAME: Office 16, Office16ProPlusVL_KMS_Client edition
LICENSE DESCRIPTION: Office 16, VOLUME_KMSCLIENT channel
BETA EXPIRATION: 1/01/1601
LICENSE STATUS:  ---LICENSED---
REMAINING GRACE: 178 days  (257459 minute(s) before expiring)
Last 5 characters of installed product key: abcde
Activation Type Configuration: ALL
        KMS machine name from DNS: x.x.x.x:1688
        Activation Interval: 120 minutes
        Renewal Interval: 10080 minutes
        KMS host caching: Enabled
---------------------------------------
---------------------------------------
---Exiting----------------------------- 

Hmmm, an old licence in there. Not sure why it would cause issues, but it's not the one they should have for O365. So, delete the licences with this command, using the last 5 characters of installed product key from the above command:

"c:\windows\system32\cscript.exe "C:\Program Files (x86)\Microsoft Office\Office16\ospp.vbs" /unpkey:abcde"

Restart Outlook and provided the user has a proper E3 / E5 licence, Online Archive should be visible.

Errors reinstalling MS Advanced Threat Analytics

When our MS ATA server was set up, we had to choose a certificate to use for the encrypted communications between domain controllers and the ATA server. So we picked the default computer cert issued by our internal PKI which was valid for another year.

What we forgot was this certificate expires and self renews. This error would have been recoverable had it been noticed before the server was rebooted, but it wasn't. So because the old cert was gone, we couldn't start the ATA services and the domain controllers were no longer talking to the ATA server. We rebuilt the ATA server and tried to reinstall the agents on the DC's. Perhaps there is an easier way....

This is where the problems started on our server-core DC's. Sometimes the agent would uninstall and sometimes not. It wouldn't upgrade however.

So, in order of things I tried to remove the agent:

With the new agent install executable in the current directory, try

"Microsoft ATA Gateway Setup.exe" /quiet /uninstall

If that didn't work, start a PowerShell and use the WMI:

$app = Get-WmiObject -Class Win32_Product -Filter "Name = 'Microsoft Advanced Threat Analytics Gateway'"
$app.uninstall()

If it's still installed, there are some log files (Microsoft Advanced Threat Analytics Gateway_yyyymmddtime.log)for each installation attempt of the agent in your %temp% (or one directory up like C:\Users\al\AppData\Local\Temp) that make for some reading like:

 .....

[0410:0C24][2018-08-07T14:10:58]i000: 2018-08-07 02:10:58.6626 1040 5   Debug [\[]DeploymentModel[\]] [\[]DeploymentAction=Upgrade[\]]
[0410:0C24][2018-08-07T14:10:59]i000: 2018-08-07 02:10:59.1313 1040 5   Error [\[]GatewayBootstrapperApplication[\]] Failed to create deployment manager [\[]exception=System.ArgumentNullException: Value cannot be null.
Parameter name: path1
   at System.IO.Path.Combine(String path1, String path2)
   at Microsoft.Tri.Gateway.Deployment.Bundle.UI.Application.GatewayUpgradeDeploymentConfiguration.GetMandatoryConfiguration(String installationPath)
   at Microsoft.Tri.Gateway.Deployment.Bundle.UI.Application.GatewayUpgradeDeploymentConfiguration..ctor(Engine engine, String installationPath)
   at Microsoft.Tri.Gateway.Deployment.Bundle.UI.Application.GatewayDeploymentModel.CreateDeploymentConfiguration(Engine engine)
   at Microsoft.Tri.Deployment.Bundle.UI.Application.DeploymentModel..ctor(BootstrapperApplication bootstrapperApplication, ProductComponent productComponent, String productFullShortName, String productFullLongName, String productFullLongDisplayName)
   at Microsoft.Tri.Gateway.Deployment.Bundle.UI.Application.GatewayDeploymentModel..ctor(BootstrapperApplication bootstrapperApplication)
   at Microsoft.Tri.Gateway.Deployment.Bundle.UI.Application.GatewayBootstrapperApplication.CreateDeploymentModel()
   at Microsoft.Tri.Deployment.Bundle.UI.Application.BootstrapperApplication`1.Run()[\]]

.....


This one was fixed by going to "C:\ProgramData\Package Cache" and searching for *.exe (dir *.exe /s). You'll see some GUID's and file names like "Microsoft ATA Gateway Setup.exe" then run the uninstall command with the path including the GUID

"C:\ProgramData\Package Cache\{GUID}\Microsoft ATA Gateway Setup.exe /uninstall"

Repeat for each version of that EXE you find under that directory. That should do it.

Finally, go and delete the contents of "C:\Program Files\Microsoft Advanced Threat Analytics\Gateway" then remove the directories "Gateway" and "Microsoft Advanced Threat Analytics" otherwise you get errors in that log like:

[13BC:1244][2018-08-07T14:15:33]i000: 2018-08-07 02:15:33.5655 5052 5   Debug [\[]DeploymentModel[\]] [\[]DeploymentAction=Install[\]]
[13BC:1244][2018-08-07T14:15:33]i000: 2018-08-07 02:15:33.8155 5052 5   Debug [\[]DeploymentModel[\]] [\[]IsAfterRestartAndConfigured=False[\]]
[13BC:1244][2018-08-07T14:15:37]i000: 2018-08-07 02:15:37.1749 5052 5   Error [\[]DeploymentManager[\]] InstallationPath is invalid [\[]directoryState=NotEmpty[\]]
[13BC:1244][2018-08-07T14:15:37]i000: 2018-08-07 02:15:37.1905 5052 5   Debug [\[]GatewayBootstrapperApplication[\]] Engine.Quit [\[]deploymentResultStatus=1602 isRestartRequired=False[\]]

You are then free to reinstall the agent with:

"Microsoft ATA Gateway Setup.exe" /quiet NetFrameworkCommandLineArguments="/q"

Error 2150858882 with Event Log Fowarding

After setting up a GPO for the event log forwarding service to use https instead of http to talk to our collector, with the recommended settings of server, refresh and issuerCA, I kept getting this 105 error in the Eventlog-FowardingPlugin log on my workstations:

The forwarder is having a problem communicating with subscription manager at address https://mycollector.alsheppard.com:5986/wsman/SubscriptionManager/WEC.  Error code is 2150858882 and Error Message is .

That's it, no error message. Frustrating. However, removing the IssuerCA, leaving the server and refresh values from the GPO's SubscriptionManager line seems to have resulted in a happy event forwarding service.

I would also check Kerberos is set up on the listening server, or at least the SPN exists.