LEARN – Getting CRC errors from ESXi hosts

I’ve been having a ton of issues with NFS datastores in one of my vcenters. They’re running Dell R640s with two NICs and everything is running fine except when you run IO speed tests. Working with VMware support they pointed out CRCerrors on one of the ports on my host.

Cyclic Redundancy Check (CRC) Error indicates when data is corrupted. Calculating from all data, CRC validates packets of information sent by devices and verifies it against the data extracted, ensuring its accuracy. When sending packets over,BACnet automatically calculates and stores a CRC value for the packet.

CRC errors are a bit vague but usually indicate one of three things. A bad port on the host, a bad cable or a bad port on the switch. Me wanting to check others found that we’re seeing CRCerrors on almost all hosts (but not all) in that particular cluster. This leads me to believe that our switch is bad but we’re waiting for network ops to verify that. 

My goal writing this script was to pull all hosts, get their NIC information, get their NIC transmit and receive errors, write the output to a CSV, then compare that CSV to the last one, and tell me which hosts have increased errors, since we don’t care about ones that haven’t. 

#Function Get-CRCErrors {
<#
.SYNOPSIS
This script is to collect NIC information from all hosts in a vcenter, exports to excell then compares latest two files.
Script then outputs the results only if CRC has increast and emails results


.NOTES
Name: Get-CRCerrors
Author: EricNeudorfer
Version: 1.2
DateCreated: 2021-Feb-23

.To Do
Add variable for path to file

 

#>

#remove-item “\\PATH TO FILE\platform_shared\CI\Eric Neudorfer\Scripts\output\get-crcerrors_OUT.csv”
$DC = Get-Datacenter
clear-host

Write-host “##################################################################################################################”
Write-host “##################################################################################################################”
Write-host “##################################################################################################################”
Write-host “############## Datacenter” $DC
Write-host
Write-host
$csvf = “\\PATH TO FILE\platform_shared\CI\Eric Neudorfer\Scripts\output\CRC_REPORTS\$($DC)_CRC_ERROR_$(get-date -f yyyyMMddTHHmmssZ).csv”

$esxName = get-vmhost *
$output = Foreach ($Name in $esxName) {

 

$esxcli = Get-EsxCli -VMHost $Name -V2
$esxcli.network.nic.list.Invoke() |

ForEach-Object -Process {

$esxcli.network.nic.stats.get.Invoke(@{nicname=$_.Name})
| Select @{N=’VMHost’;E={$esxcli.VMHost.Name}},
NICName,PacketsReceived,PacketsSent,ReceiveCRCerrors,Totalreceiveerrors,Totaltransmiterrors #| export-csv $csvf -noType -append

}

Write-Progress -Activity ‘Collecting NIC Transmission Error’ -Status “Current Host: [$Name]”;
}

 

$output | export-csv $csvf -NoTypeInformation -Force

Write-host “Colletion Completed”
Write-host “Start Comparison”

 

 

#######################################################
## Comparison portion

remove-item “\\PATH TO FILE\platform_shared\CI\Eric Neudorfer\Scripts\output\CRC_REPORTS\reports.csv”

#Importing CSV
$path = “\\PATH TO FILE\platform_shared\CI\Eric Neudorfer\Scripts\output\CRC_REPORTS\”
$file = Get-ChildItem -Path “\\PATH TO FILE\platform_shared\CI\Eric Neudorfer\Scripts\output\CRC_REPORTS\” | select -last 2
$CSVpath1 = $path + $file[0].Name
$CSVpath2 = $path + $file[1].Name

$CSV1 = Import-CSV -Path $CSVpath1
$CSV2 = Import-CSV -Path $CSVpath2

$Count = $CSV2.Count
$Results = For ($i = 0; $i -lt $Count; $i++) {
If ($CSV2[$i].Totalreceiveerrors -ne $CSV1[$i].Totalreceiveerrors) {
$Match = “Not Match”
}
Else {
$Match = “Match”
}
[PSCustomObject]@{
VMHost = $CSV2[$i].VMHost
NICName = $CSV2[$i].NICName
Old = $CSV1[$i].Totalreceiveerrors
New = $CSV2[$i].Totalreceiveerrors
Results = $Match
}
}
$Results | Where-Object {$_.Results -like “Not Match”} | Export-Csv -Path “\\PATH TO FILE\platform_shared\CI\Eric Neudorfer\Scripts\output\CRC_REPORTS\reports.csv” -NoTypeInformation

 

 

# Send MailMessage
$From = ‘vcheck <[email protected]$$$$$$.com>’
$To = ‘Eric <[email protected]$$$$$$$$.com>’
$Subjet = ‘ESXi NIC Transmit/Recieve errors’
$Body = “Report CRC errors in SHK”
$Attachemtns = ‘\\PATH TO FILE\platform_shared\CI\Eric Neudorfer\Scripts\output\CRC_REPORTS\reports.csv’

Send-MailMessage -From $From -To $To -Subject $Subjet -Body $Body -Attachments $Attachemtns -DeliveryNotificationOption OnSuccess, OnFailure -SmtpServer ‘mailer.fico.com’