vMotion Fails at 14% - Operation Timed Out

While working on a 3 host VMware cluster I was trying to vMotion some servers around and begin maintenance and updates.  I hit this error for the first time "Operation Timed Out" and the dreaded red X.

As it turns out there was a vmx-****.vswp file inside the VM folder that apparently was left over from a failed DRS migration.  After doing some research I found out that during a DRS migration the VMX file is started on both nodes.  Once the migration is completed normally one of these gets removed.  In this case it did not.

If you browse the datastore of the VM that will not move and you open up the folder of the VM you should see two vmx-****.vswp files.  If you see only one then you've got another issue and this isn't the fix.  If you do see two they will be .vswp-1 and .vswp-2.  There could be a .vswp for the VM itself.  DON'T DELETE THAT ONE!  I can't stress that enough.

To know which one to get rid of just look at the time stamps.  If the VM is running it may be hard to tell which is which so make sure you turn the VM off before you begin this process.  Once it's off then whichever file remains in the folder is the one you need to delete.

You can try deleting the file using the datastore browser but if that doesn't work then here's how you can remove it.

1. Start the SSH daemon on the host which has the VM registered and then login via SSH.  I recommend Putty.  You can download it HERE along with many other great tools for network and systems maintenance.
2. # cd /vmfs/volumes/{datastore name}
3. # cd {VM folder name}
4. # ls -lash *.vswp (this will show all the files and timestamps just to verify)
5. # rm vmx-{VM name}-[1-2].vswp

That's all there is to it.  Remember to disable the SSH daemon on your host and now you should be able to vMotion the VM again with it running.

How Long Is Left on Database (IN RECOVERY)

While working on a SQL 2008 R2 database connection error for a medical company I found the database showing (IN RECOVERY).  Naturally no connections could be made during this process.  What I truly needed to know was how to see how long the process has left.  I found this great script HERE.  For my own purposes I decided to list it here as well.

DECLARE @DBName VARCHAR(64) = 'databasename'

DECLARE @ErrorLog AS TABLE([LogDate] CHAR(24), [ProcessInfo] VARCHAR(64), [TEXT] VARCHAR(MAX))

INSERT INTO @ErrorLog
EXEC sys.xp_readerrorlog 0, 1, 'Recovery of database', @DBName

SELECT TOP 5
[LogDate]
,SUBSTRING([TEXT], CHARINDEX(') is ', [TEXT]) + 4,CHARINDEX(' complete (', [TEXT]) - CHARINDEX(') is ', [TEXT]) - 4) AS PercentComplete
,CAST(SUBSTRING([TEXT], CHARINDEX('approximately', [TEXT]) + 13,CHARINDEX(' seconds remain', [TEXT]) - CHARINDEX('approximately', [TEXT]) - 13) AS FLOAT)/60.0 AS MinutesRemaining
,CAST(SUBSTRING([TEXT], CHARINDEX('approximately', [TEXT]) + 13,CHARINDEX(' seconds remain', [TEXT]) - CHARINDEX('approximately', [TEXT]) - 13) AS FLOAT)/60.0/60.0 AS HoursRemaining
,[TEXT]

FROM @ErrorLog ORDER BY [LogDate] DESC

Update for SQL 2012.  The syntax is a bit different:

DECLARE @DBName VARCHAR(64) = 'databasename'

DECLARE @ErrorLog AS TABLE([LogDate] CHAR(24), [ProcessInfo] VARCHAR(64), [TEXT] VARCHAR(MAX))

INSERT INTO @ErrorLog
EXEC master..sp_readerrorlog 0, 1, 'Recovery of database', @DBName

SELECT TOP 5
[LogDate]
,SUBSTRING([TEXT], CHARINDEX(') is ', [TEXT]) + 4,CHARINDEX(' complete (', [TEXT]) - CHARINDEX(') is ', [TEXT]) - 4) AS PercentComplete
,CAST(SUBSTRING([TEXT], CHARINDEX('approximately', [TEXT]) + 13,CHARINDEX(' seconds remain', [TEXT]) - CHARINDEX('approximately', [TEXT]) - 13) AS FLOAT)/60.0 AS MinutesRemaining
,CAST(SUBSTRING([TEXT], CHARINDEX('approximately', [TEXT]) + 13,CHARINDEX(' seconds remain', [TEXT]) - CHARINDEX('approximately', [TEXT]) - 13) AS FLOAT)/60.0/60.0 AS HoursRemaining
,[TEXT]

FROM @ErrorLog ORDER BY [LogDate] DESC

Just copy and paste this script in, change the database name, and you're good to go.

EDIT:  One thing to note is that it may show over 85% and yet if you check the database the process has completed.  I suspect this is because once the recovery is done this information is no longer updated.

Adjust Exchange 2013 Send-Receive Message Size Limits

After setting up and Exchange 2013 SP1 server on Server 2012 R2 I ran into this small issue with a customer.  Even though I had all of the send/receive connectors in the ECP set at 100M, I was still hitting the default 10M attachment size limits.

Here's how to fix that by changing the Global Transport Settings: (short answer)

Check to see what the size limits currently are -
Get-TransportConfig | ft MaxSendSize, MaxReceiveSize

Once you know those sizes then here's the command to change them to what you need -
Set-TransportConfig –MaxSendSize 100MB –MaxReceiveSize 100MB

I know these sizes are a little large but just using them as an example.  Go ahead and bounce the Transport service just to be sure the settings take effect and you'll be good to go.

All of the other size limits can be set using the ECP under Mailflow and then Send Connectors and Receive Connectors.

Update:  I've just completed reading almost all of Mastering Microsoft Exchange Server 2013 .  This is an excellent read that will give you tons of insight into installing, maintaining, and mastering all there is to Exchange 2013.  I highly recommend you give the book a read.

How to Disable the 21 Day Timer for SBS 2003, 2008, and 2011

Here’s the process to prevent SBS2003 from shutting down after 21 days of migration have passed. The basic principle is to disable the SBS Core Service (sbscrexe.exe). The problem is when you just turn off the service, it will restart itself automatically. To achieve this goal follow the steps below:

1. Download process explorer from http://technet.microsoft.com/en-us/sysinternals/bb896653
2. Run Process Explorer and SUSPEND the sbscrexe.exe file.
3. Open regedit and navigate to HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\SBCore
4. Right click the key and click permissions. Give the Administrators group full control on both the key and child nodes.
5. Refresh the key so you can see all of the registry settings for sbcore and change the Start DWORD value from 2 to 4 to disable the service.
6. Using explorer navigate to the sbscrexe.exe file in the C:\WINDOWS\system32 directory and change permissions on the file to everyone deny.
7. Go back to process explorer and kill the sbscrexe.exe process. If it doesn’t start again then you have successfully disabled the file.

Exchange 2010: The WS-Management Service Cannot Process The Request...

PROBLEM: The WS-Management service cannot process the request. The user load quota of 1000 requests per 2 seconds has been exceeded. Send future requests at a slower rate or raise the quota for this user. The next request from this user will not be approved for at least XX milliseconds.

This can happen if a certificate on Exchange expires (SSL or UCC) and the IIS service has not been restarted since.

All you need to do is open a command prompt (Run as Administrator if using UAC) and the type in IISRESET.  Press ENTER and wait.

Now you should be able to login to the Exchange 2010 EMC with no issues.

Really simple fix.

Good luck!