Trend Deep Security Warning Message ‘Machine was unprotected during move from one esx host to another’


I wanted to post some more information on this Trend DS error message – ‘Machine was unprotected during move from one esx host to another’ as it seems to come up regularly.

The description of the error message is, ‘a virtual machine was moved to an ESX that does not have an activated Deep Security Virtual Appliance.’

In essence this warning message is saying that the ESXi host you vMotioned your VM too is not currently protecting the virtual machine.

This can be because there is no virtual appliance on the target ESXi host, the Trend Virtual Appliance is not offering Anti Malware protection, is not Activated or is Offline.

This error message will not show for unactivated virtual machines — A virtual machine has to be activated to generate this error message.

There is a known bug with this error message too – even though your VM is being protected by the appliance, the error message is always reported as an Agent error. Apparently Trend are working on this.

Back to the error message: When you receive this error message, what is the next step?

Trend is a complicated beast – An appliance can have issues for a number of reasons – whether there is a fault with the appliance or one of its dependencies is what you need to figure out. It could be something as basic as the appliance dropping off the network, losing connectivity back to the DSM or to the vShield Endpoint VMkernel port, or possibly its no  longer activated (not registered as a security appliance in vShield Manager.)

If you get this warning  message, open the virtual appliance that the VM is currently residing on and first ‘Clear Warnings/Errors’  so you remove any old status\error messages and then run ‘Check Status’ to see if there are any new issues. If there are errors reported on the appliance try and resolve them by following the patented ‘Trend DS Virtual Appliance Health Check’ below.

My main bugbear with Trend is that it is too complicated and it does not report its current state accurately and concisely. When I run a Check Status I want to know exactly what is going on. It would be most useful to have a health check screen on the appliance where the health check tests I mention below in the article are run sequentially in full view for the benefit of the administrator. Issue could be highlighted immediately and it would give us confidence that the appliance and its dependencies are all configured correctly, rather than having to check all the different components individually.

For example if you check the status of your appliance and it reports back that it is Managed and Online you would expect it to be managed, online and offering anti malware protection. In my testing after I changed the vShield VMkernel IP address on my ESXi host from 169.254.1.1 to 169.254.1.2, so the appliance could not offer anti malware protection, I ran a Check Status and the virtual appliance would still report that it was managed, online and offering anti malware protection.

On the plus side when I migrated a VM to the ESXi host with the misconfigured VMkernel port, the warning message was still generated that the VM is unprotected. What this shows is this error message is symptomatic of an underlying issue with your virtual appliance or ESXi host. While the issue may not be immediately noticable because the DSM reports that all is well, you should dig deeper following the ‘Trend DS Virtual Appliance Health Check’ below.

Bottom line — You cannot fully trust the DSM when you notice this error message. The only way to verify for sure that the appliance is actually working or not would be to drop the EICAR virus on the VM to confirm whether anti malware protection is working.

‘Trend DS Virtual Appliance Health Check’:

  1. Synchronise your Virtual Center(s) in Trend DSM
  2. Confirm your credentials for VVC and vShield are uptodate
  3. Confirm filter driver is installed on ESXi host via Trend DSM
  4. Confirm vShield driver is installed on ESXi host via vShield Manager
  5. Confirm Trend Appliance is registered as Security VM with vShield Manager
  6. Confirm the appliance is in the correct VLAN
  7. Confirm the appliance network configuration is correct
  8. Confirm you can ping the Appliance from the DSM.
  9. Confirm the VMkernel IP address for vShield Endpoint is correct on ESXi host – 169.254.1.1

and if nothing works follow my last resort:

10. Deactivate and reactivate the appliance

And if that fails…. Follow the blocksandbytes ‘Triple D’ process:

11. Deactivate, Delete and Deploy the appliance.

When I’m being lazy and I know the config hasn’t changed I will Deactivate and reactivate the appliance immediately. What I find with Trend is that as long as your environment is static, Trend will continue to stay Green, but if your environment is fairly dynamic and hosts are being rebooted, VMs are being built and vMotioned, you are performing SRM fail overs and fail backs, etc. it struggles to keep up with environment changes.

Every week I have to try and figure out why virtual machines are unhappy and do not have anti-malware protection. Hopefully this will help others stay on top of Trend DS 8.

Advertisements

Trend DS 8 Feature #873 – 300 VMs are not protected?


Argh, well I was performing some security hardening last week. One of my tasks was to tidy up the Administrators group in vCenter.

Yes, dangerous I know and it looks like a few service accounts were without vCenter admins for a while.

My fault completely, I had no one to blame this time, but I wasn’t expecting the fall out. All my other applications were fine – VUM, SRM, vShield, VC Ops,  etc but not Trend.

I got an email alert that 300 VMs were not protected. That took me by surprise. Must be some sort of mistake, so I login to the Trend Manager, and every single virtual machine and every single appliance was unmanaged.

WTF!

Looks like Trend DSM shat itself without admin privileges. According to Trend, apparently its expected behaviour. Sounds like pretty shite expected behaviour to me!

Thankfully re-activating all virtual appliances and VMs only took a few minutes, but then I noticed that none of the virtual appliances were updating. A quick check of my relay groups showed they had no members.

I deactivated and reactivated my relays again. No change. Relay groups still empty. I deactivated, uninstalled the agent, rebooted, reinstalled the agent, activated again. No change.  Both my internal, DMZ and even the default relay group remained empty with no members. WTF?

Then I installed the relay agent on brand new servers to see if they would come up in the Default Relay Group when they were activated. Nope, nothing. Weird.

At this stage I started to panic and raised a call with Trend. While investigating the issue further, we found that on the System-Updates view the relays were being shown, but when viewing the Relay Groups in the System Settings-Updates tab, they were not showing any members.

So it looks like there was some issue with the relay groups I had created. To fix the issue I had to deactivate all relays, set all VMs and virtual appliances to the Default Relay Group so I could delete my custom relay groups, and then deactivate and reactivate the agents.

Finally the relays appeared as members in the Default Relay Group and I could re-create the Internal and DMZ Relay groups and assign the members to the correct groups to recreate my update hierarchy. Lastly my virtual appliances were assigned to the internal relay group and they were able to pick up the latest definitions.

So, you’ve been warned. Trend DSM needs admin rights all the time!

vSphere 5, vShield 5, Trend DS 8 (vBlock 300HX) Upgrade


Call this the perfect storm upgrade. If you have to perform a vSphere 5, vShield 5 and Trend DS 8 upgrade (whether or not you happen to have a vBlock 300HX), read the following for what TO do and what NOT to do!

The main caveats to remember when performing this upgrade are:

  • vShield Endpoint v3.x and vShield Endpoint v5.x are NOT compatible.
  • You cannot upgrade to the latest VMware Tools if you have the old endpoint thin agent installed on your Windows VMs. It has to be removed first.

Your final approach will depend on whether you are upgrading your hosts with VUM or rebuilding them withvia ISO. I took the ISO route as I thought it would be cleaner.

Before we get started, there is some documentation you should read:

  1. vSphere 5 Upgrade Guide including vCenter, ESXi
  2.  vShield 5 Quick Start guide
  3. Trend Manager 8 Getting Started Guide

Step-by-Step Deployment Guide:

I’ll tell you what you should do to avoid the pain and suffering I went through. If you prefer testing the upgrade on a single host to ensure the process works, update accordingly. It will still work.

  1. Upgrade Trend Manager to v8
  2. Power of all your VMs except Trend appliances.
  3. De-activate your Trend Appliances from Trend Manager
    • You should see the Trend service account in Virtual Center updating the configuration (.vmx) files of all your VMs.
    • Confirm all VFILE line entries have been removed from the VMs .vmx files before continuing
  4. Power off and delete your Trend appliances from Virtual Center
  5. Put all hosts into Maintenance mode.
  6. Remove Virtual Center from Trend Manager.
  7. Login and un-register vShield Manager 4.1 from Virtual Center
    • Power off vShield Manager 4.1
  8. Disconnect and remove all hosts from cluster
  9. Upgrade Virtual Center to v5
    • If any your hosts are disconnected during the upgrade, just reconnect them.
  10. Upgrade VMware Update Manager to v5
  11. Deploy vShield Manager v5
  12. Register vShield Manager v5 with Virtual Center
  13. Rebuild hosts manually with vanilla ISO
    • Setup management IP address on each host
  14. Add hosts back into the cluster
  15. Patch hosts with VUM and apply any host profiles
  16. Add hosts back to the 1000V if present
    • Setup all vDS virtual adapters
  17. Add virtual center back into the Trend Manager
  18. Deploy vShield Endpoint v5 driver to all hosts
    • Ensure vShield Manager is reporting Endpoint is installed before continuing
  19. Deploy Trend 8 dvfilter-dsa to all hosts via Trend Manager
    • Ensure Trend Manager is reporting hosts are prepared before continuing
  20. Deploy and activate all Trend 8 virtual appliances
    • Ensure all virtual appliances are reporting as ‘vShield Endpoint: Registered’
  21. Power on your VMs
  22. Remove vShield Endpoint Thin Agent from all your Windows VMs and reboot
  23. Upgrade VMware Tools on all your VMs, ensuring vShield option is selected. Reboot required.
  24. Confirm all VMs are protected by the local virtual appliance. Anti-malware should report ‘real time’.
  25. Update all your DRS groups as all the hosts and appliances will have been removed.
If you want to upgrade, rather than rebuild, do the following between steps 3 and 4:
  1. Uninstall Trend filter (dvfilter-dsa) from all hosts
  2. Uninstall Endpoint v3 filter (epsec_vfile) from all hosts
and upgrade vShield Manager instead of deploying new version. Refer to Page 29 of the vShield Quick Start Guide.
Things to Watch Out For:
Steps 2 and 3 are crucial.
Step 2 – vShield Endpoint v3 includes a loadable kernel module (LKM) called VFILE, which loads into the kernel on a vSphere 4.1 host at boot up.  Whenever a VM is powered on, on a host running the VFILE LKM, the virtual machine’s .vmx file is updated with the following two line entries:

VFILE.globaloptions = “svmip=169.254.50.39 svmport=8888?
scsi0:0.filters = “VFILE”

vShield endpoint v5 does not do this! No VFILE LKM is loaded, no VFILE line entries are added to the .vmx files of the VMs. Therefore if you do not correctly decommission vShield Endpoint v3, your VMs will not power on, on your vSphere 5 hosts.

This is implied in the vShield 5 Quick Start guide on Page 31 under ‘Upgrading vShield Endpoint’:

2. Deactivate all Trend DSVAs. This is required to remove vShield related VFILE filter entries from the virtual machines.

What they don’t tell you above though is that all your VMs must be powered off. If you de-activate your Trend appliances while your VMs are on, well mine just had their .vmx files updated again immediately afterwards!

If you missed that step the first time around, you’ll have to manully update the .vmx file of every virtual machine to remove the vfile line entries as per KB1030463.

 Step 3 – If you don’t remove and re-add Virtual Center from Trend Manager after you have installed vShield Manager 5,  your DS virtual appliances will not register with vShield Endpoint.

Step 7 – First time I deployed vShield Manager 5 I didn’t have any issues, although I did have to re-deploy it a 2nd time as it stopped synchronising with vCenter. Unfortunately then it no longer recognised vShield Endpoint was installed and I had to rebuild all my hosts.

Besides these issues, things went relatively smoothly. Its just a matter of time.

Good Luck!