Trend Micro Deep Security and Citrix XenApp: The effect of Agentless AV on VSImax


I’ve been doing some benchmarking recently on our 2 socket 6 core 3.3GHz B200 M2’s used in our dedicated XenApp cluster (each ESXi host providing a total of 39.888GHz) to quantify the impact of AV protection on VSImax. (If you haven’t heard of LoginVSI before, it is a load testing tool for virtual desktop environments. VSImax is the maximum number of users workloads your environment can support before the user experience degrades (response times > 4 seconds) and is a great benchmark as it can be used across different platforms.)

We use Trend Micro Deep Security 9.1 in our environment providing agentless anti malware protection for our XenApp VMs. The Deep Security Virtual Appliances provides the real time scanning via the vShield Endpoint API using a custom XenApp policy that includes all the Anti Virus best practices for Citrix XenApp and Citrix PVS.

Test Summary:

  1. Testing Tool: LoginVSI 3.6 with Medium No Flash workload
  2. Citrix XenApp anti-malware policy: Real Time Scanning enabled with all the best practice directory, file and extension exclusions set as well as the recommendation to disable Network Directory Scan and only scan files on Write.
  3. Deep Security Virtual Appliance (DSVA): Deployed with the default settings: 2vCPU, 2GB RAM, no CPU reservation and a 2 GB memory reservation.

Shown below is a LoginVSI 150 user test with a medium (no Flash) workload on a single B200 M2 running 6x VMs with 4vCPU and 12GB RAM each with agentless protection disabled. The image below shows a VSImax score of 105, which is very similar to our current real user load per blade.

VSIMax with No AV

VSIMax with No AV

Shown below is the same 150 user test with a medium (No Flash) workload on a single B200 M2 running 6x VMs with 4vCPU and 12GB RAM each with agentless anti malware protection enabled. The image below shows a VSImax score of 101.

VSIMax with AV

VSIMax with AV

The impact on VSImax with Deep Security agentless protection enabled is only 4 users per blade which is only a 3.8% user penalty. Shown below is the CPU MHz usage of the DSVA during the LoginVSI test. CPU MHz peaks at 550MHz which is 1.3% of the total available MHz of the host (39888MHz).  An acceptable penalty to keep our security boys happy!

DSVA CPU MHz

DSVA CPU MHz

Advertisements

Trend Micro Deep Security 8 \ vShield Endpoint EPSEC and UNC\SMB Scanning


‘Another day another dollar’…. no no that’s not quite right…

‘Another day another Deep Security issue.’

Next up is Deep Security and UNC\SMB Scanning. This isn’t exactly Deep Security’s fault as this is another limitation of the vShield EPSEC driver.

Executables that are accessed via a SMB share will loop and need to be manually killed. This is a known limitation of the EPSEC driver as disccused in Trend Micro KB1059280.

In our case one of our applications was launching an executable from a UNC path which was crashing. We couldn’t figure out why but unmanaging the virtual machine fixed the problem.

It is relatively easy to fix this problem, but it does leave you exposed.

The EPSEC driver does not support exclusions of a particular server name, i.e. \\servername, nor can you exclude a directory on the server nor can you exclude a specific file even if you know the name. The only way to fix this problem is to exclude all UNC paths\SMB scanning, by updating your security policy and adding the exclusion ‘\\’ to your Directory exclusion list.

Even unticking ‘Scan Network Drives’ from within Trend has no effect. This has been raised as an incident and I am yet to hear back from Trend Micro.

I have been assured by VMware this will be resolved in vShield 5.1 released in Q4 2012.

New vShield Endpoint Driver available to improve Deep Security 8 performance


Thanks to http://www.joulupukki.nl/wordpress/?p=523 for alerting me to this issue.

VMware made a pre release of the new vShield Endpoint Driver (5.0.0.2 build-813867) available last week to customers who are experiencing issues with their current vShield Driver. This will be released in Q4 but if you are using an anti malware product in your virtual environment that relies on vShield Endpoint Driver I would contact VMware to get the patch.

This hotfix needs to be applied on top of vShield Endpoint Driver build 652273 which is available with the VMware tools included with ESXi 5 Express Patch 3 (build 702118).

In the words of VMware this fixes two main issues: performance issues with network files and sharing violation issues.

1. Sharing violations – It was discovered that, while you had the thin agent installed and real-time AV scanning running, if you opened a file on a network share a few times in quick succession, the 3rd or 4th attempt could result in the file being locked. This was due to the lack of caching for network files, which is the recommend AV practice, but caused this locking

2. Performance issues – This was to due to the general overhead when our thin agent called some MS filter methods.

I also found that this version fixes a BSOD issue with vsepflt.sys. More about that in my next post.

ForeFront UAG doesn’t recognise Trend Micro Deep Security 8 as compliant Anti Virus


I just noticed a new issue today with Microsoft’s ForeFront UAG and Trend Micro Deep Security.

The UAG does not recognise the Trend Micro Deep Security Agent as a compliant antivirus product  and therefore any clients using the Trend Micro Deep Security agent will not gain privileged session access to the UAG.

Interestingly enough, the UAG ForeFront Endpoint Scanner detects the Trend Firewall component.

To confirm this is from a physical desktop with the DS agent installed. The DS agent is offering anti-malware protection, not a Deep Security Virtual Appliance, so the UAG should be able to detect it.

I can understand virtual servers or desktops not being recognised there will not be way for the UAG to verify whether the client has AV services running on it.

What I have done is following the instructions here to try and customise the endpoint components detection script.

Thankfully the detection script DETECTION.VBS already has Trend Micro Office Scan so I have added a new check ‘DetectTrendMicroDeepSecurityAntiVirus’ in the script for Trend Micro Deep Security to validate whether it is installed and running but determining whether it is up to date is beyond me.

I have escalated to Trend Engineering to see if they can assist.

 

Trend DS 8 not detected in UAG Endpoint Detection

 

Trend Deep Security Warning Message ‘Machine was unprotected during move from one esx host to another’


I wanted to post some more information on this Trend DS error message – ‘Machine was unprotected during move from one esx host to another’ as it seems to come up regularly.

The description of the error message is, ‘a virtual machine was moved to an ESX that does not have an activated Deep Security Virtual Appliance.’

In essence this warning message is saying that the ESXi host you vMotioned your VM too is not currently protecting the virtual machine.

This can be because there is no virtual appliance on the target ESXi host, the Trend Virtual Appliance is not offering Anti Malware protection, is not Activated or is Offline.

This error message will not show for unactivated virtual machines — A virtual machine has to be activated to generate this error message.

There is a known bug with this error message too – even though your VM is being protected by the appliance, the error message is always reported as an Agent error. Apparently Trend are working on this.

Back to the error message: When you receive this error message, what is the next step?

Trend is a complicated beast – An appliance can have issues for a number of reasons – whether there is a fault with the appliance or one of its dependencies is what you need to figure out. It could be something as basic as the appliance dropping off the network, losing connectivity back to the DSM or to the vShield Endpoint VMkernel port, or possibly its no  longer activated (not registered as a security appliance in vShield Manager.)

If you get this warning  message, open the virtual appliance that the VM is currently residing on and first ‘Clear Warnings/Errors’  so you remove any old status\error messages and then run ‘Check Status’ to see if there are any new issues. If there are errors reported on the appliance try and resolve them by following the patented ‘Trend DS Virtual Appliance Health Check’ below.

My main bugbear with Trend is that it is too complicated and it does not report its current state accurately and concisely. When I run a Check Status I want to know exactly what is going on. It would be most useful to have a health check screen on the appliance where the health check tests I mention below in the article are run sequentially in full view for the benefit of the administrator. Issue could be highlighted immediately and it would give us confidence that the appliance and its dependencies are all configured correctly, rather than having to check all the different components individually.

For example if you check the status of your appliance and it reports back that it is Managed and Online you would expect it to be managed, online and offering anti malware protection. In my testing after I changed the vShield VMkernel IP address on my ESXi host from 169.254.1.1 to 169.254.1.2, so the appliance could not offer anti malware protection, I ran a Check Status and the virtual appliance would still report that it was managed, online and offering anti malware protection.

On the plus side when I migrated a VM to the ESXi host with the misconfigured VMkernel port, the warning message was still generated that the VM is unprotected. What this shows is this error message is symptomatic of an underlying issue with your virtual appliance or ESXi host. While the issue may not be immediately noticable because the DSM reports that all is well, you should dig deeper following the ‘Trend DS Virtual Appliance Health Check’ below.

Bottom line — You cannot fully trust the DSM when you notice this error message. The only way to verify for sure that the appliance is actually working or not would be to drop the EICAR virus on the VM to confirm whether anti malware protection is working.

‘Trend DS Virtual Appliance Health Check’:

  1. Synchronise your Virtual Center(s) in Trend DSM
  2. Confirm your credentials for VVC and vShield are uptodate
  3. Confirm filter driver is installed on ESXi host via Trend DSM
  4. Confirm vShield driver is installed on ESXi host via vShield Manager
  5. Confirm Trend Appliance is registered as Security VM with vShield Manager
  6. Confirm the appliance is in the correct VLAN
  7. Confirm the appliance network configuration is correct
  8. Confirm you can ping the Appliance from the DSM.
  9. Confirm the VMkernel IP address for vShield Endpoint is correct on ESXi host – 169.254.1.1

and if nothing works follow my last resort:

10. Deactivate and reactivate the appliance

And if that fails…. Follow the blocksandbytes ‘Triple D’ process:

11. Deactivate, Delete and Deploy the appliance.

When I’m being lazy and I know the config hasn’t changed I will Deactivate and reactivate the appliance immediately. What I find with Trend is that as long as your environment is static, Trend will continue to stay Green, but if your environment is fairly dynamic and hosts are being rebooted, VMs are being built and vMotioned, you are performing SRM fail overs and fail backs, etc. it struggles to keep up with environment changes.

Every week I have to try and figure out why virtual machines are unhappy and do not have anti-malware protection. Hopefully this will help others stay on top of Trend DS 8.

Trend DS 8 Feature #873 – 300 VMs are not protected?


Argh, well I was performing some security hardening last week. One of my tasks was to tidy up the Administrators group in vCenter.

Yes, dangerous I know and it looks like a few service accounts were without vCenter admins for a while.

My fault completely, I had no one to blame this time, but I wasn’t expecting the fall out. All my other applications were fine – VUM, SRM, vShield, VC Ops,  etc but not Trend.

I got an email alert that 300 VMs were not protected. That took me by surprise. Must be some sort of mistake, so I login to the Trend Manager, and every single virtual machine and every single appliance was unmanaged.

WTF!

Looks like Trend DSM shat itself without admin privileges. According to Trend, apparently its expected behaviour. Sounds like pretty shite expected behaviour to me!

Thankfully re-activating all virtual appliances and VMs only took a few minutes, but then I noticed that none of the virtual appliances were updating. A quick check of my relay groups showed they had no members.

I deactivated and reactivated my relays again. No change. Relay groups still empty. I deactivated, uninstalled the agent, rebooted, reinstalled the agent, activated again. No change.  Both my internal, DMZ and even the default relay group remained empty with no members. WTF?

Then I installed the relay agent on brand new servers to see if they would come up in the Default Relay Group when they were activated. Nope, nothing. Weird.

At this stage I started to panic and raised a call with Trend. While investigating the issue further, we found that on the System-Updates view the relays were being shown, but when viewing the Relay Groups in the System Settings-Updates tab, they were not showing any members.

So it looks like there was some issue with the relay groups I had created. To fix the issue I had to deactivate all relays, set all VMs and virtual appliances to the Default Relay Group so I could delete my custom relay groups, and then deactivate and reactivate the agents.

Finally the relays appeared as members in the Default Relay Group and I could re-create the Internal and DMZ Relay groups and assign the members to the correct groups to recreate my update hierarchy. Lastly my virtual appliances were assigned to the internal relay group and they were able to pick up the latest definitions.

So, you’ve been warned. Trend DSM needs admin rights all the time!

Deep Security 8 SP1 Upgrade


As you guys and girls may be aware, Trend DS 8 SP1 has been out since the 30th April.

DS 8 SP1 promises support for wildcard exclusions and also adds linux support via an agent for on-demand scanning. (no real-time scanning yet).

There is also the added benefit of fixing the HEAP_MAX_SIZE PSOD issue but still waiting confirmation on this.

We’ve been having a few ongoing issues with our Trend environment mainly due to a lack of care and attention since I installed 7.5 SP1 and upgraded to DS 8. Also Trend is not the easiest beast to get up and running correctly. A lot of this is down to the documentation. The install guide (Getting Started?) is too  simplistic and the Best Practice documentation is confidential (go figure!) so I would definitely recommend professional services if you are think about buying Trend DS. And on the plus side you get someone to blame if anything goes wrong!

I thought the release of 8 SP1 would be a good oppurtunity to get the Trend boys onsite to blow away the existing DSM + database and install DS 8.0 SP1 from scratch.

Bear in mind this was a live cluster, so we effectively split the cluster in half and kept one half on DS 8 (with all the live VMs) and the other half was upgraded to DS 8 SP1.

We deployed a new VM, installed DSM 8 SP1 on a new database, prepared the ESXi hosts and deployed the new virtual appliances. Once the infrastructure was configured, the existing virtual machines were vmotioned onto the DS 8 SP1 hosts that were managed with the new 8 SP1 DSM.

This was a little tricky as you effectively had two DSM’s in operation on a single cluster – not recommended for long! The key to managing the VMs was to change the view to sort by host, then you could easily ignore all the unmanaged VMs on half the hosts that were not prepared.

Once the VMs were vmotioned across, we waited 5 minutes for their config to update (to ensure they still didn’t think they were being protected by a DS8 appliance) and then activated them on the new DS 8 SP1 virtual appliances on the new DSM.

After all the VMs were activated we could upgrade the remaining ESXi hosts and re-enable DRS to spread the VMs back across the cluster.

All in all it was a painless upgrade with no downtime and on the plus side Trend is looking much better.

If you have been through a few iterations of  Trend DS and  you’re having issues with high maintenance, VMs being unprotected, appliances going offline, etc I recommend this approach to clear out your infrastructure and database and start off fresh.

Yes you have to reconfigure your alerting and security profiles but its a small price to pay for a healthy, stable environment.

DS 8 SP1 — well recommended!

— UPDATE 11/06/2012 —

I have had confirmation from Trend HEAP_MAX_SIZE issue has been resolved in DS 8 SP1, but for now I’ve left the HEAP_MAX_SIZE variable set on all my ESXi hosts as it is still unclear in my mind whether this setting is no longer needed.