E2EVC Rome 2016 – Part 2 (Microsoft) – Troubleshoot XenApp With Style

Hi and welcome to this second part about troubleshooting Citrix XenApp.

Last November I had the honor to speak at two well-known EUC and Citrix events. First of all at E2EVC in Rome, Italy, and the day after at DCUG TecCon in Kassel, Germany. The topic of both sessions was “Troubleshoot XenApp with Style”.

I am writing this blog for all of you that couldn´t attend one of these great conferences. The third part will cover general troubleshooting tools that didn´t fit in the session and parts 1 and 2 of the blog series.

The first part is about the Citrix tools you can use.

Why Should You Read It?

This post is not about the usual problems you might have with Certificates, Firewalls, or similar things where you might be able to find an answer by googling around or checking the knowledge base.

It is more about the methodology and the tools to use when you are not able to find anything on the whole world wide web. Check the first part to get an idea of what questions to ask and how to start the troubleshooting process.

Microsoft Tools

Let´s have a look at some useful tools from Microsoft that you can (and should) use when troubleshooting Citrix XenApp issues. Some of them need to be used in combination with the Citrix troubleshooting tools because otherwise you won´t have useful results or you might not be able to read and analyze the information, others can be used themselves. Always keep in mind the more information you get from different sources the better you will understand the issue.

Let´s go…

SvcTraceViewer.exe (from Windows 10 SDK)

The Services Trace Viewer is a very handy tool if you need to analyze Citrix StoreFront verbose logs. The general recommendation for the creation of StoreFront verbose logs can be found here. Typically you would zip the files and send the logs to Citrix support if you wouldn´t troubleshoot on your own. But because we are trying to get the issue on our own we need SvcTraceViewer.exe.

In order to get the tool you need to install the Windows 10 SDK. The download link for that is https://developer.microsoft.com/en-US/windows/downloads/windows-10-sdk.

The Trace Viewer itself can be found after installation at this location:

C:\Program Files (x86)\Microsoft SDKs\Windows\v10.0A\bin\NETFX 4.6.2 Tools

Now you are able to read the logs and you can try to find the issue on your own.

viewer.png

If you need to dig deeper you can open Sysinternals DebugView during tracing to get more information. As I said this is typically asked from Citrix Support to get a better picture of the problem. The moment you start DebugView it will collect debug output from the system it is running on.

Use SVCTraceViewer to check StoreFront Verbose log files.

Steps Recorder

Another simple but very effective tool is the Steps Recorder. It comes preinstalled in every Windows version and can be found in the start menu. I prefer to use it every time a user needs to explain or clarify an issue. Often users are not able to express perfectly what the problem is and the steps recorder gives you the opportunity to get an idea of the problem they might have.

Let the user open Steps Recorder.

steps1.png

By hitting “Start Record” every single click a user does is recorded by a screenshot and the date and time it happens along with the application name the user is acting with. This is perfect to match this data to event logs and similar things you might look into in order to resolve an issue.

steps2.png

When the user finishes he simply hits “Stop Record” and he´s able to add some extra comments. In the end, he can send it to you for further assistance.

Use Steps Recorder for situations where a user is not able to explain the issue or if you want additional information about the environment.

Process Explorer

This is the second most used tool in troubleshooting scenarios. It is the perfect tool to monitor and watch all of the processes on a Windows system or for a specific user or program.

You can see very clearly what kind of child processes are being created and if you enable the columns with User Name, Process ID, and the corresponding command line you have a good overview of what is going on in a session or during interaction with the system.

If you like you can replace the Task Manager in your system with Process Explorer. I really like this feature for my home-lab servers or systems in development environments.

sys1.png

If you want to know more about a process you can show additional information like Performance, Performance Graph, GPU Graph, Threads, or the Environment variables. The environment variables tab is sometimes helpful when troubleshooting scripts that set and use variables and you are unsure if they have the correct value.

sys2.png

Replace Task Manager with Process Explorer to get more information about running processes on your systems.

If you want to replace Task Manager with Process Explorer there is a very simple and effective way to use PowerShell:


New-Item -path ‘HKLM:\SOFTWARE\WOW6432Node\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\taskmgr.exe’
Set-ItemProperty -path ‘HKLM:\SOFTWARE\WOW6432Node\Microsoft\Windows NT\CurrentVersion\Image File Execution Options\taskmgr.exe’ -name “Debugger” -value “””C:\Program Files\Tools\SysinternalsSuite\procexp.exe”” /ACCEPTEULA”


Process Monitor

Process Monitor is one of the most famous (if not the most famous) tools out there. This is really the tool where you should be able to track down most of the issues you can have in a XenApp environment.

You can filter for registry activity, file activity, network activity, and process and thread activity.

What I personally like is the ability to show the “relative time” column. Especially when you are analyzing slow logons it shows you the time from the beginning of the trace. That makes it easy to keep track of the time a process needs to run.

Additionally, you can show columns like Architecture, Comand Line, and User and Session.

If you are searching for files and registry keys that you need to tweak with your UEM solution, this is the tool to go because you will be able to find the files, registry keys, and entries you need to set for a specific application.

proc.png

The following example shows you the creation of a shortcut through SelfService.exe from Citrix Receiver.

proc1.png

You should be aware of the fact that ProcMon can use a lot of system resources when it is running and sometimes, when you are trying to analyze timing issues during logon you will see that they are gone during analysis because of Procmon slowing down the system.

This tool can have a massive impact on the performance of the system you are starting it. Therefore don´t let it run without having a look into it, and always have a look at the amount of data you are gathering.

ProcMon is creating a lot of data…really a lot!

Netstat and TCPView

If you need to check communication from a host to specific systems and ports Netstat is a simple tool you can start from the command line.

netstat.png

If you also want to know which Process is creating connections from your host to another and the specific port it is using try TCPview from the Sysinternals Suite.

tcp1.png

TCPview will show you the process behind a connection.

BGInfo

I know that this is a no-brainer but it is simple and very effective to get some information from the system you are logging on to. You can specify and extend it with custom PowerShell scripts. The next picture shows you an example where I added Fields for the PowerShell, Citrix Receiver, App-V, and Internet Explorer Version.

The integrated PowerShell scripts simply gather the version information from the registry.

bginfo.png

If you are out of budget or need a very simple solution to get the information about the users that are logging on the system you can create a text file through BGinfo that documents these logons.

bginfo1.png

The log file creates entries with a variety of information you can import to Excel to create filters and search for specific users. This is an additional source to gather information to use in combination with other troubleshooting sources.

bginfo2.png

BGinfo – Simple but often effective.

If you want to start BGinfo in every session on a Workstation or Server there is a very simple and effective way to using PowerShell:


Set-ItemProperty -path ‘HKLM:\SOFTWARE\Microsoft\Windows\CurrentVersion\Run’ -name “BgInfo” -value  “””C:\Program Files\Tools\SysinternalsSuite\Bginfo.exe”” “”C:\Program Files\Tools\SysinternalsSuite\demolab.bgi”” /TIMER:0 /NOLICPROMPT”


The command tells BGinfo to load a custom bgi file that I created before. The files need to be present on the system. I wouldn´t recommend placing the files on a file share because this might have an impact on your logon time.

Also, take care of the fact that this BGinfo background can grow large depending on the size of the screen and a potential background image you might be using. Such a file can easily grow 3 MB and more and will reside in the user profile.

CMTrace

This tool is part of the System Center Configuration Manager Tools. If you work with Configuration Manager you know how challenging log files can be (Configuration Manager has more than 70 different logs). The cool thing about CMTrace is that it is highlighting lines with errors in red, and lines with warnings in yellow. This makes checking logfiles a much simpler task because you are quicker reading and searching through them.

cmtrace.png

You can download the System Center 2012 R2 Configuration Manager Toolkit here.

Use CMTrace to quickly find errors and warnings in log files.

rsop.msc

Group Policies or better said the configuration of those often creates problems we need to check. In big companies usually, more then one team is responsible for Group Policies and other teams might screw your environment without even knowing or noticing it.

In such cases, it might be suitable to take a look at the Group Policies that affect a user or a system. The cool thing about rsop.msc is that it will show you the configured settings and the Group Policy name it is coming from.

The user himself can start rsop.msc (if he is able to start a cmd or PowerShell console). This way you could have a quick look shadowing a user and getting some insights into his configuration.

rsop.png

As administrator, you get a lot more information.

rsop1.png

This will help you find the Group Policy that is changing a specific configured setting as last. But this won´t tell you how many times a setting might have changed before back and forth. This might be something you should additionally consider having a look at. Many policies create many problems

rsop.msc shows you a configured setting and the name of the policy it belongs to and it saved my ass more than once.

UserEnvDebugLogging

Sometimes you need to dive deeper into the login process of a user. This is the moment you should consider enabling UserEnvDegugLogging on a system.

Be aware of the fact that using ProcessMonitor in combination with UserEnvDebugLogging can have a huge impact on the performance of a system and on the log file UserEnvDebugLogging is creating.

Also, be aware of the fact that it won´t solve all of your problems. I had situations where I tried to get a picture of a slow logon but I couldn´t find any information regarding the cause of the issue.

In such a case it makes sense to have a look at Process Monitor which will show you more information.

userenvdebug.png

Enable logging by setting the above Registry Key. Logging will immediately start at the following location…

If we look into it we will see a lot of information in it. I won´t explain it in detail because there are other very good articles out there.

Do not use UserEnvDebugLogging with Process Monitor at the same time.

This is it for the second part about troubleshooting Citrix XenApp. I hope you found something useful for your next issue.

Always remember – it needs a lot of practice and experience to find some of the issues you might run into and you need to decide if it makes sense in means of the time to find the issue and the impact it might have on your business if you are contacting Citrix directly or not.

Cheers,
Sinisa

By:

Posted in: