Troubleshooting and Diagnostics in Cloud Service

As a developer working on web applications deployed on the cloud, I am always asked to make sure the applications are always up and running. Hence, it is very important to understand some of the diagnostics resources available for Azure.

Logs

The first thing that comes to my mind is reading the Event Logs and IIS Logs in the VMs. To allow this, we first need to enable Remote Desktop for all roles.

You can enable Remote Desktop for all roles before publish.

You can enable Remote Desktop for all roles before publish.

This approach should be familiar for those who are using on-premise. So, where to find the log files?

Kevin Williamson from Microsoft has listed out most commonly visited diagnostic data locations.

  • Azure Event Logs: Diagnostic output from the Windows Azure Runtime, including information such as role start/stop, startup tasks, crashes, recycles, etc.
Location of Azure Event Logs

Location of Azure Event Logs

  • Application Event Logs: Often can find w3wp.exe (a process associated with the application pool in IIS) related errors here.
Location of Application event logs.

Location of Application event logs.

  • App Agent HeartBeat Logs: Useful for determining the current state of the role within the VM. It can be used to find out the health status of the role in the past as well. These logs are generated by WindowsAzureGuestAgent.exe.
Location of App Agent Heartbeat logs.

Location of App Agent Heartbeat logs.

When I viewed the App Agent Heartbeat logs two hours after my website went down, I saw the following records in the file.

[00000011] [06/19/2015 05:59:43.37] [HEART] WindowsAzureGuestAgent Heartbeat.
[00000011] [06/19/2015 05:59:43.37] [INFO] Generating substatus from system events.
[00000011] [06/19/2015 05:59:43.37] [INFO] Substatus is Sites are being deployed. [2015-06-19T05:59:42Z]
[00000011] [06/19/2015 05:59:43.37] [INFO] Role 1234f2b2986b4e9a1234cc9268d2250c.MyWebRole_IN_0 is reporting state NotReady with sub-status Starting and details Starting role... Sites are being deployed. [2015-06-19T05:59:42Z].
[00000008] [06/19/2015 05:59:46.64] [INFO] Role 1234f2b2986b4e9a1234cc9268d2250c.MyWebRole_IN_0 has current state Unhealthy, desired state Started, and goal state execution status StartInProgress.

After my website went back online, then I found the following records which marked the role to be healthy then.

[00000011] [06/19/2015 06:00:03.70] [HEART] WindowsAzureGuestAgent Heartbeat.
[00000011] [06/19/2015 06:00:03.70] [INFO] Role 1234f2b2986b4e9a1234cc9268d2250c.MyWebRole_IN_0 is reporting state Ready.
[00000008] [06/19/2015 06:00:06.89] [INFO] Role 1234f2b2986b4e9a1234cc9268d2250c.MyWebRole_IN_0 has current state Started, desired state Started, and goal state execution status StartSucceeded.
  • App Agent Runtime Logs: If you would like to get an overview of the events happening over time to a role, you can refer to AppAgentRuntime.log in C:\Logs which logs major changes to the role without logging heartbeats.
  • IIS Logs: To determine whether a problem is caused by the website itself or comes from Azure, it is always good to check the IIS logs first.
Location of IIS logs: C:\Resources\Directory\{DeploymentID}.{Rolename}.DiagnosticStore\LogFiles\Web

Location of IIS logs: C:\Resources\Directory\{DeploymentID}.{Rolename}.DiagnosticStore\LogFiles\Web

  • HTTP.SYS Logs: Very important to troubleshoot an issue with a hosted service website not responding. The evidence of, for example, IIS not being able to process the high-volume requests can be found in HTTP.SYS logs.
HTTP.SYS logs can be found here!

HTTP.SYS logs can be found here!

Monitoring ASP .NET Web Application with Application Insights

If you would like to have a graphical dashboard with diagrams showing availability, performance, usage, failures, and other events happening on your web apps, please try out Application Insights too.

The reason I like it very much because with Visual Studio 2013, Application Insights can be added to the application with just a few clicks.

Just right click your project in Solution Explorer and choose "Add Application Insights Telemetry".

Just right click your project in Solution Explorer and choose “Add Application Insights Telemetry”.

After Application Insights is successfully added, the ApplicationInsights.config file with an instrumentation key will be added to your project.

However, by doing this, Visual Studio will not add the JavaScript SDK to the web pages of the project. If you are interested to know more about load time and user events, you can proceed to add it. After the JavaScript SDK is successfully added, you should be able to see the following lines of codes in your web pages.

    var appInsights=window.appInsights||function(config){
    function s(config){...}({instrumentationKey:"02gh149f-3ce6-47cf-b49a-6efg14d83928"});
 
     window.appInsights=appInsights;
     appInsights.trackPageView();

Now, if you right-click on the same project again, you will be able to launch the Application Insights of the project in the new Azure portal.

Application Insights on Azure.

Application Insights on Azure.

Yup, let’s start improving our web applications based on these useful insights and reports. =)

Summer 2015 Self-Learning Project

This article is part of my Self-Learning in this summer. To read the other topics in this project, please click here to visit the project overview page.

Summer Self-Learning Banner

Advertisements

One thought on “Troubleshooting and Diagnostics in Cloud Service

  1. Pingback: Summer 2015 Self-Learning | cuteprogramming

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s