Many articles can be written on Azure’s App Insights. This Application Performance Management (APM) service can be used for all aspects of KPI monitoring and dashboards of your servers.
Through the usage of analytics and machine learning, you can easily and quickly gain insights into what’s going on both on your back-end and front-end applications.
We at Kensee are using App Insights for the following tasks:
- Logging and diagnostics infrastructure
- Alerts on exceptions and failures in our services
- Monitoring of system resources (CPU, Memory, etc.)
- Usage of our platform (page views, clicks, performance)
In this 2-parts post, I’ll focus on the (long) road for using the custom events feature of App Insights for gaining insights on everything our users are doing in our platform – it can be seen as some sort of NewRelic replacement.
The first part will show my initial approach of using Power BI and why it didn’t work.
The second part will cover a better alternative that is more flexible and actually covers all our needs.
Step 1 – Creating the events
Connect to App Insights
Connecting App Insights to most types of applications is very straightforward.
For .NET applications such as ASP.Net, Web API or MVC, just include AppInsights nuget package in your project and specify the instrumentation key from code, or follow this detailed explanation on how to do it automatically from Visual Studio
For client side applications such as Angular 2, include a js snippet for the App Insights SDK. You can get it from the portal, together with the relevant instrumentation key following the steps described in this article – Application Insights for web pages.
A full description of the API can be found here.
There is also support for using App Insights from PHP, Ruby, Python, Node.js and even WordPress.
Sending events to App Insights
Sending events is easy as well.
For .NET, initialize an instance of
TelemetryClient and use any of the Track methods to log events of different types (such as exceptions, page views, metrics, traces, etc.)
TrackTrace for all our logging messages and pass it the relevant severity level and a dictionary for message related parameters that can later be used for querying.
TelemetryClient object and call the different track methods.
From our angular app, we call
trackEvent for logging any action the user does on our platform. These custom events can then be used to analyze user behavior in a very detailed way and this will be the focus of our next steps.
AppInsights also provides a built-in support for user identification by user id and account id – Call setAuthenticatedUserContext method and pass your own unique ids to be able to filter by them later.
Note that AppInsights keeps events in a local in-memory buffer until it decides to actually send them to the remote App Insights server. The decision is made by the number of events and timeframes. If you want to see the events sooner, call the Flush() method.
Step 2 – Analysis and Visualization of App Insights events
Now that our applications send events to App Insights we want to see actually monitor them, preferably in a visual and easy to use way.
App Insights Custom Events from Azure Portal
App Insights provides basic out-of-the-box support for custom events.
From Azure Portal, go to the Application Insights service and choose the relevant service from the list of services. Click on the Usage options and you’ll various information on your application. If you published custom events, you’ll be able to see a summary of them in the Custom events Tile below the Usage timeline graph. Clicking on each custom event from the list will open a blade with the list of all these events. Clicking on a specific event will show all its properties, including the custom fields added while tracking the event, as well as the user and account id if the authenticated user context method was set.
Here’s an image of some our custom events and usage statistics:
Analyzing events using Application Insights Analytics
Although the portal provides easy access to a general overview of usage and events statistics and some basic filtering options, in order to analyze the data more seriously a better tool is needed.
For that exact purpose, the Application Insights Analytics tool was created. It provides a rich query language with plenty of examples and tutorials. It also allows visualizing the results in a chart view.
Here a pie chart of custom events types from our development environment:
And the simple code to generate the above pie chart is:
customEvents | where timestamp > ago(7d) | project timestamp, name | summarize count() by name | render piechart
Visualizing custom events using Power BI
Now it is starting to get interesting. Analytics is a great tool but requires some expertise in preparing and accessing the queries and results. What if you want your CEO or product team to see a customized report of the users’ actions you meticulously sent to App Insights from the platform?
Applications Insights provides three recommended ways for exporting data to Power BI.
We’ll cover the first two options in this step and my own custom version of the third one in the next post.
Power BI adapter provides a pre-build package for using data from App Insights in Power BI. Just add the package and connect it with your service so see various reports prepared for you by the Power BI team. It includes pages for Users data, Sessions, and Page Views and can provide useful high-level information. It doesn’t have support for viewing custom events or using your own queries.
Export Analytics queries allow you to use any query you want using the Analytics tool, and export the results to Power BI. The process to achieve that is complicated and requires the usage of several tools. On each change in the query, the whole process must be repeated.
Here’s the process in short:
- Download and install Power BI Desktop.
- Prepare the query in Analytics. Make sure it includes all required columns and decide whether aggregations or projections of data are done at this stage or later in Power BI.
- Export the query as Power BI Query (M).
- From Power BI Desktop, create a blank query and copy the content of the query file into the Advanced Editor window.
- Publish the Power BI report to your Power BI cloud workspace. This will create or update a new data source in Power BI. If you want the data to be periodically refreshed you’ll need to set it yourself from the data source settings.
- Create a Dashboard from the report. Any change to the data source or report requires that you remove the old tile and add it again to the dashboard.
- Share the dashboard with your team. Note that they’ll need a Power BI license.
A few notes and issues I discovered with this process:
- App Insights data is saved only for 7 days. For longer periods you will need to set-up Continous Export (more on that in the next post).
- The Power BI report can’t combine different types of data sources. If you want to include additional data from SQL for example, it can’t be done.
- Certain formatting and sortings that are done in Power BI Desktop are lost while publishing the report.
- There are known issues with dates formats, that require some workarounds.
In this post, I’ve covered the basics steps required to start using App Insights in your application and provide you with a way to track down backend logs as well as users actions from your web platform.
In the next post, I’ll cover more advanced settings of using App Insights for a robust solution that is appropriate for a production environment.