• Home
  • About Us
  • Contact Us
  • Privacy Policy
  • Special Offers
Business Intelligence Info
  • Business Intelligence
    • BI News and Info
    • Big Data
    • Mobile and Cloud
    • Self-Service BI
  • CRM
    • CRM News and Info
    • InfusionSoft
    • Microsoft Dynamics CRM
    • NetSuite
    • OnContact
    • Salesforce
    • Workbooks
  • Data Mining
    • Pentaho
    • Sisense
    • Tableau
    • TIBCO Spotfire
  • Data Warehousing
    • DWH News and Info
    • IBM DB2
    • Microsoft SQL Server
    • Oracle
    • Teradata
  • Predictive Analytics
    • FICO
    • KNIME
    • Mathematica
    • Matlab
    • Minitab
    • RapidMiner
    • Revolution
    • SAP
    • SAS/SPSS
  • Humor

Test Automation and EasyRepro: 04 – Monitoring and Insights with EasyRepro

November 9, 2019   Microsoft Dynamics CRM

EasyRepro is an open source framework built upon Selenium allowing automated UI tests to be performed on a specific Dynamics 365 organization. In this article we will discuss monitoring execution timings and delivering these to Azure Application Insights. We will look into the various timings within an execution, how to add tracing statements and finally to push events to Application Insights.

Getting Started

If you haven’t already, please review the previous articles showing how to create, debug and extend EasyRepro. This article assumes that the reader has this knowledge and will focus specifically on monitoring.

Command Timings

Commands in EasyRepro unit tests represent an interaction with the browser to perform an action. Each command has a name, a start time, end time and other properties such as how many times it was attempted and was it successful. What’s returned to the unit test is a list of command results represented by the ICommandResult object.

 Test Automation and EasyRepro: 04   Monitoring and Insights with EasyRepro

Going to the definition for ICommandResult we see the following properties:

 Test Automation and EasyRepro: 04   Monitoring and Insights with EasyRepro

These properties represent pretty much everything we need to know about a command result. However since this is a interface, it can be extended to include any other metric needed.

Now that we have the definition we need to dig deeper to better understand how these properties are populated. But before we get into that let’s review the object that implements the ICommandResult interface: the BrowserCommandResult class.

Exploring the BrowserCommandResult class

 Test Automation and EasyRepro: 04   Monitoring and Insights with EasyRepro

The image above shows the constructor for the BrowserCommandResult class and three important attributes that represent collecting the start and stop time stamp as well as the value returned.

Exploring the BrowserCommand.cs file

The BrowserCommand object is where the actual command gets executed. It contains options describing tracing, retry policies, exception policy, etc. What’s most important is the Execute method which performs the action from the unit test.

 Test Automation and EasyRepro: 04   Monitoring and Insights with EasyRepro

When a command from a unit test is run (i.e. GetValue), it ultimately comes to this method to be performed. In the image above you can see the retry attempts are set as well as the tracing and timing (result.Start()). Once the command has finished the result and timings are returned in the BrowserCommandResult object.

Execution Times

The execution time for a command represents how long the command took to execute as shown in the Execute method in the BrowserCommand class result.Start and result.Stop methods. Since we are focusing on timings these two methods are key. They are used to calculate the execution time we will use to monitor the performance of our unit test. These methods are shown in the BrowserCommandResult image above and return a simple timestamp.

 Test Automation and EasyRepro: 04   Monitoring and Insights with EasyRepro

The execution time calculation takes the StopTime property and substracts from the StartTime to provide a number in milliseconds. This number is then set to the return value of our unit test command.

 Test Automation and EasyRepro: 04   Monitoring and Insights with EasyRepro

Think and Transition Times

You may have noticed above a couple of values for Think and Transition Times.

 Test Automation and EasyRepro: 04   Monitoring and Insights with EasyRepro

Think times are built in mechanisms to put a thread to sleep. They can be used to allow actions to complete or represent a user wait in the UI. In the image above we see that the ThinkTime value is set to 2000 which is because inside of the OpenApp method the Browser.ThinkTime method is run.

The Transition time value represents the time from the previous command end to the current command’s start time and removing think times in between.

 Test Automation and EasyRepro: 04   Monitoring and Insights with EasyRepro

How is all of this calculated?

 Test Automation and EasyRepro: 04   Monitoring and Insights with EasyRepro

Inside of the InteractiveBrowser class is a method called CalculateResults which is called from the Execute method described above. In the image above you can see think time is straight forward, it comes from the CommandThinkTimes property.

The transition time is a calculation adds the seconds and millisecond differences between the previous command and current command then subtracts the think time. All three of these timings can help us better understand where long running commands are coming from and where we could possibly tune unit tests by reducing think times between unit test commands.

Logging and Monitoring

Debug and Trace

The ability to provide output statements is essentially for any type of unit testing including ones built using the EasyRepro framework. Typically you see statements such as Console.WriteLine or Debug.WriteLine used in the unit test link in the image below.

 Test Automation and EasyRepro: 04   Monitoring and Insights with EasyRepro

Its important to note that these same lines can be added to the EasyRepro framework if you choose. Here are two images showing the output from the Debug window and the output from the test result itself.

Debug window:

 Test Automation and EasyRepro: 04   Monitoring and Insights with EasyRepro

Test Result Window:

 Test Automation and EasyRepro: 04   Monitoring and Insights with EasyRepro

You can see the test output shows only the WriteLine methods I added while removing the debug information from Visual Studio. Utilizing this technique is key to providing detailed information to the developer and testing teams to understand how a unit test is performing. This output will also be included in the test results file which will help us once we begin automating unit testing.

Application Insights

Application Insights is an extensible Application Performance Management (APM) service that can be used to monitor applications, tests, etc. Source: Application Insights Overview

In this section we will cover how to setup the Telemetry Client to be used by your unit tests and how to push command timings to Application Insights. Its assumed you have already setup an Application Insights resource and have the instrumentation key. For help on setting this up refer to this article detailing enabling and configuring Application Insights.

Setup Application Insights Telemetry Client

Begin by ensuring you have the Microsoft.ApplicationInsights NuGet package installed. As of this writing I’m using the v2.4.0 which is included in the Microsoft.Dynamics365.UIAutomation.Sample unit test project from GitHub.

 Test Automation and EasyRepro: 04   Monitoring and Insights with EasyRepro

Next, navigate to your app.settings file and add the Instrumentation Key to the AzureKey key.

 Test Automation and EasyRepro: 04   Monitoring and Insights with EasyRepro

Once that’s done we need to setup the TelemetryClient and give it our key. Once complete we are now ready to send messages to Application Insights.

 Test Automation and EasyRepro: 04   Monitoring and Insights with EasyRepro

Push Timings to Application Insights

Once the unit test has completed all of the EasyRepro actions we are going to send the CommandResults collection to Application Insights. From the previous section we detailed that each command result includes the Start and End times as well as calculated timing such as execution, think and transition times.

 Test Automation and EasyRepro: 04   Monitoring and Insights with EasyRepro

In the image above you can see how to loop through each command result and set the property and metric bags for the CustomEvent table in Application Insights. The telemetry client includes methods for each type of table in Application Insights so you can also include items like exceptions, navigation timings, etc in the appropriate table. Review this document for additional information.

Next Steps

Expanding messages sent to Application Insights

The example above shows how to collect and send timings for each command to Application Insights which is useful to understand how the test and platform are performing. In addition you may want to think about adding additional contextual details about each unit test run such as who is running the test, where its running and information about the org that the test is running on. As you can see the property and metric bags are dictionaries that can be added to to include these detail points. There are also table columns native to each specific table that can be leveraged as well to house these data points.

One area of the platform that gets overlooked is the Performance Center which is extremely useful for understanding page load timings. These markers are accurate representations of a page load. They can be extracted and sent to Application Insights as well. Check out the SharedTestUploadTelemetry sample unit test for more information. This test uses the legacy UI however EasyRepro can be extended to utilize the Performance Center in the unified interface as well. Please refer to the previous section for extending EasyRepro for techniques to accommodate this.

Conclusion

In this article we have explored EasyRepro commands and how the timings for each command are gathered. We discussed logging techniques within our unit tests. We setup Application Insights within an EasyRepro unit test and pushed messages from our test. Finally we discussed adding additional data points to our messages.

Please refer to the the previous articles for extending and debugging EasyRepro unit tests. For the next article we will work to incorporate changes to EasyRepro unit tests into an Azure Build Pipeline using Continuous Integration. This will be our first step into adding EasyRepro as a quality gate for our DevOps process!

Previous Posts:

Test Automation and EasyRepro: 01 – Overview and Getting Started

Test Automation and EasyRepro: 02 – Designing and Debugging Unit Tests

Test Automation and EasyRepro: 03 – Extending the EasyRepro Framework

Let’s block ads! (Why?)

Dynamics 365 Customer Engagement in the Field

Automation, EasyRepro, Insights, monitoring, Test
  • Recent Posts

    • ANOTHER SIMPLE EXAMPLE OF FASCIST NAZI LEFTISTS AT WORK
    • Nvidia and Harvard develop AI tool that speeds up genome analysis
    • Export with large E instead of small e
    • You’ll be back
    • Building AI for the Global South
  • Categories

  • Archives

    • March 2021
    • February 2021
    • January 2021
    • December 2020
    • November 2020
    • October 2020
    • September 2020
    • August 2020
    • July 2020
    • June 2020
    • May 2020
    • April 2020
    • March 2020
    • February 2020
    • January 2020
    • December 2019
    • November 2019
    • October 2019
    • September 2019
    • August 2019
    • July 2019
    • June 2019
    • May 2019
    • April 2019
    • March 2019
    • February 2019
    • January 2019
    • December 2018
    • November 2018
    • October 2018
    • September 2018
    • August 2018
    • July 2018
    • June 2018
    • May 2018
    • April 2018
    • March 2018
    • February 2018
    • January 2018
    • December 2017
    • November 2017
    • October 2017
    • September 2017
    • August 2017
    • July 2017
    • June 2017
    • May 2017
    • April 2017
    • March 2017
    • February 2017
    • January 2017
    • December 2016
    • November 2016
    • October 2016
    • September 2016
    • August 2016
    • July 2016
    • June 2016
    • May 2016
    • April 2016
    • March 2016
    • February 2016
    • January 2016
    • December 2015
    • November 2015
    • October 2015
    • September 2015
    • August 2015
    • July 2015
    • June 2015
    • May 2015
    • April 2015
    • March 2015
    • February 2015
    • January 2015
    • December 2014
    • November 2014
© 2021 Business Intelligence Info
Power BI Training | G Com Solutions Limited