QualityClouds Operational Scans

Operational scans complement the Quality Clouds for ServiceNow Quality and Governance scans, providing insights and alerts about the run time behaviour of SaaS instances. They are primarily oriented towards running against production environments, but they can be configured against sub-production environments as well. 

The below table summarises the differences in approach between the Quality and Governance Scans and the Operational Scans.


Quality & GovernanceOperational
Main IntentMeasure the Quality and Govern Changes on your pre-production environments. Gain insights about the run time behaviour of the instance, and about user adoption profile.
Suggested EnvironmentAny pre-production / ProductionProduction
Suggested FrequencyWeeklyDaily
Running Time HoursMinutes
Source data analysedCode and Configuration, transaction logTransaction log, events log, user profiles 
CustomisationCustom rules to raise ad-hoc issue types

Configure thresholds for alert events

Custom operational metrics on request


Metrics and Alerts

The Quality Clouds operational scans will capture the value of a number of metrics on each scan. Some of the metrics are broken down by one or more dimensions (for instance, users per role and users per department). Also, metrics, and their derived alerts, can be classified in one of three categories:

  • User adoption
  • Error detection
  • Performance
  • Resource usage

The below table lists the available metrics for each platform:

PlatformMetric NameCategoryDescriptionDimensionTime Resolution(*)
ServiceNow













Active UsersUser adoptionNumber of active users (users with the active flag set to true, regardless of their login activity)N/AT + 0

Active Users per Department

User adoptionNumber of active users, per departmentDepartmentT + 0
Active Users per RoleUser adoptionNumber of active users with the roles admin, fulfiller, requesterRoleT + 0

Average Scheduled Job Time

PerformanceThe average execution time, in seconds, for all scheduled jobs in the dayN/AT + 1

Average Transaction Time

PerformanceThe average execution time, in seconds, for all transactions in the dayN/AT + 1

Avg Email Reader Scheduled Job Duration

PerformanceThe average execution time, in seconds, for the email reader scheduled job in the dayN/AT + 1

Failed import Scheduled Jobs

Error detectionThe number of failed import jobs in the dayN/AT + 1

Inactive Users

User adoptionNumber of users with the active flag set to falseN/AT + 0

Logins per day per department

User adoptionThe total number of user logins in the day, per departmentDepartmentT + 1

Logins per day per Role

User adoptionThe total number of user logins in the day, per the roles admin, fulfiller, requesterRoleT + 1

Logins per Day

User adoptionThe total number of user logins in the dayN/AT + 1

Max Email Reader Scheduled Job Duration

PerformanceThe maximum execution time, in seconds, for the email reader scheduled job in the dayN/AT + 1

Maximum Scheduled Job Time In Seconds

PerformanceThe maximum execution time, in seconds, for all scheduled jobs in the dayN/AT + 1

Maximum Transaction Time In Seconds

PerformanceThe maximum execution time, in seconds, for all transactions in the dayN/AT + 1

Never Logged Users

User adoptionThe number of active users who have never logged in (requires access to the lastlogin field on the users table)N/AT + 0

Slow Homepage Load

PerformanceThe number of times that the homepage took over 10 seconds to load in the dayN/AT + 1

Unique users logged per day per department

User adoptionThe total number of unique users which logged in in the day, per departmentDepartmentT + 1

Unique users logged per day per role

User adoptionThe total number of unique users which logged in in the day, per the roles admin, fulfiller, requesterRoleT + 1

Unique users logged per day

User adoptionThe total number of unique users which logged in in the dayN/AT + 1
Salesforce





Active usersUser adoptionNumber of active users (users with the active flag set to true, regardless of their login activity)N/AT + 0


Active users per department

User adoptionNumber of active users, per departmentDepartmentT + 0


Active users per division

User adoptionNumber of active users, per divisionDivisionT + 0


Active users per manager

User adoptionNumber of active users, per managerManagerT + 0

Active users per profileUser adoptionNumber of active users, per profileProfileT + 0


Active users with logins in last 14 days

User adoptionNumber of users with a login in the last 14 daysN/AT + 0


Active users without logins in last 14 days

User adoptionNumber of users without a login in the last 14 daysN/AT + 0


API calls done in the last 24 hours

Resource usage

Number of API calls made in the previous 24 hour intervalN/AT + 0


Inactive users per department

User adoptionNumber of inactive users (users with the active flag set to false, regardless of their login activity), per departmentDepartmentT + 0


Inactive users per division

User adoptionNumber of inactive users (users with the active flag set to false, regardless of their login activity), per divisionDivisionT + 0


Inactive users per manager

User adoptionNumber of inactive users (users with the active flag set to false, regardless of their login activity), per managerManagerT + 0


Inactive users per profile

User adoptionNumber of inactive users (users with the active flag set to false, regardless of their login activity), per profileProfileT + 0


Licenses by Type

User adoptionNumber of available licenses, per typeLicense TypeT + 0


Logins per API type

User adoptionNumber of logins, per API type (SOAP Enterprise, SOAP Partner, N/A)API TypeT + 1


Logins per application

User adoptionNumber of logins, per application (Browser, Workbench, Trailhead Connected App, etc..)ApplicationT + 1


Logins per browser

User adoptionNumber of logins, per browserBrowserT + 1


Logins per country

User adoptionNumber of logins, per countryCountryT + 1


Logins per department

User adoptionNumber of logins, per departmentDepartmentT + 1


Logins per login type

User adoptionNumber of logins, per login type (Partner Product, Application, Remote Access 2.0...)Login TypeT + 1


Maximum capacity of data (MB)

Resource usageMaximum data storage capacityN/AT + 0


Maximum capacity of files (MB)

Resource usageMaximum files storage capacityN/AT + 0


Never logged users

User adoptionNumber of users who have never logged inN/AT + 0


Percentage of storage capacity free

Resource usagePercentage of free data storage capacityN/AT + 0


Percentage of type of files in File Storage

Resource usagePercentage of usage of file storage, per file typeFile TypeT + 0


Percentage of used licenses

User adoptionPercentage of used licenses, per license typeLicense TypeT + 0


Total API calls available per day

Resource usageMaximum number of API calls available per day N/AT + 0


Total logins per day

User adoptionThe total number of user logins in the dayN/AT + 1


Unique users logged per day

User adoptionThe total number of unique users which logged in in the dayN/AT + 1


Used  capacity of files (MB)

Resource usageThe total amount of used file storage, in MBN/AT + 0


Used capacity of data (MB)

Resource usageThe total amount of used data storage, in MBN/AT + 0


Users with department assigned

User adoptionThe number of users with the department attribute filled inN/AT + 0


Users with division assigned

User adoptionThe number of users with the division attribute filled inN/AT + 0


Users with manager assigned

User adoptionThe number of users with the manager attribute filled inN/AT + 0

Alerts

Alerts allow you to receive notification emails whenever the value of any operational metric does not match what you expect.  

Alerts can be set against any of the operational metrics described above. To configure an alert, the following information needs to be provided through an email to help@qualityclouds.com

  • Metric name for which the alert will apply
  •  Whether the alert will compare the metric value against a single threshold or against a range of values (comparisons always include the value specified - i.e. are done with equals of greater / lesser than)
    • If a threshold, the value of the threshold, and whether the alert should be raised if the actual value is above or below the threshold
    • If a range, the lower and upper bounds of the range, and whether the alert should be raised is the actual value is within or outside the threshold

Times and Time Zones

From a temporal dimension point of view, there are two kinds of operational metrics.

  1. Metrics which represent the state of a given value at a point in time (for instance, the number of active users)
  2. Metrics which represent the aggregate value of a given metric for a defined time interval (typically, one day - for instance, number of logins in a day)

Metrics in the first group are captured with a T + 0 time resolution. That is, metric values for day T will be available on that same day (which will typically be the day on which the operational scan is executed). If two operational scans are executed on the same day, the value for these metrics may change, with the latest value overriding the previous one.

Metrics in the second group are captured with a T+1 time resolution. That is, metric values for day T will be available on T+1. If two operational scans are executed on the same day, the value for these metrics will not change.

Dates always refer to the time zone configured in the ServiceNow instance. Note that if there is no time zone configured in the instance (by setting the propery glide.sys.default.tz), then the dates refer to UTC time, which is how ServiceNow stores timestamp values internally (see this link for details).