New Update Set and Application LiveCheck (from v9.6)

Introduction and background

When LiveCheck for Update Sets was first implemented, the infrastructure already in place to perform LiveChecks of individual Configuration Elements was reused.

This meant that one request was made for each Configuration Element included in the Update Set, with the last request including all the Update Set metadata.

This has proved problematic for very large update sets, even with the workarounds introduced to limit the types of CEs sent (skipping those for which no rules are available in Quality Clouds) and to make the Asynchronous execution of Update Sets mandatory based on a configurable Update Set size limit.

These limitations, and the fact that more and more customers are relying on being able to consistently scan Update Sets of any size when they are closed, in order to validate the Quality Gates threshold, made it necessary to re-think the whole architecture and process around Update Set scans.

New Approach

The new approach is based on the same architecture which is used to execute a full scan of the instance, but with the scope reduced to the Update Set or Application in question. These diagram below illustrates the steps involved, and the components which interact in each step.

A description of each step is below:

1. The Quality Clouds ServiceNow application (from version 9.6 onwards) sends a message to trigger the scan of an update set an application. The only parameters sent with the request are:
   1. Application or Update Set Sysid
   2. API key of the user triggering the LiveCheck (see API Key Management for the changes regarding API key management from version 9.6)
   3. Instance id from which the scan is triggered
2. The Quality Clouds live check service validates the API key and triggers an asynchronous message for the Quality Clouds engine to run the scan. If this is successful, it returns the Quality Clouds scan id to the ServiceNow app, which stores it in its table of LiveChecks for the Update Set or Application.
3. The Quality Clouds engine uses the ServiceNow read-only credentials used to perform the full scan to query for the contents of the Update Set or Application. This uses the standard ServiceNow REST API, which allows contents of very large applications and update sets to be retrieved efficiently. Once the scan completes, the Quality Clouds engine makes a GET request to a new component in the Quality Clouds ServiceNow application, a Scripted REST API endpoint which receives the scan id of the scan which has just completed. Note that this is a read-only call in order to avoid storing a user with write access in the Quality Clouds environment. 
4. Because the scan results can not be persisted from the read-only request, the scripted REST API endpoint which receives the request makes in turn another request to a different endpoint, with read-write access. The userid and password for this request are stored in the ServiceNow instance, in the system properties x_qucl_qc_snow.write.access.username and x_qucl_qc_snow.write.access.password. 
   1. In order for this second request to complete correctly, the user which is configured in Quality Clouds to run the full scans must be added to the "QC Users" group. This is only required if that user does not have the admin role.
5. The read-only endpoint queries the Quality Clouds REST API and retrieves the issues generated by the engine for the Update Set or Application scan. It updates the issues and live check tables to store the issues, quality gate information, and update set scan metadata in the ServiceNow application tables.

Note that the above architecture ensures that the update set and application scan information will be automatically updated once the Quality Clouds engine completes the scan. In any case, even if the read-write user is not configured, the new UI Action "Check Status" which is available in the LiveCheck form, can always be used to manually check the status of an update set or application scan. If the scan is completed, this UI Action will also retrieve the scan result data from the Quality Clouds API and update it in the ServiceNow app.