Audit Tool

Audit Tool allows Customers to have access to submitted batches of the job after they have undergone all the nodes of the taskflow.

How to get here?

From the Landing Dashboard

1. Go to the ‘Audit’ from the Solution listed on the QuickServe Dashboard (from the gear icon) of the Solutions listed.

2. User is taken to the Audit Dashboard, where they can view all the submitted Batches.

   

From Job Builder

1. Navigate to the Job Builder UI

2. Click on the 'Audit' button at the top right

From All Batches

1. Go to 'Job Details' under the gear icon beside a job on Job Builder

2. Select 'All Batches' from the left navigation.

3. Select the 'Audit' button that appears above the batch table.

Audit Dashboard

1. Select the Job from the dropdown to see all the batches with their status in the table below.

All batches will be listed under the dashboard. The table displays the following details:

• Tasks: Number of frames in one Batch

• Annotations: Total no. of Annotations created in a Batch

• Issues: No. of Mistakes marked by the auditor

• Issue Status (Open/ Closed/ Ignored): No. of Issues under each Type

• Status (of the Batch)

  • LIP (Label in Progress) - Cannot be accessed by the auditor till labelling is complete

  • RFA (Ready for Audit) - Auditor can select this sequence provided no other sequence is assigned to them

  • AIP (Audit in Progress) - Auditor is currently reviewing the sequence

• Assigned To - Auditor Name

• Last Updated - Last time the sequence was submitted. A sequence can go through multiple rounds of audit

• Action - Launch the Audit Tool

Audit Tool Interface

The tool opens in a new tab. The interface is similar to the annotation tool with all the annotation data

The 'Issues' Link on the Header is enabled in the Audit Tool. Here is where the users can document issues as and when observed.

Note:

• The Annotations are non-editable

• The only action that takes place is marking issues

On the Issues Panel, auditors can see a summary of the issues created and also filter Issues from the Issue Title row.

How to Raise Issues

To create an issue user can do either of the two:

1. Clicking on 'Issue' on the Header opens the Issue Panel. Click on the '+' icon on the issue title row.

1. Right-clicking on the annotation and select the ‘Create Issue’ option.

Create Issue Card Details

Create Issue Card appears on the Issue panel with the following data required for input before saving:

Frame

• Start: Frame number where the issue is observed. The tool will automatically populate this with the current frame and will not prompt for an input

• End: Frame number till where the issue is observed if it spans across multiple frames. The tool will automatically populate this with the current frame assuming the error doesn't span beyond the current frame. The user can update to the frame till where it is applicable.

Class, Instance, Attribute Selection

Dropdown selection from the list defined in the Project Recipe.

Severity

Leverage this field to indicates the level of impact of an issue, ranging from critical to minor.

Issue Type Selection

All the issues that can be observed are listed under default. The user must select whichever is applicable. The list of options for Issues is customisable.

Each error category is assigned a specific color, making it easier for the Auditor to quickly identify the type of issue they are looking for within the list. For example, 'blue' represents wrong annotations, 'green' indicates missing annotations, 'orange' signifies inconsistency in Track ID, and 'purple' refers to miscellaneous issues.

Comments (if any)

Issue Menu

Each Issue Card created has a few options which can be pulled up by clicking on the 3 vertical dots on the top right corner.

• Reply to the issue

• Ignore/ Close the Issue (the same gets reflected on the Audit Dashboard)
• 3D/ 2D markers to redirect the annotator's attention to a location.

  • For a Missed Annotation: In this case Select the Class & Issue type as 'Missing Object'. After saving, click on the three dots to mark the location of the expected Annotation on the Image/ point cloud using 2D/ 3D Marker respectively.

Quick Tip: To associate the card to the existing annotation, provide the class and instance details. This will allow users to click on the card to be redirected to where the issue lies.

Auditing Complete

Once reviewing is completed, click on Accept or Reject on the top right of the tool.

1. Accept will change the status of the batch to Audit Accepted in the batch lifecycle

2. Reject will change the status of the batch to Audit Rejected in the batch lifecycle

3. WithHold is used when the Auditor is unsure and would like to revisit the batch. The status of the batch turns to Withhold, letting all know that the decision is still pending.