Audit Tool
Audit Tool allows Customers to have access to submitted batches of the job after they have undergone all the nodes of the taskflow.
How to get here?
From the Landing Dashboard
Go to the ‘Audit’ from the Solution listed on the QuickServe Dashboard (from the gear icon) of the Solutions listed.
User is taken to the Audit Dashboard, where they can view all the submitted Batches.
From Job Builder
Navigate to the Job Builder UI
Click on the 'Audit' button at the top right
From All Batches
Go to 'Job Details' under the gear icon beside a job on Job Builder
Select 'All Batches' from the left navigation.
Select the 'Audit' button that appears above the batch table.
Audit Dashboard
Select the Job from the dropdown to see all the batches with their status in the table below.
All batches will be listed under the dashboard. The table displays the following details:
Tasks: Number of frames in one Batch
Annotations: Total no. of Annotations created in a Batch
Issues: No. of Mistakes marked by the auditor
Issue Status (Open/ Closed/ Ignored): No. of Issues under each Type
Status (of the Batch)
LIP (Label in Progress) - Cannot be accessed by the auditor till labelling is complete
RFA (Ready for Audit) - Auditor can select this sequence provided no other sequence is assigned to them
AIP (Audit in Progress) - Auditor is currently reviewing the sequence
Assigned To - Auditor Name
Last Updated - Last time the sequence was submitted. A sequence can go through multiple rounds of audit
Action - Launch the Audit Tool
Audit Tool Interface
The tool opens in a new tab. The interface is similar to the annotation tool with all the annotation data
The 'Issues' Link on the Header is enabled in the Audit Tool. Here is where the users can document issues as and when observed.
Note:
The Annotations are non-editable
The only action that takes place is marking issues
On the Issues Panel, auditors can see a summary of the issues created and also filter Issues from the Issue Title row.
How to Raise Issues
To create an issue user can do either of the two:
Clicking on 'Issue' on the Header opens the Issue Panel. Click on the '+' icon on the issue title row.
Right-clicking on the annotation and select the ‘Create Issue’ option.
Create Issue Card Details
Create Issue Card appears on the Issue panel with the following data required for input before saving:
Frame
Start: Frame number where the issue is observed. The tool will automatically populate this with the current frame and will not prompt for an input
End: Frame number till where the issue is observed if it spans across multiple frames. The tool will automatically populate this with the current frame assuming the error doesn't span beyond the current frame. The user can update to the frame till where it is applicable.
Class, Instance, Attribute Selection
Dropdown selection from the list defined in the Project Recipe.
Severity
Leverage this field to indicates the level of impact of an issue, ranging from critical to minor.
Issue Type Selection
All the issues that can be observed are listed under default. The user must select whichever is applicable. The list of options for Issues is customisable.
Each error category is assigned a specific color, making it easier for the Auditor to quickly identify the type of issue they are looking for within the list. For example, 'blue' represents wrong annotations, 'green' indicates missing annotations, 'orange' signifies inconsistency in Track ID, and 'purple' refers to miscellaneous issues.
Comments (if any)
Issue Menu
Each Issue Card created has a few options which can be pulled up by clicking on the 3 vertical dots on the top right corner.
Reply to the issue
Ignore/ Close the Issue (the same gets reflected on the Audit Dashboard)
3D/ 2D markers to redirect the annotator's attention to a location.
For a Missed Annotation: In this case Select the Class & Issue type as 'Missing Object'. After saving, click on the three dots to mark the location of the expected Annotation on the Image/ point cloud using 2D/ 3D Marker respectively.
Quick Tip: To associate the card to the existing annotation, provide the class and instance details. This will allow users to click on the card to be redirected to where the issue lies.
Auditing Complete
Once reviewing is completed, click on Accept or Reject on the top right of the tool.
Accept will change the status of the batch to Audit Accepted in the batch lifecycle
Reject will change the status of the batch to Audit Rejected in the batch lifecycle
WithHold is used when the Auditor is unsure and would like to revisit the batch. The status of the batch turns to Withhold, letting all know that the decision is still pending.
Last updated