Sanity Checks
Sanity Checks are intelligent guardrails that automatically monitor annotation quality in real-time, helping annotators maintain consistency and accuracy while tracking objects across frames.
Sanity Checks catch tracking errors instantly. Set realistic movement thresholds to maintain annotation quality without slowing down your workflow.
Use Case
Annotation quality depends heavily on maintaining realistic object behavior across frames. However, even experienced annotators can inadvertently create unrealistic jumps in object position or rotation when working through large datasets or during fatigue.
Sanity Checks address this by:
Detecting anomalous movements between consecutive frames that exceed physically realistic thresholds
Identifying unrealistic rotation changes (yaw, pitch, roll) that may indicate tracking errors
Alerting annotators immediately when thresholds are exceeded, allowing for instant correction
Common scenarios include:
3D cuboid annotation for autonomous vehicle datasets where objects shouldn't teleport between frames
Tracking vehicles, pedestrians, or cyclists where sudden position jumps indicate annotation errors
Maintaining smooth rotational transitions for objects changing orientation
Quality control during high-volume annotation projects
Benefits
For Annotators
Immediate Feedback: Non-intrusive alerts appear as soon as a threshold is exceeded, enabling instant correction
Reduced Rework: Catch errors during annotation rather than during QA review
Confidence: Work with the assurance that obvious errors will be flagged
Learning Tool: Understand realistic movement patterns through threshold feedback
For Project Managers
Higher Quality Output: Systematic prevention of common annotation errors
Reduced QA Time: Fewer errors reach the review stage
Consistent Standards: Enforced thresholds ensure uniform quality across annotators
Data Integrity: Training datasets maintain physical realism and consistency
Steps to Use Sanity Checks
Setup: Defining Thresholds
Access Sanity Check configuration in the recipe by navigating to your project settings
Configure Movement Thresholds
Set minimum (optional) and maximum allowed movement distance between consecutive frames for each object class
Example: Vehicles might have a 5-meter threshold, while pedestrians might have a 2-meter threshold
Consider your frame rate and typical object speeds when setting values
Configure Rotation Thresholds
Define minimum (optional) and maximum allowed changes in yaw, pitch, and roll between frames
Example: 15° for yaw (turning), 10° for pitch and roll on stable vehicles
Adjust based on object behavior and camera movement
During Annotation
Create Your Annotation by tracking objects as usual using your standard annotation workflow.
Real-Time Monitoring - As you move to each subsequent frame and adjust the annotation, Sanity Checks automatically analyze the changes.
The system compares object position and orientation against your defined thresholds.
Responding to Alerts
If a threshold is exceeded, a non-intrusive notification appears
The alert indicates which threshold was violated (movement distance, yaw, pitch, or roll)
Review the annotation to determine if:
The object genuinely moved that much (adjust threshold if needed)
An annotation error occurred (correct the current or previous frame)
Continue annotating once addressed
Best Practices
Start Conservative: Begin with stricter thresholds and relax them based on real-world feedback
Object-Specific Tuning: Different object classes may require different threshold values
Team Calibration: Discuss threshold settings with your annotation team to ensure they match actual scene dynamics
Iterative Refinement: Review flagged instances periodically to optimize threshold values
Last updated