Harnessing Vision-Language Models for Perceptual Quality Assessment and Autonomous Content Adjustment in Augmented Reality
Elias Rotondo · Lin Duan · Yanming Xiu · Sangjun Eom · Conrad Li · Maria Gorlatova
This repository accompanies the conference paper "Harnessing Vision-Language Models for Perceptual Quality Assessment and Autonomous Content Adjustment in Augmented Reality", published in the 32nd ACM Symposium on Virtual Reality Software and Technology (VRST `26). It introduces RateAR, a dataset of 321 AR images and 112 AR videos collected using various hardware platforms and user scenarios.
Figure 1: RateAR dataset examples illustrating varying quality across three features: (a) Placement plausibility: virtual desk floating feet above (poor), inches above (fair), or level with the ground (good); (b) Size appropriateness: digital car appearing far too small (poor), slightly small (fair), or realistically sized (good); and (c) Shadow realism: AR noodle bowl shadow opposing the real lighting (poor), slightly misaligned (fair), or fully aligned (good).
- Dataset Composition
- Collection Process
- RateAR Hierarchical Structure
- Dataset Download
- Supplemental Materials
- Citation
- Contacts
- Acknowledgments
The RateAR dataset comprises 321 AR images and 112 AR videos, curated to diversify the visual quality levels among influential user immersion properties. Figure 1 presents representative examples from RateAR, illustrating a range of quality levels across three key visual factors: placement plausibility, size appropriateness, and shadow realism.
AR Platforms: AR samples are sourced from a range of devices and scenarios. 321 AR images are collected from DiverseAR+ [Duan et al., 2025], selected for their varying rendering qualities in virtual content placement, shadow, and size. All RateAR image instances were captured using Android smartphones (250), Apple Vision Pro (40), and Microsoft HoloLens 2 (31). A diverse collection of environments are featured, including bedrooms, kitchens, living rooms, medical offices, reading rooms, research labs, and study rooms. The 112 AR videos were captured by us to represent a similar distribution of quality feature variations, showcasing settings such as basements, bedrooms, living rooms, and research labs. All RateAR videos are recorded using Meta Quest 3.
Context-Dependent Scenarios: Alongside common settings such as bedrooms and kitchens, where assessing visual factors primarily relies on general, publicly shared knowledge, RateAR also includes scenes that require context-dependent reasoning for evaluation. For example, medical settings feature virtual organ models placed in anatomically accurate locations on human subjects or 3D-printed models (e.g., a skull). In these scenarios, the placement plausibility and size appropriateness of virtual content is most critical, while elements like shadow realism are less relevant for perceived quality. Additionally, we include scenarios that capture dynamic user interactions, such as a hand shown bouncing a virtual basketball. As a result, some examples with observed floating AR content may be contextually appropriate, therefore receiving a high placement plausibility score.
Human Subjective Quality Score Labeling: Four graduate students with experience designing, developing, and evaluating AR systems rated placement plausibility, size appropriateness, and shadow realism using a Python notebook GUI. Before labeling samples, the annotators discussed the target quality features and expectations for handling nuanced edge cases, such as what score to assign when a virtual object without an accompanying shadow was visually appropriate (e.g., anatomical models). As needed, subsequent consensus meetings remediated annotator conceptual disagreements. Modeled after the International Telecommunication Union's (ITU) Absolute Category Rating guidelines, a 5-point discrete scale was employed. An assignment of
Inter-Annotator Reliability: To measure inter-annotator reliability, we computed the intraclass correlation coefficients (ICC) for each labeled quality feature based on a two-way random-effects model, average measures, and absolute agreement using the Pingouin open-source Python package. The model for placement plausibility yielded
The dataset follows the hierarchical file structure shown below:
dataset
└───rateAR_images
│ └───rateAR_image_metadata_mos.csv
│ └───AVP
│ │ └───AVP_can
│ │ │ └───AVP_can_1_ar.png
│ │ │ ...
│ │ ...
│ └───andriod
│ ...
└───rateAR_videos
│ └───rateAR_video_metadata_mos.csv
│ └───livingroom
│ │ └───livingroom_tire_1_1_1.mp4
│ │ ...
│ └───dorm
│ ...
The full RateAR dataset is made publicly available on Hugging Face Datasets.
Using a Python script, directly download the entire dataset (including all subdirectories) using the official Hugging Face Hub API:
from huggingface_hub import snapshot_download
snapshot_download(
repo_id="I3TDataset/RateAR",
repo_type="dataset",
local_dir="./RateAR", # Save dataset within working dir
)The RateAR dataset can also be downloaded locally using Hugging Face's official command-line interface:
- Intall the Hugging Face Hub tool
pip install huggingface_hub
- Download the dataset to the target local directory (e.g., "./RateAR")
hf download I3TDataset/RateAR --repo-type dataset --local-dir ./RateAR
In addtion to releasing the RateAR dataset, we also make publicly available supplemental materials related to the labeling of the dataset, conducted evaluations (including the user study), and ablation study. Below, we provide short document summaries introducing such materials or references to subdirectories that address related files.
rateAR_annotation_instructions.pdf: High-level instructions provided to the final dataset annotator, reflecting the consensuses held by the initial group of labelers.representative_gpt5.4_verbatim_responses.pdf: Two representative examples demonstrating how VLMs (in this case, GPT-5.4) can contextualize AR scenes and reason about a virtual object's target feature before returning a score rating (on a 1-5 scale).content_adjustment_pipeline_vlm_prompts.pdf: The placement and size task prompts provided to the VLM model for the prototype autonomous content adjustment system.- user_study_resources subdirectory: Files related to the conducted user study evaluating the proposed AR content adjustment pipeline.
- ablation_study_resources subdirectory: Prompt groupings analyzed during the prompt engineering ablation study.
If you use the RateAR dataset in any capacity, please cite:
@inproceedings{rotondo2026ratear,
author = {Rotondo, Elias and Duan, Lin and Xiu, Yanming and Eom, Sangjun and Li, Conrad, and Gorlatova, Maria},
title = {Harnessing Vision-Language Models for Perceptual Quality Assessment and Autonomous Content Adjustment in Augmented Reality},
year = {2026},
isbn = {9798400728112},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/3822517.3848685},
doi = {10.1145/3822517.3848685},
booktitle = {Proceedings of the 2026 32nd ACM Symposium on Virtual Reality Software and Technology},
location = {Sendai, Japan},
series = {VRST '26}
}For questions relating to this repository, the associated paper, or the shared dataset, please contact:
- Elias Rotondo - eli [DOT] rotondo [AT] duke [DOT] edu
We thank the participants of our user study for their invaluable help in this research and Tanish Pentakota for assistance implementing baselines.
This work was supported in part by NSF grants CSR-2312760, CNS-2112562, and IIS-2231975, NSF CAREER Award IIS-2046072, NSF NAIAD Award 2332744, a CISCO Research Award, a Meta Research Award, Defense Advanced Research Projects Agency Young Faculty Award HR0011-24-1-0001, and the Army Research Laboratory under Cooperative Agreement Number W911NF-23-2-0224. The views and conclusions contained in these works are those of the authors. They should not be interpreted as representing the official policies, either expressed or implied, of the Defense Advanced Research Projects Agency, the Army Research Laboratory, or the U.S. Government. This resource has been approved for public release; distribution is unlimited. No official endorsement should be inferred. The U.S. Government is authorized to reproduce and distribute reprints for Government purposes, notwithstanding any copyright notation herein.
This work is licensed under a Creative Commons Attribution 4.0 International License.