Point cloud data captures depth and object structure more accurately than im-ages,
making it valuable for training object detection models. Indoor point cloud data,
however, lacks the labeled datasets needed to train similar models, and manual 2D
annotation is slow and imprecise because depth is lost on a flat screen. This project
presents and compares three approaches to labeling indoor point cloud data using an
immersive head-mounted displays (HMDs): an of-fline virtual reality (VR) pipeline
that voxelizes and manually labels a point cloud before visualization, a real-time
mixed reality (MR) pipeline that uses Meta Quest's Mixed Reality Utility Kit (MRUK)
to detect and label objects au-tomatically, and a real-time augmented reality (AR)
pipeline that lets a user manually select and label raw point cloud data using a lasso
or bounding box tool. This work studies three different ways to label point cloud
data using a head-mounted display (HMD) and handheld controllers. The three approaches
are:
Approach 1: an offline VR (Virtual Reality) pipeline. The point cloud is broken into small cube-shaped
pieces and labeled by hand before the user enters VR.
Approach 2: a real-time MR (Mixed Reality) pipeline. Meta Quest's own Mixed Reality Utility Kit
(MRUK) detects objects and adds labels automatically, while the user has the HMD on.
Approach 3: a real-time AR (Augmented Reality) pipeline. A custom script loads the raw point
cloud directly into the headset, and the user selects and labels objects by hand,
using a lasso tool or a bounding box tool.
Overall system architecture, the green flow indicates online or real-time annotation
approaches.
a) image of the DVXR lab, b) mesh generated with the point cloud data, c) downsampled,
processed and labeled point cloud data (each cube represents a voxel)
Offline VR Labeling
The input data for this approach is LiDAR scanned data of a lab on a university campus,
captured using an iPhone with a LiDAR sensor. The point cloud was converted into the
.PLY format so it could be used in Unity.
a) and c) automated object detection labels overlaid over the physical world, b) and
d) the actual physical environment.
a) shows the raw point cloud data overlaid over the physical environment, b) the points
highlighted in red show the points that are inside the blue bounding box, c) and d)
two chairs annotated as object0 and object1.
GUI and Interaction Design
Each approach uses a different way for the user to move around the scene and interact
with the point cloud. The design of movement and controls was chosen to match how
each approach is meant to be used, either as a visualization tool or as a real-time
annotation tool.
a) UI panel to toggle ON and OFF certain groups, b) user pointing at a monitor to
interact, and c) the information about the monitor being displayed to the user.
user manipulating the size of the bounding box using the controller
Limitations and Future Work
This work is limited to static point cloud data; the labeling methods were not tested
on live or continually changing data.
Past Project: Point Cloud Data Visualization usingUnity 3D and MetaQuest 3 (Poster)
The goal of this project is to develop a point cloud data visualization in VR or AR
using Cesium, Unity 3D, and MetaQuest by incorporating labeling and annotation.
Preserving More Metadata for Point Clouds Using 3D Tiles.
Explore with other hardware devices such as HTC Vive, Hololens2, Magic leap 2, Apple
Vision Pro, and Meta Quest 3.
Adding relevant labeling and annotation.
Point Cloud Annotation v2 with menu (local point cloud file)
Chellatore, M.P.,Sharma, S,"Mobile application for identifying anomalous behavior and conducting time series
analysis using heterogeneous data", Springer Nature Switzerland AG, in J. Wei and
G. Margetis (Eds.): HCII 2024,https://doi.org/10.1007/978-3-031-60458-4_12, LNCS 14737, pp. 1–16, 2024.
Sharma, S, Dronavalli, S.C., Chellatore, M.P., Pesaladinne R., "Interactive Visualizations
for Crime Data Analysis by Mixed Reality", Springer Nature Switzerland AG, in J. Y.
C. Chen and G. Fragomeni (Eds.): HCII 2024,https://doi.org/10.1007/978-3-031-61047-9_19, LNCS 14708, pp. 1–18, 2024.
Katikala, J.,Sharma, K., Kumar, T.R.K., Sharma, S.,"Conversational AI for Safety-Critical
Virtual and Extended Reality: Intelligent Agents Across Health Informatics, Navigation,
and Urban Sensing", Proceedings of the 8th IEEE International Conference on Artificial Intelligence Testing, Fukuoka, Japan, July 27-30, 2026.
Omary, D., Gamineedi, S.K., Shareef, A., and Sharma, S., "Virtual Reality Fire Drill for Campus Evacuation", Proceedings of the 22nd International Conference on Information Technology - New
Generations (ITNG), Advances in Intelligent Systems and Computing, vol 1463, pp 568–579,
Las Vegas, Nevada, USA, April 13-16, 2025.
Bhatt, B., Sharma, S. "Mobile Application for Conducting Time Series Analysis on Location-Based
Spatial Data", Proceedings of the 20th International Conference on Data Science, (ICDATA'24), Springer Nature Switzerland, AG 2025, R.Stahlbockand H.R.Arabnia(Eds.), CSCE2024,CCIS2253,pp.1–14,2025,
https://doi.org/10.1007/978-3-031-85856-7_25, July 22-25, 2024.
Pesaladinne, R., Chellatore, M.P., Dronavalli,S., Sharma, S., "Situational awareness
and feature extraction for indoor building navigation using mixed reality", Proceedings
of the IEEE International Conference on Computational Science and Computational Intelligence,
(IEEE-CSCI), Research Track on Big Data and Data Science (CSCI-RTBD), Las Vegas, USA,
December 13-15, 2023.
Dronavalli,S. Pesaladinne, R., Sharma, S., "Crime Data Visualization Using Virtual
Reality and Augmented Reality", Proceedings of the IEEE International Conference on
Computational Science and Computational Intelligence, (IEEE-CSCI-RTSC), Las Vegas,
USA, December 13-15, 2023.
Tack, N, Williams, R, Holschuh, N, Sharma, S, Engel, D, "Visualizing the Greenland
ice sheet in VR using immersive fence diagrams", (ACM-PEARC 23), Conference on Practice
and Experience in Advanced Research Computing, Portland, OR, USA, ACM ISBN 978-1-4503-9985-2/23/07,
https://doi.org/10.1145/3569951.3603635, July 23–27, 2023.
Tack, N, Holschuh, N, Sharma, S, Williams, R, Engel, D, "Development and initial testing
of XR-based fence diagrams for polar science", Proceedings of the IEEE International
Geoscience and Remote Sensing Symposium (IGARSS 2023), Pasadena, California, 16-21
July 2023.
Sharma, S, "Mobile Augmented Reality System for Emergency Response", Proceedings of
the 21st IEEE/ACIS International Conference on Software Engineering, Management and
Applications (SERA 2023), Orlando, USA, May 23-25, 2023.
Sharma, S., Engel, D., "Mobile augmented reality system for object detection, alert,
and safety", Proceedings of the IS&T International Symposium on Electronic Imaging
(EI 2023) in the Engineering Reality of Virtual Reality Conference, January 15-19,
2023
Posters
Keerthana Srinivasan, "ARNav: AI-Powered Indoor Navigation and Real-Time Situational
Awareness at UNT Discovery Park", Advisor: Dr. Sharad Sharma, at UNT Research Day,
October 14, 2025.
Lavanya Nidamanuri, "Enhancing Mental Health, Stress, and Anxiety Management using
EEG Signals and Virtual Reality with Emotiv", Advisor: Dr. Sharad Sharma, at UNT Research
Day, October 14, 2025.
Keerthana Srinivasan, "ARNav: AI-Powered Indoor Navigation and Real-Time Situational
Awareness at UNT Discovery Park", Advisor: Dr. Sharad Sharma, at AI & Data Science
Executive Summit 2025, hosted by the Anuradha & Vikas Sinha Department of Data Science
at UNT Frisco, October 10, 2025.
Johannas Katikala, "Mobile App for Real-Time Multi-User Object, Pose, and Name Detection
with Location-Based Data and Time Series Analysis", Advisor: Dr. Sharad Sharma, at
AI & Data Science Executive Summit 2025, hosted by the Anuradha & Vikas Sinha Department
of Data Science at UNT Frisco, October 10, 2025.
Abhaya Ladalla, "MindYoga: A Mobile Application for Mental Health Care Management
Using Yoga-Based Interventions", Advisor: Dr. Sharad Sharma, at AI & Data Science
Executive Summit 2025, hosted by the Anuradha & Vikas Sinha Department of Data Science
at UNT Frisco, October 10, 2025.
Andrew Summitt, Geospatial Mobile Application for Navigation and Emergency Response
using Google Photorealistic 3D Tiles and Cesium for Unity, Advisor: Dr. Sharad Sharma,
Day of Health Informatics and Data Science, UNT Frisco, September 20, 2024.
Pranav Moses, Active Shooter Response Training for Discover Park Building, Advisor:
Dr. Sharad Sharma, Day of Health Informatics and Data Science, UNT Frisco, September
20, 2024.
Ian Abeyta and Jason Weinstein, Human Biomechanics as a Digital Twin & Real-time Motion
Capture in Unity 3D Game Engine, Advisor: Dr. Sharad Sharma, Day of Health Informatics
and Data Science, UNT Frisco, September 20, 2024.
Bhoj Raj Bhatt and Sharad Sharma “Mobile App for Object Tracking and Location-Based
Data for Time Series Analysis”, special celebration for the 15th anniversary event,
College of Information (COI) at the University of North Texas, November 10, 2023.
(1st Place Award).
Sri Chandra Dronavalli, and Sharad Sharma “Crime Data Analysis and Visualization through
HoloLens 2 and Oculus Quest Pro”, special celebration for the 15th anniversary event,
College of Information (COI) at the University of North Texas, November 10, 2023.
(2nd Place Award)
Maruthi Prasanna and Sharad Sharma, “Mobile Application for Identifying Anomalous
Behavior and Conducting Time Series Analysis using Parking Lot Data”, special celebration
for the 15th anniversary event, College of Information (COI) at the University of
North Texas, November 10, 2023.
Suruthi Selvam and Sharad Sharma, “Real Time Object Detection and Emotion Detection
via Camera using React Native, Python Flask, Coco Dataset and OpenCV”, special celebration
for the 15th anniversary event, College of Information (COI) at the University of
North Texas, November 10, 2023.