NSF iHARP Grant: NSF HDR Institute for Harnessing Data and Model Revolution in the Polar Regions

Current Students: Pranav Abishai Moses, Arshiya Bajaj, Om Panemangalore, Soham Sharma

Status: Current

Faculty Mentor: Pretom Roy Ovi

Point cloud data captures depth and object structure more accurately than im-ages, making it valuable for training object detection models. Indoor point cloud data, however, lacks the labeled datasets needed to train similar models, and manual 2D annotation is slow and imprecise because depth is lost on a flat screen. This project presents and compares three approaches to labeling indoor point cloud data using an immersive head-mounted displays (HMDs): an of-fline virtual reality (VR) pipeline that voxelizes and manually labels a point cloud before visualization, a real-time mixed reality (MR) pipeline that uses Meta Quest's Mixed Reality Utility Kit (MRUK) to detect and label objects au-tomatically, and a real-time augmented reality (AR) pipeline that lets a user manually select and label raw point cloud data using a lasso or bounding box tool. This work studies three different ways to label point cloud data using a head-mounted display (HMD) and handheld controllers. The three approaches are:

  • Approach 1: an offline VR (Virtual Reality) pipeline. The point cloud is broken into small cube-shaped pieces and labeled by hand before the user enters VR.
  • Approach 2: a real-time MR (Mixed Reality) pipeline. Meta Quest's own Mixed Reality Utility Kit (MRUK) detects objects and adds labels automatically, while the user has the HMD on.
  • Approach 3: a real-time AR (Augmented Reality) pipeline. A custom script loads the raw point cloud directly into the headset, and the user selects and labels objects by hand, using a lasso tool or a bounding box tool.
. Overall system architecture, the green flow indicates online or real-time annotation ap-proaches. a) image of the lab, b) mesh generated with the point cloud data, c) downsampled, processed and labeled point cloud data
Overall system architecture, the green flow indicates online or real-time annotation approaches.

a) image of the DVXR lab, b) mesh generated with the point cloud data, c) downsampled, processed and labeled point cloud data (each cube represents a voxel)

Offline VR Labeling

The input data for this approach is LiDAR scanned data of a lab on a university campus, captured using an iPhone with a LiDAR sensor. The point cloud was converted into the .PLY format so it could be used in Unity.

automated object detection labels overlaid over the physical world shows the raw point cloud data overlaid over the physical environment
a) and c) automated object detection labels overlaid over the physical world, b) and d) the actual physical environment. a) shows the raw point cloud data overlaid over the physical environment, b) the points highlighted in red show the points that are inside the blue bounding box, c) and d) two chairs annotated as object0 and object1.

GUI and Interaction Design

Each approach uses a different way for the user to move around the scene and interact with the point cloud. The design of movement and controls was chosen to match how each approach is meant to be used, either as a visualization tool or as a real-time annotation tool.

UI panel to toggle ON and OFF certain groups, user manipulating the size of the bounding box using the controller
a) UI panel to toggle ON and OFF certain groups, b) user pointing at a monitor to interact, and c) the information about the monitor being displayed to the user. user manipulating the size of the bounding box using the controller

Limitations and Future Work

This work is limited to static point cloud data; the labeling methods were not tested on live or continually changing data. 

Past Project: Point Cloud Data Visualization using Unity 3D and MetaQuest 3 (Poster)

The goal of this project is to develop a point cloud data visualization in VR or AR using Cesium, Unity 3D, and MetaQuest by incorporating labeling and annotation.

  • Preserving More Metadata for Point Clouds Using 3D Tiles.
  • Explore with other hardware devices such as HTC Vive, Hololens2, Magic leap 2, Apple Vision Pro, and Meta Quest 3.
  • Adding relevant labeling and annotation.
Point Cloud Annotation v2 with menu (local point cloud file) Point Cloud Viz in VR Meta Quest Pro
Point Cloud Annotation v2 with menu using Cesium Point Cloud Viz in VR Meta Quest Pro

Publications

Book Chapters

  1. Sharma, S., Moses, P.A., "Immersive Active Shooter Response Training and Decision-Making Environment for a University Campus Building", In: Chen, J.Y.C., Fragomeni, G. (eds) Virtual, Augmented and Mixed Reality. HCII 2025. Lecture Notes in Computer Science, pp 220–232, vol 15790. Springer, Cham. https://doi.org/10.1007/978-3-031-93715-6_14, 2025.
  2. Omary, D., Gamineedi, S.K., Shareef, A., Sharma, S. , "Virtual Reality Fire Drill for Campus Evacuation", In: Latifi, S. (eds) The 22nd International Conference on Information Technology-New Generations (ITNG 2025). ITNG 2025. Advances in Intelligent Systems and Computing, vol 1463, pp 568–579, Springer, Cham. https://doi.org/10.1007/978-3-031-89063-5_49, 2025.
  3. Chellatore, M.P., Pesaladinne, R.R., Sharma, S., "Time Series Analysis for Detecting Anomalous Behavior Using a Mobile Device",  In: Arabnia, H.R., Deligiannidis, L., Amirian, S., Ghareh Mohammadi, F., Shenavarmasouleh, F. (eds) Internet Computing and IoT and Embedded Systems, Cyber-physical Systems, and Applications. CSCE 2024. Communications in Computer and Information Science, vol 2260. Springer, Cham. https://doi.org/10.1007/978-3-031-85923-6_30, 2025.
  4. Chellatore, M.P., Sharma, S, "Mobile application for identifying anomalous behavior and conducting time series analysis using heterogeneous data", Springer Nature Switzerland AG, in J. Wei and G. Margetis (Eds.): HCII 2024, https://doi.org/10.1007/978-3-031-60458-4_12, LNCS 14737, pp. 1–16, 2024.
  5. Sharma, S, Dronavalli, S.C., Chellatore, M.P., Pesaladinne R., "Interactive Visualizations for Crime Data Analysis by Mixed Reality", Springer Nature Switzerland AG, in J. Y. C. Chen and G. Fragomeni (Eds.): HCII 2024, https://doi.org/10.1007/978-3-031-61047-9_19, LNCS 14708, pp. 1–18, 2024.

Journals

  1. M. A. S. A. Rishat, P. R. Ovi and S. Sharma, "Federated Unlearning: A Systematic Literature Review and Taxonomy of Emerging Methods, Systems, Open Challenges, and Research Agenda," in IEEE Access, doi: 10.1109/ACCESS.2026.3707516, 25 June 2026.
  2. Sharma, S and Pesaladinne R.,"Spatial Analysis and Visual Communication of Emergency Information through Augmented Reality", Journal of Imaging Science & Technology (JIST), JIST, Vol. 67, Issue 6, https://doi.org/10.2352/J.ImagingSci.Technol.2023.67.6.060401, 2023.

Conferences

  1. Afroz, U.I., Ovi, P.R., Nnaka, T.,Sharma, S., Sarker, B., "A Mobile Augmented Reality System for Personalized Dietary Feedback", Proceedings of the 13th Annual Conf. on Computational Science & Computational Intelligence (CSCI'26), December, 16-18, 2026 (Submitted)
  2. Moses,P.A., Bajaj, A., Panemangalore,O., Sharma,S., Ovi, P.R., "Immersive AR/VR Annotation of Point Clouds for Indoor Digital Twin Applications", Proceedings of the 13th Annual Conf. on Computational Science & Computational Intelligence (CSCI'26), December, 16-18, 2026 (Submitted)
  3. Sharma, S., Katikala, J., Srinivasan, K., Ovi, P. R., Holagh, S.R., Maloth, N., Cleveland, A., McMahan, T. , Kaplan-Rakowski, R., "Virtual Reality-Based AI Mental Health Companion: A Multimodal System for Therapy, Mindfulness, and Stress Detection", Proceedings of the IS&T International Symposium on Electronic Imaging (EI 2026) in the Engineering Reality of Virtual Reality Conference (ERVR), Hyatt Regency San Francisco Airport in Burlingame, California, March 1 - 5, 2026. 
  4. Alam, F., Sharma, S.,Ovi, P.R., Hossain, T., "Simulating Epidemic Response and Communication using AI-Powered NPCs in Virtual Reality", Proceedings of the IS&T International Symposium on Electronic Imaging (EI 2026) in the Engineering Reality of Virtual Reality Conference (ERVR), Hyatt Regency San Francisco Airport in Burlingame, California, March 1 - 5, 2026. 
  5. Katikala, J., Ovi, P.R., Sharma, S., "Real-Time Multimodal Sensing and Spatiotemporal Event Logging on Mobile Devices" Proceedings of the IEEE SoutheastCon, Huntsville, Alabama, 13-15 March 2026.
  6. M. A. S. A. Rishat, P. R. Ovi and S. Sharma, "A Scalable Hybrid Model for EEG-Based Stress Detection Toward Real-Time Deployability"  Proceedings of the IEEE SoutheastCon, Huntsville, Alabama, 13-15 March 2026. 
  7. Sharma, S., Ovi, P. R., Katikala,J. M., "Virtual Reality-Assisted Building Fire Evacuation Training using Digital Twin and Brain  Computer Interface Devices", Proceedings of the 28th International Conference on Human-Computer Interaction (HCI International 2026), Thematic Area: Virtual, Augmented and Mixed Reality, Montreal Convention Centre, Montreal, Canada, 26 - 31 July 2026. 
  8. Sarkar, A., Ovi, P.R., Sharma, S., Rahnemoonfar, M., Gangopadhyay, A. "Consistent and Robust Visual Explanations for Remote Sensing Image Classification", Proceedings of the 2026 IEEE International Geoscience and Remote Sensing Symposium (IGARSS) , Washington, D.C, 9 - 14 August 2026. 
  9.  Katikala, J.,Sharma, K., Kumar, T.R.K., Sharma, S.,"Conversational AI for Safety-Critical Virtual and Extended Reality: Intelligent Agents Across Health Informatics, Navigation, and Urban Sensing", Proceedings of the 8th IEEE International Conference on Artificial Intelligence Testing, Fukuoka, Japan, July 27-30, 2026. 
  10. Sharma, S, Summitt, A., "A Mobile Augmented Reality Application for Indoor Emergency Evacuation and Navigation", Proceedings of the IS&T International Symposium on Electronic Imaging (EI 2025) in the Engineering Reality of Virtual Reality Conference, pp 170-1 - 170-6,  https://doi.org/10.2352/EI.2025.37.13.ERVR-170, 2025.
  11. Summitt, A., Bhatt, B.J., Sharma, S.,"Emergency Assistive Mobile Application with Digital Twin for Real-Time 3D Navigation in Indoor Environments", Proceedings of the 23rd IEEE/ACIS International Conference on Software Engineering, Management and Applications (SERA 2025), Las Vegas, USA, pp. 233-238, doi: 10.1109/SERA65747.2025.11154602, 2025. 
  12. Bhatt, B.J., Summitt, A., Sharma, S.,"Immersive Virtual Reality Data Visualization for Urban Parking Management", Proceedings of the 23rd IEEE/ACIS International Conference on Software Engineering, Management and Applications (SERA 2025), Pages: 23-30,DOI: 10.1109/SERA65747.2025.11154584, Las Vegas, USA, May 29-31, 2025. 
  13. Omary, D., Gamineedi, S.K., Shareef, A., and Sharma, S., "Virtual Reality Fire Drill for Campus Evacuation", Proceedings of the 22nd International Conference on Information Technology - New Generations (ITNG), Advances in Intelligent Systems and Computing, vol 1463, pp 568–579, Las Vegas, Nevada, USA, April 13-16, 2025. 
  14. Bhatt, B., Sharma, S. "Mobile Application for Conducting Time Series Analysis on Location-Based Spatial Data", Proceedings of the 20th International Conference on Data Science, (ICDATA'24), Springer Nature Switzerland, AG 2025, R.Stahlbockand H.R.Arabnia(Eds.), CSCE2024,CCIS2253,pp.1–14,2025, https://doi.org/10.1007/978-3-031-85856-7_25, July 22-25, 2024.
  15. Chellatore,M.P., Pesaladinne R., and Sharma, S., "Time Series Analysis for Detecting Anomalous Behavior using a Mobile Device", Proceedings of the 22nd International Conference on Embedded Systems, Cyber-physical Systems, & Applications (ESCS'24), IEEE-CSCI conference, Las Vegas, USA, July 22-25, 2024.
  16. Chellatore, M.P., Sharma, S, "Mobile application for identifying anomalous behavior and conducting time series analysis using heterogeneous data", Proceedings of the 26th International Conference on Human-Computer Interaction (HCI International 2024), Thematic Area: Virtual, Augmented and Mixed Reality, Copenhagen, Washington Hilton Hotel, Washington DC, USA, 29 June - 4 July 2024.
  17. Sharma, S., Pesaladinne R., "Spatial Analysis and Visual Communication of Emergency Information through Augmented Reality", Proceedings of the IS&T International Symposium on Electronic Imaging (EI 2024) in the Engineering Reality of Virtual Reality Conference, DOI: 10.2352/J.ImagingSci.Technol.2023.67.6.060401, January 21-25, 2024.
  18. Pesaladinne, R., Chellatore, M.P., Dronavalli,S., Sharma, S., "Situational awareness and feature extraction for indoor building navigation using mixed reality", Proceedings of the IEEE International Conference on Computational Science and Computational Intelligence, (IEEE-CSCI), Research Track on Big Data and Data Science (CSCI-RTBD), Las Vegas, USA, December 13-15, 2023.
  19. Dronavalli,S. Pesaladinne, R., Sharma, S., "Crime Data Visualization Using Virtual Reality and Augmented Reality", Proceedings of the IEEE International Conference on Computational Science and Computational Intelligence, (IEEE-CSCI-RTSC), Las Vegas, USA, December 13-15, 2023.
  20. Tack, N, Williams, R, Holschuh, N, Sharma, S, Engel, D, "Visualizing the Greenland ice sheet in VR using immersive fence diagrams", (ACM-PEARC 23), Conference on Practice and Experience in Advanced Research Computing, Portland, OR, USA, ACM ISBN 978-1-4503-9985-2/23/07, https://doi.org/10.1145/3569951.3603635, July 23–27, 2023.
  21. Tack, N, Holschuh, N, Sharma, S, Williams, R, Engel, D, "Development and initial testing of XR-based fence diagrams for polar science", Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS 2023), Pasadena, California, 16-21 July 2023.
  22. Sharma, S, "Mobile Augmented Reality System for Emergency Response", Proceedings of the 21st IEEE/ACIS International Conference on Software Engineering, Management and Applications (SERA 2023), Orlando, USA, May 23-25, 2023.
  23. Sharma, S., Engel, D., "Mobile augmented reality system for object detection, alert, and safety", Proceedings of the IS&T International Symposium on Electronic Imaging (EI 2023) in the Engineering Reality of Virtual Reality Conference, January 15-19, 2023

Posters

  1. Keerthana Srinivasan, "ARNav: AI-Powered Indoor Navigation and Real-Time Situational Awareness at UNT Discovery Park", Advisor: Dr. Sharad Sharma, at UNT Research Day, October 14, 2025.
  2. Lavanya Nidamanuri, "Enhancing Mental Health, Stress, and Anxiety Management using EEG Signals and Virtual Reality with Emotiv", Advisor: Dr. Sharad Sharma, at UNT Research Day, October 14, 2025.
  3. Keerthana Srinivasan, "ARNav: AI-Powered Indoor Navigation and Real-Time Situational Awareness at UNT Discovery Park", Advisor: Dr. Sharad Sharma, at AI & Data Science Executive Summit 2025, hosted by the Anuradha & Vikas Sinha Department of Data Science at UNT Frisco, October 10, 2025. 
  4. Johannas Katikala, "Mobile App for Real-Time Multi-User Object, Pose, and Name Detection with Location-Based Data and Time Series Analysis", Advisor: Dr. Sharad Sharma, at AI & Data Science Executive Summit 2025, hosted by the Anuradha & Vikas Sinha Department of Data Science at UNT Frisco, October 10, 2025. 
  5. Abhaya Ladalla, "MindYoga: A Mobile Application for Mental Health Care Management Using Yoga-Based Interventions", Advisor: Dr. Sharad Sharma, at AI & Data Science Executive Summit 2025, hosted by the Anuradha & Vikas Sinha Department of Data Science at UNT Frisco, October 10, 2025.
  6. Andrew Summitt, Geospatial Mobile Application for Navigation and Emergency Response using Google Photorealistic 3D Tiles and Cesium for Unity, Advisor: Dr. Sharad Sharma, Day of Health Informatics and Data Science, UNT Frisco, September 20, 2024.
  7. Pranav Moses, Active Shooter Response Training for Discover Park Building, Advisor: Dr. Sharad Sharma, Day of Health Informatics and Data Science, UNT Frisco, September 20, 2024.
  8. Ian Abeyta and Jason Weinstein, Human Biomechanics as a Digital Twin & Real-time Motion Capture in Unity 3D Game Engine, Advisor: Dr. Sharad Sharma, Day of Health Informatics and Data Science, UNT Frisco, September 20, 2024.
  9. Bhoj Raj Bhatt and Sharad Sharma “Mobile App for Object Tracking and Location-Based Data for Time Series Analysis”, special celebration for the 15th anniversary event, College of Information (COI) at the University of North Texas, November 10, 2023. (1st Place Award). 
  10. Sri Chandra Dronavalli, and Sharad Sharma “Crime Data Analysis and Visualization through HoloLens 2 and Oculus Quest Pro”, special celebration for the 15th anniversary event, College of Information (COI) at the University of North Texas, November 10, 2023. (2nd Place Award) 
  11. Maruthi Prasanna and Sharad Sharma, “Mobile Application for Identifying Anomalous Behavior and Conducting Time Series Analysis using Parking Lot Data”, special celebration for the 15th anniversary event, College of Information (COI) at the University of North Texas, November 10, 2023. 
  12. Suruthi Selvam and Sharad Sharma, “Real Time Object Detection and Emotion Detection via Camera using React Native, Python Flask, Coco Dataset and OpenCV”, special celebration for the 15th anniversary event, College of Information (COI) at the University of North Texas, November 10, 2023.