Skip Section Navigation

iHARP- NSF HDR Institute for Harnessing Data and Model Revolution in the Polar Regions

Current Students: Kanishk Sharma, Maimuna Patwary, Rishat, Md Al Samiul Amin, Ananya Agarwal, S M Saiful Islam Badhon, Abhignya Jagathpally, Umme Israt Afroz, Khawja Imran Masud, Gowtham Vuppaladhiam

Past Students: Johannas Meela Katikala,  Tharun Swaminathan Ravi Kumar, Abhaya Ladalla, Andrew Summit, Bhoj Bhatt, Rishitha Reddy Pesaladinne, Suruthi Selvam, Maruthi Prasanna, Kavya Gundla, Archith Sharma (TAMS)

Status: Current

Annotation and Visualization of Heterogeneous Data using VR/AR

The goal of this project is to develop intelligent data-driven techniques, accurate labeled data is needed for training and evaluation of deep learning techniques. To better engage scientists in the labeling and to facilitate the labeling we will use AR/VR tools. This work seeks to build upon the previous efforts by creating an integrated tool suite for enhancing the value, situational awareness, accessibility, and understanding of science data connected with Arctic science using virtual reality (VR) and augmented reality (AR) technologies. This work will design, develop, and evaluate AR/VR tools to explore and annotate, using tablets/mobile devices, and HoloLens. The objective of the effort will be to use the AR displays on mobile devices and headsets to guide the users with virtual overlays, paths, and way-points. It will also involve development of algorithms for layering, situational awareness, and location sensing.

Real-Time Multi-user Object Detection and Time Series Analysis using Mobile Devices (Poster) APK Files: Version 6

Accurate labeled data is essential for training and evaluating deep-learning models. This work aims to develop an integrated VR/AR tool suite that enhances the value, accessibility, situational awareness, and overall understanding of scientific data. In our parking-lot case study, users place pins to mark vehicles and capture associated information such as latitude, longitude, time, date, text, images, and videos. By allowing users to drop a pin where their car is parked, the system collects temporal data that can be leveraged for time-series analysis. This project presents a real-time, mobile-based system that integrates object detection, pose estimation, face recognition, emotion detection, and audio alerts into a unified app. Designed using Unity and OpenCV, the system enables contextual logging with time, location, and identity for smart surveillance and behavioral analysis. It incorpodatres the follwing modules:
1) Data Collection Module: Utilizes GPS, cameras, microphones, and manual inputs to capture comprehensive data.
2) Data Visualization Module: Applies time series to visualize and interpret the data, revealing patterns and predictions.
3) Time Series Analysis Module: Enhance decision-making processes

Use Cases:

usecases

Real-time Multi-user data collection for time series analysis Time series analysis to interpret data, revealing patterns and predictions.
AI-Assistant for parking lot in VR environment Object & Pose detection, face detection & recognization
iharp iharp
iharp iharp
   
iharp iharp
Muti-user login page Face dtection and recoginization
iharp iharp
Login System that enables users to signup and capture faces  Live database updates -Admin

Project 1: Development of mobile application for identifying anomalous behavior and conducting time series analysis using heterogeneous data (Poster)

We have incorporated a use case of a parking lot to develop and test our tool. Understanding anomalous behavior and spatial changes in an urban parking area can enhance decision-making and situational awareness insights for sustainable urban parking management. Decision-making relies on data that comes in overwhelming velocity and volume, that one cannot comprehend without some layer of analysis and visualization. This work presents a mobile application that performs time series analysis and anomaly detection on parking lot data for decision-making. The mobile application includes two modules: 1) Information gathering module and 2) Time series analysis module. In the information gathering module, users can add pins at the parking lot and in the time series analysis module, users can analyze the pins they added over a period. Our approach uses parking pins to identify each vehicle and then collect specific data, such as temporal variables like latitude, longitude, time, date, and text (information from the license plate), as well as images and videos shot at the location. Users have the option of placing pins at the location where their car is parked, and the information collected can be used for time series analysis. By examining the data pattern, we may quickly identify vehicles parked in restricted spaces but without authorization and vehicles parked in disabled spaces but owned by regular users. This time series analysis enables the extraction of meaningful insights, making it useful in the identification of recurring patterns in parking lot occupancy over time. This information aids in predicting future demands, enabling parking administrators to allocate resources efficiently during peak hours and optimize space usage. It can be used in detecting irregularities in parking patterns, aiding in the prompt identification of unauthorized or abnormal parking and parking violations which includes parking of the wrong type of vehicle, and parking at restricted or reserved areas.

Information Gathering Module Information Gathering Module Time Series Analysis Module Time Series Analysis Module

Videos (Time Series Analysis Module )

Project 2: Time Series Analysis for Detecting Anomalous Behavior using Unity 3D and MetaQuest 3 (Poster)

The proposed mobile application is developed using the Unity framework and seamlessly integrates image capture, note-taking, GPS tracking, and a robust SQLite database, offering users a comprehensive memory management system. This work focuses on performing time series analysis techniques for detecting anomalous behavior in urban parking lots.

Detecting Anomalous Behavior Detecting Anomalous Behavior Detecting Anomalous Behavior Detecting Anomalous Behavior

Mobile Application using Unity 3D for time series analysis for Geospatial data                      Visualization of Geospatial data in MetaQuest 3

Project 3: Real Time Object Detection and Emotion Detection via Camera using React Native, Python Flask, Coco Dataset and OpenCV (Poster)

This project presents a real-time object detection and emotion recognition system implemented through React Native for mobile application development, Python Flask for backend support, and leveraging the Coco dataset and OpenCV for robust and accurate detection. The integration of these technologies enables seamless camera-based object recognition and emotion analysis, offering a versatile and responsive user experience. Our object detection model is trained on the Coco Dataset and powered by OpenCV, showcases accurate and responsive detection capabilities. Additionally, our emotion recognition module provides a seamless user experience, highlighting the project's potential in various practical applications. As we look ahead, there is immense scope for further enhancements and broader utilization of this technology in diverse domains. The work includes:
1) Coco Dataset: The Coco dataset serves as a foundational element of our object detection system.  Its extensive and varied content is crucial for training a robust object detection model capable of recognizing a wide range of objects in real-time scenarios.
2) OpenCV Integration:  It provides a comprehensive set of functions for image processing and computer vision tasks.
3) Emotion Detection: The core of our emotion detection system is a deep learning model trained to recognize human emotions from images and video frames. We employed a convolutional neural network (CNN) architecture for this purpose. The model was trained on a diverse dataset of labeled facial expressions, allowing it to identify a spectrum of emotions, including happiness, sadness, anger, and more.
4) Camera Integration: To capture and analyze user emotions in real-time, we seamlessly integrated the device's camera into our application. This feature enables users to interact naturally with our system without the need for additional sensors or hardware. Image frames from the camera are processed on the device, ensuring privacy and real-time responsiveness. Emotion predictions are then displayed to the user or transmitted to the backend for further analysis if needed.

Mobile Application using React Native, Python Flask, Coco Dataset, and OpenCV                      Visualization of Geospatial data

Project 4: Point Cloud Data Visualization using Unity 3D and MetaQuest 3 (Poster)

The goal of this project is to develop a point cloud data visualization in VR or AR using Cesium, Unity 3D, and MetaQuest by incorporating labeling and annotation.

  • Preserving More Metadata for Point Clouds Using 3D Tiles.
  • Explore with other hardware devices such as HTC Vive, Hololens2, Magic leap 2, Apple Vision Pro, and Meta Quest 3.
  • Adding relevant labeling and annotation.
Point Cloud Annotation v2 with menu (local point cloud file) Point Cloud Viz in VR Meta Quest Pro
Point Cloud Annotation v2 with menu using Cesium Point Cloud Viz in VR Meta Quest Pro

Publications

Book Chapters

  1. Sharma, S., Moses, P.A., "Immersive Active Shooter Response Training and Decision-Making Environment for a University Campus Building", In: Chen, J.Y.C., Fragomeni, G. (eds) Virtual, Augmented and Mixed Reality. HCII 2025. Lecture Notes in Computer Science, pp 220–232, vol 15790. Springer, Cham. https://doi.org/10.1007/978-3-031-93715-6_14, 2025.
  2. Omary, D., Gamineedi, S.K., Shareef, A., Sharma, S. , "Virtual Reality Fire Drill for Campus Evacuation", In: Latifi, S. (eds) The 22nd International Conference on Information Technology-New Generations (ITNG 2025). ITNG 2025. Advances in Intelligent Systems and Computing, vol 1463, pp 568–579, Springer, Cham. https://doi.org/10.1007/978-3-031-89063-5_49, 2025.
  3. Chellatore, M.P., Pesaladinne, R.R., Sharma, S., "Time Series Analysis for Detecting Anomalous Behavior Using a Mobile Device",  In: Arabnia, H.R., Deligiannidis, L., Amirian, S., Ghareh Mohammadi, F., Shenavarmasouleh, F. (eds) Internet Computing and IoT and Embedded Systems, Cyber-physical Systems, and Applications. CSCE 2024. Communications in Computer and Information Science, vol 2260. Springer, Cham. https://doi.org/10.1007/978-3-031-85923-6_30, 2025.
  4. Chellatore, M.P., Sharma, S, "Mobile application for identifying anomalous behavior and conducting time series analysis using heterogeneous data", Springer Nature Switzerland AG, in J. Wei and G. Margetis (Eds.): HCII 2024, https://doi.org/10.1007/978-3-031-60458-4_12, LNCS 14737, pp. 1–16, 2024.
  5. Sharma, S, Dronavalli, S.C., Chellatore, M.P., Pesaladinne R., "Interactive Visualizations for Crime Data Analysis by Mixed Reality", Springer Nature Switzerland AG, in J. Y. C. Chen and G. Fragomeni (Eds.): HCII 2024, https://doi.org/10.1007/978-3-031-61047-9_19, LNCS 14708, pp. 1–18, 2024.

Journals

  1. M. A. S. A. Rishat, P. R. Ovi and S. Sharma, "Federated Unlearning: A Systematic Literature Review and Taxonomy of Emerging Methods, Systems, Open Challenges, and Research Agenda," in IEEE Access, doi: 10.1109/ACCESS.2026.3707516, 25 June 2026.
  2. Sharma, S and Pesaladinne R.,"Spatial Analysis and Visual Communication of Emergency Information through Augmented Reality", Journal of Imaging Science & Technology (JIST), JIST, Vol. 67, Issue 6, https://doi.org/10.2352/J.ImagingSci.Technol.2023.67.6.060401, 2023.

Conferences

  1. Afroz, U.I., Ovi, P.R., Nnaka, T.,Sharma, S., Sarker, B., "A Mobile Augmented Reality System for Personalized Dietary Feedback", Proceedings of the 13th Annual Conf. on Computational Science & Computational Intelligence (CSCI'26), December, 16-18, 2026 (Submitted)
  2. Moses,P.A., Bajaj, A., Panemangalore,O., Sharma,S., Ovi, P.R., "Immersive AR/VR Annotation of Point Clouds for Indoor Digital Twin Applications", Proceedings of the 13th Annual Conf. on Computational Science & Computational Intelligence (CSCI'26), December, 16-18, 2026 (Submitted)
  3. Sharma, S., Katikala, J., Srinivasan, K., Ovi, P. R., Holagh, S.R., Maloth, N., Cleveland, A., McMahan, T. , Kaplan-Rakowski, R., "Virtual Reality-Based AI Mental Health Companion: A Multimodal System for Therapy, Mindfulness, and Stress Detection", Proceedings of the IS&T International Symposium on Electronic Imaging (EI 2026) in the Engineering Reality of Virtual Reality Conference (ERVR), Hyatt Regency San Francisco Airport in Burlingame, California, March 1 - 5, 2026. 
  4. Alam, F., Sharma, S.,Ovi, P.R., Hossain, T., "Simulating Epidemic Response and Communication using AI-Powered NPCs in Virtual Reality", Proceedings of the IS&T International Symposium on Electronic Imaging (EI 2026) in the Engineering Reality of Virtual Reality Conference (ERVR), Hyatt Regency San Francisco Airport in Burlingame, California, March 1 - 5, 2026. 
  5. Katikala, J., Ovi, P.R., Sharma, S., "Real-Time Multimodal Sensing and Spatiotemporal Event Logging on Mobile Devices" Proceedings of the IEEE SoutheastCon, Huntsville, Alabama, 13-15 March 2026.
  6. M. A. S. A. Rishat, P. R. Ovi and S. Sharma, "A Scalable Hybrid Model for EEG-Based Stress Detection Toward Real-Time Deployability"  Proceedings of the IEEE SoutheastCon, Huntsville, Alabama, 13-15 March 2026. 
  7. Sharma, S., Ovi, P. R., Katikala,J. M., "Virtual Reality-Assisted Building Fire Evacuation Training using Digital Twin and Brain  Computer Interface Devices", Proceedings of the 28th International Conference on Human-Computer Interaction (HCI International 2026), Thematic Area: Virtual, Augmented and Mixed Reality, Montreal Convention Centre, Montreal, Canada, 26 - 31 July 2026. 
  8. Sarkar, A., Ovi, P.R., Sharma, S., Rahnemoonfar, M., Gangopadhyay, A. "Consistent and Robust Visual Explanations for Remote Sensing Image Classification", Proceedings of the 2026 IEEE International Geoscience and Remote Sensing Symposium (IGARSS) , Washington, D.C, 9 - 14 August 2026. 
  9.  Katikala, J.,Sharma, K., Kumar, T.R.K., Sharma, S.,"Conversational AI for Safety-Critical Virtual and Extended Reality: Intelligent Agents Across Health Informatics, Navigation, and Urban Sensing", Proceedings of the 8th IEEE International Conference on Artificial Intelligence Testing, Fukuoka, Japan, July 27-30, 2026. 
  10. Sharma, S, Summitt, A., "A Mobile Augmented Reality Application for Indoor Emergency Evacuation and Navigation", Proceedings of the IS&T International Symposium on Electronic Imaging (EI 2025) in the Engineering Reality of Virtual Reality Conference, pp 170-1 - 170-6,  https://doi.org/10.2352/EI.2025.37.13.ERVR-170, 2025.
  11. Summitt, A., Bhatt, B.J., Sharma, S.,"Emergency Assistive Mobile Application with Digital Twin for Real-Time 3D Navigation in Indoor Environments", Proceedings of the 23rd IEEE/ACIS International Conference on Software Engineering, Management and Applications (SERA 2025), Las Vegas, USA, pp. 233-238, doi: 10.1109/SERA65747.2025.11154602, 2025. 
  12. Bhatt, B.J., Summitt, A., Sharma, S.,"Immersive Virtual Reality Data Visualization for Urban Parking Management", Proceedings of the 23rd IEEE/ACIS International Conference on Software Engineering, Management and Applications (SERA 2025), Pages: 23-30,DOI: 10.1109/SERA65747.2025.11154584, Las Vegas, USA, May 29-31, 2025. 
  13. Omary, D., Gamineedi, S.K., Shareef, A., and Sharma, S., "Virtual Reality Fire Drill for Campus Evacuation", Proceedings of the 22nd International Conference on Information Technology - New Generations (ITNG), Advances in Intelligent Systems and Computing, vol 1463, pp 568–579, Las Vegas, Nevada, USA, April 13-16, 2025. 
  14. Bhatt, B., Sharma, S. "Mobile Application for Conducting Time Series Analysis on Location-Based Spatial Data", Proceedings of the 20th International Conference on Data Science, (ICDATA'24), Springer Nature Switzerland, AG 2025, R.Stahlbockand H.R.Arabnia(Eds.), CSCE2024,CCIS2253,pp.1–14,2025, https://doi.org/10.1007/978-3-031-85856-7_25, July 22-25, 2024.
  15. Chellatore,M.P., Pesaladinne R., and Sharma, S., "Time Series Analysis for Detecting Anomalous Behavior using a Mobile Device", Proceedings of the 22nd International Conference on Embedded Systems, Cyber-physical Systems, & Applications (ESCS'24), IEEE-CSCI conference, Las Vegas, USA, July 22-25, 2024.
  16. Chellatore, M.P., Sharma, S, "Mobile application for identifying anomalous behavior and conducting time series analysis using heterogeneous data", Proceedings of the 26th International Conference on Human-Computer Interaction (HCI International 2024), Thematic Area: Virtual, Augmented and Mixed Reality, Copenhagen, Washington Hilton Hotel, Washington DC, USA, 29 June - 4 July 2024.
  17. Sharma, S., Pesaladinne R., "Spatial Analysis and Visual Communication of Emergency Information through Augmented Reality", Proceedings of the IS&T International Symposium on Electronic Imaging (EI 2024) in the Engineering Reality of Virtual Reality Conference, DOI: 10.2352/J.ImagingSci.Technol.2023.67.6.060401, January 21-25, 2024.
  18. Pesaladinne, R., Chellatore, M.P., Dronavalli,S., Sharma, S., "Situational awareness and feature extraction for indoor building navigation using mixed reality", Proceedings of the IEEE International Conference on Computational Science and Computational Intelligence, (IEEE-CSCI), Research Track on Big Data and Data Science (CSCI-RTBD), Las Vegas, USA, December 13-15, 2023.
  19. Dronavalli,S. Pesaladinne, R., Sharma, S., "Crime Data Visualization Using Virtual Reality and Augmented Reality", Proceedings of the IEEE International Conference on Computational Science and Computational Intelligence, (IEEE-CSCI-RTSC), Las Vegas, USA, December 13-15, 2023.
  20. Tack, N, Williams, R, Holschuh, N, Sharma, S, Engel, D, "Visualizing the Greenland ice sheet in VR using immersive fence diagrams", (ACM-PEARC 23), Conference on Practice and Experience in Advanced Research Computing, Portland, OR, USA, ACM ISBN 978-1-4503-9985-2/23/07, https://doi.org/10.1145/3569951.3603635, July 23–27, 2023.
  21. Tack, N, Holschuh, N, Sharma, S, Williams, R, Engel, D, "Development and initial testing of XR-based fence diagrams for polar science", Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS 2023), Pasadena, California, 16-21 July 2023.
  22. Sharma, S, "Mobile Augmented Reality System for Emergency Response", Proceedings of the 21st IEEE/ACIS International Conference on Software Engineering, Management and Applications (SERA 2023), Orlando, USA, May 23-25, 2023.
  23. Sharma, S., Engel, D., "Mobile augmented reality system for object detection, alert, and safety", Proceedings of the IS&T International Symposium on Electronic Imaging (EI 2023) in the Engineering Reality of Virtual Reality Conference, January 15-19, 2023

Posters

  1. Keerthana Srinivasan, "ARNav: AI-Powered Indoor Navigation and Real-Time Situational Awareness at UNT Discovery Park", Advisor: Dr. Sharad Sharma, at UNT Research Day, October 14, 2025.
  2. Lavanya Nidamanuri, "Enhancing Mental Health, Stress, and Anxiety Management using EEG Signals and Virtual Reality with Emotiv", Advisor: Dr. Sharad Sharma, at UNT Research Day, October 14, 2025.
  3. Keerthana Srinivasan, "ARNav: AI-Powered Indoor Navigation and Real-Time Situational Awareness at UNT Discovery Park", Advisor: Dr. Sharad Sharma, at AI & Data Science Executive Summit 2025, hosted by the Anuradha & Vikas Sinha Department of Data Science at UNT Frisco, October 10, 2025. 
  4. Johannas Katikala, "Mobile App for Real-Time Multi-User Object, Pose, and Name Detection with Location-Based Data and Time Series Analysis", Advisor: Dr. Sharad Sharma, at AI & Data Science Executive Summit 2025, hosted by the Anuradha & Vikas Sinha Department of Data Science at UNT Frisco, October 10, 2025. 
  5. Abhaya Ladalla, "MindYoga: A Mobile Application for Mental Health Care Management Using Yoga-Based Interventions", Advisor: Dr. Sharad Sharma, at AI & Data Science Executive Summit 2025, hosted by the Anuradha & Vikas Sinha Department of Data Science at UNT Frisco, October 10, 2025.
  6. Andrew Summitt, Geospatial Mobile Application for Navigation and Emergency Response using Google Photorealistic 3D Tiles and Cesium for Unity, Advisor: Dr. Sharad Sharma, Day of Health Informatics and Data Science, UNT Frisco, September 20, 2024.
  7. Pranav Moses, Active Shooter Response Training for Discover Park Building, Advisor: Dr. Sharad Sharma, Day of Health Informatics and Data Science, UNT Frisco, September 20, 2024.
  8. Ian Abeyta and Jason Weinstein, Human Biomechanics as a Digital Twin & Real-time Motion Capture in Unity 3D Game Engine, Advisor: Dr. Sharad Sharma, Day of Health Informatics and Data Science, UNT Frisco, September 20, 2024.
  9. Bhoj Raj Bhatt and Sharad Sharma “Mobile App for Object Tracking and Location-Based Data for Time Series Analysis”, special celebration for the 15th anniversary event, College of Information (COI) at the University of North Texas, November 10, 2023. (1st Place Award). 
  10. Sri Chandra Dronavalli, and Sharad Sharma “Crime Data Analysis and Visualization through HoloLens 2 and Oculus Quest Pro”, special celebration for the 15th anniversary event, College of Information (COI) at the University of North Texas, November 10, 2023. (2nd Place Award) 
  11. Maruthi Prasanna and Sharad Sharma, “Mobile Application for Identifying Anomalous Behavior and Conducting Time Series Analysis using Parking Lot Data”, special celebration for the 15th anniversary event, College of Information (COI) at the University of North Texas, November 10, 2023. 
  12. Suruthi Selvam and Sharad Sharma, “Real Time Object Detection and Emotion Detection via Camera using React Native, Python Flask, Coco Dataset and OpenCV”, special celebration for the 15th anniversary event, College of Information (COI) at the University of North Texas, November 10, 2023.