InstructAR - UNIHACK 2024

InstructAR is an innovative VR app that streams live camera feeds from a VR headset to a website, allowing remote users to place interactive AR markers in 3D space. These markers appear in real-time in a VR user's field of view, guiding them through complex tasks with precision and ease. Developed over 2 days as an entry into UNIHACK 2024, we definitely bit off more than we could chew technically, but it made for some very interesting learnings about the emerging technology of VR/AR.

We built InstructAR using a combination of technologies and frameworks. The VR encoding and streaming uplink is powered by Python with OpenCV and NumPy, while the AR overlay is implemented using Unity (C#) and native Meta Quest 3 APIs. The front-end of the website was developed with TypeScript using Next.js (React), and the back-end integration was achieved with Node.js running TypeScript behind Cloudflare and an NGINX reverse proxy. Each of these 4 applications communicates with each via Socket.IO except for the headset and the steaming uplink which communicate via ADB.

Design & Technical Challenges

Throughout the development of InstructAR, we encountered several challenges:

• Real-Time Streaming Latency: Ensuring low-latency, high-quality video streaming was critical. We tackled this by establishing low-latency connections via Socket.IO from the browser to the server and the server to the headset video processing uplink. Using compression and variable frame timing, we are able to stream frames within milliseconds to a remote browser anywhere in the world.
• AR Marker Accuracy: Achieving precise alignment of AR markers in the VR environment required an intricate understanding of realtime 3D to 2D projection and vice-versa. This involves raycasting into a 3D mesh of the real world, using the headset's position, orientation, marker location on the 2D plane and corrections for rotation and lens distortion of the video livestream.
• User Interface Design: Designing an intuitive UI for both web and VR users presented a unique challenge as InstructAR needs to make collaboration easier, not harder. This means designing a system no hidden tricks and a simple interface.

Future Objectives

InstructAR as a project has a lot of room for growth and certainly some polish. More specifically, the following are some immediate steps both feature and product wise to make the technology more accessible and useful to a broarder market:

• Expand the functionality of InstructAR to support customisable markers, text annotations and a native voice link.
• Enhance the scalability of our platform to accommodate larger user bases.
• Explore partnerships with headset and wearables creators such as Meta, Apple and Rayban to broaden the application scope and platform availability of InstructAR.

In all, there is definitely the potential of InstructAR to transform remote collaboration by bringing expert knowledge to where it's needed most.