Inspiration
Working with industrial machines got us interested in working with Digital Shadows and Twins. Based on our previous knowledge of how machines work, we wanted to create a MR experience of a digital shadows.
What it does
The way it works is each part of the robot arm machine can be detached and inspected with hand interactions. There is also a visualization mode which turns on X-Ray mode. Here the internals of the robot arm can be seen and the moving parts are highlighted. Animations of the robot arm effect the visualization mode, meaning that machine parts change colors.
How we built it
The app has been build using Unity3D 2021. The interactions are based on Oculus SDK. The app uses Oculus Hand tracking and passthrough feature configured using Unity.
Challenges we ran into
Setting up the mesh in a way that we can effectively visualize parts using Blender. Animating the game-object using Unity's animator proved challenging.
Accomplishments that we're proud of
Socket interactions of the Robot arm are synced with animations and work in a generic way, the system was tested with other meshes which could be disassembled and reassembled. One other advanced feature of the socket system is that both the parent and child socket are movable, which makes it a lot harder to keep track the state of the robot arm.
What we learned
Team work and coordination is key to victory! Making custom socket implementation would be better than trying to make the built-in sockets work in a way that they aren't mean to.
What's next for MR Robot Arm Shadow
Using hand tracking to animate the robot arm and visualize each part! This is doable using custom gestures and would allow a user to move a real robot arm based on MR hand tracking!
Log in or sign up for Devpost to join the conversation.