Thomas Tracks XR

GIF
Thomas our mascot
In App Recording
Live stylized map of the yard including live colors for track status

Inspiration

Are missed signals, speeding, and human errors causing accidents on Swiss railways? Absolutely, according to the statistics. But what if we could pave the way for a transformative future in rail transport? Imagine a solution that not only enhances navigation on railways but also redefines the way we guide drivers through shunting yards. This innovation is on the horizon!

What it does

Thomas quickly builds your desired path, eliminating the need to search for the right track segment on a map. With Thomas, you save valuable time. He reserves your route, highlights it while you drive, and ensures you never miss crucial signals in augmented reality. Plus, you can easily track your train's location.

How we built it

By utilizing XR (Extended Reality) and HoloLens, we've created an immersive and intuitive experience. Unity, a powerful development platform, is the backbone of our solution to serve XR applications. Moreover, we've harnessed the potential of generative AI to provide dynamic communication. The XR App for HoloLens is built with Unity and the help of MRTK3. The App also uses the socket.io API from Siemens and Matterhorn Gotthard Bahn. For the Speech-to-text functionality, we used the Windows-Build in API, and the conversion is modulated through ChatGPT and vocalized by the state-of-the-art Coqui Text-to-Speech model, served by FastAPI. We utilized the Google Cloud Compute Services to serve the TTS model.

Challenges we ran into

Mapping the XR and HoloLens into the real-world railway environment, where aligning data points with actual rail tracks was no small feat, posed a significant challenge in itself. While trying to link the yard-state data to the real-world location we noticed that even the good openstreet map data is not detailed enough. Optimizing the generative AI system to meet the needs for real-world communication presented its own set of hurdles. Deciphering these blueprints and translating them into an intuitive, user-friendly interface was a formidable task.

Accomplishments that we're proud of

We successfully implemented a real-time shunting yard data visualizer and provided a virtual assistant for train drivers to speed up their decision-making. During the course of our journey, we found a number of useful extensions for our application to build the future of Swiss railways.

What we learned

We learned the challenges of the railway industry, the very nature of track-ops, and the safety measures to avoid incidents.

What's next for ThomasTrackXR

We found a chance to analyze and propose enhancements in this special shunting yard. With collaboration with Siemens, more can be achieved on a bigger scale to further improve the quality of railway maintenance.