Smart Glove System in Mixed Reality Scenarios

Numerous students were forced to take online classes from home during the COVID-19 pandemic. The challenge is that many harmful and only practicable offline chemical or biological experiments are challenging to carry out in a domestic setting. Fortunately, the advent of mixed reality technology has made it possible to overcome the time and physical limitations of laboratory instruction and has given students an efficient way to carry out safe experimental exercises. As a result, the primary contribution of this work is to develop the MRLab, a mixed reality smart experiment system, which enables the input of sensor information, speech information, and scene visual information. Simultaneous, this study also develops a multimodal hybrid fusion model called GVVS based on the input data in order to determine the user's experimental intention. The user can make use of experimental instruments or drugs in the real world while also interacting with objects in the virtual world and observing the associated experimental phenomena.