Everything is possible on the holodecks created in the Star Wars films. Not only galactic beings come to life using holograms but entire landscapes and chapters in time can be visualised. Captain Picard is often heard alerting his crew “This is the wrong chapter”. The band Abba also uses holography projection – the musicians will be on stage as avatars during the 2022 world tour. Whether Star Wars or Abba, mixed reality appears ready-made for the big stage, for creating imaginative ideas and immersive 360-degree experiences. However, those looking for more significant benefits of MR beyond the performance stage have had to think a little bit further. Ironically, it could be the field of medicine that might be the beneficiary of this innovative technology. Mixed reality for a longer life!
Microsoft Hololens in medicine | Image Source: https://news.microsoft.com/hololens2_healthcare2/
“Teams” is the new virtual conference table. Thousands of meetings are held every day with Microsoft software. Switch on the camera, microphone and you’re all ready to take part. And it is just as simple when the device used is an augmented reality headset instead of a laptop or smartphone. The only difference with the “Hololens” from Microsoft is that you start the team meeting with a leap into an empty space. Slightly unusual perhaps, but definitely possible.
Glasses and Teams are now being tested in laboratories for their use in operating theatres. Doctors are using augmented reality to invite other experts to virtually join them in the operating room. Or they permit virtual operating assistants to appear at the real-life operating table. Everything is possible. And the quality of the headsets is improving by the day: On the Hololens 2 smartglasses, each eye has three lasers in front of it, producing extremely colour-intensive and true-to-detail 3D holographs.
And other innovations are also finding their way into medical treatment such as virtual reality glasses, 3D motion captures inside the body and virtual models of body organs. Clinical innovation teams are increasingly experimenting with mixed reality. This article highlights five possible areas of application for mixed reality in the future.
AR glasses have been around in the industry for some time now. Let’s take for example a large industrial facility that has broken down somewhere in the world. The technician responsible is several flight hours away and so a wireless computer headset is used instead. Someone on location puts on the glasses and calls up the remote expert for assistance. Four eyes, i.e. the off-site expert and the headset user both see directly inside the machine. This saves both time and money.
What makes sense for expensive equipment, makes even more sense for expensive medical surgery. Remote surgeons not only see what on-site doctors see. They can also draw in the virtual space, for example they can sketch the perfect incision to be made using the scalpel. The on-site doctor sees both the patient in front of them as well as the virtual incision that the remote doctor has made. Everything is performed in the mixed reality space, making this technology the perfect assistant.
Although the on-site team is always highly qualified and fully capable of performing the surgery, not every operation belongs to the standard repertoire of every doctor, such as knee joint implants. Some indications and operations are very seldom. Very few experts in the field have the necessary experience to perform this work. Mixed reality also enables doctors to consult with experts from other medical fields during the operation.
Although the co-working principle has not been favoured by these “demigods in white” in the past, times are changing and AR technology is proving to be a perfect assistant. You can watch a video of the Hololens 2 in a test OP environment which we recorded at the Think Tank HealthCare Futurists:
Even if no other colleague is looking virtually through the AR headset, the benefits of AR can still be enormous. Surgeons can call up additional patient data in the virtual room: They can use holography to project three-dimensional MRT or CT images into the room. For example, with an artificial hip joint. AR technology overlays the X-ray onto the patient’s real – and still closed – thigh. Surgeons then see inside the leg although the incision has yet to be made. They can see the exact position of the joint and can work more precisely.
The Imperial College in London has been experimenting for some time with the application of AR in operating rooms. Radiologists and surgeons overlay images of 3D data onto the patient’s leg after they have suffered serious accidents. By overlaying an image of the patient’s internal anatomy onto the body, surgeons can see the safest place for the incision where no veins or arteries can be punctured. Each bone, each blood vessel, each tendon and every individual injury can be visualised at the exact position inside the body.
However, 3D holograms require a lot of computational power. The image files also have to be programmed so precisely that there is not a single millimetre of error to the original visualised under the skin. It will take some time before this technology is in standard practice. One step closer to technological maturity are however AR systems that are currently being tested at the Munich University Hospital. IT students have installed a type of user’s manual for surgical instruments into the glasses. The software explains by voice command how the instruments work and shows videos to explain their use by virtual reality. That all sounds quite strange for outsiders. But new instruments with which the surgical team have not had much practice are used more often than you’d think in modern operating theatres.
Our body is built like a Swiss clock: it has 80 organs, 200 bones and several 1,000 kilometres of blood vessels, all neatly accommodated in a very compact space. Doctors require years of teaching and training to understand the body’s complexity. Technology is assisting doctors increasingly more often. With the Da Vinci surgical robot, a 3D-HD camera rather than the doctor now looks directly into the body. Measuring a diameter of just a few millimetres, it acts as the surgeon’s technical eye. Using a minimal invasive approach, it delivers high-definition TV views, magnified 10 times to what the human eye sees. This makes the surgeon’s work easier.
Especially because the cameras now in use produce three-dimensional images. So surgeons can now, for example, see what’s inside the patient’s heart. The doctors perform surgery like under the microscope and can detect the tiniest structures in regions which were once hard to see by the human eye. Rather than holding the instruments, the surgeon now lets the robotic arms on the Da Vinci system perform the job. Doctors now navigate the instruments by remote control using a joystick.
If mobile 5G or even 6G transmission rates become reality, it’s theoretically possible for a doctor in Berlin to perform surgery on a patient in Paris using Da Vinci. From a technical perspective, this wouldn’t pose a problem. But going back to holography. The screen which the medical team uses to see inside the body will soon be a thing of the past thanks to mixed reality. The camera image will then be projected into the surgeon’s field of vision.
Mixed reality has been used for the teaching and training of trainee medical practitioners for some time now. The Magic Mirror for Anatomy Learning from the Munich University Hospital has already achieved a certain claim to fame. Medical students stand in front of a mirror and suddenly see their inner organs and structures reflected back at them – in both lengthwise and cross-sectional views. Although the mirror only reflects a standard model of the inner human body and not of each individual, it still offers a fascinating insight.
It still remains common practice to work and practice surgery on deceased bodies that have been donated to science. However, virtual technologies are now offering new simulation-based training options. For example, the Ulm University Hospital has developed a three-dimensional heart for VR training. Trainee doctors can use virtual reality glasses to practice inserting an artificial mitral valve into the human heart. And they don’t even have to be at the same location.
What sounds even more amazing is a test application project developed at Chemnitz University of Technology. Aspiring surgeons implant artificial hip replacements, in virtual reality – and are provided with haptic feedback at the same time. Wearing VR glasses, they operate in virtual surgery while using real medical instruments. Users stand in front of a manipulated milling machine which they operate to drill the necessary channel into the bone. They see through the glasses how the milling machine drills into the hip bone while moving the robotic arm.
The milling machine simulates the haptic responses of the patient’s body. It shakes and responds just like a drill would when drilling a hole into a wall. One of the most delicate and critical steps in this operation is “drilling the hole” where the artificial prosthetic hip is to be implanted later. Every single step of the operation can be achieved using this technology. This has enabled the scientists in Chemnitz overcome an obstacle that has always eluded mixed reality technology. They combine virtual reality with haptic feedback, an all important factor for doctors.
Not only medical staff but also patients could be using VR more frequently. Numerous start-up companies are now using VR glasses to assist in the recovery of stroke patients suffering from hemiplegia. While the patient performs a specific, therapeutic exercise, virtual reality is used to simulate the movement of the paralysed arm.
This illusion tricks you into believing it is a healthy arm. In training, this arm moves towards the cup and it appears as if the paralysed arm is doing the same. It is as if the patient is moving both arms. This claims to assist in the recovery of the paralysed arm. Damaged areas of the brain would be stimulated into forming new connections.
Text: Ron Voigt
Most popular