The brain surgeon cuts into your hologram

It started with robots slipping off the wall, on the three-seater sofa and the sideboard. Live holograms in the living room, only visible if you look through computer glasses.

Microsoft's first demonstration of the Hololens in 2015 looked like a futuristic game. Three years later, gambling became a serious tool: in the Netherlands, three brain patients were operated on using holograms. The test will soon be expanded.

Neurosurgeon Tristan van Doormaal of UMC Utrecht projects a hologram of a brain scanner on the real brain of a real patient, on a real operating table. On his head, he wears the Hololens, a Windows computer wrapped in a headband with a transparent 3D screen for the eyes. This is how you really get together the biological organ and digital representation.

Three Dimensions

When he sees the underlying structure of a brain, Van Doormaal can move more safely and remove tumors without affecting vulnerable areas. "We often operate deep under the brain, through the nose or through the skull, and then it's important to know exactly where you are."

To find the way in someone's brain – "neuronavigation "- The surgeon is now working with a pointer that shows CT or MRI on an external monitor. This does not offer any real stereoscopic experience – the feeling that you can grab objects. While a brain surgeon "thinks" in three dimensions, says Van Doormaal.

Existing brain navigation systems are expensive. "Soon 3 tons, while a Hololens costs around 3,000 euros." For computers in developing countries, computer glasses would be a boon. "They have to do it now with our discarded equipment." The surgeon points out that he is not committed to Hololens and does not have a financial agreement with Microsoft. He talks about holograms. They will conquer the operating room

Own tools

In a laboratory on the Utrecht Uithof, Tristan van Doormaal works with the Brain Technology Institute on new tools for brain surgery. In this way, he looks for better types of glue to seal the brain's membrane after surgery – the leakage of the wound makes many patients stay unnecessarily long in the hospital.

He is trained, a surgeon who prefers to design his own tools because practice is the best teacher. Van Doormaal is always looking for new gadgets, provided they are practical and non-futuristic. "For example, I would like to test a robot that helps to more accurately bridge the brain in a young child's brain.These blood vessels have a thickness of 0.6 or 0.7 millimeters."

But, according to him, this robot is not realistic. "You must be a surgeon in the operating room to solve problems and work in a team. This course is about people. "

Read also: Robot surgery? The operation remains manual work

The plan to experiment with a Hololens comes from his tube.Van Doormaal designed, in collaboration with the Institute of Brain technology and developer Tom Mensink, the software that translates and projects 3D scans into holograms on hololens.Hiking a hologram of a patient now takes only ten minutes.

It took six months to rebuild the Hololens navigation machine and to measure whether the holograms remain exactly in place.The hologram has a deviation of a few millimeters, while the expensive equipment has a millimeter clearance.

Since April, he has operated three patients with Hololens on his head. "They were not guinea pigs," says Van Doormaal because he used as a guide to the traditional method of neuro-navigation and compared it to holograms.

This month, Van Doormaal submits a request to the Ethics and Medical Review Committee: he wants to be allowed to operate from thirty to fifty other patients with both techniques. These surgeries are performed by several surgeons

A tangible brain

In a meeting room, he evokes the brain of his first "holo patients". The brain, blood vessels and tumors are almost tangible, even for a layman. "Nice, is not it?" Said Van Doormaal.

Brain navigation through holograms could work on phones and other computer glasses in addition to Hololens. Traditional manufacturers of medical equipment are also interested in the hologram.

The surgeon thinks aloud: you can project the images through an ex microscope – which enlarges the functioning of the brain on a screen above the patient. The surgeon no longer looks through the eyepiece of a microscope, but through 3D glasses.

This exoscope offers new possibilities, says Van Doormaal: "Then I can stand up or sit down during the operation – it makes a lot of difference and a herniated neck."

The Neurosurgeon uses Hololens to better explain to patients what he will do in their head and to explain to the assistants how the operation should be performed safely. In a few years, holography is the traditional neuronavigation alongside, he expects. As a surgeon, you think in three dimensions

Tristan van Doormaal, brain surgeon

Computer glasses determine its position on the basis of its environment, via depth cameras and d & # 39; Other sensors, and place the hologram on the patient's head. But if many things change in the space – more wizards or an additional table with tools, the software must "calibrate" again; calculate how the hologram is anchored. When the head is turned, the hologram rotates.

And to be able to work really accurately, computer glasses should also be able to handle brain change . Van Doormaal explains, "Before you operate, you let cerebrospinal fluid run out, so that a space is created in the brain. As a result, the brain surface falls to an inch and the 3D scan no longer exactly matches. Even if a large tumor is removed, the holographic image deviates from reality.

Measuring changes in the "soft parts" of the human body – and correcting the software during operation – is the holy grail of the human body. neurosurgery, says Van Doormaal. You need glasses with a more powerful processor.

Microsoft will provide the new Hololens – which should appear in early 2019 – with a chip that also performs artificial intelligence calculations – this could be the hologram in brain shift can correct.

Van Doormaal has more wishes. The glasses should remain wireless, lighter (now over half a kilo), with a wider field of view and a longer battery life. The glasses should also automatically "filter" your hands in view – now the holograms are projected onto the surgeon's hands. This makes the mixed reality less realistic.

(The text continues after the video)

Tristan van Doormaal also showed the Hololens in the "World of Tomorrow"

Microsoft learned its lesson

At Microsoft they are surprised by the eagerness with which doctors go to the Hololens. Simon Kos, who heads Microsoft's medical branch: "At the first presentation, we had a demonstration of a stereoscopic look into the heart. The doctors wanted something more practical and designed holograms on the dummies to train the students, "he says by phone at a medical conference in London.

Hololens were used to see ultrasounds of pregnant women in Japan. computer glasses used in liver operations, to avoid blood vessels. "It's going faster than expected," says Kos, who has long worked in a hospital

. Technology companies like Microsoft are turning their backs on the medical sector, as health spending will increase in the coming years. For example, Google has its own branch of health, in truth, Amazon is working on its own health insurance for staff. Apple, in turn, turns the Apple Watch into a medical tracker that informs your doctor of your activities.

Read also: This discovery of Apple may be more important than the new iPhone

Microsoft had to just look before determining its strategy. For example, the company stopped using the Microsoft Band, a fitness tracker that went missing after two versions.

The software manufacturer has invested in various medical companies in 2008 and 2009, but has already resold it to GE. Kos: "It's a lesson we've learned.We are good at inventing.But Microsoft is not good in medical services.We do not have to compete with our partners and our customers. "

In the cloud

Microsoft achieved a business turnover of $ 110 billion ($ 93 billion) last year and did not say how much medical customers. However, health care facilities are expected to spend 25% more money on storage and analysis in the cloud over the next few years. This is the market on which Microsoft, like Google and Amazon, has set its sights.
Hospitals will collect more data on patients and want to discover trends in these data. For example, the computer helps doctors in their diagnoses and the hospital can better estimate how long patients will stay.
A hundred scientists work for the health branch of Microsoft. "A serious investment," says Kos. According to him, Hololens is a good way to exploit a new market and at the same time attract hospitals to Microsoft's servers and software: "We do not want to develop our own health services. We want to build tools for doctors. "