In the wake of Microsoft's Windows 10 event, HoloLens, Microsoft's pitch to bring augmented reality into the mainstream, has stolen the spotlight.
While Microsoft insists that HoloLens will be available within a 'Windows 10 timeframe' (the OS hits later this year), the company won't give any further launch details. In fact, the company is doing all it can to simultaneously hint at the potential of the device and keep the physical nature of the thing under wraps. Here's what we know so far:
Alex Kipman, the Microsoft executive driving development of the device, describes it as featuring "see-through holographic lenses." That could be a 720p or 1080p display, but you can bet if they managed to cram 4K resolution on the lenses, they'd be shouting about it.
Again, Microsoft won't talk specifics, but the publicity photos clearly show four sensors at the front of the headset. This will almost certainly build upon the technology used in the Microsoft Kinect, specifically the sensor array featured in the version 2 for the Xbox One console. That included a 1080p wide-angle colour camera, three infra-red illuminators and an accompanying infra-red sensor for light-independent data gathering and accurate depth perception, using proprietary and highly cutting edge time-of-flight technology. Presumably all of that will have been further refined. And what of that fourth sensor? An additional IR sensor for wider angle data gathering makes sense for a device so reliant on spatial recognition – tracking within a field of 120 x 120 degrees has been mooted.
Microsoft is promising 3D audio placement, so that users will hear as well as see holographic content in 3D space. The prototype unit seems to feature just two speakers, though. Audio recognition is also crucial to the unit's success. The original Kinect featured a 4-way microphone array, with six on the Kinect 2, vastly improving voice recognition. For a head-mounted device, good audio capture shouldn't be such an issue.
Rather than develop an untethered WiFi or even tethered peripheral for a PC or console system, Microsoft has instead designed HoloLens as a standalone Windows 10 device. That means the headset contains its own CPU and GPU, alongside a 'Holographic Processing Unit' (HPU). Rumour has it the CPU and GPUs will be based on Intel's new Cherry Trail 14-nanometer chips. Cherry Trail is already known to support wires-free power charging and wireless video streaming via Wi-Di. The HPU is more of an unknown quantity. This will be the part that analyses and processes pixels in 3D space. It could be an advancing graphics processing unit related to Intel's Xeon Phi processor, though many pundits believe both the HPU and Liquid Crystal on Silicon (LCOS) will be provided by Himax, which has also been linked to Google Glass.
Testers have commented that the actual portion of vision occupied by the augmented reality of the HoloLens is surprisingly tiny, "akin to a 16x9" monitor situated around 7" from the face." That's not necessarily a bad thing, as it still enables users to interact with the real world relatively easily, with augmented interactions confined to the main area of focus. This isn't a device intended to block out incoming stimuli to present a virtual reality alternative, after all. What matters is how well the pixels are blended into the real-world view. Word is HoloLens has those crucial tracking latency issues cracked.
The development/prototype kits tested by journalists so far are extremely bulky. The HPU is the size of small laptop, the unit is mains powered and the headset is heavy and uncomfortable. Microsoft claims the sleeker ones shown in presentations are working units, but when it becomes commercially available, we can still expect some hiccups. Even Microsoft co-founder Bill Gates admits that "it will take a few years of software applications being built to realise the full promise."
The ultra-secretive Magic Leap has already hinted at a similar augmented reality device. The secrecy suggests more vapourware, but a $500 million investment and involvement from Google, as well as luminaries from vfx studio Weta, suggests otherwise. Then there's SpaceGlasses from Meta, the3 AiR Smartglasses from fellow AR startup Atheer Lab, devices from ODG, and even a headset from Epson, though none of these look as advanced as Microsoft's creation.
The withdrawal of Google Glass might suggest this is a dead-end, but a widely available device that seamless blends of pixels and real-world surrounding, while boasting advanced hand and voice recognition, has the potential to fundamentally change the way we interact with computers and CG displays and, in doing so, to change all manner of practical and creative tasks. OnSight, the collaboration between NASA's Jet Propulsion Laboratory and the HoloLens team that allows scientists to interactively and collaboratively explore landscapes constructed using data from the Mars Curiosity rover, is certainly impressive and provides the HoloLens with a stamp of credibility.
Picture on-set previz seamlessly blended with the user's view of the live movie set or live footage recording with simultaneous infra-red scene gathering. Or how about asset modelling that allows for sculpting in 'real space' using live objects as points of reference? The 3D scene data gathering abilities of the HoloLens even point to a portable environment scanning device.
Coupled with its on-board camera, there should be no reason why fully digital 3D environments can't be reconstructed from a simple movie set walkabout and a spot of image-based modelling... ditto for quick 3D scans of objects or even people, for that matter. Add a 3D printer and things get really interesting.
Given the myriad uses the hacking community has found for Microsoft's far more basic Kinect motion sensing technology, we can only begin to ponder what innovative uses will be found for a full-featured augmented reality device. Whether the HoloLens will be the one to kickstart the AR revolution does, however, remain to be seen.