So many articles these days start with “Covid lockdowns greatly accelerated the development of tech and its adoption for remote and virtual work and meetings, blah blah.” We start, though, with a true world-first. With live performances an impossibility, many musicians – and their supply chain ecosystems – had to be resourceful and innovative. Gaming has been a sector leader for using virtual reality technology, but it made headlines in the music world when top UK band Massive Attack were special guests in a VR music festival created to take place inside a Minecraft game. This article takes a look at other recent AR/VR technology trends and development.
Being far more than a 2D performance streamed via a video conferencing platform or Facebook Live, the Minecraft virtual concert was so popular the servers crashed and it was rescheduled for three weeks later. For many people it is perhaps the simplest single example to convey the benefits and opportunities of harnessing augmented and virtual reality technology to deliver more than day-to-day experiences. Also, the festival built a community, bringing together a crowd of people who appear to have bonded through the shared experience. It developed as a collaborative effort, and a second concert is in the pipeline.
Market value projection
What size of market potential is everyone looking at? AR and VR first came to many people’s attention when the Oculus Rift creators generated $2.5 million of orders through crowdfunding in 2012. Four years later, Facebook bought the business for $2 billion. There are several projections of how far the estimated global market value of $22.9 billion near the end of 2019 will grow. Statista reckons $300 billion by the close of 2024. Evolve Business Intelligence reckons it will be a market worth $530 bn a year by 2027.
Do a search and you will find numerous other predictions. Though with the big tech firms of Apple, Microsoft, Facebook and Alphabet/Google all vying to develop the market’s “next big thing,” who can really tell what the pace of developments and rate of user uptake will be?
AR and VR component technology providers
Facebook, Microsoft, Apple and the rest are creating apps that enable customers to enjoy the benefits of augmented, virtual, mixed and extended reality technology. Those apps will be the bridge that links users to the tech. Let’s look at the nature of the various component technologies, and who are the companies that are actually creating the tech?
LiDAR (Laser Diodes for Automotive & Industrial)
Lidar is an acronym for Laser Imaging, Detection And Ranging. It uses a reflected laser to locate objects in space and determine their distance, speed and direction. It is a widely used sensor for autonomous vehicles to see surrounding traffic, geographers and archeologists to map the ground, and smartphones and tablets to provide vision for augmented reality apps.
Haptic technology, also known as 3D touch, is any technology that can create a sensation of touch by applying forces, vibrations, or motions to the user.
- Ultraleap uses ultrasound waves to create the experience of touching something in mid-air.
- Tapwithus makes a hand-wearable device that we can use to operate our devices through tapping and air gestures.
- Motion Engine provides highly accurate dead reckoning software solutions for intelligent navigation systems in indoor robots, including robot cleaners and Autonomous Mobile Robots (AMR) used in industrial or commercial settings.
- Motion Gestures teaches devices with motion, touch and vision monitors to recognise and respond to a range of human gestures for interactions without a need to physically touch them. Non-touch tech will be more important as we emerge from Covid.
In a distinct category of its own, away from spatial tech, motion capture (or mo-cap / mocap) digitally records people’s movements. It is used in entertainment, sports, medical applications, ergonomics and robotics. In film-making and game development it is the tech that enables animated/CGI characters to follow a wired-up actor’s actions.
The use of cameras and infrared LEDs at each end of a telephone connection allow hand movements to be replicated in a different location. A current UK tv commercial shows an actor up a mountain being shaved by a robot using a cut throat razor, and the robot is operated by a barber 250 miles away.
Appearance embedding interpolation
This technology is for creating 3D images of buildings or other landmarks for insertion in to AR/VR apps and platforms.
Google researchers have developed a NeRF-W system that uses a neural network with crowdsourced photos of a location to convincingly replicate landmarks in 3D. They can be used in games, mapping, or for any other AR and VR technology application. The light and shadow of images recorded at different times of day allow a realistic display of a passage of time, and they collect images recorded to match all types of weather conditions.
Google reverses the capture technology to collect images. Mobile/cell users of Google Maps are asked to scan buildings and other landmarks when they seek directions. GPS can pinpoint their location, informing Google where the images were taken from. Google then provides directions to where the person wants to go, and the service generates a stream of crowdsourced incoming images
Intel researchers are advancing work in generating synthesized versions of real world locations, again using multiple photographs coupled with tech to ‘fill in the gaps’ of uncaptured angles. Intel’s system appears to outperform numerous alternatives on pixel-level sharpness and temporal smoothness, though it doesn’t offer variable lighting capabilities. They are restricted by not using crowdsourced photos to be able to recreate as many real-world locations.
Headsets and smart glasses
Google made headlines when it launched a cardboard headset at a starting price of $15. Google was very enthusiastic about VR for a few years, developed higher tech products, though started shutting down its operations in 2019. The cardboard set was finally taken off the market in March 2021.
Facebook has similarly discontinued the iconic Oculus Rift. Though as a different stepping stone to its continued AR/VR technology ambitions, and in partnership with Ray-Ban, in September 2021 Facebook unveiled smart sunglasses that record video.
By contrast, at the cutting edge of headset tech, the Taiwanese computer manufacturer HTC has just announced in October 2021 its latest VR headset called the Vive Flow. It weighs just 189g and is worn like glasses. It has dual-built-in speakers, dual microphones with echo and noise cancellation, support for Bluetooth audio, 64GB of memory on-board and 4GB of RAM. It pairs with a smartphone and is controlled by an app. It is available now for pre-order for $499 with shipments estimated to arrive in November.
AR has been a staple talking point for Apple in recent years, and anecdotal evidence suggests their next wearable may be a set of augmented reality glasses. Called “Apple Glass,” users will interact with information and digital objects via transparent glass worn over the eyes. They can be fitted with prescription lenses, and may have a starting price in the region of $500.
Nreal, the China-based maker of AR glasses and mixed reality (MR) equipment and technology, has very recently raised $100M in a Series C funding round. The business is now valued at $700M and plans to go public within five years. It currently operates in South Korea, Japan, Germany and Spain, and has 8,000 developers on its platform. Its products sometimes come in a higher spec and more expensive version for enterprise customers. They help to improve employees’ productivity.
Will universal or disparate AR and VR systems emerge, and could one big tech firm become a clear market leader?
While the commercial and societal benefits of the technology are clear, a plethora of incompatible tools and systems have made it more difficult for both service providers and users to decide on an entry point to invest.
Let’s start a look at an AR/VR technology that’s a short step from commonplace current reality – digital mapping. Several companies are bringing products to the marketplace.
In 2019, UK-based Scape displayed a city-scale AR mapping solution that would let mobile devices tap into 3D maps and cloud-stored digital assets ranging from AR guidance to advertisements — ambitious technology. Facebook acquired Scape in February 2020.
Finnish software company Immersal created a free Android and iOS spatial mapping and visual positioning app that captures crowdsourced images of real-world locations. As phone users walk through them it tags each image with GPS and orientation data, creates maps with AR recognition points from multiple images, then joins the small maps together to represent a larger area. In July 2021, Immersal was bought by Hexagon, a company in neighbouring Sweden developing sensors and software for the autonomous vehicle era.
In September 2020, Apple announced new ways in which iPhone users in London, Los Angeles, New York City, and San Francisco can explore major cities in 3D. The biggest ever update of Apple Maps included immersive walking directions shown in augmented reality. Users will be able to scan nearby buildings and Maps will generate a highly accurate position and deliver detailed directions to where the user wants to go. Later on in what’s left of 2021 it will be available in Philadelphia, San Diego, and Washington, D.C. Support for additional cities, including Montreal, Toronto, and Vancouver, will be available in 2022.
Over at Facebook, CTO Andrew Bosworth is at the forefront of the social networking giant’s efforts to reinvent itself and divert attention away from a string of scandals. It is developing augmented reality goggles that project holographic images into the wearer’s field of vision, and wristbands that read nerve signals from the brain.
Zuckerberg sees these as crucial steps toward owning the “metaverse.” A Facebook blog defines the metaverse as “virtual spaces where you can create and explore with other people” that you’re not physically with, spread out over a variety of products and services. Described as an “embodied internet,” it will allow us to put on headsets and send avatars of ourselves anywhere to work, present, collaborate, buy or sell, study, be entertained or do the entertaining, or just socialise in a virtual reality setting, all while we physically remain where we are.
Facebook will work with policymakers, researchers, and industry partners to build the metaverse responsibly, and announced a $50M fund to last two years. No single company will own and run the metaverse, it will be operated by many different players in a decentralized way. Accordingly, there must be collaborative agreements on operating systems between highly competitive rivals.
They have got as far as launching Horizon Workrooms meeting service. It is an app that we won’t just use as spectators, we will ‘get inside it’ and become part of it. People gather remotely wearing headsets and meet in a virtual workspace that gives users numerous options, from switching conference rooms to presentation settings, as well as creating their own avatars. Users can employ hand gestures to interact with their colleagues, and move about the virtual space or sit around a table as though in an in-person meeting.
Microsoft appears to be developing different services to operate for B2B and B2C markets. For business users, they talk about the great benefits they will enjoy from a convergence of several technology developments. Some are already creating an alternative reality that delivers previously unimaginable experiences. We’ve already mentioned virtual versions of building designs to test in simulations and rectify future issues at the planning stage, and surgeons who can go through intricate procedures before making the first incision.
Microsoft are working with clients and AR and VR technology to pressure test delivery systems to virtual stores so that they never run out of anything; virtual production lines to learn how to cope with any supply disruption to refine contingency planning; and testing energy supplies under unusual combinations of climate or customer demands will prevent power failures (ideally).
At a more individual person level, Microsoft unveiled its Microsoft Mesh in March 2021. This is their new mixed-reality platform, powered by Azure, that allows people in different physical locations to join collaborative and shared holographic experiences on many kinds of devices. People will initially be able to express themselves as avatars in shared virtual experiences, and over time use holoportation to project themselves as their most lifelike, photorealistic selves.
It will allow geographically distributed teams to have more collaborative meetings, conduct virtual design sessions, assist others, learn together and host virtual social meetups. Microsoft Mesh’s open standards will give developers freedom to build solutions that work across many different devices: HoloLens 2, a range of virtual reality headsets, smartphones, tablets and PCs.
AR and VR technology, 3D imaging and mapping are at the convergence point of autonomous motoring and commercial drone services in the ‘real world’, and the departure point for the metaverse world. That will in itself spawn a raft of new businesses in an AR/VR ecosystem. As an example, what will your avatar wear in your important 3D business meetings? Virtual clothing can be created and marketed as NFTs, with payment in crypto. We have crypto fashion!
We’d really like to hear what other ideas you’ve heard of, or maybe come up with yourself.