AR will mean dystopia if we don’t act today

The martial arts actor Jet Li turned down a role in the Matrix and has been invisible on our screens because he does not want his fighting moves 3D-captured and owned by someone else. Soon everyone will be wearing 3D-capable cameras to support augmented reality (often referred to as mixed reality) applications. Everyone will have to deal with the sorts of digital-capture issues across every part of our life that Jet Li avoided in key roles and musicians have struggled to deal with since Napster. AR means anyone can rip, mix and burn reality itself.

Tim Cook has warned the industry about “the data industrial complex” and advocated for privacy as a human right. It doesn’t take too much thinking about where some parts of the tech industry are headed to see AR ushering in a dystopian future where we are bombarded with unwelcome visual distractions, and our every eye movement and emotional reaction is tracked for ad targeting. But as Tim Cook also said, “it doesn’t have to be creepy.” The industry has made data-capture mistakes while building today’s tech platforms, and it shouldn’t repeat them.

Dystopia is easy for us to imagine, as humans are hard-wired for loss aversion. This hard-wiring refers to people’s tendency to prefer avoiding a loss versus an equal win. It’s better to avoid losing $5 than to find $5. It’s an evolutionary survival mechanism that made us hyper-alert for threats. The loss of being eaten by a tiger was more impactful than the gain of finding some food to eat. When it comes to thinking about the future, we instinctively overreact to the downside risk and underappreciate the upside benefits.

How can we get a sense of what AR will mean in our everyday lives, that is (ironically) based in reality?

When we look at the tech stack enabling AR, it’s important to note there is now a new type of data being captured, unique to AR. It’s the computer vision-generated, machine-readable 3D map of the world. AR systems use it to synchronize or localize themselves in 3D space (and with each other). The operating system services based on this data are referred to as the “AR Cloud.” This data has never been captured at scale before, and the AR Cloud is 100 percent necessary for AR experiences to work at all, at scale.

Fundamental capabilities such as persistence, multi-user and occlusions outdoor all need it. Imagine a super version of Google Earth, but machines instead of people use it. This data set is entirely separate to the content and user data used by AR apps (e.g. login account details, user analytics, 3D assets, etc.).

The AR Cloud services are often thought of as just being a “point cloud,” which leads people to imagine simplistic solutions to manage this data. This data actually has potentially many layers, all of them providing varying degrees of usefulness to different use cases. The term “point” is just a shorthand way of referring to a concept, a 3D point in space. The data format for how that point is selected and described is unique to every state-of-the-art AR system.

The critical thing to note is that for an AR system to work best, the computer vision algorithms are tied so tightly to the data that they effectively become the same thing. Apple’s ARKit algorithms wouldn’t work with Google’s ARCore data even if Google gave them access. Same for HoloLens, Magic Leap and all the startups in the space. The performance of open-source mapping solutions are generations behind leading commercial systems.

So we’ve established that these “AR Clouds” will remain proprietary for some time, but exactly what data is in there, and should I be worried that it is being collected?

The list of data that could be saved is long. At a minimum, it’s the computer vision (SLAM) map data, but it could also include a wireframe 3D model, a photo-realistic 3D model and even real-time updates of your “pose” (exactly where you are and what you are looking at), plus much more. Just with pose alone, think about the implications on retail given the ability to track foot traffic to provide data on the best merchandise placement or best locations for ads in store (and at home).

The lower layers of this stack are only useful to machines, but as you add more layers on top, it quickly starts to become very private. Take, for example, a photo-realistic 3D model of my kid’s bedroom captured just by a visitor walking down the hall and glancing in while wearing AR glasses.

There’s no single silver bullet to solving these problems. Not only are there many challenges, but there are also many types of challenges to be solved.

Tech problems that are solved and need to be applied

Much of the AR Cloud data is just regular data. It should be managed the way all cloud data should be managed. Good passwords, good security, backups, etc. GDPR should be implemented. In fact, regulation might be the only way to force good behavior, as major platforms have shown little willingness to regulate themselves. Europe is leading the way here; China is a whole different story.

A couple of interesting aspects to AR data are:

• Similar to Maps or Streetview, how “fresh” should the data be, and how much historical data should be saved. Do we need to save a map with where your couch was positioned last week? What scale or resolution should be saved. There’s little value in a cm-scale model of the world, except for a map of the area right around you.
• The biggest aspect that is difficult but doable is no personally identifying information leaves the phone. This is equivalent to the image data that your phone processes before you press the shutter and upload it. Users should know what is being uploaded and why it is OK to capture it. Anything that is personally identifying (e.g. the color texture of a 3D scan) should always be opt-in and carefully explained how it is being used. Homomorphic transformations should be applied to all data that leaves the device, to remove anything human readable or identifiable, and yet still leave the data in a state that algorithms can interpret for very specific relocalization functionality (when run on the device).
• There’s also the problem of “private clouds” in that a corporate campus might want a private and accurate AR cloud for its employees. This can easily be hosted on a private server. The tricky part is if a member of the public walks around the site wearing AR glasses, a new model (possibly saved on another vendor’s platform) will be captured.

Tech challenges the AR industry still needs to solve

There are some problems we know about, but we don’t know how to solve yet. Examples are:

• Segmenting rooms: You could capture a model of your house, but one side of an inner apartment wall is your apartment while the other side is someone else’s apartment. Most privacy methods to date have relied on something like a private radius around your GPS location, but AR will need more precise ways to detect what is “your space.”
• Identifying rights to a space is a massive challenge. Fortunately, social contracts and existing laws are in place for most of these problems, as AR Cloud data is pretty much the same as recording video. There are public spaces, semi-public (a building lobby), semi-private (my living room) and private (my bedroom). The trick is getting the AR devices to know who you are and what it should capture (e.g. my glasses can capture my house, but yours can’t capture my house).
• Managing the capture of a place from multiple people, and stitching that into a single model and discarding overlapping and redundant data makes ownership of the final model tricky.
• The Web has the concept of a robots.txt file, which a website owner can host on their site, and the web data collection engines (e.g. Google, etc.) agree to only collect the data that the robots.txt file asks them to. Unsurprisingly this can be hard to enforce on the web, where each site has a pretty clear owner. Some agreed type of “robots.txt” for real-world places would be a great (but maybe unrealistic) solution. Like web crawlers, it will be hard to force this on devices, but like with cookies and many ad-tracking technologies, people should at least be able to tell devices what they want and hopefully market forces or future innovations can require platforms to respect it. The really hard aspect of this attractive idea is “whose robots.txt is authoritative for a place.” I shouldn’t be able to create a robots.txt for Central Park in NYC, but I should for my house. How is this to be verified and enforced?

Social contracts need to emerge and be adopted

A big part of solving AR privacy problems will come from developing a social contract that identifies when and where it’s appropriate to use a device. When camera phones were introduced in the early 2000s, there was a mild panic about how they could be misused; for example, cameras used secretly in bathrooms or taking your photos in public without a person’s permission. The OEMs tried to head off that public fear by having the cameras make a “click” sound. Adding that feature helped society adopt the new technology and become accustomed to it pretty quickly. As a result of having the technology in consumers hands, society adopted a social contract — learning when and where it is OK to hold up your phone for a picture and when it is not.

Companies added to this social contract, as well. Sites like Flickr developed policies to manage images of private places and things and how to present them (if at all). Similar social learning took place with Google Glass versus Snap Spectacles. Snap took the learnings from Glass and solved many of those social problems (e.g. they are sunglasses, so we naturally take them off indoors, and they show a clear indicator when recording). This is where the product designers need to be involved to solve the problems for broad adoption.

Challenges the industry cannot predict

AR is a new medium. New mediums come along only every 15 years or so, and no one can predict how they will be used. SMS experts never predicted Twitter and Mobile Mapping experts never predicted Uber. Platform companies, even the best-intentioned *will* make mistakes.

These are not tomorrow’s challenges for future generations or science fiction-based theories. The product development decisions the AR industry is making over the next 12-24 months will play out in the next five years.

This is where AR platform companies are going to have to rely on doing a great job of:

1. Ensuring their business model incentives are aligned with doing the right thing by the people whose data they capture; and
2. Communicating their values and earning the trust of the people whose data they capture. Values need to become an even more explicit dimension of product design. Apple has always done a great job of this. Everyone needs to take it more seriously as tech products become more and more personal.

What should the AR players be doing today to not be creepy?

Here’s what needs to be done at a high level, which pioneers in AR believe is the minimum:

10. Personal Data Never Leaves Device, Opt In Only: No personally identifying data required for the service to work leaves the device. Give users the option to opt in to sharing additional personal data if they choose for better apps feedback. Personal data does NOT have to leave the device in order for the tech to work; anyone arguing otherwise doesn’t have the technical skills and shouldn’t be building AR platforms.

11. Encrypted IDs: Coarse Location IDs (e.g. Wi-Fi network name) are encrypted on the device, and it’s not possible to tell a location from the GPS coordinates of a specific SLAM map file, beyond generalities.

12. Data Describing Locations Only Accessible When Physically at Location: An app can’t access the data describing a physical location unless you are physically in that location. That helps by relying on the social contract of having physical permission to be there, and if you can physically see the scene with your eyes, then the platform can be confident that it’s OK to let you access the computer vision data describing what a scene looks like.

13. Machine-Readable Data Only: The data that does leave the phone is only able to be interpreted by proprietary homomorphic algorithms. No known science should be able to reverse engineer this data into anything human readable.

14. App Developers Host User Data On Their Servers, Not The Platforms: App developers, not the AR platform company, host the application and end user-specific data re: usernames, logins, application state, etc. on their servers. The AR Cloud platform should only manage a digital replica of reality. The AR Cloud platform can’t abuse an app user’s data because they never touch or see it.

15. Business Models Pay for Use Versus Selling Data: A business model based on developers or end users paying for what they use ensures the platform won’t be tempted to collect more than necessary and on-sell it. Don’t create financial incentives to collect extra data to sell to third parties.

16. Privacy Values on Day One: Publish your values around privacy, not just your policies, and ask to be held accountable to them. There are many unknowns, and people need to trust the platform to do the right thing when mistakes are made. Values-driven companies like Mozilla or Apple will have a trust advantage over other platforms whose values we don’t know.

17. User and Developer Ownership and Control: Figure out how to give end users and app developers appropriate levels of ownership and control over data that originates from their device. This is complicated. The goal (we’re not there yet) should be to support GDPR standards globally.

18. Constant Transparency and Education: Work to educate the market and be as transparent as possible about policies and what is known and unknown, and seek feedback on where people feel “the line” should be in all the new gray areas. Be clear on all aspects of the bargain that users enter into when trading some data for a benefit.

19. Informed Consent, Always: Make a sincere attempt at informed consent with regard to data capture (triply so if the company has an ad-based business model). This goes beyond an EULA, and IMO should be in plain English and include diagrams. Even then, it’s impossible for end users to understand the full potential.

Even apart from the creep factor, remember there’s always the chance that a hack or a government agency legally accesses the data captured by the platform. You can’t expose what you don’t collect, and it doesn’t need to be collected. That way people accessing any exposed data can’t tell precisely where an individual map file refers to (the end user encrypts it, the platform doesn’t need the keys), and even if they did, the data describing the location in detail can’t be interpreted.

Blockchain is not a panacea for these problems — specifically as applied to the foundational AR Cloud SLAM data sets. The data is proprietary and centralized, and if managed professionally, the data is secure and the right people have the access they need. There’s no value to the end user from blockchain that we can find. However, I believe there is value to AR content creators, in the same way that blockchain brings value to any content created for mobile and/or web. There’s nothing inherently special about AR content (apart from a more precise location ID) that makes it different.

For anyone interested, the Immersive Web working group at W3C and Mozilla are starting to dig further into the various risks and mitigations.

Where should we put our hope?

This is a tough question. AR startups need to make money to survive, and as Facebook has shown, it was a good business model to persuade consumers to click OK and let the platform collect everything. Advertising as a business model creates inherently misaligned incentives with regard to data capture. On the other hand, there are plenty of examples where capturing data makes the product better (e.g. Waze or Google search).

Education and market pressure will help, as will (possibly necessary) privacy regulation. Beyond that we will act in accordance with the social contracts we adopt with each other re: appropriate use.

The two key takeaways are that AR makes it possible to capture everything, and that the platform doesn’t need to capture everything in order to deliver a great AR UX.

If you draw a parallel with Google, in that web crawling was trying to figure out what computers should be allowed to read, AR is widely distributing computer vision, and we need to figure out what computers should be allowed to see.

The good news is that the AR industry can avoid the creepy aspects of today’s data collection methods without hindering innovation. The public is aware of the impact of these decisions and they are choosing which applications they will use based on these issues. Companies like Apple are taking a stand on privacy. And most encouragingly, every AR industry leader I know is enthusiastically engaged in public and private discussions to try to understand and address the realities of meeting the challenge.