Alibaba made a smart screen to help blind people shop and it costs next to nothing

Just a few years ago, Li Mengqi could not have imagined shopping on her own. Someone needed to always keep her company to say aloud what was in front of her, who’s been blind since birth.

When smartphones with text-to-speech machines for the visually impaired arrived, she immediately bought an iPhone. “Though it was expensive,” Li, a 23-year-old who grew up in a rural village in eastern China’s Zhejiang province, told me. Cheaper smartphone options in China often don’t have good accessibility features.

Screen readers opened a plethora of new opportunity for those with visual impairments. “I felt liberated, no longer having to rely on others,” said Li, who can now shop online, WeChat her friends, and go out alone by following her iPhone compass.

Reading out everything on the screen is helpful, but it can also be overwhelming. Digital readers don’t decipher human thoughts, so when Li gets on apps with busy interfaces, such as an ecommerce platform, she’s bombarded with descriptions before she gets to the thing she wants.

Over the past two years, Alibaba’s $15 billion R&D initiative Damo Academy has been working to improve smartphone experience for the blind. Its latest answer, a joint effort with China’s prestigious Tsinghua University, is a cheap silicone sheet that goes on top of smartphone screens.

Li is among the first one hundred visually impaired or blind users to trial the technology. Nothing stands out about the plastic film – which cost RMB 0.25 or 3.6 American cents each to produce – until one has a closer look. There are three mini buttons on each side. They are sensory-enabled, which means pressing on them triggers certain commands, usually those that are frequently used like “go back” and “confirm”.

“It’s much easier to shop with the sheet on,” said Li. Having button shortcuts removes the risk of misclicking and the need for complex interactions with screens. Powering Smart Touch is human-machine interaction, the same technology that makes voice control devices possible.

“We thought, human-machine interaction can’t just be for sighted people,” Chen Zhao, research director at Damo Academy told TechCrunch. “Besides voice, touch is also very important to the blind, so we decided to develop a touch feature.”

Smart Touch isn’t just for fingers. It also works when users hold their phones up to the ears. This lets them listen to text quickly in public without having to blast it out through speakers or headphones. Early trials of ear touch show a 50 percent reduction in time needed to complete tasks like taking calls and online shopping, Alibaba claims.

Emotions also matter. People with visual disabilities tend to be more cautious as they fumble through screens, so Smart Touch takes that into account. For instance, users need to double-click on the silicone button before a command goes through.

At the moment, Smart Touch works only on special editions of Alibaba’s two flagship apps, e-commerce marketplace Taobao and payment affiliate Alipay . The buttons automatically take on different functions when users switch between apps.

But Zhao said she wanted to make the technology widely available. Some tinkering with existing apps will make Smart Touch compatible. The smart film requires more testing before it officially rolls out early 2019, so Damo and Tsinghua have been recruiting volunteers like Li for feedback.

“Unlike with regular apps, it’s hard to beta test Smart Touch because the blind population is relatively small,” observed the researcher, but embedding the technology in popular apps could speed up the iteration process.

There’s also the issue with distributing the physical sheets. According to state census, China had around 13 million visually impaired people in 2012. That’s about one in a hundred people. However, they are rarely seen in public, as a post on China’s equivalent of Quora points out.

One oft-cited obstacle is that most roads in China aren’t disability-friendly, even in major cities. (In my city Shenzhen, blind lanes are common but they often get cut off abruptly to make way for a crossing or a bus stop.)

Damo doesn’t plan to monetize the initiative, according to Zhao. She envisions a future where her team could give out the haptic films — which can be mass produced at low costs — for free through Alibaba’s expanding network of brick-and-mortar stores.

Time will tell whether the accessibility scheme is more than public relations fluff. Initiatives around corporate social responsibility have mushroomed in China in recent years. They have come under fire, however, for being transient because many merely pander to the government’s demand (link in Chinese) for corporate ethics overlook long-term impact.

“The technology is ready. It just takes time to test it on different smartphones and bring to users at scale,” said Zhao.