Instagram adds automatic alternative text powered by object recognition

Instagram announced Wednesday, in a short article on its blog, that this would make the application more accessible to the visually impaired.

"With over 285 million visually impaired people around the world, we know that many people could benefit from a more accessible Instagram," wrote the company before describing the new feature: alternative description of the photos in the stream and the Explore sections of the application, which would allow a screen reader software to automatically describe pictures aloud to users.

Users can manually add descriptions to advanced settings for their messages, which is a small adjustment from what many accessibility-conscious users were. do already – put more detailed or literal descriptions of their photos directly in the picture legend. More importantly, Instagram will use its object recognition features to automatically generate image descriptions that the poster does not provide.

The responses to the addition were generally warm, with the company receiving Thank you visually impaired users in response to the tweet announcement. But the change has also been widely discussed on Twitter as a simple start, especially in the hashtag # a11y – a shortcut for an online accessibility movement called Project A11Y, which shares open source accessibility software information on GitHub and runs a blog on design. and the internet community.

The Royal Netherlands Institute for the Blind, a UK non-profit organization, congratulated Instagram on its congratulations for its commitment to accessibility, asked that the functionality be extended to Stories – it's important, given that 40% of the 1 billion users of the platform publish their article every day. Louise Taylor, software engineer at the BBC, tweeted that the feature might be a little more usable "if the option of adding alternative text was easier to find than clicking a button with a very questionable contrast".

Others were slightly confused as to why the addition took so long, reacting with a friendly "finally."

Another thing that is a little strange here: Instagram's object recognition capabilities apparently would have allowed this feature to exist for a while.

The story of how Instagram polishes its algorithm, identifying the most relevant or interesting content for its users, has always been a little opaque, but we know that object recognition is at least part of the puzzle . Facebook uses Instagram's many public photos to form its machine learning models, as technology director Mike Schroepfer revealed at the company's annual developer conference in May. He did not specify how long the company had done it, but he had already processed 3.5 billion images from Instagram and said the company had "produced results at the cutting edge of technology improved by 1 to 2% than any other ImageNet test system. (ImageNet is the most used online visual database for object recognition software testing.)

Object recognition is obviously a priority for Instagram and its parent company. This will not be useless for their advertising business, and their moderation protocols clearly require that it be very good.

For its part, Facebook has been offering since April 2016 an alternative text generated by artificial intelligence as an accessibility feature and allows users to manually enter text for a certain time. Even putting aside the challenges of AI, manual entry options for alternative texts are more or less standard on the Web, and Twitter added them in 2016. If Instagram took so long to paste a new text entry box on his application, it's nice It's hard to believe that it's because it was technically difficult, and not afterwards.

Instagram

Reaching a comment, an Instagram spokesperson said the accessibility features were "a project in progress for some time" and they could not be more precise on the timeline.

"The challenge was to ensure that we could provide meaningful descriptions on a large scale," said the spokesman. Essentially: the system can not recognize the content of the photo, it is difficult to determine what is important about it and to establish the context.

I was also directed to a research paper posted on Facebook last February on Facebook, explaining the issues solved when creating the automatic replacement text feature for Newsfeed, and how this feature had been received during of his first 10 months of use:

… It's a pretty easy task for a person to choose the most interesting part of a photo, while it can be quite difficult even for the most intelligent AI. The social context and the good amount of comments are what would give this service a magical experience, and we hope to make it happen! From our interviews, we found that it can often be worse to provide incorrect information on a photo rather than leaving out things we are not sure about. For example, if the service said that the photo contained a child, it accidentally misidentified a little woman.

Just! However, when I asked why it took two and a half years for the Facebook feature to be optimized for Instagram, the rep simply replied that Instagram was "essentially visual" and that the investments in the "Instagram" feature were "good. accessibility took time. This is an important change that deserves to be celebrated; it's also a change that took two and a half years.