FB developing AI that can see, hear, remember whatever you do
New Delhi
15-October-2021
Facebook is developing a new artificial intelligence (AI)-based system that can analyse your lives using first-person videos -- recording what they see, do and hear in order to help you with daily tasks.
Imagine your AR device displaying exactly how to hold the sticks during a drum lesson, guiding you through a recipe, helping you find your lost keys, or recalling memories as holograms that come to life in front of you.
To realise this new AI potential, Facebook AI has announced 'Ego4D' -- a long-term project aimed at solving research challenges in 'egocentric perception' (the perception of direction or position of oneself based on visual information).
"We brought together a consortium of 13 universities and labs across nine countries, who collected more than 2,200 hours of first-person video in the wild, featuring over 700 participants going about their daily lives," the social network said in a statement.
This dramatically increases the scale of egocentric data publicly available to the research community by an order of magnitude, more than 20x greater than any other data set in terms of hours of footage.
"Next-generation AI systems will need to learn from an entirely different kind of data -- videos that show the world from the center of the action, rather than the sidelines," said Kristen Grauman, lead research scientist at Facebook.
In collaboration with the consortium and Facebook Reality Labs Research (FRL Research), Facebook AI has developed five benchmark challenges centred on first-person visual experience that will spur advancements toward real-world applications for future AI assistants.
Ego4D's five benchmarks are episodic memory, forecasting, hand and object manipulation, audio-visual 'diarization' and social interaction.
"These benchmarks will catalyse research on the building blocks necessary to develop smarter AI assistants that can understand and interact not just in the real world but also in the metaverse, where physical reality, AR, and VR all come together in a single space," Facebook elaborated.
The data sets will be publicly available in November this year for researchers who sign Ego4D's data use agreement.
As a supplement to this work, researchers from FRL used Vuzix Blade smart glasses to collect an additional 400 hours of first-person video data in staged environments. This data will be released as well.
While it's easy for people to relate to both first- and third-person perspectives, AI today doesn't share that level of understanding.
"For AI systems to interact with the world the way we do, the AI field needs to evolve to an entirely new paradigm of first-person perception," Grauman said. "That means teaching AI to understand daily life activities through human eyes in the context of real-time motion, interaction, and multi-sensory observations - IANS
More Headlines
CPI(M) Reels from Internal Conflict as EP Jayarajan Meets BJP Leader, Meeting Called on Monday
Brand Owners Eye Chess Grandmaster Gukesh and Peers for Endorsement Deals
Gupta Brothers Propel Boult to Rs 750 Crore Turnover, Aim Rs 1,000 Crore This Fiscal
Supreme Court to Rule on Mandatory EVM-VVPAT Vote Verification on Friday
PM Modi Ups Ante Against Congress, Says Rajiv Gandhi Abolished Inheritance Law To 'Save Own Interests'
CPI(M) Reels from Internal Conflict as EP Jayarajan Meets BJP Leader, Meeting Called on Monday
Brand Owners Eye Chess Grandmaster Gukesh and Peers for Endorsement Deals
Gupta Brothers Propel Boult to Rs 750 Crore Turnover, Aim Rs 1,000 Crore This Fiscal
Supreme Court to Rule on Mandatory EVM-VVPAT Vote Verification on Friday
PM Modi Ups Ante Against Congress, Says Rajiv Gandhi Abolished Inheritance Law To 'Save Own Interests'