JavaScript is not available.

Jun 28, 2021

My Ph.D. Thesis on Building Intelligent Autonomous Navigation Agents is now available at: arxiv.org/abs/2106.13415 Thesis Defense recording and slides: youtu.be/rJ7tGT5cHtU Thanks

for being an amazing advisor! And I am very excited to join Facebook AI Research!

GIF

358

Dec 9, 2022

For more details PDF: arxiv.org/abs/2212.00922 Webpage: theophilegervet.github.io/projects/real- Presentation: youtu.be/zZ6nlkTZVds Led by

, w.

(14/N)

Analysis: Why does modular learning work even better in reality than simulation? Because failure modes in sim and reality have almost no overlap! Takeaway: Navigation methods must be evaluated in the real world. (13/N)

The SemExp policy of the modular learning approach takes a semantic map as input, while the end-to-end policy directly operates on the RGB-D frames. The semantic map space is invariant between sim and reality, while the image space exhibits a large domain gap. Example👇 (12/N)

GIF

Analysis: Why does Modular Learning transfer well while End-to-end does not? To answer this question, we reconstructed one real-world home in simulation and conducted experiments with identical episodes in sim and reality. (11/N)

GIF

Key Result 3: End-to-end Learning Fails to Transfer The end-to-end policy collides often, revisits the same places, and even fails to stop in front of goal objects when they are in sight. Examples👇 (10/N)

GIF

Key Result 2: Modular Learning explores more efficiently than Classical methods This is because Modular Learning is goal-oriented, it learns semantic exploration priors to find the goal object efficiently. Classical Frontier exploration is goal-agnostic. Example👇 (9/N)

GIF

Key Result 1: Modular Learning is Reliable We find that modular learning is very reliable on a robot, with a 90% success rate. Examples👇 (8/N)

GIF

Example👇: All policies start at the same location and need to find a couch. Left: Modular Learning first reaches the couch successfully. Center: End-to-end learning fails after colliding too many times. Right: Classical policy succeeds after a detour through the kitchen. (7/N)

GIF

Results: In simulation, all approaches perform comparably, at around 80% success rate. But in the real world, End-to-end learning drops to 23% success rate while Modular learning achieves 90% success! (6/N)

We conduct the first large-scale real-world empirical evaluation for semantic visual navigation: - 3 methods - 6 unseen homes - 6 goal object categories All methods are trained in simulation and directly transferred to the real world without any fine-tuning. (5/N)

GIF

We compare Modular Learning with 2 other classes of methods: - End-to-end Learning (IL + RL) - Classical (Frontier Exploration) (4/N)

GIF

Humans can do this effortlessly: when looking for a glass of water at a friend's house we're visiting for the first time, we can easily find the kitchen without going to bedrooms or storage closets. But teaching this kind of common sense to robots is challenging. (3/N)

Task: The robot starts in a completely unseen environment and is asked to find an object, let's say a toilet. (2/N)

Robot visual navigation in unseen homes is hard: end-to-end RL works well in sim but gets only 23% real-world success. Today, in the first real-world empirical study of visual navigation, we show Modular Learning achieves 90% success in unseen homes! theophilegervet.github.io/projects/real- 1/N

GIF

206

We are announcing Habitat-Matterport 3D Semantics Dataset! 216 real-world 3D spaces with dense semantic annotations available to download now: aihabitat.org/datasets/hm3d-

Quote Tweet

Meta AI

@MetaAI

Oct 12, 2022

(1/3) Today we’re releasing the Habitat-Matterport 3D Semantics dataset, the largest public dataset of real-world 3D spaces with dense semantic annotations. HM3D-Sem is free and available to use with FAIR's Habitat simulator: bit.ly/3EymX8x

Show this thread

GIF

Mustafa Mukadam

@mhmukadam

Jul 20, 2022

Thrilled to release Theseus, a library to build custom nonlinear optimization layers in PyTorch. sites.google.com/view/theseus-ai Nonlinear least squares is at the heart of many robotics and vision problems. Now we can add such structures in our end-to-end architectures.

241

1,362

1/4 I was watching

interview w/t

- it reminded me of a couple fun stories about Geoff and how he got me into deep learning. Back in 2005, I was not thinking about PhD & had a good career as ML engineer. One morning, on my way to work I bumped into Geoff.

622

Co-organizing the 3rd Embodied AI Workshop at #CVPR2022. We have challenge results, posters, and an amazing set of speakers:

@carolina_parada

@RoozbehMottaghi

@DhruvBatraDB

, Katerina Fragkiadaki,

@drfeifei

& Jitendra Malik! Check out embodied-ai.org for full schedule

GitHub - srama2512/PONI: PONI: Potential Functions for ObjectGoal Navigation with Interaction-free...

Jun 18, 2022

PDF: arxiv.org/abs/2201.10029 Webpage with oral talk and demo videos: vision.cs.utexas.edu/projects/poni/ Blog: ai.facebook.com/blog/new-resea Code and pre-trained models: github.com/srama2512/PONI Led by

@srama2512

, w. Ziad Al-Halah, Jitendra Malik, Kristen Grauman 4/4

github.com

PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning. CVPR 2022 (Oral). - GitHub - srama2512/PONI: PONI: Potential Functions for ObjectGoal Navigation with Interaction...

The Potential function prediction network can be trained offline with just top-down map data without requiring any simulation or environment interaction. This makes training PONI easy and efficient. 3/4

Our key insight is that “where to look?” can be treated purely as a perception problem and learned without interactions. Our model predicts two complementary potential functions conditioned on a semantic map and uses them to decide where to look for an unseen object. 2/4

New #CVPR2022 Oral paper on PONI: Potential functions for Object goal Navigation with Interaction-free learning. PONI achieves SOTA on ObjectNav without using any simulation just using top-down map data! Also 1,600x reduction in training time! Web: vision.cs.utexas.edu/projects/poni/ 1/4

GIF

Santhosh Kumar Ramakrishnan

@srama2512

Jun 17, 2022

Our work "PONI: Potential Functions for ObjectGoal Navigation with Interaction-free Learning" was accepted as an Oral at CVPR 2022. Code and models are available. Code: github.com/srama2512/PONI Website: vision.cs.utexas.edu/projects/poni/ Blog:

ai.facebook.com

New advances in embodied AI research to better understand and navigate physical and virtual worlds.

Luca Weihs

@LucaWeihs

Jun 16, 2022

The 3rd Embodied-AI Workshop is being held

@CVPR

in just 3 days (June 19th, starting at 9am) we have 6 amazing invited speakers, a poster session, multiple Q&A panels, and lots of challenge results! See our website embodied-ai.org for our day-of timeline and more details

Meta AI

@MetaAI

Jun 16, 2022

And we’re continuing to push for efficiency with a novel formulation for object-goal navigation tasks using PONI that obtains state-of-the-art results while achieving a 1,600x reduction in training time compared with prior methods. arxiv.org/abs/2201.10029

New advances in embodied AI research to better understand and navigate physical and virtual worlds.

Meta AI

@MetaAI

Jun 16, 2022

Today, we're announcing new advances in embodied AI research to better understand & navigate physical and virtual worlds.

ai.facebook.com

Georgia Gkioxari

@georgiagkioxari

Jun 15, 2022

Object recognition should happen in 3D because the world is 3D. On the heels of Mesh R-CNN, we build a model that detects all objects, predicts their 3D position in space (layout) and their 3D shape *from a single image of a novel complex scene*. At CVPR 2022. (1/n)

0:09

35.5K views

136

786

Guillaume Lample

@GuillaumeLample

May 24, 2022

Excited to release our latest work: arxiv.org/abs/2205.11491 We present a new algorithm, HyperTree Proof Search (HTPS) inspired by the recent success of AlphaZero. Our model is able to prove mathematical theorems in a fully automated way and significantly outperforms the SOTA. 1/n

203

795

Yilun Du

@du_yilun

May 5, 2022

Today (5/5) 4pm EDT, Ruslan Salakhutdinov

from CMU

@SCSatCMU

@mldcmu

will be giving a talk at the Embodied Intelligence Seminar

@MIT_CSAIL

on building autonomous embodied agents! Live-streaming from: youtube.com/channel/UCnXGb…, co-hosted with

@EpisodeYang

AI Habitat

@ai_habitat

Mar 18, 2022

We are excited to announce the Habitat Challenge 2022 on ObjectNav for the #CVPR22 Embodied AI workshop: aihabitat.org/challenge/2022/ We are introducing the brand new Habitat-Matterport 3D Semantics dataset in the challenge: aihabitat.org/datasets/hm3d-

GIF

123

So Yeon (Tiffany) Min

@SoYeonTiffMin

Feb 21, 2022

The code for FILM, my ICLR 2022 paper with

@ybisk

, and

, is now publicly released as the starting code for the ALFRED challenge held at the #CVPR2022 EAI 2022 Workshop! Excited to see people push the SotA even further github.com/soyeonm/FILM

Quote Tweet

Luca Weihs

@LucaWeihs

Feb 14, 2022

Happy to announce that the 3rd Embodied AI Workshop is happening on June 19th at #CVPR2022. Lots of exciting speakers and 13 (!!!) challenges. Check out our website (embodied-ai.org), dive into a challenge, and save the date!

Show this thread

102

Soumith Chintala

@soumithchintala

Jan 19, 2022

It’s been 5 years since we launched

@pytorch

. It’s much bigger than we expected -- usage, contributors, funding. We’re blessed with success, but not perfect. A thread (mirrored at soumith.ch/posts/2022/01/ ) about some of the interesting decisions and pivots we’ve had to make 👇

330

2,563

Mustafa Mukadam

@mhmukadam

Dec 19, 2021

Our recent work LEO shows how to differentiate through an optimizer via sampling instead of unrolling. We apply this to learn observation models end-to-end, for instance, tactile sensor models in object state estimation during pushing.

Quote Tweet

Russ Salakhutdinov

Dec 18, 2021

An Energy-based Perspective on Learning Observation Models – Machine Learning Blog | ML@CMU | Carnegie Mellon University blog.ml.cmu.edu/2021/12/17/lea

2/3:

: Devendra Chaplot: Building Intelligent Autonomous Navigation Agents devendrachaplot.github.io

@shrimai_

Shrimai Prabhumoye: Controllable Text Generation cs.cmu.edu/~sprabhum/ Emilio Parisotto: Meta Reinforcement Learning through Memory cs.cmu.edu/~eparisot/

Russ Salakhutdinov

Dec 18, 2021

1/3: 2021 has been an incredible year: 5 of my PhD students (some co-advised) have graduated this year. It's a privilege to have worked with such talented people! Devendra Chaplot, Lisa Lee, Emilio Parisotto, Shrimai Prabhumoye & Hubert Tsai Check out their theses/papers here:

179

Saurabh Gupta

@_saurabhg

Dec 13, 2021

Checkout our new work at DeepRL / OfflineRL Workshops at #NeurIPS: Learning Value Functions From Undirected State-Only Experience. We study how we can derive useful behavior from undirected state-only (i.e. w/o action labels) data. Website: matthewchang.github.io/latent_action_ See 🧵below.

Victoria Dean

@vdean314

Dec 9, 2021

How should agents explore and interact with the world in order to quickly learn tasks? Learn how in our

@NeurIPSConf

paper “Interesting Object, Curious Agent” - oral session 5, tomorrow at 7pm ET! neurips.cc/virtual/2021/o With Simone Parisi,

@pathak2206

, and Abhinav Gupta

GIF

If you are attending #NeurIPS2021, visit our poster on Wednesday, Dec 8 at 4.30pm PST! neurips.cc/virtual/2021/p

Quote Tweet