JavaScript is not available.

First look - GPT-3 in Roblox - Emulating the functionality of Google...

How can robots perform a wide variety of novel tasks from natural language? Execited to present Code as Policies - using language models to directly write robot policy code from language instructions. See paper, colabs, blog, and demos at code-as-policies.github.io long

Show this thread

0:20

36.1K views

10

52

Great way to start the year with 2/2 papers accepted at #ICRA23 on language based skill learning and navigation! VLMaps: vlmaps.github.io HULC++: hulc2.cs.uni-freiburg.de Thanks to my wonderful collaborators

@andyzengtweets

,

@huang_chenguang

,

@wolfram_burgard

and Jessica!

GIF

11

57

SayCan is available in Roblox now! youtube.com/watch?v=cLFrxJ You can play with an interactive agent using supported by GPT-3

youtube.com

Play now: https://www.roblox.com/games/11462889413/GPT-3-SayCan-NPCJames: https://github.com/JavaFXpertBased on the CoT paper: https://arxiv.org/abs/2201.119...

4

33

92

SayCan wins the Special Innovation Award at #CoRL2022!! ✅ LLMs for robotics ✅ real world tasks at scale ✅ 13 robots, MANY collaborators, 17 months ✅ 3 live demos from across the world

3

17

158

Come hang out at the #CoRL2022 Google booth at 3:45PM today! We will showcase our work Inner Monologue that turns SayCan into a closed loop system by using language as a Universal API. With

innermonologue.github.io

Inner Monologue: Embodied Reasoning through Planning with Language Models.

Project page for Project page for Inner Monologue: Embodied Reasoning through Planning with Language Models

8

29

📢 Interested in learning to use sight and touch to model deformable objects? Check out our paper VIRDO++ arxiv.org/abs/2210.03701 #CoRL2022 to see how implicit representations seamlessly integrate multimodal sensing in the real-world.

@WiYoungsun

@andyzengtweets

@peteflorence

GIF

1

8

32

Join our technical session in-person in Auckland, or virtually, to hear from

, Cynthia Matuszek

, Dieter Fox

, Jean Oh

, Nakul Gopalan

1

3

19

The virtual Habitat Rearrangement workshop at #NeurIPS2022 is this Thurs, Dec 8, 1-4pm PST! Hear from winners of the Habitat Rearrangement and ObjectNav challenge along with a great lineup of speakers:

,

,

! Schedule

Habitat Rearrangement Challenge 2022

Overview For NeurIPS 2022, we are hosting the Object Rearrangement challenge in the Habitat simulator 1 2. Object rearrangement focuses on mobile manipulation, low-level control, and task planning....

1

6

12

What's the best part of virtual workshops at #NeurIPs2022? Engaging with a huge variety of top researchers without the overwhelming venue. Video optional, no stress, high insight. Here's where you can learn with the best at the Robot Learning workshop on Trustworthiness. 🧵

1

11

29

Stop by our booth today at 4pm to have a video chat with a robot arm! Ask it to move around objects, or just say hello. Powered by Code as Policies, the system uses the latest language models to write robot code. Read more → goo.gle/3U5CmCg

4

41

152

Looking for an easy-to-read review paper on Grasp Learning? I just wrote one for Annual Reviews that I'm excited about. Check it out:

arxiv.org

Grasp Learning: Models, Methods, and Performance

Grasp learning has become an exciting and important topic in robotics. Just a few years ago, the problem of grasping novel objects from unstructured piles of clutter was considered a serious...

3

16

75

You can play with Code as Policies on

@huggingface

now! 👇

Quote Tweet

Nov 10, 2022

Just released our Code as Policies robot demo using @Gradio on @huggingface!! (thx @_akhaliq for the help!) You need an @OpenAI API key and ideally Codex access, but GPT-3 also works fine! The sim/video rendering may take some time so pls be patient

huggingface.co/spaces/jackyli

Show this thread

0:26

6K views

9

47

Check out the first(?) robotics demo on 🤗

Quote Tweet

Nov 10, 2022

Just released our Code as Policies robot demo using @Gradio on @huggingface!! (thx @_akhaliq for the help!) You need an @OpenAI API key and ideally Codex access, but GPT-3 also works fine! The sim/video rendering may take some time so pls be patient

huggingface.co/spaces/jackyli

Show this thread

0:26

6K views

6

44

Just released our Code as Policies robot demo using

@Gradio

on

@huggingface

!! (thx

@_akhaliq

for the help!) You need an

@OpenAI

API key and ideally Codex access, but GPT-3 also works fine! The sim/video rendering may take some time so pls be patient 🙏 huggingface.co/spaces/jackyli

0:26

6K views

9

19

101

Can't wait for the #NeurIPS2022 Workshop on Robot Learning? 🤖👨‍🏫 We got you! In the upcoming weeks, we will post talks of our panelists starting with

@andyzengtweets

&

Language as robot middleware

who have cool insights on "Language as Robot Middleware" 🤖💬🚀

youtube.com

Andy Zeng & Jacky Liang will join us during the workshop (December 9), where we bring together experts with diverse perspectives to highlight the ways curren...

20

45

Code & Data for Semantic Abstraction is out! semantic-abstraction.cs.columbia.edu We also hosted a HuggingFace demo for you to try out our multi-scale CLIP relevancy extractor: huggingface.co/spaces/huy-ha/

The following media includes potentially sensitive content. Change settings

Quote Tweet

Shuran Song

@SongShuran

Jul 26, 2022

Semantic abstraction -- give CLIP new 3D reasoning capabilities, so your robots can find that “ rapid test behind the Harry Potter book.”

w. Huy Ha twitter.com/_akhaliq/statu…

View

4

15

From our demo floor at AI@, check out Code as Policies at work. This helper robot is able to compute and execute a task given via natural language. Read more → goo.gle/3U5CmCg

1:17

15.1K views

15

81

270

This is an incredible paper! Excited about control as program induction. Reminds me of en.m.wikipedia.org/wiki/Symbolic_ It also shows most robot tasks studied are implementable with a few lines of scripted codes, so shouldn't need much human demos or RL (given good CV/NLP)

Quote Tweet

How can robots perform a wide variety of novel tasks from natural language? Execited to present Code as Policies - using language models to directly write robot policy code from language instructions. See paper, colabs, blog, and demos at code-as-policies.github.io long

Show this thread

0:20

36.1K views

1

21

Just ask for the source code

Quote Tweet

How can robots perform a wide variety of novel tasks from natural language? Execited to present Code as Policies - using language models to directly write robot policy code from language instructions. See paper, colabs, blog, and demos at code-as-policies.github.io long

Show this thread

0:20

36.1K views

7

34

It's out! Even though natural language is a very useful planning medium, it can be limiting. Generating code fixes many of these limitations: code-as-policies.github.io ai.googleblog.com/2022/11/robots See

great🧵describing the work in detail👇 A few examples below:

Quote Tweet

How can robots perform a wide variety of novel tasks from natural language? Execited to present Code as Policies - using language models to directly write robot policy code from language instructions. See paper, colabs, blog, and demos at code-as-policies.github.io long

Show this thread

0:20

36.1K views

1

3

18

Excited this is out! CaP uses code generation LLMs to plan and execute robotics tasks, since code is more expressive and can represent complex logics, loops, conditions, etc. I want to highlight a few contributions of this work that goes beyond vanilla program synthesis:

Quote Tweet

How can robots perform a wide variety of novel tasks from natural language? Execited to present Code as Policies - using language models to directly write robot policy code from language instructions. See paper, colabs, blog, and demos at code-as-policies.github.io long

Show this thread

0:20

36.1K views

1

12

54

“What if we just had the LLM write the code that runs the robot?” Also, a generic way (hierarchical code-gen) to improve code-writing LLM performance on OpenAI’s benchmark used for Codex — and applicable for code-gen in any domain. Great thread by

with more!

Quote Tweet

How can robots perform a wide variety of novel tasks from natural language? Execited to present Code as Policies - using language models to directly write robot policy code from language instructions. See paper, colabs, blog, and demos at code-as-policies.github.io long

Show this thread

0:20

36.1K views

1

7

Beyond task planning, can LLMs generate robot policy code that exhibits spatial-geometric reasoning ("draw 5cm hexagon around apple"), and leverages code logic ("go in a 1.5m square until you see a coke"), all given a language instruction and without any additional training? 🧵👇

Quote Tweet