I will attempt to explain the basic idea of how diffusion models work!
... in only 15 tweets! 😲
Let's get started ↓
Follow
Tanishq Mathew Abraham, PhD
@iScienceLuvr
PhD at 19 |
Founder and CEO at |
Part-time at |
Notebooks GM |
Biomed. engineer @ 14 |
TEDx talk
bit.ly/3tpAuan
bit.ly/3tpAuanTanishq Mathew Abraham, PhD’s Tweets
Very excited to share the news that I successfully defended my PhD research today! 🥳🎉
After 4 years 8 months in the graduate program,
I am now Dr. Tanishq Mathew Abraham (at 19 years old)!!
Awesome and surprising things you can do with Jupyter Notebooks ⬇
I got to try GPT-4's multimodal capabilities and it's quite impressive! A quick thread of examples...
Let's start out with solving a CAPTCHA, no big deal
This is currently the most important network in deep learning!
From helping to power search for billions of users to better understanding proteins, it does it all!
Here are 10 of the best resources to help you learn about the attention mechanism & Transformer network ⬇⬇⬇
Are you wondering how large language models like ChatGPT and InstructGPT actually work?
One of the secret ingredients is RLHF - Reinforcement Learning from Human Feedback.
Let's dive into how RLHF works in 8 tweets!
Just discovered this AMAZING website - "Deep Learning Drizzle"
It is a constantly-updated list of machine learning/deep learning course materials that are taught by domain experts and are available for FREE!
Check it out here → deep-learning-drizzle.github.io
ICLR 2023 (a top ML/AI conference) submissions have been released, and do you know what that means?
Time for mind-blowing papers! 🤯↓
I had a great time chatting with yesterday! We discussed a range of topics, from synthetic biology to ML conferences. What's funny is that we are about 2 hrs from each other but we got to meet up in Amsterdam airport! 😄
How the machine learning community feels after PaLM and DALL·E 2 during this week:
Research in AI is surprisingly more accessible to people with different backgrounds compared to other fields.
Anyone (w/ relevant experience) can contribute to impactful research.
Here are 5 research orgs you can join to contribute to real, open research in deep learning ↓
So has various options and controls and one of the main ones is the sampler used for generation. Let's talk a little bit about these samplers since this has some interesting and unexpected effects on generated image quality (below image from subreddit)🧵
The Tesla team discussed how they are using AI to crack Full Self Driving (FSD) at their Tesla AI Day event.
They introduced many cool things:
- HydraNets
- Dojo Processing Units
- Tesla bots
- So much more...
Here's a quick summary 🧵:
After you train a machine learning model, the BEST way to showcase it to the world is to make a demo for others to try your model!
Here is a quick thread🧵on two of the easiest ways to make a demo for your machine learning model:
What matters most when training a neural network is how well it generalizes to unseen data.
For neural networks, it turns out there's a simple principle that can allow you to understand model generalization. (1/18)
A thread ↓
GPT-4 release
Med-PaLM2 announcement
PaLM API release
Claude API release
Annotated PyTorch Paper Implementations by is an AMAZING resource:
• Deep learning papers explained in-depth with code side-by-side
• Constantly updated with some of the latest papers!
• 100% free and open-source!
Check it out here → nn.labml.ai
Claude, 's powerful ChatGPT alternative, was trained with "Constitutional AI".
Constitutional AI is particularly interesting since it uses less human feedback than other methods, making it more scalable.
Let's dive into how Constitutional AI works in 13 tweets!
So, I've heard people say anyone could have built ChatGPT. I think this is disingenuous.
ChaGPT isn't just GPT-3 w/ a chat interface on top of it.
The closest base model on the OpenAI API is probably text-davinci-003, but it was only released a day before ChatGPT! (1/9)
Proud and happy to announce I finally am a Notebooks Grandmaster! 🥳🎉
Thank you to all who have supported me over the past ~4 years and I am glad you all have been enjoying my notebooks!
Today, announced Whisper, an automatic speech recognition model. Plus, they released it open-source!
Blog post → openai.com/blog/whisper/
Research paper → cdn.openai.com/papers/whisper
Open-source code and models → github.com/openai/whisper
Quick thread about it (1/10) ↓
I'm really excited to share 's first paper since our public launch 🥳
🧠👁️ MindEye!
Our state-of-the-art fMRI-to-image approach that retrieves and reconstructs images from brain activity!
Project page: medarc-ai.github.io/mindeye/
arXiv: arxiv.org/abs/2305.18274
Have you been seeing artwork like this on your timeline and wondered how it was created?
Let's learn about one of the most popular algorithms for AI-generated art! ⬇ ⬇ ⬇
Happy to finally announce that I'm one of the first recipients of the PhD fellowship!🥳👨🎓
Remainder of my PhD is funded with this fellowship.
It's been great to be part of Stability AI as a PhD fellow for the past 4 months. Glad to be supported by this amazing org!
OpenAI has announced GLIDE, a new 3.5 billion parameter text-to-image generation model that is even better than DALL-E!
Link: arxiv.org/abs/2112.10741
Take a look at some of these results from the paper!
Read on to see how you can try it out for yourself!⬇
Hi 👋 if you are interested in:
🐍Python
🤖Machine/Deep Learning
😂ML-related memes
🔬STEM
Consider following me. ✔
I have lots of great content planned for the new year! Stay tuned!🎉
Materials for a comprehensive course on Geometric Deep Learning are available here: geometricdeeplearning.com/lectures
• 12 lectures
• Taught by pioneers in the field (, , , )
• 100% free
Check it out! 🚀
Stability AI has launched their StableLM suite of language models. Right now, 3B and 7B param models have been released, with 15B and 65B models coming soon.
Additionally, RLHF-tuned models have been released for research use.
Try it out here! → huggingface.co/spaces/stabili
The data is the most important part of any machine learning project!
Time spent on improving the data >>> Time spent on trying many model archs or even hyperparameter tuning.
I saw that recently followed me on Twitter! It's amazing that one of my deep learning idols from whom I have learned a lot (especially from the RNN, NN training recipe blog posts) is interested in the content I post here! Thanks for the follow, Andrej!
Happy to share the fast.ai Part 2 course is now released!
Wow what a long and fun journey it's been!
I think this is one of the most cutting-edge courses about deep learning and specifically about diffusion models! (1/6)
Quote Tweet
Our new course, "From Deep Learning Foundations to Stable Diffusion", is finally done after 8 months of work!!!
With >30 hours of video content (all free, no ads!), you'll learn how to create and train a Stable Diffusion model starting from pure Python 
fast.ai/posts/part2-20
fast.ai/posts/part2-20After knowing him for >9 months, I got to meet in person at the launch event this week!
And on this note, I'm happy to share that I became part of the Stability AI team since August! 🧵
How does GPT-4 do in the medical domain?
I got to play around with its multimodal capabilities on some medical images!
Plus a recent Microsoft paper examined its text understanding and got SOTA results on USMLE medical exams!
A quick thread ↓
It can explain memes quite well! Here it is explaining an AI-generated meme I shared recently.
(The AIs will create their own memes and explain it to us humans 😂)
If you haven't seen this before, is a great tool for exploring related papers, displayed in a nice graph fashion!
I've used it previously for some literature reviews and for discovering new and relevant papers
Check it out → connectedpapers.com
I'm glad to be part of the team developing the latest iteration of the Part 2 course, From Deep Learning Foundations to Stable Diffusion.
We've released the first few lectures as a preview, specifically talking about diffusion models!
Link → fast.ai/posts/part2-20
1. Write a full-fledged Python library!
You can write all of your code, documentation, & tests with Jupyter Notebooks & nbdev.fast.ai, all while maintaining best software practices and implementing CI/CD!
fastai deep learning library is entirely written in notebooks!
Given that this ML competition recently started, I thought it would be a good opportunity to share my approach to Kaggle competing
A quick thread (1/7) 👇
Quote Tweet
A state-of-the-art text-to-image generation model (like DALL-E) from Google Brain! 🚀
Imagen - unprecedented photorealism × deep level of language understanding 🔥
Website: gweb-research-imagen.appspot.com
Paper: gweb-research-imagen.appspot.com/paper.pdf
Yesterday, I got access to Copilot! 🥳
And it really is quite amazing!
Here's a quick demo where I get GitHub Copilot to write most of the code for a script to fine-tune a ResNet50 on a custom dataset in <5 mins! (video is 2x)
Silver medal🥈in Cassava #DeepLearning competition! From 1138th place public to 36th place private LB (top 0.9%)! What a jump!
Check out our solution here, which uses the libraries , , 's Tez, and 's timm:
kaggle.com/c/cassava-leaf
I have compiled a list of some of the best resources
Check out awesome-fastai!
github.com/tmabraham/awes
WOW: The 'godfather of AI' quits Google in order to speak out against the risks of AI
It's interesting to see how his views on AI has changed. He once said, regarding AI research, "the prospect of discovery is too sweet.”
This explanation on by is actually one of best conceptual/layperson explanations of diffusion models and text-to-image models:
A quick thread on my 2021 wins and 2022 goals ↓
My 2021 wins:
• Became a 2x Kaggle Master
• Passed my Ph.D. qualifying exam
• Open-source contributions (fastai, DALL-E mini, etc.)
• Started blogging
• Submitted my 1st first-author paper
• Reached >20k followers on Twitter
PhD student ➡️ PhD candidate.
I passed my Qualifying Exam today!
And it's #CincodeMayo today, it's fiesta time 🥳🎉
read image description
ALT
It's finally done!
A big day today (June 15)! Walked the stage with my PhD advisor and received my doctoral hood from the Vice Provost and Dean of Grad Studies
June was a very busy month full of milestones from my PhD graduation to my 20th birthday few weeks ago! Can't believe two decades has passed so quickly🎂🎉🥳
Just within the past decade I graduated high school, graduated community college, obtained my bachelor's degree at UC… Show more
10 years ago, I presented my first research project at the age of 9 at the NASA Lunar Science Forum!
It was a fun experience, and from then I have grown so much as a researcher 😄
Mixture of expert denoisers might be the next major trick/advancement for diffusion models.
Both Baidu's ERNIE-ViLG 2.0 and NVIDIA's eDiffi do this.
The idea is to have different models specialized for different noise levels:
Einstein summation or einsum notation is a concise way to represent tensor operations & is often used to simplify deep learning code.
I've been going through this nice tutorial by recently. It explains einsum notation quite well w/ many examples!
rockt.github.io/2018/04/30/ein
I did it! After a year-long journey I have met the entire AK trinity of ML 😂😂😂
1. (met last Fri at HF meetup)
2. (met at NeurIPS in Dec)
3. (met in Amsterdam airport a year ago)
#KeepTwitterPositiveBy being a positive role model for my generation #educationmatters #EducationIsCool
Saw this meme on #Facebook !
Had implemented RLHF for diffusion models 2 weeks back, replicating the DDPO paper.
Here's a before/after comparison training with aesthetics reward.
This is actually the 1st RL algorithm I've coded from scratch!
Code linked in next tweet.
Explanatory blog post coming soon!
An adventurous 18 years from being a child genius to official adulthood!
Climbing my way to bigger and better things!
Celebrating my 18th birthday today!🎂🎈🎉🥳
Applying deep learning to pathology is quite challenging due to the sheer size of the slide images (gigapixels!).
A common approach is to divide images into smaller patches, for which deep learning features can be extracted & aggregated to provide a slide-level diagnosis (1/9)
Beginners often get confused about why ReLU actually works as a good activation func. that can be used in a neural network to universally approximate any function.
The animation by is quite good at providing a visual intuition behind this. (1/3)
A useful warning regarding downloading and trying deep learning models in general
After training an ML model, the BEST way to showcase it to the world is to make a demo for others to try!
The easiest way to do so is w/ , hosted on Spaces.
Read my new blog post to learn how to do this (w/ appearance by )!
I am excited to share what I have been working on for the last several months in collaboration with Stability AI!
Check out my new venture, !
Please support and share! ↓↓↓
Quote Tweet
Announcing the launch of the MedARC!
MedARC is a novel, open, & collaborative approach to medical AI research.
It was created to develop large-scale AI models for medicine & build interdisciplinary teams to address clinical needs. (1/10)
medarc.ai/blog/announcem
A new text-to-image generation AI is coming out soon! 👀
#StableDiffusion is a new model soon to be released . It will be completely open-source as well! It will hopefully be a good contribution to the #AIart space.
Here are a bunch of images I generated:
App-integrated LLMs can be jailbreaked:
showed how prompt injections can be incorporated in webpages or other content that may be retrieved by LLM systems to result in nefarious behavior.
Here, text is embedded in a webpage to direct BingChat to perform a scam.
Thanks for sending my Master-level mug! Hot chocolate will be tastier now😄
Reached Master level, thanks to the support from the #kaggle & communities, and folks like , , , , and many other Kagglers!
In 2021, we saw many ML/AI models that tackle seemingly impossible problems:
• Photorealistic text-to-image generation → DALL-E & GLIDE
• Protein structure prediction → AlphaFold2
• Programming w/ natural language → Codex
I wonder what problems ML/AI will tackle in 2022👀
EfficientNetV2: Smaller Models and Faster Training
arxiv.org/abs/2104.00298
❗️Beating ViT, NFNets, and other SOTA models on ImageNet
❗️Improved training efficiency and much smaller models
Google knows!
Last year as a teenager...and so begins Chapter 19 🎉
Yet another state-of-the-art method for text-to-image generation, this time from researchers at !
Link: arxiv.org/abs/2203.13131
How does it work? A short thread on this paper ⬇
Have 2 surprises for my 20k milestone!
First: A BOOK GIVEAWAY🎁
I'm autographing and giving 3 special color editions of 's book "Approaching Almost Any Machine Learning Problem" (which I helped review)
To enter: like, RT this tweet & follow me by Jan 14th 10am PST
Recently, released the Diffusers library for training and inference of diffusion models. 🧨
I helped put together the official training example notebook (further expanded by ).
Check it out here! →
Quote Tweet
I thought it was just my family getting excited about the unexpected showers. But looks like the whole of California is hyped up 'cos #rain is trending on #Twitter!
GIF
read image description
ALT
It's kinda wild that two of the components of currently popular image generation AIs were actually invented for medical AI:
1. U-net - originally developed for cell segmentation
2. CLIP - a scaled-up version of ConVIRT, which was for learning medical visual representations
You may have seen surreal and absurd AI-generated images like these ones...
These are all generated with an AI tool known as DALL·E mini
Let's talk about the history of #dallemini, and also *how* it works! ↓↓↓🧵
BREAKING: PaLM 2 was announced! It is now being used in Google Bard.
It is a SOTA LLM with better multilingual capabilities and more compute-efficient. 🔥
Read the paper here → ai.google/static/documen
Looks like RLHF and RLAIF for text-to-image diffusion models is here!
A quick thread about this impressive work ↓
If you want to explore this further, check out the lectures where Jeremy Howard explores this in a little more detail than a Twitter thread 😅
fast.ai/posts/part2-20
I'm up at 4:30am in our family room to do an interview on , my second interview in a period of 24 hrs 😮💨😄
You can watch here, interview in about 15min:
nbcnews.com/now
OpenAI has released a 35-page paper on Codex (the model that powers GitHub Copilot)!
Happy 20th Mother's Day to !
My mom quit her own PhD to raise me, 20 years later my gift this Mother's Day is finishing my PhD! 😉
Thankful for two decades of love, support, encouragement, & guidance. Without it, I wouldn't be here.
My mom is truly a super-mom! ♥️
Had a great time at the #WoodstockAI meetup on Friday.
There was a literal hugging face and llamas roaming around!
The place was packed with AI researchers, developers, and enthusiasts!
I think there is some confusion in the #AiArt community about Stable Diffusion weights. Some folks think non-EMA is for inference, while EMA is for fine-tuning.
Use EMA weights for generation!
You can use non-EMA for fine-tuning but probably fine to use EMA for fine-tuning too
BREAKING: There's not just one, but two announcements from today!
has released StableVicuna, an RLHF-trained version of Vicuna-13B.
Announcement → stability.ai/blog/stablevic
Here are some examples of responses:
Hi 👋 if you are interested in:
🐍Python
🤖Machine Learning
😂ML-related memes
👨🔬STEM
Follow me. ✔
I use Twitter to share a lot of content that you won't want to miss. 🎉
1. DreamFusion by
Text-to-3D generation starting from a pretrained text-to-image diffusion model and not needing any 3D training data:
Quote Tweet
Did you know it's easy to mix plain w/ ?
My starter code for SETI comp. shows how to use a custom PyTorch Dataset w/ fastai
It also shows the power of using pretrained CNNs on completely different datasets (spectrograms)!
What I did this decade:
Started college @ 7
talk @ 9
High school grad @ 10
Participated on Child Genius
Appeared on
Community college grad @ 11
grad @ 14
Joined PhD program @ UCD
Looking forward to the next decade! Happy New Year #2020NewYear
3. Create dashboards, interactive tools, and web apps!
You can use Jupyter widgets (ipwidgets) to make your notebooks even more interactive, and you can turn your Jupyter notebooks into dashboards and web apps with voila.readthedocs.io.
Kind of awesome and exciting to be cited in a Google Research paper 🥳
(This is the new text-to-image model, Parti)
I guess it's a new achievement unlocked! 😄
A quick thread on my 2022 wins and 2023 goals ↓
My 2022 wins:
• Presented my PhD research at 2 conferences (first in-person confs after pandemic)
• Co-first-author paper was published
• Became a Grandmaster
• Joined
• Reached >35k followers on Twitter
2. Create a blog!
Platforms like fastpages.fast.ai easily allow you to create blog posts from your Jupyter Notebooks, with the code cells and outputs in your post, and can even be made interactive.
I have my own such blog at tmabraham.github.io/blog
I thought maybe I could use it for getting LaTeX code but it's not that great. I was really hoping to be able to use it as a tool to get LaTeX of random equations. It may be good enough for starting out though.
Proximal Policy Optimization is the RL technique often used for RLHF.
has provided a PPO implementation with equations and explanation side-by-side.
Check it out here → nn.labml.ai/rl/ppo/index.h
