JavaScript is not available.

11h

just discovered the most useless function in pytorch pytorch.org/docs/stable/ge

3

1

17

Feb 2

PyTorch nightly wheels for Python 3.11 are now available for all OSes pip install --upgrade --pre torch==2.0.0.dev20230202 --extra-index-url download.pytorch.org/whl/nightly/cpu github.com/pytorch/pytorc download.pytorch.org/whl/nightly/to

3

8

why would we use torch's built-in flash attention, given how it's (slightly) slower than xformers? because it supports attention masks. github.com/Birch-san/diff

xformers currently treats attn_bias as a boolean (i.e. "enable causal mask"). doesn't yet support applying a BoolTensor.

read image description

ALT

3

17

okay, you can't use torch.compile() on Python 3.11 yet. github.com/pytorch/pytorc

Exception has occurred: KeyError
'JUMP_ABSOLUTE'
File "/home/birch/git/pytorch/torch/_dynamo/bytecode_analysis.py", line 8, in <module>
dis.opmap["JUMP_ABSOLUTE"]

read image description

ALT

1

Measured torch built-in flash attention:

Quote Tweet

tried out pytorch's built-in Flash Attention on 4090. no flash attention: 4.10it/s torch.nn.MultiheadAttention: 6.20~6.23it/s xformers flash attention: 6.32~6.35it/s batch-of-5 images, 528x768 res, 22 steps, 6 batches github.com/Birch-san/diff

Show this thread

1

a less noisy commit demonstrating how to replace diffusers' CrossAttention with torch.nn.MultiheadAttention. you'll want to update MultiheadAttention#forward() to match how I did it in the commit above though. github.com/Birch-san/diff

tried out pytorch's built-in Flash Attention on 4090. no flash attention: 4.10it/s torch.nn.MultiheadAttention: 6.20~6.23it/s xformers flash attention: 6.32~6.35it/s batch-of-5 images, 528x768 res, 22 steps, 6 batches github.com/Birch-san/diff

2

7

31

ERA-Solver seems to absolutely slap. Its FID at 10 steps is lower than what the other samplers achieved at 100 steps. Note: their FID measures only the *older* DPM-Solvers; DPM-Solver++ is only measured in context of computation time. DEIS not mentioned at all.

Quote Tweet

AK

@_akhaliq

Jan 31

ERA-Solver: Error-Robust Adams Solver for Fast Sampling of Diffusion Probabilistic Models abs: arxiv.org/abs/2301.12935

2

12

`No operator found for this attention` when `attn_bias` is a `torch.Tensor` · Issue #576 · facebo...

Jan 31

I should clarify: I don't mean that attention got 50% faster. I mean that *image generation* got 50% faster; attention is one part of that. On Mac I measured that attention accounts for just under half of where the Unet spent its time, for 512x512 images.

I also wanna check out pytorch's built-in flash attention and see how it compares to xformers. compiled pytorch 2.0 alpha with USE_FLASH_ATTENTION=ON; will give it a spin soon

pytorch build flags indicating USE_FLASH_ATTENTION: ON

read image description

ALT

5

25

one of the first things I want to do now that I have CUDA is try out flash attention. xformers gave me a 50% speed increase (non-rigorous measurement). doesn't support attention masks though (except if you have an A100)

github.com

Bug I get the error No operator found for this attention Command Run the code below To Reproduce from einops import rearrange import xformers.ops as xops def attention(q, k, v, attn_bias, D,...

1

6

here's how I encrypted my home directory on Ubuntu with ZFS. at login: password authenticates user *and* decrypts the home directory. once all sessions log out: home directory is unmounted, encrypted again. works over SSH too

gist.github.com

ZFS home encryption Ubuntu 22.10

ZFS home encryption Ubuntu 22.10. GitHub Gist: instantly share code, notes, and snippets.

4

Jan 31

finally got the Nvidia 4090 PC set up! Python 3.11 + CUDA 11.8 + PyTorch 2.0 alpha + xformers + Ubuntu 22.10 it's so faaaassst thanks again

@cafeai_labs

for sponsoring the build! gist.github.com/Birch-san/8ec1

prompt was something like:
flandre scarlet, reddizen, 1girl, ascot, blonde hair, blush, bow, closed mouth, collared shirt, hair between eyes, hat, hat bow, looking at viewer, medium hair, mob cap, one side up, orange background, puffy short sleeves, puffy sleeves, red bow, red eyes, red vest, shirt, short sleeves, simple background, sketch, smile, solo, upper body, vest, white headwear, white shirt, yellow ascot

read image description

ALT

2

3

15

Jan 19

I never found the pen, so I did with numpy and a footswitch

Quote Tweet

Jan 19

it actually worked! you can split floats into integer mantissae and exponents, and compute matmul via elementwise products. exponents can be multiplied cheaply (it's just addition!) new datatype for ML training? gist.github.com/Birch-san/4f49

# two formulations which achieve the same result:

# regular matmul
mat = np.array([[1.3e2, 8.e2], [1.6e1, 5.e-1]])
vec = np.array([2.1e2, 3.6e-4])
mat @ vec

# matmul via separate elementwise products of mantissa and exponent, then recombining when we want to return to canonical representation
mat_m = np.array([[13, 8], [16, 5]])
mat_e = np.array([[1, 2], [0, -1]])
vec_m = np.array([21, 36])
vec_e = np.array([1,-5])
mantissa_elementwise_product = mat_m * vec_m
exponent_elementwise_product = mat_e + vec_e
elementwise_product = mantissa_elementwise_product * np.array([10.])**exponent_elementwise_product
elementwise_product.sum(1)

read image description

ALT

1

2

it actually worked! you can split floats into integer mantissae and exponents, and compute matmul via elementwise products. exponents can be multiplied cheaply (it's just addition!) new datatype for ML training? gist.github.com/Birch-san/4f49

read image description

ALT

5

6

50

Jan 19

have a potentially-significant linear algebra idea but can't find a bloody pen

1

2

2 npm packages printed adverts during postinstall one author was looking for a job the other was hiring I should introduce them

11

Jan 16

Did you know about named tensors? Name your tensor's dimensions, for better error messages pytorch.org/tutorials/inte

1

6

Jan 16

added wacky undocumented batching support to diffusers-play, to get ready to make animations on the new GPU. took me a while to work out PyTorch (MPS) was broken rather than my CFG implementation. will try out CUDA soon though! github.com/Birch-san/diff

don't submit a batch of images to the VAE for decoding. you just get disappointing noise.
submit them one-at-a-time.
or try to work out whether there's a less-broken version of PyTorch, or work out which line of diffusers code was too ambituous to work on Mac

read image description

ALT

9

Apple Announces Full Swift Rewrite of the Foundation Framework

Jan 16

I actually prefer the undersampled image to the converged image. this one is the same seed with a 15-step Karras schedule running the full gamut of sigmas, rho=7. k-diffusion DPM-Solver++ (2M) with (2S) LMS warmup

read image description

ALT

1

5 Unet steps is (occasionally) all you need. sigmas: 7.0796, 2.8739, 1.0514, 0.3394, 0.0936, (0.0000) #stablediffusion

WD1.4
5 Unet steps, k-diffusion DPM-Solver++ (2M)
sigmas:
7.0796, 2.8739, 1.0514, 0.3394, 0.0936, 0.0000
seed (CPU):
2772908837
artoria pendragon (fate), carnelian, 1girl, general content, upper body, white shirt, blonde hair, looking at viewer, medium breasts, hair between eyes, floating hair, green eyes, blue ribbon, long sleeves, light smile, hair ribbon, watercolor (medium), traditional media

read image description

ALT

1

7

Windows Update now schedules updates for when lower-carbon electricity is available support.microsoft.com/en-us/windows/

1

2

Apple just launched AI audio narration for books. This quality of this AI narration is 🤯

0:34

384.1K views

160

844

6,188

What does this mean for Nativescript and cross-platform,

@LinguaBrowse

?

infoq.com

The Swift team has started to work on a new open-source implementation of the Foundation framework. To be written entirely in Swift, the new Foundation aims to improve performance by getting rid of...

1

4

Finally, on-par quality with Dreambooth, updated + optimized PTI CLI, SVD distillation CLI, flexible dataset and CLIP metrics utilities, wandb logging, v0.1.0 is finally out! github.com/cloneofsimo/lo

6

32

174

this is actually canon for Kudryavka's storyline

1

fumo jail

read image description

ALT

1

6

ensure Ram is well-seated

Ram plushie from Re:zero inside Fractal Torrent PC case

read image description

ALT

8