I feel like this is my fault but I'm not sure what I was supposed to do? Persistent disks are really slow, so the experiments did need a scratch disk. I guess I should have added a persistent disk as well, so the state can be backed up periodically?
-
-
Prikaži ovu nitHvala. Twitter će to iskoristiti za poboljšanje vaše vremenske crte. PoništiPoništi
-
-
-
Same thing happened with me a few months back on an AWS EC2 backed by NVMe - felt like an idiot when I read it was mentioned in the docs itself. Two things I did then - stopped using EC2 with NVMe and moved to Jupyterlab on EMR with auto sync to S3.
Hvala. Twitter će to iskoristiti za poboljšanje vaše vremenske crte. PoništiPoništi
-
-
-
You may be interested to check out
@arrikto's Rok software. It is designed to solve exactly this problem. We allow you to run on NVMe instances, but at the same time incrementally snapshot the disks, so you can restore them anywhere, instantly, even on other clouds@manasrnkarHvala. Twitter će to iskoristiti za poboljšanje vaše vremenske crte. PoništiPoništi
-
-
-
do you think dvc(version control) or MLflow would have helped.
Hvala. Twitter će to iskoristiti za poboljšanje vaše vremenske crte. PoništiPoništi
-
-
-
Hi
@honnibal use Google Cloud Filestore rather then SSD if you need persistence and high IOPS. Or store the data in Google Storage and run GPU intensive Jobs with AI platform jobs. I experience a similar case in the past and got my learnings already
Hvala. Twitter će to iskoristiti za poboljšanje vaše vremenske crte. PoništiPoništi
-
-
-
Domino Data Lab trivializes every component of this. We've feel the pain of configuring and using cloud compute with all the struggles around sharing and persisting before. Domino had been great to solve all of that.
Hvala. Twitter će to iskoristiti za poboljšanje vaše vremenske crte. PoništiPoništi
-
Čini se da učitavanje traje već neko vrijeme.
Twitter je možda preopterećen ili ima kratkotrajnih poteškoća u radu. Pokušajte ponovno ili potražite dodatne informacije u odjeljku Status Twittera.
Author of the
Founder
. Put up a GPU instance for a remote intern on GCP. Needs to do lots of i/o, so add NVME disk. Server gets terminated after a week or so -- expected. But now the disk is wiped and a bunch of state is lost on a short internship.