You can add location information to your Tweets, such as your city or precise location, from the web and via third-party applications. You always have the option to delete your Tweet location history. Learn more
The bigger your master set is, the more aggressive filters you can apply and still get a big enough dataset at the end. This also gives you the freedom to use sloppy but good enough filtering techniques that have a lot of false positives such as blurry image detection. 3/
For the master you can use a huge and fairly slow/cheap drive as opposed to a fancy ssd/nvme. Save the speedy drive space for the resulting filtered and processed datasets. 4/
Thanks for sharing this approach, very useful.
This sounds a lot like your own image data lake pattern. Raw, transformed, curated, published images. On cheap storage. How do you track the models, versions and what inputs / parameters affected the model? How are outcomes measured / what metrics would you review?
Model versions: Github! Inputs/Parameters/Outcomes: Well documented experiment Jupyter notebooks that I save in a separate backup drive to freeup space along with tensorboard outputs and model checkpoints. Point is I can reproduce with ease and not think too much about it.
Would a metadata database be helpful here? Store each image's location plus the metadata like grayscale, resolution, etc? I'd imagine if yoh have many images on a slow mechanical hard drive, that this could speed up the filtering quite a bit?
Sure that would be a really smart idea actually and I have yet to do it LOL. I'd just suggest to keep it simple and just make a simple local file with perhaps key/value entries. i.e. just treat it as a cache. Then you can just as easily delete it if you want to start fresh.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.
