Let say I want to use Firebase Realtime Database. What is the needed setup to publish automatically the scrapper result to Firebase?
-
-
Show this threadThanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
Depends how valuable the data is. Firebase is not safe for very important stuff, because the data that you store there, won't be fully yours. Other than that it's great for simple PoCs and prototypes.
-
The data is not very valuable, the idea is really to make a quick POC
-
Then by all means. Go for it! Using it can really cut down on development time. I personally only had 1-2 issues with Firebase. In one case it was pretty annoying when they pulled down their services every hour. My Android App did not work at those times.
- 1 more reply
New conversation -
-
-
Many choices but imho the best way to do this cleanly with python backend: scrapy + Celery for your scheduled scraper task. Django + Django Rest Framework for the mobile app API. Redis as the message broker for Celery + cache for Django. Postgresql as the database.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
BeautifulSoup + MySQL for the scrapper part, API + Json payload for the communication part
-
Swap MySQL for PostgreSQL. PG is amazing.
-
yup, actually I said MySQL but the database doesn't really matter, could even be NoSQL db like mongodb
-
Absolutely, agree 100%.
End of conversation
New conversation -
-
-
@ParsePlatform Parse Server use this. -
New conversation -
-
-
Firebase could be a really simple solution, as a hosted, real-time database. It has downsides, especially regards to authentication and compliance, but for side project it might be good enough.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
You can build the scrapper using scrapy and then use SQLAlchemy for DB ORM/or any other ORM. Then using django-rest or flask you can provide a REST API , which mobile app can access using token.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
Django, Django Rest framework and Celery will do just fine.
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
Any ideas on whitleblower app, shielding the Id of user ?
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
Create replica of CA
Thanks. Twitter will use this to make your timeline better. UndoUndo
-
-
-
if you're hosting on heroku Heroku Postgres is a good option otherwise firebase *might* have you covered
-
But firebase is not at all trusted one .. your data are not at all yours when u used that . Issues in security
-
tf lmao
End of conversation
New conversation -
Loading seems to be taking a while.
Twitter may be over capacity or experiencing a momentary hiccup. Try again or visit Twitter Status for more information.