These are chat archives for django/django

6th
Feb 2017
Miracle Ayodele
@iammiracle
Feb 06 2017 08:09
hello guys
anyone worked on scrapping website?
i scrapped a website and the result its in json format
but i want it in html file
format*
tulpn
@tulpn
Feb 06 2017 09:18
Hi guys, I have a quick question. I am running Django in my dev env with WSGI and Apache, i have to restard apache everytime I modify the python code to take changes effect. This is quite annoying. Just out of curiosity, would that be resolved when I swap over to gunicorn ?
Sopan
@sopanshewale
Feb 06 2017 10:02
@tulpn - gunicorn should be able to help you. Out of curiosity: In Dev, Django's own style (manage.py) is best. Did you face any issues to use that in your dev environment?
tulpn
@tulpn
Feb 06 2017 10:06
@sopanshewale thanks, Well, no. My setup is a bit custom though. I am running my dev env in a custom vagrant box and connect to it via Pycharms vagrant feature. All I want is to get rid of a second terminal where I run commands on the vagrant box. But maybe you are correct that the easiest is to just use the manage.py and bind it to the VMs IP, but then again, I'd need to leave it running while I develop.
Sopan
@sopanshewale
Feb 06 2017 10:09
@tulpn - so your development environment is accessed by other developers/testers when you are developing your code, right?
tulpn
@tulpn
Feb 06 2017 10:10
Well, no, each one has their own vagrant box running
Sopan
@sopanshewale
Feb 06 2017 10:13
@tulpn - i am still not sure if i understand your custom environment. I do code frequently using Vagrant box. My VM is always running. I connect via "vagrant ssh" & "vi" is still my IDE. I am happy with manage.py. I can think of using gunicorn or uwsgi, ngnix or apache only in production environment. BTW - these days Docker containers are best friends :)
tulpn
@tulpn
Feb 06 2017 10:19
@sopanshewale I run a Windows environment in which I develop but I want to be as close as possible with the setup to the production env, that I don't run into unexpected issues for whatever reason. To keep it simple, and also for new developers in the team, I created a custom vagrant template in which I set everything up for our django & api development. All that a developer then has to do is to install the box, git clone any of the projects and run their personal editor. I personally use Pycharm, in which i can directly connect the virtualenv via vagrant and it figures all out for the IDE project support.
The django app is then disclosed via the vagrant ip and thats it. Happy coding ^.^ In theory. But because it is setup as an apache wsgi it is annoying to run the restart command to reflect the changes. Tbh, I am just experimenting with this setup so far - and the current setup is definately not convenient.
Sopan
@sopanshewale
Feb 06 2017 11:08
@tulpn - That's great! Thank you! Why don't you replace only one part! Instead of uwsgi - use manage.py (off-course you need to check port vs socket stuff)! I think everything should work properly in your environment
@tulpn - else small script using https://pypi.python.org/pypi/watchdog can do magic of your restart stuff. Any code changed will restart your apache
Sopan
@sopanshewale
Feb 06 2017 11:16
@koderNeptune - are you looking for cloning the sites? If you are looking for only scrapped data and want to push that into DB's or want to create csv or other formats then JSON is best. I am not understanding your use-case
Miracle Ayodele
@iammiracle
Feb 06 2017 11:16
@sopanshewale the issue i have is, i scraped the website, and it showed in json format
i also tried csv format, and i got the result
but for an html file, i was not able to do that
Sopan
@sopanshewale
Feb 06 2017 11:20
@koderNeptune - may be website detected your client as script or scraper. Try setting 'User-Agent', 'Mozilla/5.0' in your scraper. I assume your scrapper is using Python's urllib2 or urllib
Miracle Ayodele
@iammiracle
Feb 06 2017 11:29
you mean adding USER_AGENT = "Mozilla/5.0 (Windows NT 6.2; WOW64)" to my settings.py?
Sopan
@sopanshewale
Feb 06 2017 11:30
@koderNeptune - we are talking two different things.
@koderNeptune - can you share part of your view
Miracle Ayodele
@iammiracle
Feb 06 2017 11:31
this is the result of my json file
{'link': [u'/hp-15-6-inches-4gb-ram-500gb-hdd-windows-8-laptop-1601562'],
'name': [u'HP - 15.6 inches - 4GB RAM - 500GB HDD - Windows 8 Laptop'],
'price': [u'\u20a6134,000'],
'qty_sold': [u'257 SOLD'],
'seller': [u'Maxitech Global']}
tulpn
@tulpn
Feb 06 2017 11:31
@sopanshewale thanks! I ended up doing it via manage.py and reverse proxy it via nginx - works fine. The gunicorn solution didnt solve my initial problem, but at least all 3 methods are working and are configured so a developer can choose their preferred method
Miracle Ayodele
@iammiracle
Feb 06 2017 11:32
but i need it to show like i can view on my browser
Sopan
@sopanshewale
Feb 06 2017 11:32
@tulpn - very nice to hear that ;) Good Job
@koderNeptune - i am confused :)
Miracle Ayodele
@iammiracle
Feb 06 2017 11:33
smiles
Sopan
@sopanshewale
Feb 06 2017 11:33
@koderNeptune - are you developing website using Django. The same site you are scrapping but the resulting data is displayed as JSON format
Miracle Ayodele
@iammiracle
Feb 06 2017 11:36
i want to develop a website using django, and i want the data that will show on my website to be the data i scrapped from another site
the first thing i did is to create the scrapy project
Sopan
@sopanshewale
Feb 06 2017 11:37
@koderNeptune - so which data you see in JSON. The data scraped from other site?
Miracle Ayodele
@iammiracle
Feb 06 2017 11:38
exactly
Sopan
@sopanshewale
Feb 06 2017 11:38
@koderNeptune -that scrapping handled by some script or code, right? - in that script you are using urllib, right?
Miracle Ayodele
@iammiracle
Feb 06 2017 11:39
not really
i am new to it, just followed a tutorial and urllib was not used
unless i use beautiful soup
Sopan
@sopanshewale
Feb 06 2017 11:40
@koderNeptune - hmm... so this seems like not Django related problem. May be you should explore appropriate forum or room to solve that issue. It's not Django issue
@koderNeptune - last attempt from me :) - share that tutorial URL
its a pdf, i trying to upload directly but not going through
Sopan
@sopanshewale
Feb 06 2017 11:46
@koderNeptune - don't upload here
Miracle Ayodele
@iammiracle
Feb 06 2017 11:46
ok
that's the link, page 10
Sopan
@sopanshewale
Feb 06 2017 11:49
ok - will check that later.
@koderNeptune - good luck at this moment
Miracle Ayodele
@iammiracle
Feb 06 2017 11:51
smiles, alright, thanks
Sopan
@sopanshewale
Feb 06 2017 11:51
@koderNeptune - try following:
[settings]
default = myproject.settings
USER_AGENT = "Mozilla/5.0 (Windows NT 6.2; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/27.0.1453.93 Safari/537.36"
in settings file
@koderNeptune - take care
Miracle Ayodele
@iammiracle
Feb 06 2017 11:52
you too
Mohamed Chedli Ben Yaghlane
@medchedli
Feb 06 2017 13:42
Hi! I am new to Django architecture and currently working on a projet where I want to make a generic/flexible batch to manage multiple streaming APIs ( like Twitter, Facebook, Weather APIs, ... ) and frequently update the feed using javascript. I find difficulty in finding 3rd party packages which I can build on top of them a flexible solution like GetStream.io ( http://getstream.io/ ). Any hints or tips on how achieve such feature will be highly appreciated.
Nadir
@nadirollo
Feb 06 2017 16:06
Hello!
If I have 3 models with FK relations A->B->C, is there a way to define a form where I can have formsets for B, that include formsets of A? Something like nested formsets, or inheritance formsets..? I have a multilevel hierarchy model’s that I would like to present in a form… and I have no clue where to begin