These are chat archives for mirumee/saleor

20th
Aug 2018
Saqib khan
@saqibkhan2523
Aug 20 2018 01:13
Can anyone please tell me the exact process of setting up S3 bucket for media files. I have done as said in saleor.readthedocs.ioand have set AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY and AWS_STORAGE_BUCKET_NAME My site is in production But on image upload it open my local machine directory and when I update, it gives Server Error (500).
asalt
@asalt
Aug 20 2018 03:30

Hey guys. I'm trying to add a custom free text to products pages (customer enters their own text before add to cart). I've been able to to successfully give this association to products with a new, optional product attribute (maxlen, which indicates to add a text field) and gotten it added to my product forms (since it's associated with the product). However, I'm unable to get this custom text object returned when "add to cart" is clicked. It seems to be something to do with the ajax return (since the response.POST is ajax). I've looked around at the various javascript files but haven't made much headway - any suggestsions on what to look for?

For example, I've added within the product form page: <form id=id_custom_text>. But in my response.POST data I'm only getting quantity and variant.

Ken
@sjkenstone
Aug 20 2018 04:36
Hi everyone, I just want to know whether the SECRET_KEY is needed to set once the server system restarts everytime?
Saqib khan
@saqibkhan2523
Aug 20 2018 05:55
@sjkenstone everytime unless you set envvar through script
Mateusz Barlikowski
@Kappa7
Aug 20 2018 05:58
@sjkenstone If you have on your mind that secret key is cleared all the time you close the terminal - yes, you have to set it everytime with export. If you want to set secret key permanently you can just set it manually at settings.py
Filip Kucharczyk
@Pacu2
Aug 20 2018 05:59

@stephenmoloney Django settings are loaded at the start and are supposed to be immutable. For dynamic settings we are using SiteSettings instance that can be modified on the fly. You can use DJANGO_SETTINGS_MODULE env variable to point to a different settings file.

Celery is used mostly for asynchronous actions, like sending emails, generating thumbnails and analytics
Static folder is loaded at the server start and not supposed to be changed afterwards, all of the user-generated content goes into the /media/ directory

@Ken you could add an SECRET_KEY permanently to your os environment settings, I don't recommend changing that in the settings.py file directly as it could be easily overridden by an accident
@saqibkhan2523 Please provide a complete stacktrace, as far as I remember this guide was pretty accurate. Have you set STATIC_URL env variable to be used by the webpack?
Saqib khan
@saqibkhan2523
Aug 20 2018 06:08
@Pacu2 I have just added the above three vars. what url should be added?
Filip Kucharczyk
@Pacu2
Aug 20 2018 06:10
Complete url to your static bucket/cloudfront instance, like https://xxxxxxx.cloudfront.net/static/assets/
Filip Kucharczyk
@Pacu2
Aug 20 2018 06:17

@stephenmoloney Totally agree with your point of view on the upcoming changes. At some point we want to completely split the logic between different repositories, new storefront will be started as a separate repo for sure, I think that we will keep backend + dashboard logic at mirumee/saleor for now.

Some sort of saleor-cli would be helpful in glueing it all together, but that's still a problem we will need to face in the near future.

Saqib khan
@saqibkhan2523
Aug 20 2018 06:22
[Mon Aug 20 01:20:02.668899 2018] [wsgi:error] [pid 9539:tid 139933306869504] [remote 39.53.63.36:6909] return self._save(name, content) [Mon Aug 20 01:20:02.668902 2018] [wsgi:error] [pid 9539:tid 139933306869504] [remote 39.53.63.36:6909] File "/home/googleuser/ecom/ecom/lib/python3.6/site-packages/django/core/files/storage.py", line 236, in _save [Mon Aug 20 01:20:02.668905 2018] [wsgi:error] [pid 9539:tid 139933306869504] [remote 39.53.63.36:6909] os.makedirs(directory) [Mon Aug 20 01:20:02.668907 2018] [wsgi:error] [pid 9539:tid 139933306869504] [remote 39.53.63.36:6909] File "/home/googleuser/ecom/ecom/lib/python3.6/os.py", line 210, in makedirs [Mon Aug 20 01:20:02.668910 2018] [wsgi:error] [pid 9539:tid 139933306869504] [remote 39.53.63.36:6909] makedirs(head, mode, exist_ok) [Mon Aug 20 01:20:02.668912 2018] [wsgi:error] [pid 9539:tid 139933306869504] [remote 39.53.63.36:6909] File "/home/googleuser/ecom/ecom/lib/python3.6/os.py", line 220, in makedirs [Mon Aug 20 01:20:02.668915 2018] [wsgi:error] [pid 9539:tid 139933306869504] [remote 39.53.63.36:6909] mkdir(name, mode) [Mon Aug 20 01:20:02.668927 2018] [wsgi:error] [pid 9539:tid 139933306869504] [remote 39.53.63.36:6909] PermissionError: [Errno 13] Permission denied: '/home/googleuser/ecom/saleor/media' [Mon Aug 20 01:20:02.668940 2018] [wsgi:error] [pid 9539:tid 139933306869504] [remote 39.53.63.36:6909]
I added the link http://testbuck.s3-website-us-east-1.amazonaws.com
Filip Kucharczyk
@Pacu2
Aug 20 2018 06:24
As you can see there's a problem with your local media bucket PermissionError: [Errno 13] Permission denied: '/home/googleuser/ecom/saleor/media'
Saqib khan
@saqibkhan2523
Aug 20 2018 06:25
I am sorry I am doing this all for the first time. No idea what I am doing. can you please explain in noob terms
I created a directory saleor/media it lets me upload images but they are not displayed on site and they are not getting added in my s3 bucket
Filip Kucharczyk
@Pacu2
Aug 20 2018 06:37

If you want to upload media to s3 bucket, then you should set the appropriate env variables in the first place, take a look at the Serving media files with a S3 bucket section in the https://saleor.readthedocs.io/en/latest/deployment/s3.html

Right now it looks like you've created a new folder and didn't set the proper ownership for the folder and it's files (can be done with chown command on unix systems)

Stephen Moloney
@stephenmoloney
Aug 20 2018 09:39
@Pacu2 Thanks for answers.
Pete Dermott
@petedermott
Aug 20 2018 10:04
Hey everyone, I was looking through the dashboard 2.0 issues on GitHub and I've realised that you are looking to make more substantial changes going forward than I was really expecting. I'm currently writing an import script to bring products into Saleor and an integration to the SagePay payment gateway but I'm starting to have concerns that a lot of this will not be compatible when 2.0 launches. Are you expecting any major changes to the way products, checkout and payments etc are structured? Should I switch from using the master to 2.0 branch for my development?
Filip Kucharczyk
@Pacu2
Aug 20 2018 10:56
Hi @petedermott, payments and checkout will be definitely reworked to make them compatible with our new SPA architecture, you can track the progress in https://github.com/mirumee/saleor/projects/10
Pete Dermott
@petedermott
Aug 20 2018 10:59
@Pacu2 OK, that's good to know (even if it's a little frustrating since I got 90% of my integration done!) Do you know if there are other areas that are expecting large changes? I know there was talk at one point of merging Categories and Collections
Stephen Moloney
@stephenmoloney
Aug 20 2018 11:01
Im wondering if I should just git checkout dashboard-2.0 or wait, thoughts ?
when do you think you will create a new repo for it... that would be the best time perhaps.
Filip Kucharczyk
@Pacu2
Aug 20 2018 11:02
We have planned dropping categories, refactoring payments and checkout process(might be Cart as well) and shipping (working on it!). Think that products should be quite stable
dashboard-2.0 should be merged into the master, today or day after if everything goes as planned
It's hard to say when it becomes a separate repo as dashboard still has lots of in common with the backend
Stephen Moloney
@stephenmoloney
Aug 20 2018 11:58
I see, so I might just go with master and endure the transitions as they arrive.
Pete Dermott
@petedermott
Aug 20 2018 13:16
Thanks @Pacu2 that doesn't have the removed categories in it though? Has there being any more talks about what is going on there?
Filip Kucharczyk
@Pacu2
Aug 20 2018 13:48
As far as I remember we've agreed that categories should be eventually dropped, but don't think we've set a timeframe yet.
Khalifa Lame
@khalibloo
Aug 20 2018 14:23
Hi. I'm just getting started with saleor, evaluating it for a project in the coming months. I was wondering if someone could perhaps give me a brief description of how one might go about extending saleor. Does it have a sort of plugin system? Or do we extend its classes to add or customize its behavior? For example, our client wants a wishlist and a loyalty point system with ranks (gold, silver, etc). They also want different prices based on customer role (individual, wholesaler). What would be the recommended approach generally?
If it uses plugins, is there a central place I could find them? Ah, and our client wants a custom Payment provider as well.
Stephen Moloney
@stephenmoloney
Aug 20 2018 16:42
just a quick question about uwsgi workers
ENV PROCESSES 4 tells it to spawn 4 workers i presume
but what are they - some sort of web server (completely separate from celery - right?)
also, why does uwsgi need 4 workers ? is one worker per client connection ?
or one worker per cpu thread?
Sorry, don't know much about uwsgi
Patryk Zawadzki
@patrys
Aug 20 2018 17:10
django is fully synchronous (unfortunately) so you need one worker per connection
they are typically either processes, threads, or thread-like objects (greenlets and so on)
NyanKiyoshi
@NyanKiyoshi
Aug 20 2018 17:12

The workers are their to improve the concurrency, thus the throughput. uwsgi is (pre)forking the WSGI application at launch time and dispatch the requests to the workers that will serve it. As the fork process is heavy, instead of forking on demand (e.g.: in some implement of fastcgi, they would fork the application whenever a request was made, as long as the number of concurrent process is respected, which is time consuming).

You can also set a number of thread per process (workers * threads), which are spawned inside the worker whenever a request is done. Depending on the resources you have on your server (mostly memory), you may or may not prefer threads as they are lighter.

4 workers with 1 or 2 thread is most of the time enough to serve the major load. Which can be combined with a load balancer. The values really depend on the system though (IO throughput, cores, memory size and throughput, etc.)
Patryk Zawadzki
@patrys
Aug 20 2018 17:16
where asynchronous servers would use an event loop, django uses a synchronous worker pool so each worker is busy until its particular request completes (even if it’s waiting for external resources like a database)
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:18
ok, thanks for explanations. I'm trying to figure out how many processes/threads for a default installation on kubernetes... because kubernetes itself will launch replicas... and those are loadbalanced typically. Hence, I think a low number of workers/threads is reasonable in that case. It's just a question of finding the balance. I'll just get the helm chart written for now and those tweaks can be done later.
Patryk Zawadzki
@patrys
Aug 20 2018 17:18
there’s no way to tell without measuring
if your typical request takes 100ms to complete then even a single worker will give you around 10 RPS
if it takes around 500ms it will only give you 2 RPS
but
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:19
RPS = ?
Patryk Zawadzki
@patrys
Aug 20 2018 17:19
requests per second
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:20
oh, yea
Patryk Zawadzki
@patrys
Aug 20 2018 17:20
but then
if it takes 100ms and 90ms of that is spent waiting for a database to respond then you are underusing your CPU
so you could probably then run 10 such workers on a single CPU core before running out of processing power
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:21
The thing with kubernetes is that it has horizontal autoscaling
Patryk Zawadzki
@patrys
Aug 20 2018 17:21
contrary if your response takes 100ms to generate but it spends 90ms in Python code then you’re probably already near maxing out a single core
I know
I am fully aware how that works
I’m trying to give you something to work with as there is no single silver bullet
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:22
yea, sure. I'm just gonna make it configurable, let the user decide.
Patryk Zawadzki
@patrys
Aug 20 2018 17:22
make sure you are using the full power allocated to a docker container
so if you give each node 1 CPU core make sure it will use a big chunk of that
this usually means running 2-4 workers per node
and then using kubernetes to spawn additional nodes when needed
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:23
nodes -- you mean pods
kubernetes spawns pods on top of a cluster of nodes.
Patryk Zawadzki
@patrys
Aug 20 2018 17:24
yes
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:24
ok, so the resources for a pod can be set under
resources:
  requests:
    cpu: 
    memory:
  limits:
    cpu:
    memory
this will define the allocation per pod
Patryk Zawadzki
@patrys
Aug 20 2018 17:25
again, I’m trying to help you measure your own app
not to provide numbers
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:26
sure... I think for that something like prometheus is needed.... but yea, as a starting point, i'd go for 2 - 4 workers sounds good.
Patryk Zawadzki
@patrys
Aug 20 2018 17:27
it’s much cheaper in terms of resources to spawn five containers with four workers each than to spawn twenty containers
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:27
With helm, you configure it in the values.yaml so I will try to make it as configurable as possible. So if someone wanted:
Patryk Zawadzki
@patrys
Aug 20 2018 17:27
so you’d want to see how much output you can squeeze out of a single container limited to a single CPU core
it is better to scale horizontally than to scale vertically but you don’t want to build very small containers as you’d spend most of your resources on the overhead of running the containers
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:30
I agree with that :point_up:
Most users will probably not have heavy traffic - i don't
I'll focus on making it configurable early on.
Patryk Zawadzki
@patrys
Aug 20 2018 17:30
that’s why I suggest trying to max out a single CPU before scaling and then work with the assumption that one container = one logical (or physical) CPU
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:31
yea, i can run some tests on it when its written...
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:33
handy, thanks.
Patryk Zawadzki
@patrys
Aug 20 2018 17:33
also everything would be much easier if Django finally adopted the asynchronous event loop model :)
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:35
don't know enough about django/python, elixir is my thing - we never really worry about async as erlang was built from the ground up to handle those problems.
Patryk Zawadzki
@patrys
Aug 20 2018 17:35
yeah but erlang runs in its own virtual computer
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:36
it does, erlang VM.
Patryk Zawadzki
@patrys
Aug 20 2018 17:36
and it has its own problem when it comes to dynamically scaling the VM
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:36
not really, you make a cluster of nodes in erlang if you want to scale.
Patryk Zawadzki
@patrys
Aug 20 2018 17:36
ie. you are responsible of moving resources between nodes that join and leave the VM and for restarting resources that disappear
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:37
the developer uses OTP for that.
So you have supervisors... if they fail, they will auto restart -- or do whatever you set for to happen.
Patryk Zawadzki
@patrys
Aug 20 2018 17:38
I am familiar with erlang, OTP and elixir
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:38
If they crash enough times, then it will crash and take down a supervision tree or even the entire application - at which point you'd need to fix it.
Patryk Zawadzki
@patrys
Aug 20 2018 17:39
I’m talking about dynamically adding nodes to a VM to scale out, they will join the VM but they won’t be assigned any work to do until you explicitly restart some services on the newly added nodes
and you need to always have a master node where your main supervisors live
it’s a great platform but every advantage comes at a cost
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:41
I haven't required auto horizontal scalability... but I think it is possible to have a script to do that... but I'm not 100% sure...
TBH, I'm at a point where I don't think any app should be written in one language. which is why i was harping on about that up there :point_up: :laughing:
Patryk Zawadzki
@patrys
Aug 20 2018 17:43
microservices also have advantages and disadvantages :)
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:43
agree.
changing a monolith at a certain point gets hard... also reusability is reduced, its kinda take it all or take nothing
Patryk Zawadzki
@patrys
Aug 20 2018 17:44
I know more projects that migrated back to monolith than those that thrived as microservices
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:45
but microservices is not with out it's issues as you say.
For me, microservices is more geared towards services required very high scalability primarily (not only).
Patryk Zawadzki
@patrys
Aug 20 2018 17:46
you get high availability for core features but you also lose some performance because everything needs to talk to everything else
and synchronizing releases becomes a pain point because team A cannot deploy as they are waiting for team B which in turn is waiting for team C
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:47
I hear ya!
Patryk Zawadzki
@patrys
Aug 20 2018 17:47
and in the worst case A depends on B depends on A :D
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:47
but still, on the balance, if the resources are there :moneybag: , then I think it's preferable in the long run.
Trying to maintain one app with a large team is hard.... even just getting agreement on codebase changes, too diffuse responsiblity.
Patryk Zawadzki
@patrys
Aug 20 2018 17:48
it is but it’s much easier to optimize a single app
I recall an application where business wanted to allow users to reorder their most recent purchases as long as the project was still available
this was not required previously so auth, catalog and orders were separate microservices
and the initial implementation resulted in dozens of internal API calls
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:51
Well, it is possible to make a mess of a microservice too...
For all the communications, documentation is key.
Patryk Zawadzki
@patrys
Aug 20 2018 17:52
I don’t think that implementation was particularly bad it was just particularly tied to the initial requirements
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:52
But I agree with you in the end, monolith is not something to be gotten rid of.
neither is microservices.
Patryk Zawadzki
@patrys
Aug 20 2018 17:52
yup
there are also some obvious candidates for microservices, paypal is a microservice, stripe is a microservice and so on, the domains are well separated and it would not make sense to roll our own payment gateway as part of a monolith :D
Stephen Moloney
@stephenmoloney
Aug 20 2018 17:54
:thumbsup:
Don't separate it if it don't make sense.
Another day, I'd like to ask about how auth is handled but for now just wanna focus on getting this running in helm/kubernetes. I'm gonna open a WIP PR soon, and probably put some questions in there as part of the review.
Patryk Zawadzki
@patrys
Aug 20 2018 17:58
alright, off the top of my head, when logging in django handles auth by comparing the password to salted hashes stored in the db (using PBKDF2, can’t recall the number of iterations)
then you are either assigned a session cookie (in the traditional MVC) or a JWT token (when using the GraphQL API)
it’s 8 PM here so I need to leave for today
Khalifa Lame
@khalibloo
Aug 20 2018 18:03
Just reposting this in case it got buried :)
"Hi. I'm just getting started with saleor, evaluating it for a project in the coming months. I was wondering if someone could perhaps give me a brief description of how one might go about extending saleor. Does it have a sort of plugin system? Or do we extend its classes to add or customize its behavior? For example, our client wants a wishlist and a loyalty point system with ranks (gold, silver, etc). They also want different prices based on customer role (individual, wholesaler). What would be the recommended approach generally?
If it uses plugins, is there a central place I could find them? Ah, and our client wants a custom Payment provider as well."
NyanKiyoshi
@NyanKiyoshi
Aug 20 2018 18:19

As of now, you would have to edit the core.

and our client wants a custom Payment provider as well.

Depends what you mean by "custom", if it is a non already implemented payment provider, you would have to implement it which would be fairly quick and easy to do. You have the list of the existing providers here and two more here, in addition to razorpay.

Khalifa Lame
@khalibloo
Aug 20 2018 18:21
Edit the core... that basically means once the project takes off, there would be little possibility of updating and getting access to new features, right? sounds like a big trade-off
with the custom payment backend, they have a specific bank whose API they want us to interface with. It's a very tiny island country here, and that one bank serves the majority of the population, so I can't blame them for that choice
so at least the payment part looks pluggable from the way you described it. Good to know
NyanKiyoshi
@NyanKiyoshi
Aug 20 2018 18:26
It would depend how heavy are the changes. You could rebase your changes to the upstream or cherry pick every change (features or fixes) that you want to apply to your version.
Alrighty
Khalifa Lame
@khalibloo
Aug 20 2018 18:26
ok, thanks very much
Stephen Moloney
@stephenmoloney
Aug 20 2018 18:59
@patrys thanks, have a good evening.