Erudika/scoold

An open-source StackOverflow clone, optimized for Heroku and the cloud

Alex Bogdanovski

@albogdano

you could perhaps delete their Scoold accounts and if their accounts have no associated posts or reputation, they probably won't notice and would be able to log back in

pikrakpzu

@pikrakpzu

@albogdano Before support of nested fields user accounts were created with invalid name/email.
Now accounts are created with correct info, but there are two cases:

Users already created - yes, while they have no posts or customized profile photo/settings they won't notic.
Existing users info changed (for example get married and chcnged last name, or changed department included in user name), even email could change.

I think it should get updated at Scoold on every sign in, because that user info is owned by OIDC/LDAP and they are the only source of truth about user.
Worth to notice that we have no picture config and it's only customizable by user. It would be evil to reset their image to default on every sign in :smiling_imp:
But when our OIDC let us fill para.security.oauth.parameters.picture = "picture" i would set para.avatar_edits_enabled = false and expect pictures to get updated on sign in just like other data from identity providers.

Alex Bogdanovski

@albogdano

@pikrakpzu that's exactly how Para does it - user info is updated on every sign in request. Users are found by their email address, so if their email has changed in LDAP the user will no longer be found and a new user will be created

pikrakpzu

@pikrakpzu

@albogdano What is use of para.security.oauth.parameters.id = "sub"? That may be the case why users were not updated or recreated trough OIDC, as theirs No Name account had no email specified.

Alex Bogdanovski

@albogdano

@pikrakpzu yes, the sub should not change because when the user is created Para will create a special identifier object with id oa2:sub and will try to find the user based on that id later

pikrakpzu

@pikrakpzu

@albogdano Ok, the sub is same, but email was empty, and now it's set.
User sign in to old account, but his info is not updated.
Is that suspected behaviour?
What if I fill correct email on user in mongo, should it fix, or is email somehow mixed with sub?

Alex Bogdanovski

@albogdano

@pikrakpzu the info should be updated if it has changed since the last login but check the logs and make sure that Para doesn't create a new user

I'm a bit confused - is user data inside LDAP or MongoDB?

you can try to clean up identifier objects in Para - search for oa2:sub objects where sub is the sub ID

you can safely delete those objects in Para for the problematic users

also you can check that the user info is reflected in each Para object of type user for each problematic user

the data flow is: OIDC -> Para user object -> Scoold profile object

Alex Bogdanovski

@albogdano

for each user object there is only one Scoold profile object and multiple identifier objects - one for each identity provider

I guess what happened is this - user with No Name logged in with sub 123 then you changed settings and that same user logged in but now with sub 345 and a new user was created so you now have "dead" user objects which need to be deleted

pikrakpzu

@pikrakpzu

@albogdano AD is a storage for user data, LDAP and OIDC are just different ways to get them.
I could have been wrong about the Scoold users being stored in Mongo (which we use with Para), as You say they are stored in Para. So there's no mongo in process.
OIDC data for para between versions was like:

scoold v. | sub | email            | name
1.45.0    | 123 |                  |
1.46.3    | 123 | user@company.com | Best User (His Department)

Also users were able to sign in their No Name accounts using LDAP wchich has a valid user info.
In fact that was our workaround - first login with username/pass trough LDAP to create Scoold account, then use OIDC wchich supports SSO.
In result they got full user data in Scoold, and sign in with OIDC didn't clear user name or email.

While signed in with opposite order, No Name account were created for Scoold and no info were updated after LDAP sign in to the same account.

I'll try to investigate this more after the weekend.
Thank you for your help and have a nice weekend :smiley:

Perfect, had to edit multiple times to get table rendered right :smile:

Alex Bogdanovski

@albogdano

@pikrakpzu ok, let me know what you find. And, yes MongoDB is used by Para to store data so you are in fact using it.

I am not quite sure how we can fix your problematic users in this situation but definitely check the user objects in Para and if they have the correct user data from OIDC

pikrakpzu

@pikrakpzu

@albogdano At last I had some time to investigate Scoold users.
Looks like there is a problem with profile update.
While user was updated with correct name, profile have old wrong name and new correct originalName.

Method I used to check this.

Search query:
para-cli search "creatorid:ID* OR id:ID*"
Where ID is a id field value from result of:
para-cli search "type:user AND email:greatuser*"

For users that used ldap and oa2 to login, result is (excluding posts/queries/votes etc.):

sysprop | identifier | id:ldap:...
sysprop | identifier | id:oa2:...
type | profile
type | user

On profile we have name and originalName.
profile where identifier for ldap was created before oa2 got valid name and originalName seems ok.
profile where identifier for oa2 was created before ldap (and before last update which fixed oa2) got invalid name but originalName seems ok.

Also I have found that long time ago I had different name pattern from ldap like User Name (login) and oa2 is User Name (Office).
My own profile is one of oldest and it has name in old format, but originalName have been updated from oa2.
Last time I used oa2 to signin.

Seems that Scoold uses name field from profile which is not updated with data from identity providers (ldap, oa2).
I didn't write about user object as it contains valid name.
If it would help I can paste this objects.

Alex Bogdanovski

@albogdano

@pikrakpzu yes, there's an option for that - para.name_edits_enabled = false this way, you disable name editing on Scoold and force name updates to be passed on from the identity provider (LDAP or OA2)

the idea is that once a Scoold account is created (type: profile) users can edit their name independently from what's provided by the IDP. For better synchronization, you can turn off that feature and always sync the names

I'm sorry if I forgot to mention that earlier in our conversation... :neutral_face:

could've saved you some time

pikrakpzu

@pikrakpzu

@albogdano That's exactly what I need, i'm glad we finally get to it :smile:

rafik777

@rafik777

@albogdano How to extend the default session duration? Token expiration? I am using LDAP login. I have to sing in too often...

Alex Bogdanovski

@albogdano

@rafik777 for example, set para.session_timeout = 21600 for 6h session duration (in seconds) and restart. In 6h the auth cookie will
expire but the JWT inside the cookie may still be valid. So make sure you also configure Para to issue JWTs with the same
validity period using para.jwt_expires_after = 21600 or by updating your Para app object to have a property tokenValiditySec: tokenValiditySec

se-alexnsa

@se-alexnsa

Hey there. Whenever we restart the para service (in AWS Fargate, using RDS mysql to store database), we have a the server is unhealthy and must be re-indexed/re-built. Each time, we use para-cli rebuild-index and it works - it gets re-indexed and the server is healthy again.
To save us from running that para-cli command every time, will changing the para config to use the para elasticsearch plugin work to prevent that re-indexing error coming up? and will we have to also spin up an AWS Elasticsearch service to use alongside the plugin?

Alex Bogdanovski

@albogdano

@se-alexnsa that is not normal - there must be something wrong with the Par configuration or the search index storage. Are you sure the server is unhealthy because the index has been lost? What's the exact message in the logs?
If you use the ES plugin, you'll definitely need to connect to some Elasticsearch server of your own

se-alexnsa

@se-alexnsa

Yes, will set up an Elasticsearch server

Regarding the server unhealthy message, it was this

2021-11-02 10:56:11 [WARN ] Server is unhealthy - the search index may be corrupted and may have to be rebuilt.

then after I ran para-cli rebuild-index, the log message was

[INFO ] Server is healthy.

Alex Bogdanovski

@albogdano

@se-alexnsa I'm pretty sure that your search index is not stored to disk properly. check if the ./data folder is mounted properly as a volume if using docker.

Alex Bogdanovski

@albogdano

@pikrakpzu in the next release of Scoold, you'll be able to make the feedback page available only to registered users by setting para.is_default_space_public = false

se-alexnsa

@se-alexnsa

@se-alexnsa I'm pretty sure that your search index is not stored to disk properly. check if the ./data folder is mounted properly as a volume if using docker.

We're using AWS Fargate.

When running locally using docker, data is at /para/data
I didn't physically mount it, but it was already there when I spun up the container. Do we need to do something differently there?
And how can that be mounted when using Fargate too?

Alex Bogdanovski

@albogdano

@se-alexnsa I am no expert in Fargate but there should be a volumes section in the settings. my guess is that Fargate doesn't allow local disk writes and all writable volumes must be attached explicitly.

se-alexnsa

@se-alexnsa

OK. Also, if I install the Elasticsearch plugin as per the docs https://github.com/erudika/para-search-elasticsearch, do we also have to add the erudikaltd/para-search-elasticsearch image to our already existing dockerfile?

Alex Bogdanovski

@albogdano

@se-alexnsa yes, you need to add the plugin image as a layer on top of the Para Docker image. Also pay attention to where the ./data volume is mounted - without it the indexing won't work

se-alexnsa

@se-alexnsa

locally the /data volume is mounted in /para directory
/para/data. That was without me having to manually mount it.
Should /data be mounted in /para for the Fargate container too?

Alex Bogdanovski

@albogdano

@se-alexnsa yes, that is the default location and it should work on Fargate as well

pikrakpzu

@pikrakpzu

@pikrakpzu in the next release of Scoold, you'll be able to make the feedback page available only to registered users by setting para.is_default_space_public = false

Thanks, I've just done tests and going to prod with 1.46.5 :smile:

pikrakpzu

@pikrakpzu

@albogdano Hello! :)
My organization's security team found few bugs in Scoold.

First is with avatar url set by users.

There's no check of what user set, and its served to all other users as avatar.
User can setup malicious link, containing js code which for exapmle could steal auth tokens from cookies.
As example this URL were given:

https://suvroc.github.io/security-demos/XSS/reflectedXSS.html?name=%3Cscript%3Ealert%3C/script%3E

Recommendations:

User input data filtering, for example content-type check.
Implementing CORS(Cross-Origin Resource Sharing)
Secure headers
Implementing CSP(Content Security Policy)

References:

https://owasp.org/www-community/attacks/Code_Injection

Second is about session.

Cookie scoold-auth with JWT can be stolen and it's valid for 7 days.
Also user session is not closed after logout, with stolen cookie it's possible to get access without login.
Recommendations:

Proper session closing according to the OWASP: https://cheatsheetseries.owasp.org/cheatsheets/Session_Management_Cheat_Sheet.html
Allowing only one session per user, by closing older sessions while establishing a new one.

References:

I've seen para.jwt_expires_after few posts before.
Correct me if i'm wrong, when set to 300 it will make JWT expire after 5min? It should fix 7 days problem.
But it won't solve existing session after logout.

Both problems were marked as high security risk.

Is there anything that can be done to fix them?

Alex Bogdanovski

@albogdano

@pikrakpzu Hey, thanks for this! I will work on the code to address these points.

On the first topic: I don't think that's even possible because Scoold has a pretty strict CSP - https://cspvalidator.org/#url=https://live.scoold.com
Security headers are also in place - https://securityheaders.com/?q=https%3A%2F%2Flive.scoold.com&followRedirects=on
There's always room for improvement here and I will soon add filtering of the actual URLs for avatars
On the second topic: Scoold does not use sessions at all - it uses JWTs instead. It's a similar concept and JWTs can be made to expire after a configurable time period.
para.session_timeout is the validity period in seconds for the auth cookie itself, para.jwt_expires_after is the validity period in seconds for the JWT token inside the auth cookie.
Again, here we can tighten security by only allowing one valid JWT per user/browser. It should be pretty straightforward to implement this.

se-alexnsa

@se-alexnsa

yes, you need to add the plugin image as a layer on top of the Para Docker image

If I do that, do I still need to download this jar file as well:
https://github.com/Erudika/para-search-elasticsearch/releases

Alex Bogdanovski

@albogdano

@se-alexnsa no need - the dockerfile for the ES plugin will download the JAR into the lib folder next to para.jar

se-alexnsa

@se-alexnsa

great
also since mounting the volume, we no longer get indexing errors, but now instead have this error

Application run failed org.springframework.context.ApplicationContextException: Failed to start bean 'webServerStartStop'; nested exception is org.springframework.boot.web.server.WebServerException:

could this be because something has changed location now?

Perhaps something else needs to be mounted manually now as well?

Alex Bogdanovski

@albogdano

@se-alexnsa I will need to see the full stack trace to tell you what caused that exception.

se-alexnsa

@se-alexnsa

      ____  ___ _ ____ ___ _ 
     / __ \/ __` / ___/ __` /
    / /_/ / /_/ / /  / /_/ / 
   / .___/\__,_/_/   \__,_/  v1.40.1-SNAPSHOT
  /_/                        

2021-11-24 14:08:45 [INFO ] --- Para.initialize() [production] ---
2021-11-24 14:08:45 [INFO ] Loaded new DAO, Search and Cache implementations - SqlDAO, LuceneSearch and CaffeineCache.
2021-11-24 14:08:46 [INFO ] HikariPool-1 - Starting...
2021-11-24 14:08:46 [INFO ] HikariPool-1 - Start completed.
2021-11-24 14:08:51 [INFO ] Server is healthy.
2021-11-24 14:08:51 [INFO ] Found root app 'para' and 0 existing child app(s).
2021-11-24 14:08:55 [INFO ] Queue 'para-default' could not be found: software.amazon.awssdk.services.sqs.model.QueueDoesNotExistException: The specified queue does not exist for this wsdl version. (Service: Sqs, Status Code: 400, Request ID: e7c38864-0824-594a-91fd-d27c419227a6, Extended Request ID: null)
2021-11-24 14:08:55 [ERROR] null
java.util.concurrent.ExecutionException: software.amazon.awssdk.services.sqs.model.SqsException: Access to the resource https://sqs.eu-west-1.amazonaws.com/ is denied. (Service: Sqs, Status Code: 403, Request ID: 08a4be23-07df-519e-9d7e-e43b4e68cf72, Extended Request ID: null)
    at java.base/java.util.concurrent.CompletableFuture.reportGet(Unknown Source)
    at java.base/java.util.concurrent.CompletableFuture.get(Unknown Source)
    at com.erudika.para.queue.AWSQueueUtils.createQueue(AWSQueueUtils.java:95)
    at com.erudika.para.queue.AWSQueue.getUrl(AWSQueue.java:89)
    at com.erudika.para.queue.AWSQueue.startPolling(AWSQueue.java:65)
    at com.erudika.para.ParaServer.initialize(ParaServer.java:157)
    at com.erudika.para.ParaServer.runAsJAR(ParaServer.java:423)
    at com.erudika.para.ParaServer.main(ParaServer.java:432)
    at com.erudika.para.Run.main(Run.java:26)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.base/java.lang.reflect.Method.invoke(Unknown Source)
    at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:49)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:108)
    at org.springframework.boot.loader.Launcher.launch(Launcher.java:58)
    at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:467)
Caused by: software.amazon.awssdk.services.sqs.model.SqsException: Access to the resource https://sqs.eu-west-1.amazonaws.com/ is denied. (Service: Sqs, Status Code: 403, Request ID: 08a4be23-07df-519e-9d7e-e43b4e68cf72, Extended Request ID: null)
    at software.amazon.awssdk.services.sqs.model.SqsException$BuilderImpl.build(SqsException.java:95)
    at software.amazon.awssdk.services.sqs.model.SqsException$BuilderImpl.build(SqsException.java:55)
    at software.amazon.awssdk.protocols.query.internal.unmarshall.AwsXmlErrorUnmarshaller.unmarshall(AwsXmlErrorUnmarshaller.java:99)
    at software.amazon.awssdk.protocols.query.unmarshall.AwsXmlErrorProtocolUnmarshaller.handle(AwsXmlErrorProtocolUnmarshaller.java:102)
    at software.amazon.awssdk.protocols.query.unmarshall.AwsXmlErrorProtocolUnmarshaller.handle(AwsXmlErrorProtocolUnmarshaller.java:82)
    at software.amazon.awssdk.core.http.MetricCollectingHttpResponseHandler.lambda$handle$0(MetricCollectingHttpResponseHandler.java:52)
    at software.amazon.awssdk.core.internal.util.MetricUtils.measureDurationUnsafe(MetricUtils.java:64)
    at software.amazon.awssdk.core.http.MetricCollectingHttpResponseHandler.handle(MetricCollectingHttpResponseHandler.java:52)
    at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler.lambda$prepare$0(AsyncResponseHandler.java:89)
    at java.base/java.util.concurrent.CompletableFuture$UniCompose.tryFire(Unknown Source)
    at java.base/java.util.concurrent.CompletableFuture.postComplete(Unknown Source)
    at java.base/java.util.concurrent.CompletableFuture.complete(Unknown Source)
    at software.amazon.awssdk.core.internal.http.async.AsyncResponseHandler$BaosSubscriber.onC

at the top, it says a sqs queue is missing (para-default?), but we've never needed to make a queue before in order to get it working

Alex Bogdanovski

@albogdano

@se-alexnsa do you set para.webhooks_enabled = true or para.q = "sqs" anywhere in your configuration?

se-alexnsa

@se-alexnsa

this only is appearing after mounting the volume, so I wondered if I have to now manually mount other things as well

I'll check...

Where communities thrive

People

Repo info

Activity

Method I used to check this.

First is with avatar url set by users.

Second is about session.

Both problems were marked as high security risk.