Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Sep 11 2018 10:21

    asahd on v2.5.2

    (compare)

  • Sep 11 2018 10:21

    asahd on v2.5.2

    (compare)

  • Sep 11 2018 10:19

    asahd on master

    Restore nginx.conf.example (compare)

  • Sep 11 2018 10:05

    asahd on v2.5.2

    (compare)

  • Sep 11 2018 10:04

    asahd on feature-symantic

    (compare)

  • Sep 11 2018 10:04

    asahd on master

    Added full document toggle to s… Added post and patch metadata. Updated Metadata to update stat… and 6 more (compare)

  • Sep 11 2018 10:04
    asahd closed #1232
  • Sep 11 2018 09:05
    asahd labeled #1232
  • Sep 11 2018 09:05
    asahd unlabeled #1232
  • Sep 11 2018 08:42
    asahd synchronize #1232
  • Sep 11 2018 08:42

    asahd on feature-symantic

    Fixes statement creations being… (compare)

  • Sep 10 2018 15:07

    happy-machine on Statement_component

    fixed default naming in Stateme… (compare)

  • Sep 10 2018 12:45
    asahd synchronize #1232
  • Sep 10 2018 12:45

    asahd on feature-symantic

    Changes Forwarding UI, updates … (compare)

  • Sep 10 2018 11:45
    ht2 synchronize #1232
  • Sep 10 2018 11:45

    ht2 on feature-symantic

    Start of shareable dashboards u… Removed filter required paramet… Added validation for shareable … and 49 more (compare)

  • Sep 04 2018 16:29
    cutz opened #1246
  • Sep 04 2018 14:20
    samjaved closed #1245
  • Aug 30 2018 22:20
    ryansmith94 assigned #1242
  • Aug 30 2018 16:43

    happy-machine on Statement_component

    working as Map (compare)

Brandon Hixson
@TheDonOfBran_twitter
Anyone know the current password for demouser@learninglocker.net to login to this demo: https://demo.learninglocker.net/login ?
Jesus M Bianco T
@jbiancot_twitter
@ryansmith94 NodeJS 10 has already some time on the market now, Learning Locker should support on NodeJS, maybe some work has to be done in there.
Shane Crockett
@Blumanfry_twitter
@ryansmith94 you mentioned pulling the env files from S3; is this with the community AMI? If so, how did you make the AMI aware of the S3 env files?
Tejovathi Consulting
@tejovathiconsul_twitter
@Ian247 Dear Mr. Blackburn, Thanks for your quick reply. I did not install Learning Locker using AMI. I installed using custom installation on a CentOS 7 VPS. I searched thoroughly for ll_credentials.txt and i could not locate it in my system anywhere. Please assist in getting to know the username and password.
Ryan Smith
@ryansmith94
@Blumanfry_twitter no that's with our own AMI for instances of LL that we host
@tejovathiconsul_twitter you'll need to use the createSiteAdmin CLI operation http://docs.learninglocker.net/guides-cli/
@jbiancot_twitter for sure, surprised we don't already support Node 10, doubt there were any breaking changes for us, do you know why it didn't work on Node 10?
@TheDonOfBran_twitter demo.learninglocker.net is a demo of Learning Locker Enterprise and is invite only, please get in touch with @Ian247 to request a demo
Joost van Doremalen
@doender

Hi all, I've just posted my question on github, but this might be a better place to ask it.

As far as I understand Learning Locker uses Redis to cache statements that are not stored yet, so its size should remain relatively constant over time (?).
However, on our instance, the Redis store has grown to be around 11GB. This in turn makes redis-rdb-bgsave take up a long time. INFO keyspace is db0:keys=16788661,expires=11,avg_ttl=16841

Is this a normal size or is there some way to decrease it?

Agli Pançi
@aglipanci

@ryansmith94 I have moved Mongo on a separate server as @ht2 suggested and the UI seems to work really great but when it comes to the xAPI I am getting this error on the logs:
5|xAPI | 2019-08-13 08:04:25:215 - error: Failed Mongo connection: failed to connect to server [localhost:27017] on first connect [MongoNetworkError: connect ECONNREFUSED 127.0.0.1:27017]

I have updated the xAPI .env file with the new mongo server endpoint, and it looks like this: MONGODB_PATH=mongodb://user:pass@lrs-db-internal.server.com:27017/learninglocker_v2

I am missing anything obvious here ?

Seems like it still tries to connect locally.
Agli Pançi
@aglipanci
Checked this: https://docs.learninglocker.net/guides-configuring/ and the mongo variabe for the xAPI seems to be different. MONGO_URL and not MONGODB_PATH but still is trying to connect with localhost.
Agli Pançi
@aglipanci
Hmm never mind. The MONGO_URL seems to work but I have “illegal” characters on my password and they are not supported.
Ryan Smith
@ryansmith94

Hi @doender, the part I've quoted below from your comment is not true. We use Redis as a pub/sub so that the app can subscribe to new statements for worker jobs (personas, query builder cache, etc).

As far as I understand Learning Locker uses Redis to cache statements that are not stored yet, so its size should remain relatively constant over time (?).

We also use Redis to cache the results of aggregation queries. If you're making heavy use of the dashboards and visualisations in user interface it's possible it could be growing because of that, I think we put TTLs on the caches so they should expire. If you're Redis instance is growing just with statement IDs then it's possible you don't have workers running or configured correctly.

Joost van Doremalen
@doender

Thanks @ryansmith94 !

I've ran redis-cli --bigkeys to gain more insight into the content of the store:

Sampled 16788658 keys in the keyspace!
Total key length in bytes is 742012940 (avg len 44.20)

Biggest string found 'bull:DEV_STATEMENT_QUERYBUILDERCACHE_QUEUE:3337928:lock' has 36 bytes
Biggest   list found 'bull:STATEMENT_QUERYBUILDERCACHE_QUEUE:wait' has 4904921 items
Biggest    set found 'bull:STATEMENT_FORWARDING_QUEUE:completed' has 3 members
Biggest   hash found 'bull:STATEMENT_FORWARDING_QUEUE:784130' has 11 fields

16 strings with 207 bytes (00.00% of keys, avg size 12.94)
8 lists with 16788623 items (00.00% of keys, avg size 2098577.88)
3 sets with 8 members (00.00% of keys, avg size 2.67)
16788631 hashs with 139532114 fields (100.00% of keys, avg size 8.31)
0 zsets with 0 members (00.00% of keys, avg size 0.00)

I've checked a number of values for the keys formed as bull:STATEMENT_FORWARDING_QUEUE:* as hashes seem to take up all of the space. They all contain information on statements that are present in the LRS. For example:

redis-cli HGETALL bull:STATEMENT_FORWARDING_QUEUE:814131
 1) "data"
 2) "{\"statementId\":\"5bbdd45b5c9b410fc6d22e8c\"}"
 3) "opts"
 4) "{\"removeOnFail\":false,\"removeOnComplete\":false}"
 5) "progress"
 6) "0"
 7) "delay"
 8) "0"
 9) "timestamp"
10) "1539167323294"
11) "attempts"
12) "1"
13) "attemptsMade"
14) "0"
15) "failedReason"
16) ""
17) "stacktrace"
18) "[]"
19) "returnvalue"
20) "null"

and statement 5bbdd45b5c9b410fc6d22e8c is stored in Mongo.

What's the reason these keys are in Redis? And could they all be removed to shrink the size?

karlmanning1
@karlmanning1
Similar to @tejovathiconsul_twitter I have just installed Learning Locker using the script/custom installation. Everything seems to be ok, except that I cannot login to the main page of Learning Locker - I get login incorrect. I have run the createSiteAdmin from the cli and also tried to reset the password from the main learning locker page. I've not received a reset password email (although that could be blocked by our email system). Any suggestions please
Jesus M Bianco T
@jbiancot_twitter
Hi folks, I am seeing LL v4.x, what is new?
ysguy
@ysguy
Hello team, i am supporting a company that has Learning Locker set up on AWS. The CPU usage is very high when a few users start a course and progress through it
we tried increasing the VM size (CPU capacity) and the problem persists
I dont see much in the documentation on troubleshooting / optimization for CPU.
ysguy
@ysguy
Seems MongoDB is running at 100%
M1sterM1ke
@M1sterM1ke
@ysguy did you try the suggested indexing suggested for increased mongo performance? We had some performance issues before running a few of those. might be good (and easy to try) https://docs.learninglocker.net/guides-indexing/
thekaister
@thekaister

Hi folks,

One of my clients is interested in Learning Locker. I saw that for production it's recommended to have the following:
1 Load balancer (e.g. AWS Elastic Load Balancing);
2 Learning Locker servers (e.g. AWS EC2);
3 Mongo servers in a replica set (e.g. Atlas);
1 Redis server (e.g. AWS ElastiCache).

We currently host their Moodle on a standalone server running mySQL and Apache. I don't think our client is ready to allocate resources for the described production infrastructure above. So I have 3 questions:
1-Would it be possible to host Learning Locker within the same Moodle server ?
2-If not, would it be possible to host Learning Locker on its own standalone server, and what are the recommended server specs ?
3-If not, what's the minimum requirements you can suggest ?

Thank you

ysguy
@ysguy
I did apply the indexes
interestingly we are on a old release (2.6.1) and thats likely a good step. As well the AWS server ran out of disk space shortly after troubleshooting so i added another volume and moved the LRS (/usr/local/learninglocker) to the new drive. Everything started up and when the users reconnected the CPU is low. It appears that something triggers a massive amount of activity in Mongo and then crushes the DB going forward.
ysguy
@ysguy
@thekaister you can run it on any hardware and the disk usage is low depending on how many courses/users you are using. I dont see a scaling recommendation so i suggest running it and doing some testing on any equipment you might have. The core system is NodeJS + MongoDB so that wont compete with your apache + mysql config. I'd estimate allocating at least 4GB ram and 20GB storage to start with and a dual CPU @ 2Ghz for resourcing. As with anything, when its not under load it its not going to exploit your other services.
thekaister
@thekaister
@ysguy Thank you! I appreciate your assistance :)
Tejovathi Consulting
@tejovathiconsul_twitter
@ryansmith94 Thanks. I was able to create the SiteAdmin using the CLI and access the Dashboard. Really appreciate your support.
Ryan Smith
@ryansmith94
Hi @doender, thanks for the extra info. Seems odd that one of your queues has that many items. I think it's possible that your workers aren't running. When you run pm2 status on the server what do you get back?
Hi @karlmanning1, sounds like something might be misconfigured. I would clear your Mongo database, retry the createSiteAdmin CLI, go to Learning Locker in your browser, open your network panel in your dev tools, try to login as the new admin, then check what the response is from the server in your network panel
Ryan Smith
@ryansmith94
Hey @jbiancot_twitter, yep, nothing particularly exciting in v4, we just made a breaking change in our API where we removed the ability to patch org settings on a user because the logic was quite complex for that part due to the permissions. To make it simpler to understand we created a new org settings API, it's not documented just yet but you can check it out in the network panel when you change org settings. Since we follow semantic versioning we released this as v4 so it was clear we made a breaking change. The change may affect people that are automatically inviting users to organisations via the API
Ryan Smith
@ryansmith94
Thanks for helping out thekaister @ysguy :+1: With regards to your own problem, glad you've applied the indexes. Upgrading Learning Locker might also help since the high Mongo activity could simply be caused by a bug that we've since fixed. I'd be interested to know if you can track down what it is that seems to trigger a massive amount of activity in Mongo. We have instances processing millions of statements each day on the recommended production setup mentioned above, so I'm guessing there might be something specific to you that is triggering the heavy load on Mongo
Thanks for trying to help out @M1sterM1ke :+1:
Hey @tejovathiconsul_twitter glad you got in! No problem
ysguy
@ysguy

@ryansmith94 i'll check into upgrading in the near future, we need to plan and schedule and outage. It seems after the LRS is restarted the CPU usage returns to a humane amount. When checking transactions in mongo there was about a 6000ms delay in running each transaction and cpu was 100%, but when i checked the current transactions and the logs, they all seemed normal.

When checking PM2 monit those processes were <8% CPU at any given moment and usually 0-1% CPU, even when mongodb was at 197% (we have 2 cpu's allocated).

Ryan Smith
@ryansmith94
Hey @ysguy, sounds good, we've been upgrading LL without downtime for at least a couple of years now so you should be able to upgrade without any too if that helps. Interesting, does the Mongo CPU return to normal levels at some stage or is it just maxed out all the time?
diversityedu
@diversityedu
Hi All. We are getting some errors trying to migrate the ll database. Please see below. The xAPI service is restarting non-stop, and that there are two observed errors:
1 - can't migrate database, running yarn migrate at /usr/local/learninglocker/current/webapp yields error Error: Db migration (20171008104700_personas_indexes) does not match (20171122100800_common_indexes) and stops
2 - xAPI logs show error error: uncaughtException: ENOENT: no such file or directory, open '/usr/local/learninglocker/releases/ll-20190114-dcb84057e8d3aea11a00cbf8bb52d42a7d16fedd/xapi/storage/state/5ca0691ddc3536056ffe31fc/states/5d59d8067760e9efdc8a923c.bin' with different state name each time and restarts
ysguy
@ysguy
@ryansmith94 it appears to run normally for a period of time then suddenly spike and hold there until restarted. When i check the backlog of queries there are no special statements looping for any long period of time, it feels like the queue is just hammered by incoming requests. It usually happens during our classes with about 10-20 people, the LL instance is on a T2.Large so it doesnt appear to be a resource problem in terms of cpu. I believe the update is the best course of action.
Nic Johnson
@nicjohnson
@ryansmith94 I'm seeing the same behavior as @ysguy on our Mongo server. CPU Pegged at 100% for several minutes then falls off. It's traffic-related because it goes quiet during the night with few or no users. I'm going to run the index commands to see if we get some love.
Nic Johnson
@nicjohnson
@ryansmith94 So, the initial changes on the Mongo server look amazing after adding those indexes. I'll watch it and see how it looks tomorrow. My question now, is: Why doesn't the LL install script offer to create those indexes automatically?
DellaStapleton
@DellaStapleton
Hi there, please can someone assist with the below error I'm getting on all my visual charts. This has happened since upgrading our xapi server. Clients are currently using this system for reporting so this is unfortunately an urgent request. Please HELP!
Screenshot 2019-08-19 at 22.29.21.png
Tejovathi Consulting
@tejovathiconsul_twitter
@ryansmith94 we have a self assessment that results in 16 scores. We would like to store the scores in the LRS and then retrieve it to customize the content of the course. We plan to create the self assessment using HTML5, CSS and JS (scores are not stored in any other database). We plan to package it as TINCAN API package and are using Drupal based Opigno as the LMS , Learning Locker as the LRS. Any links to how this can be implemented will be greatly appreciated.
Ryan Smith
@ryansmith94
Hi @diversityedu, thanks for getting in touch, please can you add that information to issue #1288, that will be investigated very soon
Okay thanks for the extra info @ysguy. You probably need some more worker processes running (potentially on separate machines). Sounds like the worker process is just getting too busy
Hey @nicjohnson, to be honest I thought it did, please feel free to raise this as an issue on the install script repository https://github.com/LearningLocker/deploy
Hi @DellaStapleton, please report your issue on our Github issue tracker and send me a link to that in here so I can get someone to investigate https://github.com/LearningLocker/learninglocker/issues
Ryan Smith
@ryansmith94
Hi @tejovathiconsul_twitter, we have a lot of videos (linked below) and also some code examples in the xapi-demos repository
ETL part 1: https://vimeo.com/273458734
ETL part 2: https://vimeo.com/273458727
ETL part 3: https://vimeo.com/273652618
ETL part 4: https://vimeo.com/273652648
ETL part 5: https://vimeo.com/273652598
Launching content: https://vimeo.com/273848934
ADL Launch part 1: https://vimeo.com/272792123
ADL Launch part 2: https://vimeo.com/272792141
ADL Launch part 3: https://vimeo.com/272792152
Tejovathi Consulting
@tejovathiconsul_twitter

Hi @ryansmith94 , thanks a lot. In the meanwhile, I tried using the TinCANJs by Rustici Software. From Javascript, I tried to send a Test Statement. The TinCan object is able to connect to the LRS.

Followed the code given in the link: http://rusticisoftware.github.io/TinCanJS/

However, when we try to insert statement, we get an error "net::ERR_CONNECTION_CLOSED
nativeRequest @ tincan-min.js:3
sendRequest @ tincan-min.js:1
saveStatement @ tincan-min.js:1
send_statement @ (index):50
create @ (index):99
onload @ (index):126

Please help us to point to any resource.