Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
    Ben Yoshino
    @byosh_gitlab

    It's been a real long time since I came back to gitter. I updated my installation of beer garden with the latest version, (this time in a virtual environment) but it seems the old systems I wrote no longer works. The error I am receiving is "Backend request timed out" when the system starts up.

    This is on a CentOS 7.5, on Python 3.7. Thanks!

    Logan Jones
    @loganasherjones
    Hi @byosh_gitlab welcome back! :) The error you're receiving seems to indicate that bartender is down. Are you sure bartender is running?
    Logan Jones
    @loganasherjones
    Hi @mshirb_gitlab glad you like the tool! The documentation for authentication/authorization is a work in progress, but yes, you can do AA in Beer Garden up to a point
    burzikw
    @burzikw
    Hello @loganasherjones,
    I attended the Open Space talk at Pycon and was inspired to give Beer Garden a go, especially with the intention of spinning up quick plugins with Docker to help get some of our plugins accessibility across our organization. We've had luck with the docker-compose you've provided on the site, along with the example plugins compose. The issue I'm having is creating a plugin from scratch. I've attempted to follow the docs provided, but I think I'm a little lost. I have the generic Hello World plugin in the my_plugin-1.0.0.0.tar.gz saved into the folder referenced in the compose. I've composed up and when I go into the container, I see it there and untar/compress it.... That's about as far as I can get. I try to rescan the system directory... but it's not going so well. So at the risk of looking foolish in this chat, I thought I would pose the question and see what I get.
    Logan Jones
    @loganasherjones
    Hey @burzikw! Good to hear from you again. No need to worry about sounding foolish. We are here to help :)
    Let me make sure I have your problem correct. You've stood up BG using our docker-compose file from our GitHub repo. After you've stood that up, you have a plugin that you've created and put into a .tar.gz. You've transferred the .tar.gz to the correct folder in the compose, exploded the file, and then tried to rescan but to no avail
    I would be interested in seeing what the bartender logs had to say when you rescanned
    you can find that out with: docker-compose logs -f bartender
    Matt Patrick
    @hazmat345
    Hi @burzikw, just a quick check, did you change the BG_PUBLISH_HOSTNAME in the docker-compose.yml file?
    burzikw
    @burzikw

    I found the issue. Seems I didn't read enough :)

    TLDR: I was missing some required fields in the beer.conf

    After I ran the docker-compose logs (thanks @loganasherjones ) and saw the error messages, I went back and read the entry for the required fields below in the tutorial. Might want to consider updating the "Configure your plugin" portion on the plugin local guide. For a noob like myself, it can be misleading. I was under the impression all I needed was the PLUGIN_ENTRY.
    That all said, I have my first plugin running. Now the real fun begins.
    Thanks again!

    Logan Jones
    @loganasherjones
    Great! Glad you got it figured out. We are looking at the docs now anyway.
    Thanks so much
    Matt Patrick
    @hazmat345
    Yeah that’s great, thanks for the feedback. Glad you got it working 😁
    burzikw
    @burzikw
    Thanks again for the help. I've been busy working for a while and I think what I have is stable enough, I would like to move my containers from one of our testing environments to a more stable one. The issue is that now that I'm moving forward, I need to consider more about the administrative setting for beer-garden itself. I was looking for a bit of documentation on this but was coming up short. I apologize that I don't have real specific questions just yet, mainly because I was hoping there would be something for me to reference that could lead me to ask meaningful questions. and if I'm oblivious to where it could be found, I apologize, but this time I tried to do my homework. :)
    Logan Jones
    @loganasherjones
    Hi @burzikw we are still working on some of the documentation.
    would you mind checking this out: https://beer-garden.io/docs/user_manual/ and telling us what you think?
    burzikw
    @burzikw
    I think I really appreciate the speedy response. You'll have to give me a minute to digest what you posted, but at a cursory glace it looks good. If I could ask, do you have examples of the syntax for calling the REST API as a specific user? I'm also curious if the request page would have any identification as to who ran each request?
    Logan Jones
    @loganasherjones
    Hmm, I don't think we have any examples of that posted anywhere, but the simplest thing you can do to get that behavior is use the SystemClient and pass in the username/password you want to use. It will automatically take care of making the correct auth requests for you.
    The process goes something like this:
    1. Get a token by using your username/password
    2. Use the token in the POST on /requests
    But if you don't mind using the SystemClient I highly recommend that instead. That way you can just avoid all the headache other than just passing username/password.
    Bill
    @gershwinlabs
    https://github.com/gershwinlabs/pylint-beergarden
    @loganasherjones let me know if you want to pull it in with the other repos in the beer-garden space
    Logan Jones
    @loganasherjones
    Thanks I will take a look
    Peter Crampton
    @cramppet
    Hello, this is perhaps an obvious question; but is it possible for me to configure Beer Garden such that the cleartext credentials shown in the screenshot don't get included in API responses? Are they needed in any way? https://imgur.com/a/1MOqUDI
    TheBurchLog
    @TheBurchLog
    Hi there Peter, I was able to reproduce the picture you sent over. The connection information is included for the response object because when the Plugin connects, it uses that API call to get the RabbitMQ config information (username/password). I agree that it probably isn't the best practice to include that with the UI calls. Anyone with System_read permissions can call the API for this information. We also have cert authentication options if you are interested in that.
    Peter Crampton
    @cramppet
    @TheBurchLog Thanks for the info. It is an acceptable risk for my use case, just wasn't sure if it was configurable. Also, if you have client cert options available, I'd be interested in reading about it.
    TheBurchLog
    @TheBurchLog
    @cramppet We are working on getting the Quick Start page finalized for securing your environment. Once we get that up, I will post the link.
    Matt Patrick
    @hazmat345
    quick question - is anyone running plugins with python 3.4? i ask because the latest version of a transitive brewtils dependency (pyyaml) no longer supports it. i'm going to add a constraint for this one, but it'd be good to get a feel for how people would be affected if we need to eventually drop support for 3.4
    Keith Ancowitz
    @keith.ancowitz_gitlab

    Trying out BG for the time. The docker-compose gives me the following error for brew-view container's logs.

    brew-view_1 | 2020-03-28 21:03:49,535 - bg_utils.mongo.util - WARNING - No roles found: creating convenience roles
    brew-view_1 | 2020-03-28 21:03:49,540 - bg_utils.mongo.util - WARNING - Creating missing admin user...
    brew-view_1 | 2020-03-28 21:03:49,540 - bg_utils.mongo.util - INFO - Creating username "admin" with password "password"
    brew-view_1 | 2020-03-28 21:03:49,960 - tornado.application - ERROR - Exception in callback functools.partial(<function wrap.<locals>.null_wrapper at 0x7fd32810d598>, <Future finished exception=NotUniqueError('Tried to save duplicate unique keys (E11000 duplicate key error collection: beer_garden.principal index: unique_index dup key: { : "admin" })',)>)
    brew-view_1 | Traceback (most recent call last):
    brew-view_1 | File "/usr/local/lib/python3.6/site-packages/mongoengine/document.py", line 378, in save
    brew-view_1 | object_id = self._save_create(doc, force_insert, write_concern)
    brew-view_1 | File "/usr/local/lib/python3.6/site-packages/mongoengine/document.py", line 434, in _save_create
    brew-view_1 | object_id = collection.save(doc, write_concern)
    brew-view_1 | File "/usr/local/lib/python3.6/site-packages/pymongo/collection.py", line 3158, in save
    brew-view_1 | to_save, True, check_keys, manipulate, write_concern)
    brew-view_1 | File "/usr/local/lib/python3.6/site-packages/pymongo/collection.py", line 612, in _insert
    brew-view_1 | bypass_doc_val, session)
    brew-view_1 | File "/usr/local/lib/python3.6/site-packages/pymongo/collection.py", line 600, in _insert_one
    brew-view_1 | acknowledged, _insert_command, session)
    brew-view_1 | File "/usr/local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1491, in _retryable_write
    brew-view_1 | return self._retry_with_session(retryable, func, s, None)
    brew-view_1 | File "/usr/local/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1384, in _retry_with_session
    brew-view_1 | return func(session, sock_info, retryable)
    brew-view_1 | File "/usr/local/lib/python3.6/site-packages/pymongo/collection.py", line 597, in _insert_command
    brew-view_1 | _check_write_command_response(result)
    brew-view_1 | File "/usr/local/lib/python3.6/site-packages/pymongo/helpers.py", line 221, in _check_write_command_response
    brew-view_1 | _raise_last_write_error(write_errors)
    brew-view_1 | File "/usr/local/lib/python3.6/site-packages/pymongo/helpers.py", line 202, in _raise_last_write_error
    brew-view_1 | raise DuplicateKeyError(error.get("errmsg"), 11000, error)
    brew-view_1 | pymongo.errors.DuplicateKeyError: E11000 duplicate key error collection: beer_garden.principal index: unique_index dup key: { : "admin" }
    brew-view_1 |
    brew-view_1 | During handling of the above exception, another exception occurred:
    brew-view_1 |
    brew-view_1 | Traceback (most recent call last):
    brew-view_1 | File "/usr/local/lib/python3.6/site-packages/tornado/ioloop.py", line 758, in _run_callback
    brew-view_1 | ret = callback()
    brew-view_1 | File "/usr/local/lib/python3.6/site-packages/tornado/stack_context.py", line 300, in null_wrapper
    brew-view_1 | return fn(*args,     kwargs)
    brew-view_1 | File "/usr/local/lib/python3.6/site-packages/tornado/ioloop.py", line 779, in _discard_future_result
    brew-view_1 | future.result()
    brew-view_1 | File "/usr/local/lib/python3.6/site-packages/tornado/gen.py", line 1141, in run
    brew-view_1 | yielded = self.gen.throw(*exc_info)
    brew-view_1 | File "/usr/local/lib/python3.6/site-packages/brew_view/init.py", line 106, in startup
    brew-view_1 | partial(setup_database, config), "Unable to connect to mongo, is it started?"
    brew-view_1 | File "/usr/local/lib/python3.6/site-packages/tornado/gen.py", line 1133, in run
    brew-view_1 | value = future.result()
    brew-view_1 | File "/usr/local/lib/python3.6/site-packages/tornado/gen.py", line 326, in wrapper
    brew-view_1 |

    Keith Ancowitz
    @keith.ancowitz_gitlab
    the errors appear to be related to brew-view running tornado.application
    Matt Patrick
    @hazmat345
    Hi @keith.ancowitz_gitlab - does this happen again if you restart the brew-view container? I think this may be a race condition where both brew-view and bartender are attempting to create the admin user at the same time.
    Keith Ancowitz
    @keith.ancowitz_gitlab
    Yes, that's right. Not sure of the expected ordering of events so both way was tried. Starting brew-view container first before others and then just stopping, removing, and restarting the brew-view after all the containers started. I don't know. I pivoted to the RPM and it worked smoothly. I'm on AWS should I start from scratch in case the its related to the dependencies.
    Matt Patrick
    @hazmat345
    I don't think you need to start from scratch, and after the initial run you shouldn't need to worry about starting containers in any particular order.
    I'm pretty sure what happened is that both bartender and brew-view check to see if certain users exist in the database and create them if they don't.
    We didn't have any easy way to make that check+create atomic, so i think the timing was just wrong enough that both containers tried to create that user
    but now that the user already exists that process should be skipped from now on, so you shouldn't have to worry about the startup ordering
    Keith Ancowitz
    @keith.ancowitz_gitlab
    will try... thanks
    Keith Ancowitz
    @keith.ancowitz_gitlab

    Questions about running beer-garden/bartender and plugins in different Python versions.

    1. From the documentation, It looks like bartender install requirement is 2.7 or 3.4. What controls the version used? The default "/usr/bin/python" version.

    2. Looking at Plugin_Runner.py, bartender spawns plugins in a equivalent python version using sys.executable. Because of 3rd party libraries, it looks like i'm limited to python2.7 for the plugin. Would you consider a enhancement to allow the "setup_args" to specify a python version like setuptools? For example 'python_requires' : '~=2.7'.

    Logan Jones
    @loganasherjones
    @keith.ancowitz_gitlab Beer Garden should work on any Python version >= 3.4. If you're installing Beer Garden from the docker images, it's just the way the docker images are built that decide which Python version gets installed. Otherwise, if you're installing from pip, it depends on what version of Python you're using.
    If you need Python 2.7 for some reason, and you're using the docker images, you could always use our latest-python2 tag.
    Specifying Python version isn't something we've really looked at that hard, but I imagine it could be done. @hazmat345 @TheBurchLog thoughts?
    TheBurchLog
    @TheBurchLog

    @keith.ancowitz_gitlab & @loganasherjones
    I agree with Logan. Right now the python environment that Bartender uses is based on the version that it is running. This is why we support both libraries. If you plugins are not using 3.4, then I the interim I would suggest installing the 2.7 docker image as Logan stated.

    It would be interesting to see what would happen if we update the subprocess command to use the local environment or a user specified configuration (python version). @hazmat345 you have been working in this area for our next version. The biggest concern I have so far is making sure the users are aware that they are running something in a different version of python. That way they don't get into weird scenarios where they are debugging the wrong version.

    Another option is to run your plugins are Remote Plugins. These you would have to manage independent from what is located in the plugin directory, but this gives you the most control over the python environment. This will work if you have a 3.4 Bartender running and 2.7 Remote Plugins.

    Keith Ancowitz
    @keith.ancowitz_gitlab
    Yes, the best solution now seems to run it as a remote plugin using 2.7. Because as a plugin author, I may not have control over the beer-garden/bartender configuration. Thinking ahead, it would be a very nice option to have a local plugin with the same feature. I'd be very supportive of a configurable executable for the subprocess command. Cheers
    TheBurchLog
    @TheBurchLog
    Just in case anyone else is tracking the request from @keith.ancowitz_gitlab . We have a ticket for that issue ( beer-garden/beer-garden#492 ) and a PR ( beer-garden/beer-garden#500 ) for it for our v3 branch.
    zach valenta
    @zachvalenta
    newb question:
    • Are there other frameworks inspired by Beer Garden?
    • Was Beer Garden inspired by preceding frameworks? If so, which?
    Logan Jones
    @loganasherjones
    To my knowledge there aren’t any frameworks inspired by beer garden.
    Beer Garden was really inspired by the fact that I didn’t want to write 15 of the same interfaces. But it wasn’t really based on anything that existed already
    TheBurchLog
    @TheBurchLog
    When I first started to contribute to Beer Garden I thought it was just going to be another BPM engine. But once you start using it, it really does feel like traditional python that just happens to be executable by a framework. It really doesn't line up nicely with any of the other frameworks that I have seen (when I focused on Java dev work). If you do find similarities in other frameworks we would be super excited to hear about it, especially if someone says they were inspired by Beer Garden. Logan is definitely the authority on the inspiration.
    James Nichols
    @jnichols0
    I am trying to understand brewtils -> beer-garden version compatibility. I'm sure this is answered somewhere but I could not find it quickly. Can one expect a higher major version of brewtils (e.g. 3.9.0) to communicate successfully with an instance of a lower major version of beer-garden (e.g. 2.4.9)? My project would like to upgrade to the latest brewtils internally but would like to know if this'll be a breaking change for users who we expect to manage their own beer-garden instances.
    TheBurchLog
    @TheBurchLog
    In the initial design, the 2.x and 3.x versions of brewtils were supported by 3.x release of beer-garden. It has been close to a year since I last touched to the code, so I can't remember if using a 3.x brewtils would work with a 2.x beer-garden. My suggestion would be to spin up a docker image of beer-graden 2.4.9 and attempt to use the latest brewtils. If there is a failure it will occur during the initial API exchange to configure the plugin. I have a gut feeling it might be, but it has been a while and I know the framework is always evolving.
    1 reply