These are chat archives for FRosner/cluster-broccoli

25th
Mar 2018
Greg Fausak
@lgfausak
Mar 25 15:31

I have been using the broccoli for a while, really like it! I run it from docker. I would like to try using the couchdb interface. It is not clear to me how to configure this. I see that a .conf file can add and delete configuration, because you can't select couchdb with a -D in the docker args. There is documentation stating that I have to select the model, which I have tried by creating a /couchdb.conf file in the docker executable before I start bin/cluster-broccoli. However, my messages on stdout still indicate that the file storage module is being used. My couchdb.conf file looks like this:

play.modules.disabled += "de.frosner.broccoli.instances.storage.filesystem.FileSystemStorageModule"
play.modules.enabled += "de.frosner.broccoli.instances.storage.couchdb.CouchDBStorageModule"

but that doesn't seem to work. I don't know scala, maybe this is obvious, but I can't figure out how to configure it!

Frank Rosner
@FRosner
Mar 25 15:31
Hi!
Greg Fausak
@lgfausak
Mar 25 15:32
hey
Frank Rosner
@FRosner
Mar 25 15:32
Yeah we changed the way how to select couchdb around in the last release and I didn't really document it. May I ask why you want to use it?
Greg Fausak
@lgfausak
Mar 25 15:32
ya, well, it is a long story, but, mostly because of persistence.
Frank Rosner
@FRosner
Mar 25 15:32
It was added some time ago because the file system storage had the risk of corrupting the data if broccoli gets killed in the middle of a write.
But we fixed this
Greg Fausak
@lgfausak
Mar 25 15:33
you know, a nomad file to start with an ephemeral disk would be nice.
Frank Rosner
@FRosner
Mar 25 15:33
We now use an atomic move operation so this is not an issue anymore.
Greg Fausak
@lgfausak
Mar 25 15:33
i tried everything.
Frank Rosner
@FRosner
Mar 25 15:33
ah
do you have access to NFS?
this is how we do it
Greg Fausak
@lgfausak
Mar 25 15:33
i mean, eventually, i created a new docker instance from yours, did some hocus pocus before starting broccoli, and then it worked.
no
no nfsw
Frank Rosner
@FRosner
Mar 25 15:33
kk
then couchdb sounds like an option
Greg Fausak
@lgfausak
Mar 25 15:34
i did manage to get things started with persistence.
but, it was a real buggar
Frank Rosner
@FRosner
Mar 25 15:34
sorry to hear
tbh I never actually used the couchdb back-end in the end
Greg Fausak
@lgfausak
Mar 25 15:34
well, it would be good. except, now i need a hook to do a 'backup'
Frank Rosner
@FRosner
Mar 25 15:34
It also doens;t work with couchdb 2 I think
Greg Fausak
@lgfausak
Mar 25 15:34
so i have something to restore.
the way i have it, i can tar the instances directory and save it on ftp/github/etc.
then i can start broccoli with persistence by loading the saved backup.
Frank Rosner
@FRosner
Mar 25 15:35
So you managed to get it running with CouchDB?
Greg Fausak
@lgfausak
Mar 25 15:35
not optimal.
no, files.
Frank Rosner
@FRosner
Mar 25 15:35
Ah I see.
Greg Fausak
@lgfausak
Mar 25 15:35
i thought maybe couchdb would give me persistence.
Frank Rosner
@FRosner
Mar 25 15:35
It will
Should we see what's not working? Which version of Broccoli and CouchDB are you using?
Greg Fausak
@lgfausak
Mar 25 15:36
i am using couchdb 1.7.1 i think
the thing is, i don't know what .conf file to update
it isn't obvious to me how to load the couchdb driver instead of the file
Frank Rosner
@FRosner
Mar 25 15:37
Yes I see
Greg Fausak
@lgfausak
Mar 25 15:37
i am using the current version of broccoli (0.8.0 i think).
Frank Rosner
@FRosner
Mar 25 15:37
Ok
The change in the config file above seems great. Were you able to locate the application.conf file inside the container?
Greg Fausak
@lgfausak
Mar 25 15:38
nope.
do i find that and change it, or appen?
Frank Rosner
@FRosner
Mar 25 15:38
So there are multiple ways to success.
One way, which is what we do, is to create a brand new application.conf file (e.g. using the template stanza if you like) and then pointing Broccoli to it:
          "-Dconfig.file=/local/application.conf",
I think the "default" file is in /opt/docker/conf/application.conf but I'm not sure.
Greg Fausak
@lgfausak
Mar 25 15:41
ya, i just exec'ed in and found it.
Frank Rosner
@FRosner
Mar 25 15:41
You can find the default config values in the reference.conf
Greg Fausak
@lgfausak
Mar 25 15:42
perfect. i can create an inline template for nomad to pop this in to place.
Frank Rosner
@FRosner
Mar 25 15:42
The application.conf overrides those. That's the config hierarchy from typesafe conf.
Greg Fausak
@lgfausak
Mar 25 15:42
it was pretty frustrating not being able to use the 'artifact' to restore the file system, go-getter has a bug on that... always makes it read only.
Frank Rosner
@FRosner
Mar 25 15:43
So you can put the lines you pasted above into the reference.conf, specify the path to the generated conf file on launch using the config.file option and that should do the trick.
Greg Fausak
@lgfausak
Mar 25 15:43
well, thanks! i wil give it a shot now...brb
Frank Rosner
@FRosner
Mar 25 15:43
Yeah. I'm pretty happy with the NFS solution we have. Another guy was suggesting to use S3 but he never came back with a PR/
GL! Tell me how it went. I'll also go fetch some food soon :)
Greg Fausak
@lgfausak
Mar 25 15:44
:-)
Greg Fausak
@lgfausak
Mar 25 16:19
i can't figure out the markdown. anyway, doesn't start now, i get this errr
Oops, cannot start the server.
@77cn47am0: Configuration error
at play.api.libs.crypto.CryptoConfigParser.get$lzycompute(Crypto.scala:498)
at play.api.libs.crypto.CryptoConfigParser.get(Crypto.scala:465)
at play.api.libs.crypto.CryptoConfigParser.get(Crypto.scala:463)
at com.google.inject.internal.ProviderInternalFactory.provision(ProviderInternalFactory.java:81)
at com.google.inject.internal.BoundProviderFactory.provision(BoundProviderFactory.java:72)
at com.google.inject.internal.ProviderInternalFactory.circularGet(ProviderInternalFactory.java:61)
at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:62)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.BoundProviderFactory.get(BoundProviderFactory.java:61)
at com.google.inject.internal.SingleParameterInjector.inject(SingleParameterInjector.java:38)
at com.google.inject.internal.SingleParameterInjector.getAll(SingleParameterInjector.java:62)
at com.google.inject.internal.ConstructorInjector.provision(ConstructorInjector.java:110)
at com.google.inject.internal.ConstructorInjector.construct(ConstructorInjector.java:90)
at com.google.inject.internal.ConstructorBindingImpl$Factory.get(ConstructorBindingImpl.java:268)
at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205)
at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199)
at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199)
at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180)
at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
at com.google.inject.Guice.createInjector(Guice.java:99)
at com.google.inject.Guice.createInjector(Guice.java:84)
at play.api.inject.guice.GuiceBuilder.injector(GuiceInjectorBuilder.scala:181)
at play.api.inject.guice.GuiceApplicationBuilder.build(GuiceApplicationBuilder.scala:123)
at play.api.inject.guice.GuiceApplicationLoader.load(GuiceApplicationLoader.scala:21)
at play.core.server.ProdServerStart$.start(ProdServerStart.scala:47)
at play.core.server.ProdServerStart$.main(ProdServerStart.scala:22)
at play.core.server.ProdServerStart.main(ProdServerStart.scala)
i have also tried one with the reference.conf file.
Frank Rosner
@FRosner
Mar 25 17:11
Can you format the code plz?
with three backticks, like on github
and then three to close it
Do you also mind cating the generated /local/application.conf file?
Ok I think it might be related to the crypto secret being missing
Can you include play.crypto.secret = "IN_PRODUCTION_CHANGE_THIS_TO_A_LONG_RANDOM_STRING" also? Please replace the string if you want to use this somewhere important.
Greg Fausak
@lgfausak
Mar 25 20:27
  group "broccoli" {
    task "broccoli" {
      driver = "docker"
      config {
        image = "frosner/cluster-broccoli"
        args = [
          "-Dconfig.file=/local/application.conf",
          "-Dbroccoli.instances.storage.couchdb.url=http://couchdb.dev.domain.com:5984",
          "-Dbroccoli.instances.storage.couchdb.database=couchdb",
          "-Dbroccoli.nomad.url=http://1.2.30.53:4646",
          "-Dbroccoli.consul.url=http://1.2.31.55:8500",
          "-Dbroccoli.templates.path=/local/templates"
        ]
        port_map = {
          http = 9000
        }
      }
      template {
        data = <<ENV
play.modules.disabled += "de.frosner.broccoli.instances.storage.filesystem.FileSystemStorageModule"
play.modules.enabled += "de.frosner.broccoli.instances.storage.couchdb.CouchDBStorageModule"
broccoli {
  auth {
    mode = none
    conf {
      accounts = [
        {username:admin, password:admin, instance-regex=".*", role:"administrator"},
        {username:operator, password:operator, instance-regex=".*", role:"operator"},
        {username:user, password:user, instance-regex=".*", role:"user"},
        {username:test, password:test, instance-regex="^test.*", role:"administrator"}
      ]
    }
  }
}
ENV
        destination = "local/application.conf"
      }
      artifact {
        source = "https://github.vcp.vzwnet.com/VCP/broccoli/blob/master/templates.tar.gz?raw=true"
        destination = "local/templates"
      }
...
between the <<ENV and ENV is the local/application.conf file. I'll try inserting the play.crypto.secret as you suggest.
Greg Fausak
@lgfausak
Mar 25 20:35
that did give me a better error message. Something about
5) Error in custom provider, java.lang.IllegalArgumentException: Unable to aquire lock for CouchDB (http://couchdb.dev.domain.com:5984/_config/broccoli/couchdb)
  at de.frosner.broccoli.instances.storage.couchdb.CouchDBStorageModule.provideCouchDBInstanceStorage(CouchDBStorageModule.scala:33) (via modules: com.google.inject.util.Modules$OverrideModule -> de.frosner.broccoli.instances.storage.couchdb.CouchDBStorageModule)
  at de.frosner.broccoli.instances.storage.couchdb.CouchDBStorageModule.provideCouchDBInstanceStorage(CouchDBStorageModule.scala:33) (via modules: com.google.inject.util.Modules$OverrideModule -> de.frosner.broccoli.instances.storage.couchdb.CouchDBStorageModule)
  while locating de.frosner.broccoli.instances.storage.InstanceStorage
Caused by: java.lang.IllegalArgumentException: Unable to aquire lock for CouchDB (http://couchdb.dev.domain.com:5984/_config/broccoli/couchdb)
    at de.frosner.broccoli.instances.storage.couchdb.CouchDBInstanceStorage.<init>(CouchDBInstanceStorage.scala:53)
    at de.frosner.broccoli.instances.storage.couchdb.CouchDBStorageModule.provideCouchDBInstanceStorage(CouchDBStorageModule.scala:34)
    at de.frosner.broccoli.instances.storage.couchdb.CouchDBStorageModule$$FastClassByGuice$$cee6be31.invoke(<generated>)
    at com.google.inject.internal.ProviderMethod$FastClassProviderMethod.doProvision(ProviderMethod.java:264)
    at com.google.inject.internal.ProviderMethod$Factory.provision(ProviderMethod.java:401)
    at com.google.inject.internal.ProviderMethod$Factory.get(ProviderMethod.java:376)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter$1.call(ProviderToInternalFactoryAdapter.java:46)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1092)
    at com.google.inject.internal.ProviderToInternalFactoryAdapter.get(ProviderToInternalFactoryAdapter.java:40)
    at com.google.inject.internal.SingletonScope$1.get(SingletonScope.java:194)
    at com.google.inject.internal.InternalFactoryToProviderAdapter.get(InternalFactoryToProviderAdapter.java:41)
    at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:205)
    at com.google.inject.internal.InternalInjectorCreator$1.call(InternalInjectorCreator.java:199)
    at com.google.inject.internal.InjectorImpl.callInContext(InjectorImpl.java:1085)
    at com.google.inject.internal.InternalInjectorCreator.loadEagerSingletons(InternalInjectorCreator.java:199)
    at com.google.inject.internal.InternalInjectorCreator.injectDynamically(InternalInjectorCreator.java:180)
    at com.google.inject.internal.InternalInjectorCreator.build(InternalInjectorCreator.java:110)
    at com.google.inject.Guice.createInjector(Guice.java:99)
    at com.google.inject.Guice.createInjector(Guice.java:84)
    at play.api.inject.guice.GuiceBuilder.injector(GuiceInjectorBuilder.scala:181)
    at play.api.inject.guice.GuiceApplicationBuilder.build(GuiceApplicationBuilder.scala:123)
    at play.api.inject.guice.GuiceApplicationLoader.load(GuiceApplicationLoader.scala:21)
    at play.core.server.ProdServerStart$.start(ProdServerStart.scala:47)
    at play.core.server.ProdServerStart$.main(ProdServerStart.scala:22)
    at play.core.server.ProdServerStart.main(ProdServerStart.scala)
Greg Fausak
@lgfausak
Mar 25 20:40
$ curl http://couchdb.dev.domain.com:5984/
{"couchdb":"Welcome","uuid":"9f3d1b5b53e34da3bfa9f17a5bc60521","version":"1.7.1","vendor":{"version":"1.7.1","name":"The Apache Software Foundation"}}
Frank Rosner
@FRosner
Mar 25 20:42
It wants to write a config property
http://couchdb.dev.domain.com:5984/_config/broccoli/couchdb
Is it possible that it cannot for permission reasons or something?
can you try to insert this config property yourself using curl
and look at the error?
Frank Rosner
@FRosner
Mar 25 22:12
Or could it be that you started Broccoli and then terminated it ungracefully?
Then the lock might still be there.
You can delete it by hand using curl as well.
I'm locking because I didn't implement any logic for handling multiple Broccolis accessing the same couchdb
CouchDB itself has a nice API for optimistic locking but I'm a pessimist :(
:D