Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    naturzukunft
    @naturzukunft:matrix.org
    [m]

    This here seems to be the biggest problem:

    public void save(String subject, PostalAddressLoa postalAddressLoa) {
      PostalAddressKomma created = entityManager.createNamed(URIs.createURI(subject),PostalAddressKomma.class);
      created.setAddressCountry(postalAddressLoa.getAddressCountry());
      created.setAddressLocality(postalAddressLoa.getAddressLocality());
      created.setAddressRegion(postalAddressLoa.getAddressRegion());
      created.setPostalCode(postalAddressLoa.getPostalCode());
      created.setStreetAddress(postalAddressLoa.getStreetAddress());
    }

    because each line seems to do a lot http things ?! Each setter takes ~ 10 sec.
    I think there is another way to create objects, but haven't found it yet.

    kenwenzel
    @kenwenzel
    OK. Do you already use transactions? You should wrap the creation of your entities within a transaction by using
    ITransaction tx = entityManager.getTransaction();
    tx.begin();
    try {
      // create beans here
      tx.commit();
    } catch (KommaException e) {
      // rollback tx if necessary
    }
    BTW, 10s for each setter is way too long. Which KOMMA version are you using? Do you use the IModelSet-API or only the entity manager?
    naturzukunft
    @naturzukunft:matrix.org
    [m]
    Transaction is a good idea, test runtime before 55.853sec with transaction 12.379
    I use 1.4.0, copied it from an example.
    switching to 1.5.2 end in applicationContext errors while initialization.
    is there somewhere an example with 1.5.2 ?
    kenwenzel
    @kenwenzel
    I try to upgrade https://github.com/komma/komma-examples to 1.5.2 tomorrow. But we're already using it in our linked data platform enilink https://github.com/enilink/enilink without any problems... hm...
    @naturzukunft:matrix.org Can you provide your examples/unit tests somehow?
    kenwenzel
    @kenwenzel

    I think version 1.4.0 uses eager change tracking leading to many HTTP requests as observed by you. This is fixed in version 1.5.2.
    You can try:

    DataChangeTracker changeTracker = injector.getInstance(DataChangeTracker.class);
    changeTracker.setEnabled(null, false);

    Use the code above once before executing any modifications with an entity manager.
    The enabled state of the change tracker is a thread-local.

    1 reply
    naturzukunft
    @naturzukunft:matrix.org
    [m]
    @kenwenzel: we currenly struggeling with another problem. do you have experience with using rdf4j in a cluster?
    In the moment one of our node seems to lock the repo directory and the other node fails.
    The same problem would we have using komma.
    kenwenzel
    @kenwenzel
    @naturzukunft:matrix.org Are you trying to directly share the repo directory between multiple instances? Are you using the NativeStore?
    naturzukunft
    @naturzukunft:matrix.org
    [m]
    yes, we do ;-) maybe not a good idea ?
    we was using memory store, but changed to native store. Same problem with both.
    i got confused by the "The Repository API supports multithreaded access" statement because i assumed there were threads on several nodes.
    kenwenzel
    @kenwenzel
    You can't share the files between different NativeStore instances. If you need clustering then you have to switch to GraphDB https://www.ontotext.com/products/graphdb/ that AFAIK only supports full replication or Stardog https://www.stardog.com/
    naturzukunft
    @naturzukunft:matrix.org
    [m]
    that sounds awful. there is no open source solution?
    kenwenzel
    @kenwenzel
    You could also try Virtuoso or Blazegraph (discontinued, now Amazon Neptune). Do you really need clustering for performance reasons or only for HA?
    naturzukunft
    @naturzukunft:matrix.org
    [m]
    In the IESE environment, we very definitely need fail-safety. It will be a kind of activityPub server, but it is still unclear which features will actually be implemented and which will be publicly accessible. We are starting very small and want to gain experience with rdf and activityPub. We would also use it as a kind of message bus, exchanging messages/tasks between systems. If we open it up to users, the number of users can reach ~ 30,000 quite quickly. However, each user would have their own repository! So load balancing could also be achieved without a cluster.
    kenwenzel
    @kenwenzel
    I would also suggest to start with the NativeStore and then switch the database technology if necessary. We also combine RDF4J stores e.g. with LevelDB for high-performance (> 100.000 values per second) collection of time-series data.
    naturzukunft
    @naturzukunft:matrix.org
    [m]
    i did not really sea the issue with the cluster, if there is a shared file system. What is the difference to multithreaded app on one cluster. Am I missing something?
    kenwenzel
    @kenwenzel
    A multi-threaded app is able to use locks to ensure that the database file(s) are only updated by one thread at a time and hence race conditions, like overwriting data by two simultaneous threads, are prevented. Two separate processes need to synchronize their work by other mechanisms, e.g. by locking whole files. This is exactly what you are oberserving with the NativeStore. But this is the same for (most) other database systems. They all need to ensure that the data files are only updated by one process at a time - it doesn't matter if the database is distributed by nature or not.
    BTW, using a network file system for a database is only a good idea if it is connected via a really fast network.
    Did you make any progress with KOMMA?
    naturzukunft
    @naturzukunft:matrix.org
    [m]
    Unfortunately there was no time for Komma today.
    kenwenzel
    @kenwenzel
    Let me know if you've made any progress :-)
    naturzukunft
    @naturzukunft:matrix.org
    [m]
    ^ found it !
    naturzukunft
    @naturzukunft:matrix.org
    [m]
    kenwenzel
    @kenwenzel

    Cool!
    BTW, to speed up the retrieval of the bean at this line

    https://git.fairkom.net/fairsync/development/loa-suite/-/blob/master/loa-repository-KOMMA/src/main/java/org/linkedopenactors/repository/PostalAddressRepository.java#L47

    you can replace em.find(...) by something like:

    em.createQuery("construct { ?s a <komma:Result> ; ?p ?o } where { ?s ?p ?o }").setParameter("s", uri).getSingleResult(PostalAddressKomma.class);