Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
    David Ribeiro Alves
    @dralves
    first post!
    Mike Percy
    @mpercy
    hallo!
    hmm does gitter have apps?
    David Ribeiro Alves
    @dralves
    kind of a bummer that you need to login to github to talk, lemme see if you can change that
    @mpercy dunno, you mean ios and the like?
    Mike Percy
    @mpercy
    yeah
    also no jira integration
    David Ribeiro Alves
    @dralves
    yeah it does
    yeah, missing jira is a bummer
    Misty Stanley-Jones
    @mstanleyjones
    I'm skeptical
    David Ribeiro Alves
    @dralves
    yeah forcing people to have a github account isn't very good
    happy to try some alternatives :)
    Stefano Tondo
    @tondostefano_twitter
    hello everyone
    I'm trying to index more than one hbase collection with lily indexer using only 1 morphline.conf
    does anyone know if this is possible?
    S. Matthew English
    @s-matthew-english
    zizhec
    @zizhec
    can I ask other cloudera related questions here?
    krishnadevaraya
    @design-trails
    Hello Any body has idea about when DR capabilities will be part of Kudu
    fabio fumarola
    @fabiofumarola
    Hi all, is it possible to setup row based acl in KUDU similar to the cell base acl of HBASE? I've googled, but with no results
    Aaron Hiniker
    @hindog

    I have a Spark job that appears to be hung (Kudu 1.4.0-cdh5.12.0):
    In the driver stacks, I see threads stuck here:

    java.lang.Object.wait(Native Method)
    java.lang.Object.wait(Object.java:502)
    com.stumbleupon.async.Deferred.doJoin(Deferred.java:1136)
    com.stumbleupon.async.Deferred.join(Deferred.java:1019)
    org.apache.kudu.client.KuduClient.joinAndHandleException(KuduClient.java:340)
    org.apache.kudu.client.KuduClient.tableExists(KuduClient.java:196)
    org.apache.kudu.spark.kudu.KuduContext.tableExists(KuduContext.scala:102)
    ...

    and in the driver logs, I'm seeing these messages get logged:

    20:19:07 WARN  ConnectToCluster - Unable to find the leader master (10.50.3.56:7051,10.50.3.209:7051,10.50.3.202:7051), will retry
    20:19:07 ERROR TabletClient - [Peer master-10.50.3.56:7051] Unexpected exception from downstream on [id: 0xf129dffb, /10.50.3.72:37744 => /10.50.3.56:7051]
    java.lang.RuntimeException: Could not deserialize the response, incompatible RPC? Error is: step
            at org.apache.kudu.client.KuduRpc.readProtobuf(KuduRpc.java:383)
            at org.apache.kudu.client.Negotiator.parseSaslMsgResponse(Negotiator.java:282)
            at org.apache.kudu.client.Negotiator.handleResponse(Negotiator.java:235)
            at org.apache.kudu.client.Negotiator.messageReceived(Negotiator.java:229)
    Running kudu cluster ksck <master> reports all tables as healthy
    I should probably mention that we have multiple threads operating on independent KuduContext's in this case, but this same code has been running fine for months until recently
    Aaron Hiniker
    @hindog
    Oh, and also this in the Kudu logs: W0105 19:58:16.417731 18382 negotiation.cc:310] Unauthorized connection attempt: Server connection negotiation failed: server connection from 10.50.3.72:53896: authentication token expired
    Dan Burkert
    @danburkert
    @hindog You'll probably have more luck on the Kudu slack channel, discussion is much more active there: https://getkudu-slack.herokuapp.com/ . KUDU-2013 is probably your issue, especially if that job took more than 6 or 7 days
    Ricardo Gaspar
    @ricardogaspar2

    hey, can someone help me with Kudu API?

    If I wand to use more than one KuduPredicate in a Scanner, the scanner will use both together as an AND expression or an OR?

    https://kudu.apache.org/apidocs/org/apache/kudu/client/KuduPredicate.html (edited)

    e.g.

    ColumnSchema columnSchemaUpdatedTs = table.getSchema().getColumn(TABLE_HOUSE_INFO_ACTIVE_TIPS_COLUMN_UPDATED_TS);
    
    ColumnSchema columnSchemaDeletedTs = table.getSchema().getColumn(TABLE_HOUSE_INFO_ACTIVE_TIPS_COLUMN_ACTIVE_TIP_DELETED_TS);
    
    
    
    KuduScanner scanner = kc.newScannerBuilder(table).addPredicate(KuduPredicate.newComparisonPredicate(columnSchemaUpdatedTs, KuduPredicate.ComparisonOp.GREATER, lastExecutionTs)).addPredicate(KuduPredicate.newComparisonPredicate(columnSchemaDeletedTs, KuduPredicate.ComparisonOp.GREATER, lastExecutionTs)).build();

    In this example the scanner will filter records using columnSchemaUpdatedTs AND columnSchemaDeletedTs ? Like a

     WHERE TABLE_HOUSE_INFO_ACTIVE_TIPS_COLUMN_UPDATED_TS > lastExecutionTs AND TABLE_HOUSE_INFO_ACTIVE_TIPS_COLUMN_ACTIVE_TIP_DELETED_TS > lastExecutionTs

    or it will use an OR ?

    Dan Burkert
    @danburkert
    @ricardogaspar2 they will be combined with AND. Kudu doesn't support OR predicates yet
    BTW you'll find our slack instance is much more active than this room (https://getkudu-slack.herokuapp.com/)
    oh woops, just saw you already asked and got an answer there :)
    Ricardo Gaspar
    @ricardogaspar2
    thanks @danburkert ;)
    Ricardo Gaspar
    @ricardogaspar2

    Hi, I'm having difficulty using a Kudu predicate and a lowerBound in a kudu scanner.
    My goal is to get a batch of rows (after applying a predicate ) and storing in a Set(LinkedHashSet) to return to my program which will then process them.
    For this, in each call to my method I give the last record I've read and I use the lowerBound function.

     scannerUpdated = kc.newScannerBuilder(table).addPredicate(predicate).limit(recordsToRead).lowerBound(lastRecordPartialRow).build();

    From my execution logs, I've seen that for a table with 7000 rows (all satisfy the predicate) using a batch of 1000 I sometimes get less of a 1000 records/rows. And also the most intriguing thing is that sometimes: the last record (the last record present in the last set requested to the calling class) is not the same as the first record returned by the RowResult

    How I'm doing it :
     private Set<? extends Record> getRecentlyUpdatedRecords(String tableName, KuduPredicate predicate,
                                                                int recordsToRead, Record lastRecord,
                                                                PartialRow lastRecordPartialRow) throws
                InvalidRecordException,
                IOException {
            long start = System.currentTimeMillis();
    
            Set<Record> recordsSet = new LinkedHashSet<>(recordsToRead);
    
            try {
                if (kc.tableExists(fullTableName(tableName))) {
                    KuduTable table = kc.openTable(fullTableName(tableName));
    
                    KuduScanner scannerUpdated;
    
                    if (lastRecord != null) {
                        // start from the last record processed
                        scannerUpdated =
                                kc.newScannerBuilder(table).addPredicate(predicate).limit(recordsToRead).lowerBound(
                                        lastRecordPartialRow).build();
                    } else {
                        scannerUpdated = kc.newScannerBuilder(table).addPredicate(predicate).limit(recordsToRead).build();
                    }
                    LOGGER.info("Kudu Scanner read rows limit: " + scannerUpdated.getLimit());
    
    
                    int rowCounter = 0;
                    while (scannerUpdated.hasMoreRows() && rowCounter < recordsToRead) {
                        RowResultIterator results = scannerUpdated.nextRows();
                        while (results != null && results.hasNext() && rowCounter < recordsToRead) {
                            RowResult rowData = results.next();
                            Record record = createRecordFromRow(rowData, tableName);
    
                            if (lastRecord != null && lastRecord.equals(record)) {
                                LOGGER.debug("Skiping current record (previously processed): " + record.toString() + " lastRecord: " +
                                        lastRecord.toString());
                            } else {
                                recordsSet.add(record);
                                LOGGER.debug("record added: " + record.toString());
                                rowCounter++;
                            }
                        }
                    }
    
                }
            } catch (KuduException e) {
                closeSession();
                e.printStackTrace();
                throw new IOException(e);
            } finally {
                String methodName = new Object() {
                }.getClass().getEnclosingMethod().getName();
                LOGGER.info("KUDU " + methodName + " run time = " + (System.currentTimeMillis() - start) + " ms");
            }
    
            return recordsSet;
        }
    Ricardo Gaspar
    @ricardogaspar2
    It seems to me that the lowerBound is being applyed before the predicate. Is it?
    Is there any way I can specify the order the restrictions are being applied (predicate before the lowerBound)?