Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Activity
  • 02:14
    yaooqinn closed #1772
  • 02:14
    yaooqinn closed #1771
  • 02:14

    yaooqinn on master

    [KYUUBI #1771] Change rate unit… (compare)

  • 02:14
    yaooqinn milestoned #1772
  • 02:13
    turboFei synchronize #1769
  • 02:12
    yaooqinn commented #1707
  • 02:12
    yaooqinn closed #1707
  • 02:12
    zhenjiaguo edited #1703
  • 02:12
    yaooqinn closed #1696
  • 02:12

    yaooqinn on master

    [KYUUBI #1696] Add fetch logs a… (compare)

  • 02:12
    yaooqinn milestoned #1707
  • 02:11
    yaooqinn assigned #1770
  • 02:11
    yaooqinn milestoned #1770
  • 02:11
    yaooqinn commented #1770
  • 02:10
    yaooqinn closed #1770
  • 02:10
    yaooqinn closed #1704
  • 02:10

    yaooqinn on master

    [KYUUBI #1704] Complete Flink-H… (compare)

  • 02:07
    yaooqinn closed #1768
  • 02:07
    yanghua edited #1322
  • 02:07
    yaooqinn closed #1645
Pham Nguyen
@akizminet
Thank you @yaooqinn. I find it's difficult to config limit for resultset and driver cores. Although collect stage finishes sucessfully (very fast), driver still has GC overhead or heartbeat error when data is being fetched.
kmsantanu
@kmsantanu
Hello-------
How can i access jdbc:hive2://<host>:<port>/ with user name and password to kyuubi

Kent Yao
@yaooqinn
-n username
Hi @akizminet there is a PR there under review to limit the resultset
feel free to share you thought there
Pham Nguyen
@akizminet
I don't think we need a config to limit resultset. This config doesn't respect the real size of queries's result. Sometime, I only select a few columns (such as a group by count query) and the query result is small enough for the Spark engine driver to handle.
I think users should insert limit clause explictly or use other SQL clients have auto limit like Superset.
Pham Nguyen
@akizminet
I see a PR in apache spark repo help reduce GC limit and OOM errors in engine driver caused by large resultset size. apache/spark#22219
I don't know much about metadata of AQE. I wonder if we could use that metadata to switch between incrementalCollect and collect
Pham Nguyen
@akizminet
I also have a question about spark.sql.optimizer.finalStageConfigIsolation.enabled. Does it apply to all queries or insert only? I don't think I need a large partition for non-insert queries?
Pham Nguyen
@akizminet
Hi @yaooqinn, How can I set infinite duration for user?
Kent Yao
@yaooqinn
Probably not
ckesslau
@ckesslau
Hi,
I'm new to Kyuubi and I'm trying to evaluate it as an alternative to Spark Thrift Server to serve data to MS Power BI. I have a Spark Cluster up and running and this cluster can connect to Hive Meta Store. How do I set up the Spark connection in Kyuubi? Do I need anything else for serving Power BI users with Spark data?
Thanks for your advice.
Kent Yao
@yaooqinn
刘智超
@zhichaoleo

Hi Team.
We are trying to use kyuubi(kyuubi-1.2.0-bin-spark-3.1-hadoop3.2) with delta-lake (Azure blob storage). But I can't find any documents to set up related configurations on kyuubi offical doc site or google search.
My kyuubi-defaults.conf:

spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension
spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog
fs.azure.account.key.<account>.blob.core.windows.net=<token>

But it doesn't work as expected. Am I configure kyuubi right? Is there any docs and examples about how to use kyuubi with delta-lake (Azure blob storage) ?
Thanks!

pizhihui
@pizhihui
Hi Team.
We are using Spark3.0 in hive v1.2,But kyuubi use hive v2.3,now We can't how to sovle the conflict of hive version.
Thanks!
roychen11232357
@roychen11232357

各位好,

我现在在尝试ACL Management (https://kyuubi.apache.org/docs/stable/security/authorization.html)

我想使用SQL-Standard Based Authorization,

我也在spark home放了submarine-spark-security-0.6.0.jar, 但启动了beeline后
想执行GRANT/REVOKE, 却得到Operation not allowed: GRANT,

请问是否缺少什么步骤呢?

或是说我对SQL-Standard Based Authorization的操作方式认知有误呢?

=============

Hello, everyone,

I am now trying ACL Management (https://kyuubi.apache.org/docs/stable/security/authorization.html)

I want to use SQL-Standard Based Authorization,

I also put submarine-spark-security-0.6.0.jar in spark home, but after starting beeline
I want to execute GRANT/REVOKE, but get Operation not allowed: GRANT,

Are there any steps missing?

santanu mohanty
@km_santanu_twitter
how to fetch 10milinum records by kuubi ..Any pagination or tune required
Ramakrishna Chilaka
@RamakrishnaChilaka
Hi
does kyuubi also suffer from the above issue ?
Ben Roubicek
@brickyard
Hi Team, great work on the project. I have Kyuubi up and running in my K8s environment and love it so far. I have LDAP authentication enabled. My question is whether passwords are safe over-the-wire in this configuration. Can someone please explain or give some hints? We are not using Kerberos and do not intend to. Thanks!
Kent Yao
@yaooqinn

does kyuubi also suffer from the above issue ?

Are you collecting the many results to driver?

how to fetch 10milinum records by kuubi ..Any pagination or tune required

yes, try the latest version of apache kyuubi 1.4.0-incubating, I shall have a incremental collection support

Hi Team, great work on the project. I have Kyuubi up and running in my K8s environment and love it so far. I have LDAP authentication enabled. My question is whether passwords are safe over-the-wire in this configuration. Can someone please explain or give some hints? We are not using Kerberos and do not intend to. Thanks!

Hi, Ben. I don't know much about LDAP, this part is mostly copied from Apache Hive. Based on the widely use of Hive, I guess it's OK.

Kent Yao
@yaooqinn

各位好,

我现在在尝试ACL Management (https://kyuubi.apache.org/docs/stable/security/authorization.html)

我想使用SQL-Standard Based Authorization,

我也在spark home放了submarine-spark-security-0.6.0.jar, 但启动了beeline后
想执行GRANT/REVOKE, 却得到Operation not allowed: GRANT,

请问是否缺少什么步骤呢?

或是说我对SQL-Standard Based Authorization的操作方式认知有误呢?

=============

Hello, everyone,

I am now trying ACL Management (https://kyuubi.apache.org/docs/stable/security/authorization.html)

I want to use SQL-Standard Based Authorization,

I also put submarine-spark-security-0.6.0.jar in spark home, but after starting beeline
I want to execute GRANT/REVOKE, but get Operation not allowed: GRANT,

Are there any steps missing?

this module will be maintained in Kyuubi project soon. The plugin only support ACL checking now, and the DCL is not support yet

Hi Team.
We are using Spark3.0 in hive v1.2,But kyuubi use hive v2.3,now We can't how to sovle the conflict of hive version.
Thanks!

Hi, please search hive in our online doc, there is a solution there

Kent Yao
@yaooqinn
Hi Kyuubiers, Kindly ask to leave a simple powered by note on https://github.com/apache/incubator-kyuubi/discussions/925, it really helps get more people involved and make kyuubi better
@brickyard @RamakrishnaChilaka @km_santanu_twitter @roychen11232357 thank you very much
Ben Roubicek
@brickyard
Is there a way to trigger a session shutdown from Spark SQL?
Kent Yao
@yaooqinn
what kind of session?client session?spark session?
Ben Roubicek
@brickyard
spark session. For users to be able to easily set new configurations
Kent Yao
@yaooqinn
spark session is a thread level,users can have a isolated session state,including SQL configs, for each connection
if you want to manage all configs including resource oriented ones like memory,cores, etc,I suggest you use kyuubi.engine.share.level=connection,the entire runtime spark context will shutdown when the connection closes
Kent Yao
@yaooqinn
other levels will cache the spark contexts for a while when all conns disconnected
you can make the TTL shorter
or there is also a button on spark web UI to kill it manually
Razvan Vacaru
@rvacaru
Hi all!
I'm trying to fix issue apache/incubator-kyuubi#1610 . For this I'm trying to mvn install the root project but I'm getting an error:
java.lang.NoClassDefFoundError: Could not initialize class org.apache.kyuubi.service.authentication.PlainSASLServer$

if I do an mvn compile it can't resolve the imports below:

import org.apache.kyuubi.jdbc.hive.logs.InPlaceUpdateStream;
import org.apache.kyuubi.shade.org.apache.hadoop.hive.common.log.InPlaceUpdate;

I'd like to run the kyuubi server to test the fix, and it's my first contribution to the project, do you know how to build / run this correctly?

Ramakrishna Chilaka
@RamakrishnaChilaka
apache/submarine#861 Can someone please check this, this is a PR to enable spark security on spark 3.2, Please note that this is very rough PR, I want to get one initial review before I polish it up.. Please check thanks.
santanu mohanty
@km_santanu_twitter
Is zookeper is delaying our responce time in kyuubi?

2021-12-24 06:09:08.534 DEBUG zookeeper.ClientCnxn: Got ping response for sessionid: 0x104f9c40d280000 after 0ms

2021-12-24 06:09:23.515 DEBUG server.FinalRequestProcessor: Processing request:: sessionid:0x104f9c40d280009 type:ping cxid:0xfffffffffffffffe zxid:0xfffffffffffffffe

exactly 15 second after Processing request is processing...will this spark sql query will delay by 15 seconds?We are getting bad performance compare to thrift server in kyuubi.is it the reason?
santanu mohanty
@km_santanu_twitter
can anyone please answer on it?
santanu mohanty
@km_santanu_twitter
is zookeper is eating of response time for kyuubi durin database query
jia
@zhenjiaguo
No, zookeeper is used for service discovery. Once the connection is established with the kyuubi servers and engines, the query does not need to go through zookeeper
Kent Yao
@yaooqinn
what were the steps you did your test?and any other additional information you can provide to us?
btw better to ask questions in mailinglist to get more people involved