Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
jatcwang
@jatcwang:matrix.org
[m]
@AlexITC: I think this may be due to how fs2 resource cleanup works. It typically only cleans up after the whole stream ends unless you add a "scope" like you did in onFinalize.
I'm a bit fuzzy on doobie+fs2 since I've never used it that way. But I think the default behaviour is a bit surprising so it's worth looking into. Perhaps we should add the onFinalize always, as you did there.
Alexis Hernandez
@AlexITC
Indeed I think the behavior is very surprising, still, adding onFinalize before returning the stream would break referential transparency, wouldn't it?
Maxence Cramet
@max5599

Hi everyone, I'm trying to test an issue due to concurrent access to the database

val find = sql"SELECT id FROM person WHERE name='max'".query[Long]
OptionT(find.option).getOrElseF {
  for {
    sp <- HC.setSavepoint
    _ <- // do some stuff that needs to be rollbacked if the following insert fail
    id <- sql"INSERT INTO person(name) VALUES('max')".update.withUniqueGeneratedKeys[Long]("id").recoverWith {
      case _: SQLIntegrityConstraintViolationException => HC.rollback(sp) *> find.unique // This happen when a concurrent process insert "max" after the select and before the insert
    }
  } yield id
}

How can I do the insertion of the value "max" between the select and the insert? Thx!

Sölvi Páll Ásgeirsson
@solvip
Is there a shorthand for enforcing that an update only touches a single row? I'm thinking about something like .update.withUniqueGeneratedKeys but I'm not really interested in the return result
Sölvi Páll Ásgeirsson
@solvip
update.withUniqueGeneratedKeys[Long]("id") is good enough and does what I want :)
jatcwang
@jatcwang:matrix.org
[m]

To start, will need two IOs running in parallel. Something like (insertSelector, inserter).parTupled where insertSelector is the logic you have above and inserter is just another IO that inserts (to trigger the unique constraint violation)

To ensure that the order of operation happens as intended, there are two ways I can think of to reliably test this:

  1. Use https://typelevel.org/cats-effect/docs/core/test-runtime and add in some IO.sleep
  2. Use some sort of synchronization (Ref) to control the exact order of operation

Let me know if you need more info on either approach :)

5 replies
jatcwang
@jatcwang:matrix.org
[m]
Yep that's most likely the case. Each ConnctionIO when run is wrapped in a transaction (by default) so transaction isolation should kick in, but each DB has their own default isolation level
Rafał Sierkiewicz
@RafalSierkiewicz

Hey all,
I've have a problem with database queries. It looks like something is wrong with database connection, it gets stuck while executing query. I have no idea what I am doing wrong.
I am using following libraries versions:

  • doobie = "1.0.0-RC2"
  • http4sVersion = "0.23.10"
    I am creating hikari transactor in the following way
      for {
        ec <- ExecutionContexts.fixedThreadPool(32)
        transactor <- HikariTransactor
          .newHikariTransactor[F]("org.postgresql.Driver", dbConfig.url, dbConfig.user, dbConfig.password, ec)
      } yield transactor
    Here there is my query
    sql"""INSERT INTO articles(title,body) values ($title, $body)""".update.withGeneratedKeys[Id]("id")
    And code gets stuck in here and any other code using database.
    def insert(title: String, body: String): fs2.Stream[F, Article.Id] =
      articleDao.insert(title, body).transact(xa)

I am using Flyway for migrations and those are executed correctly with the same configuration.
Do you have any idea how I can debug it? Or what might be the problem?

Rafał Sierkiewicz
@RafalSierkiewicz
Moreover I found when I create transactor with unsafeRunSync it works, but when it is created in that way it seems to hangs forever
  def stream[F[_]: Async]: Stream[F, Nothing] = {
    for {
      client <- Stream.resource(EmberClientBuilder.default[F].build)
      conf <- Stream.eval(config)
      _ <- Stream.eval(runMigrations(conf.db))
      transactor <- Stream.resource(createTransactor(conf.db))
      httpApp = routes(BlogModule.make(transactor)).orNotFound
      finalHttpApp = Logger.httpApp(true, true)(httpApp)
      exitCode <- Stream.resource(
        EmberServerBuilder
          .default[F]
          .withHost(ipv4"0.0.0.0")
          .withPort(port = Port.fromInt(conf.app.port).get)
          .withHttpApp(finalHttpApp)
          .build >>
          Resource.eval(Async[F].never)
      )
    } yield exitCode
  }.drain
Rafał Sierkiewicz
@RafalSierkiewicz
I found the problem. It is correlated with opaque type Id = Int when I switched to Int it is working. I will look closer on the custom mappings !
Toby Thain
@toby5box
Hi! idk if this is a frequent and annoying comment but the github still points here and not Discord
Toby Thain
@toby5box
i'm not on discord, so i'll ask my question here, is there a way to get better context for query exceptions than is provided by the implicit logging?
the backtrace obviously does not reflect the call site. it's always been a paper cut so i just thought i should ask
also: Should I move to Skunk
(our service is large already but if there's a payoff, I'm just crazy enough to maybe do it!)
Eric K Richardson
@ekrich
Do you need an invite for Discord typelevel? I don't think you'll find much help here.
@toby5box
Toby Thain
@toby5box
is it invitation only?
Toby Thain
@toby5box
ok, i'm signed up. thanks @ekrich
Eric K Richardson
@ekrich
Sorry, was in the process of pasting the link but got distracted.
Eric von Dike
@ericvondike1_gitlab
Hello, I am new to doobie and i am not sure if this is the good place to ask this question: When I click on the links provided in doobie documentation for the source code I get a 404 response back which is not of course what I expect ;-) I think the page has been moved but the links are not updated properly. Can anybody send me the correct link to the source codes please?
jatcwang
@jatcwang:matrix.org
[m]
@ericvondike1_gitlab: https://github.com/tpolecat/doobie
Eric K Richardson
@ekrich
Doobie is now in the Typelevel discord - https://discord.gg/s6ks324t
For questions
Leo Uzcategui
@leou

I'm following the documentation trying to connect to a database but keep getting an authentication failure when I run program1.transact(xa).unsafeRunSync(). I'm running Ubuntu 20.04 in WSL 2 with PostgreSql 12 installed. I can successfully connect to and query the 'world' sample database using 'psql' but not in the scala REPL (see output below).
I would appreciate any help so I can connect and continue learning Doobie. Thanks in advance.
Leo
'''
scala> val program1 = 42.pure[ConnectionIO]
val program1: doobie.free.connection.ConnectionIO[Int] = Free(...)

scala> val xa = Transactor.fromDriverManagerIO
val xa: doobie.util.transactor.Transactor.Aux[cats.effect.IO, Unit] = doobie.util.transactor$$anon$13@6a16a078

scala> val io = program1.transact(xa)
val io: cats.effect.IO[Int] = IO(...)

scala> io.unsafeRunSync()
org.postgresql.util.PSQLException: FATAL: password authentication failed for user "leou"
at org.postgresql.core.v3.ConnectionFactoryImpl.doAuthentication(ConnectionFactoryImpl.java:623)
at org.postgresql.core.v3.ConnectionFactoryImpl.tryConnect(ConnectionFactoryImpl.java:163)
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:215)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:51)
at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:225)
at org.postgresql.Driver.makeConnection(Driver.java:466)
at org.postgresql.Driver.connect(Driver.java:265)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:677)
at java.sql/java.sql.DriverManager.getConnection(DriverManager.java:228)
at doobie.util.transactor$Transactor$fromDriverManager$.apply

KaTeX parse error: Can't use function '$' in math mode at position 8: anonfun$̲2(transactor.sc…: anonfun$2(transactor.scala:387)
  at doobie.util.transactor$Transactor$fromDriverManager$.$anonfun$1(transactor.scala:355)
  at blocking @ doobie.util.transactor$Transactor$fromDriverManager$.create
anonfun$1(transactor.scala:355)
at make @ doobie.util.transactor$Transactor$fromDriverManager$.create
KaTeX parse error: Can't use function '$' in math mode at position 8: anonfun$̲1(transactor.sc…: anonfun$1(transactor.scala:357)
  at make @ doobie.util.transactor$Transactor$fromDriverManager$.create
anonfun$1(transactor.scala:357)
at use @ doobie.util.transactor$$anon$4.apply(transactor.scala:167)
'''

2 replies
jatcwang
@jatcwang:matrix.org
[m]
Hm that's weird. The error indicates that you're trying to connect as the user leou
@leou: Can you post the full code you're trying to run (use code block please otherwise they get lost)

The example code you're following are actually run during doc compilation, so it's interesting that it's breaking for you.

What you can try (if you haven't alreay) is to clone the doobie repo, and run docker compose up -d which should setup the postgres DB like it is used in the doc compilation

Leo Uzcategui
@leou
Thanks. I managed to get past the authentication failure by changing the configuration in 'pg_hba.conf' to 'trust' instead of 'peer'
The examples from the documentation now work as described using Scala 2.13.7, but when tried using Scala 3.1.1, the macro expansions fail, e.g. 'sql"select 42"'
jatcwang
@jatcwang:matrix.org
[m]
That's weird. I tried both latest of 0.13.x and 1.x and they compile fine. Here's 0.13.x https://scastie.scala-lang.org/3tYXD4VJSr21dilXn9jiNA
@leou: If you can reduce it to a smaller example would love to take a look at what's causing the macro expansion to fail
Leo Uzcategui
@leou

@jatcwang In build.sbt I have scalaVersion := "3.1.1" and lazy val doobieVersion = "1.0.0-RC1" and this is what happens in the REPL
"""
scala> val program1 = 42.pure[ConnectionIO]
scala> val xa = Transactor.fromDriverManagerIO
scala> val io = program1.transact(xa)
scala> io.unsafeRunSync()
val res0: Int = 42

scala> val program2 = sql"select 42".query[Int].unique
-- Error: ----------------------------------------------------------------------
1 |val program2 = sql"select 42".query[Int].unique
| ^
| Exception occurred while executing macro expansion.
| java.lang.NullPointerException
| at doobie.util.PosPlatform$.originImpl(PosPlatform.scala:21)
|
| This location contains code that was inlined from rs$line$9:1
1 error found
"""

jatcwang
@jatcwang:matrix.org
[m]
@leou: I think it might be a Scala repl issue actually. If you try it in an actual codebase it should work :)
Leo Uzcategui
@leou
@jatcwang: Okay thanks, you're right, it works with Scala 3 in regular code.
Ghost
@ghost~6047c92a6da03739846691e0
is it possible to set a query/statement timeout on a HikariDataSource. ? I am using doobie with the tranzactio library and I want to put in some fail safes against long running queries eating up connections
balthz
@balthz
I'm reading https://tpolecat.github.io/doobie/docs/13-Unit-Testing.html about testing: it's great how easy it is to type-check queries against the DB. Is there a way I can get the actual, interpolated SQL query back (meaning, the SQL query with ? replaced by the actual values sent to the DB) to run assertions on the SQL strings as well? Or is there a better practice for this?
jatcwang
@jatcwang:matrix.org
[m]
@balthz: The underlying DB (and also the JDBC API) will need to support it. I'm not aware of any DB that does but if you find out let me know :)
Jeff
@eceppda

Hi all, I think I'm missing a key detail about the use of Meta. I have a simple program consisting of a few case classes each with a few parameters. The parameters are all simple types: Int, String, some Options. I have defined an implicit Reads and Writes in the companion object of each. Did I need to define the implicit Read and Write for each, or can they be derived using Meta?

I read in the docs the inductive cases, We can Read or Write a product type (case class or tuple) if its shapeless Generic representation (i.e., its fields as an HList) can be read or written, respectively. This suggests to me that it can, but I haven't found a convincing example that this is the intended use.

Can I do something like this?

case class Thing(a: Option[Int], b: String, c: Long, d: Option[String])

object Thing {
    implicit val thingMeta: Meta[Thing] = ???
}
Jeff
@eceppda
My ultimate use case is a legacy system with a large, 130 column table. I want to minimize the number of times I have to write the column list.
Jeff
@eceppda
It looks like FirstExample.scala covers this.
ivan-klass
@ivan-klass
Hi guys! I'm sorry if I ask the question which has been probably asked multiple times already - I saw a lot of pain about timezone-aware datatype, but still can't clearly understand how do I deal with problem - it seems like a bug to me. My postgres DB is executed in UTC, while the app is local (UTC +2h). Doobie version is 1.0.0-RC2, scala 3.1.1.
Using Transactor.fromDriverManager in test scope.
The following assertion fails - there's a 2 hours time-shift in round-trip:
val instant: Instant = Instant.now().truncatedTo(ChronoUnit.SECONDS)
val dateTime: ZonedDateTime = instant.atZone(ZoneOffset.UTC)
assert(
  sql"""
    SELECT $dateTime
  """
    .query[ZonedDateTime]
    .unique
    .transact(xa)
    .unsafeRunSync()
    .toInstant == instant,
  "Expected identical instant, got time-shifted!"
)
ivan-klass
@ivan-klass
Currently I end up with following
transactorResource
  .preAllocate(
    Sync[F].delay(TimeZone.setDefault(TimeZone.getTimeZone("UTC")))
  )
jatcwang
@jatcwang:matrix.org
[m]

@ivan-klass: Most popular databases do not support a time type that stores the timezone, so you don't really want to be using ZonedDateTime.

You want to use either Instant (if this is an instant in time), or LocalDateTime (if you care about the wall clock time, e.g. maybe for a calendar event)

I don't recommend changing your timezone for the whole JVM like what you're doing with TimeZone.setDefault. Its global and can have nasty impact on other parts of your app
ivan-klass
@ivan-klass
@jatcwang:matrix.org thank you for the answer. I'm afraid most of the codebase in the project already bound to ZonedDateTime - the intent was to know both user wall-clock and exact moment. I understand the meaning of JVM-global timezone, but the app I'm working on is basically a standalone web service which production is hosted in UTC, so setting default doesn't seem dangerous for running app/tests locally. As for doobie support- I wonder why do we have dedicated typeclasses for ZonedDateTime then, even in postgres-specific scope? https://github.com/tpolecat/doobie/blob/409a51cd95406f0ea332a9649253c6df12f2ec15/modules/postgres/src/main/scala/doobie/postgres/JavaTimeInstances.scala#L45 - is there any chance to fix it there, or is it a JDBC design problem?
jatcwang
@jatcwang:matrix.org
[m]
@ivan-klass: I think it's a mistake to have that typeclass instance in core because as you've discovered it has quite surprising behaviour for 90% of the DBs out there 😓
Certain DBs does have proper ZonedDateTime support (storing the instant + timezone) but most dont
Diego Casella
@diego.casella_gitlab
Hi all,
I'm quite puzzled by the behavior of nested SELECT operations. I have a query defined like so
val query = fr"""SELECT I.id, I.tx_id FROM ($innerQuery) AS I""" // ++ whereAnd(fr"""I.tx_id = '$id'""")
which correctly returns a list of (Long, String) tuples.
However, if I uncomment the whereAnd(...) part, at runtime I get this exception:
org.postgresql.util.PSQLException: The column index is out of range: 3, number of columns: 2
what is going on here?
jatcwang
@jatcwang:matrix.org
[m]
@diego.casella_gitlab: Yeah that seems off. Once you've turned it into a Query/Query0 you can call .sql to see the actual SQL. Perhaps that'll help you figure out what's wrong?
Also I don't don't think you need/want the single quotes around $id. So this: whereAnd(fr"""I.tx_id = $id""")