Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
kannar
@KannarFr
Correct. I can serialize to text[].
Rob Norris
@tpolecat
Maybe I misunderstand. I have been assuming you wish to store a List[Array[Byte]] as a single value.
kannar
@KannarFr
If I could yes. But seems complicated, so maybe I can serialize List[Array[Byte]] to something easier on database side.
Rob Norris
@tpolecat
See this code and the blocks below that define mappings for other array types.
You may not care about the nullable case, in which case it will be simpler.
kannar
@KannarFr
Yup, I don't care about it. Will check it.
kannar
@KannarFr
Thanks
jatcwang
@jatcwang:matrix.org
[m]
Rob, FYI I've gone ahead and merged tpolecat/doobie#1469 since I don't think there's anything stopping us from publishing doobie-scalatest for Scala 3
Drew Boardman
@drewboardman
Hi, I'm trying to migrate to cats-effect 3 and I'm seeing some issue with the following:
[error]         * co.fs2:fs2-core_2.13:3.0.4 (early-semver) is selected over {2.5.3, 2.5.0, 3.0.3}
[error]             +- org.tpolecat:doobie-free_2.13:0.10.0               (depends on 2.5.0)

[error]         * org.typelevel:cats-effect_2.13:3.1.1 (early-semver) is selected over {3.1.0, 3.1.0, 2.3.3, 2.1.3, 2.3.1}
[error]             +- org.tpolecat:doobie-free_2.13:0.10.0               (depends on 2.3.1)

[error]         * co.fs2:fs2-io_2.13:3.0.4 (early-semver) is selected over 2.5.0
[error]             +- org.tpolecat:doobie-postgres_2.13:0.10.0           (depends on 2.5.0)
with is with my val doobieV = "1.0.0-M3"
Rob Norris
@tpolecat
The error says you're asking for version 0.10.0
Drew Boardman
@drewboardman
well I'm not directly declaring doobie-free in my build file at all
Rob Norris
@tpolecat
The last line of the error says org.tpolecat:doobie-postgres_2.13:0.10.0
You're certainly declaring that. Free is pulled in as a dependency.
Drew Boardman
@drewboardman
weird, this is how it's written in my dependencies
val doobieV              = "1.0.0-M5"

    "org.tpolecat"       %% "doobie-hikari"       % doobieV, //Streaming JDBC w/ Hikari support
    "org.tpolecat"       %% "doobie-postgres"     % doobieV, //Doobie Postgres support (replace if needed)
    "org.tpolecat"       %% "doobie-refined"      % doobieV //Doobie Postgres support (replace if needed)
Rob Norris
@tpolecat
did you reload your project after changing it?
Drew Boardman
@drewboardman
yeah i'm not sure why these errors popped up like this. I had to update a library that changed orgs and the errors went away
Greg Fisher
@gnfisher

I am processing data in streams. I have a Stream[ConnectionIO, (A, B)]. I am trying to save the A's and B's to the database. The B can't be saved unless it can find the local id of the A, so the As are processed first. This is the step I have and it is not working. The As save, but the Bs do not. I am not sure if its a misunderstanding of streams and evaluations (maybe inside the flatMap the two new streams are being evaluated concurrently, so B cant find local A to save?), or if there is something with doobie, transactions, etc perhaps I am doing something very wrong.

This is the code snippet if anyone can point me in the right direction to debug this further. processA and processB are Pipe[Connection, A, C]

    input.chunks.flatMap { chunk =>
      Stream.eval(Stream.chunk(chunk).map(_.a).through(processA).compile.drain) >>
        Stream.chunk(chunk).map(_.b).through(processB)
    }
Its not really a tuple (A, B) but a value, hence the map step there... but you get the idea.
If run in isolation both process steps work as expected, but when doing things this way, only processA has expected results.
Fabio Labella
@SystemFw
I didn't fully get your reply to my reply in the fs2 channel :)
Greg Fisher
@gnfisher
:wave: The "why do we have pipes"?
Its how its been, its the pattern I followed. I don't have a better explanation unfortunately, this is all pushing past where I have any sort of confidence in what i know and understand
What would an alternative to a pipe look like? One the pipe for parts is just an abstraction that wraps doobies updateManyWithGeneratedKeys. What else might it be if not a pipe? Or, why does using a pipe here cause trouble? Someone else mentioned refactoring the pipes to Chunk[A] => IO[Chunk[A]] might help but I'm not quite sure what they mean by that or where the change would happen. I think they thought that perhaps having these all in ConnectionIO could be a source of issues?
(also, happy to continue chatting in fs2 if this fits better there)
Grzegorz walen
@GrzegorzWalen_twitter
Hi,
I'm using Quill + Doobie to access db. What is the DoobieContext equivalent of QuillContext translate, or any other option I can enforce logging the
queries with parameter values ? I'm in the dead end any help appropriated.
Dmitry Polienko
@nigredo-tori

@gnfisher, with chunks the above can be simplified to

input.evalMapChunk { chunk =>
  processA(chunk.map(_.a)) >> processB(chunk.map(_.b))
}

Here processA: Chunk[A] => ConnectionIO[Chunk[B]] and so on. It's a little easier to reason about than streams.
Upd: removed the .chunks call.

Also less opportunity to confuse .eval and .eval_.
Greg Fisher
@gnfisher
Thanks @nigredo-tori , but I am still not sure why my more hard to read version doesn't behave as expected.
I will try that refactor to see if it makes a difference
Oh, I am on 2.0.1 of fs2 so i dont have evalMapChunk :(
Dmitry Polienko
@nigredo-tori
foo.evalMapChunk(bar) is foo.chunks.evalMap(bar).flatMap(Stream.chunk(_)).
Greg Fisher
@gnfisher
Would that example result in processA and processB running concurrently?
Dmitry Polienko
@nigredo-tori
No.
Greg Fisher
@gnfisher
I didn't think so. But that was the only explanation I could think of for why the processB records didnt persist to the DB while the As did. And if they are run in isolation, they each work as expected.
Fabio Labella
@SystemFw
are you sure you want >> and not ++?
Greg Fisher
@gnfisher
I tried ++ (by compiling and draining both) but that didnt work either. Although now all the cases I tried are blurring together, so I can go back and try again
but processA does not return a nything that processB takes as input
Fabio Labella
@SystemFw
yeah
Greg Fisher
@gnfisher
I feel very dumb these past days! :sweat:
Fabio Labella
@SystemFw
if the stream returned by processA is empty
I need to go, but text me later, and I can explain you >> vs ++
Greg Fisher
@gnfisher
Ok, thank you so much for all your help throughout this saga
Fabio Labella
@SystemFw
it's not hard but if you don't clearly know the difference it can be confusing, nothing to feel dumb about :)
Kristian Nordal
@nor

Hi, I'm having an issue with a mssql query. I try to fetch all ~25k rows from a table, and get this:

Jun 04, 2021 9:05:24 AM com.microsoft.sqlserver.jdbc.TDSReader throwInvalidTDS
SEVERE: ConnectionID:3 ClientConnectionId: 3c8eb486-fe5f-4e72-b267-69e79e780b93 got unexpected value in TDS response at offset:7992
Jun 04, 2021 9:05:24 AM com.microsoft.sqlserver.jdbc.TDSParser throwUnexpectedTokenException
SEVERE: ConnectionID:3 ClientConnectionId: 3c8eb486-fe5f-4e72-b267-69e79e780b93: getNextResult: Encountered unexpected TDS_FEATURE_EXT_UTF8SUPPORT (0x0A)

which to me looks like an encoding issue, but when I paginate the query; e.g. order by foo offset 0 rows fetch next 5000 rows only – I can query all the pages fine.
I'm using doobie, but this might be a pure jdbc issue?

jatcwang
@jatcwang:matrix.org
[m]
@nor: yep looks like it. Give raw jdbc a go and see if you get the same issue
Roberto Leibman
@rleibman

ok.... here's a question.... I have a bunch of ConnectionIO's strung together in a for comprehension.... half way through these I need to save something to the file system and base on where it ends up update a database. Because I'm using zio, this is somewhat what I have:

def save(saveMe: MyComplicatedObject, saveToFileSystem: MyComplicatedObject => ZIO[A,B,File]): ConnectionIO[MyComplicatedObject] = {
  for {
     res1 <- savePart1(saveMe) //returns ConnectionIO[MyComplicatedObject]
     res2 <- savePart2(res1) //returns ConnectionIO[MyComplicatedObject]
     res3 <- saveToFileSystem(res2) // <<<<< here... how to convert ZIO to ConnectionIO
     res4 <- savePart3(res3) //returns ConnectionIO[MyComplicatedObject]
  } yield res4
}

As you can see, the basic question, is how to convert a ZIO to a ConnectionIO? I know I can use .pure to get a ConnectionIO out of a normal thing, but I doubt .pure on a ZIO IO would give me what I want.

Rob Norris
@tpolecat
You probably don’t want to do that because if the transaction rolls back the filesystem operation will persist.
But in any case if you use cats-effect IO you can say myIO.to[ConnectionIO]. It may or may not work with ZIO, I don’t know.