by

Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • 05:14
    dependabot-preview[bot] labeled #680
  • 05:14
    dependabot-preview[bot] opened #680
  • 05:14

    dependabot-preview[bot] on cargo

    Bump raft from `d71daff` to `0c… (compare)

  • Sep 11 19:09
    dependabot-preview[bot] labeled #679
  • Sep 11 19:09
    dependabot-preview[bot] opened #679
  • Sep 11 19:09

    dependabot-preview[bot] on cargo

    Bump serde from 1.0.115 to 1.0.… (compare)

  • Sep 08 16:11

    hntd187 on master

    Fix test (compare)

  • Sep 08 15:00
    dependabot-preview[bot] commented #641
  • Sep 08 15:00
    dependabot-preview[bot] edited #641
  • Sep 08 14:50

    hntd187 on master

    Added sync client for toshi cli… (compare)

  • Sep 08 14:48
    dependabot-preview[bot] edited #641
  • Sep 08 14:48

    hntd187 on master

    Added sync client for toshi cli… (compare)

  • Sep 06 19:48
    dependabot-preview[bot] commented #641
  • Sep 06 19:48
    dependabot-preview[bot] edited #641
  • Sep 06 19:37

    dependabot-preview[bot] on cargo

    (compare)

  • Sep 06 19:37
    dependabot-preview[bot] closed #676
  • Sep 06 19:37

    dependabot-preview[bot] on cargo

    (compare)

  • Sep 06 19:37
    dependabot-preview[bot] edited #676
  • Sep 06 19:37
    dependabot-preview[bot] closed #673
  • Sep 06 19:37
    dependabot-preview[bot] commented #676
Stephen Carman
@hntd187
and distributed index summaries
I've written I think the client side for both
I think both need their serverside rpc calls
blitzerr
@blitzerr
okay
oh and the bulk insert too
I just stubbed it out I didn't think about it at all yet
but I'd imagine it's the harder of what's left but it's reasonable
blitzerr
@blitzerr
Thanks a lot for the code pointer.
Stephen Carman
@hntd187
I mean I'm just open to wherever people wanna dig in it's really up to anyone
blitzerr
@blitzerr
Is there a patch/commit you can point me to ?
For some of what you mentioned ?
Stephen Carman
@hntd187
what specifically ar eyou looking for?
blitzerr
@blitzerr
I built and ran toshi it just works. Awesome.
@hntd187 I will browse the source and ask you questions (probably a lot :) )
Stephen Carman
@hntd187
Okay no probs just lemme know how you fare
blitzerr
@blitzerr
👍
blitzerr
@blitzerr
Hi @hntd187, I am going through the code. You have done quite a lot single handedly. kudos. I am wondering how do you debug or step through.
Stephen Carman
@hntd187
Depends what I am working with
Some of the futures stuff is somewhat...difficult to step through at times
I've kind of learned how to debug it by seeing it fail a million times
but maybe you could be more specific
blitzerr
@blitzerr
Let's say, you want to step through the code. It is a single machine system at the moment, so that might be doable, right ?
I don't have a specific use case. I was just wondering how to.
Stephen Carman
@hntd187
Oh yea, my unit tests give me pretty good insight into individual parts
but I run locally with high debugging
or I have a lot of extra hardware to run clusters on in my apartment
blitzerr
@blitzerr
thanks a lot
Stephen Carman
@hntd187
and if you're working on something and need help investigating things myself or Paul the creator of tantivy has been incredibly helpful to me
blitzerr
@blitzerr
thanks a lot @hntd187
orangesoup
@orangesoup
Hey! It's probably been asked a lot, but... is there a way to use a custom tokenizer?
Stephen Carman
@hntd187
Actually it's not @orangesoup but no there is no current way, but I'm open to suggestions for how you might want to do this in toshi
and by that I mean how you would want to define it during index creation or something of the sort
blitzerr
@blitzerr
@hntd187 do you nhave
Sorry did not mean to hit send yet
Stephen Carman
@hntd187
Yes I have it
blitzerr
@blitzerr
:)
I was thinking to making Toshi scale and be more elastic than elastic search and I was thinking we would need to split hot shards and move them over to different machines etc. I was thinking that for this I would have to know the Tantivy internals well. So I was wondering if you have something on that. Their github does not contain any design docs
Stephen Carman
@hntd187
So I don’t know about design docs but I know a bit about it from actual usage of tantivy. The segments in tantivy the api for it is somewhat locked down a bit I want to talk to Paul about opening the api up a bit
I think toshi “shuffling” shards is kinda hard right now
blitzerr
@blitzerr
hmm..
Stephen Carman
@hntd187
Yup so I was going to do some tantivy work for their next release to open up that api a bit
so we could shuffle individual segments
Franz Heinzmann
@Frando
@hntd187 so the thing is, src/query in toshi is as far as i can see currently the only json impl for tantivy's query structs. so when wanting to write queries for a tantivy index in json, you'd either have to rewrite it, copy paste it, or factor it out into a crate
what would stand against having the toshi-query crate and thus a split between the parts of toshi that rely on an http server and those that just directly extend tantivy?
Stephen Carman
@hntd187
or you can just use toshi isn't of rewriting all the code yourself?
Franz Heinzmann
@Frando
@hntd187 yes sure we're still thinking about that. we didn't start with toshi because we don't need a builtin http server
Stephen Carman
@hntd187
you're essentially taking the json types to write your own http server and manage indexes yourself, which toshi provides all that
Franz Heinzmann
@Frando
well I have tantivy running as a binary that's running as a subprocess in node and there is fed with data from p2p replicated data structutes (via hyperdrives/Dat)
so it would be an option to use all of toshi, but it would be nicer for me to not have an unneeded http server on the localhost for it. we want this to run not only on servers but regular laptops. its pros and cons, not completely sure yet either