These are chat archives for rust-lang/rust

25th
Sep 2018
Kelly Thomas Kline
@kellytk
Sep 25 2018 01:36
If a binary uses two local unpublished libraries A and B, and A also uses B directly, how can the binary disambiguate them so cargo build doesn't fail with the error https://github.com/rust-lang/cargo/blob/8201560b6955426d887a3621ee4b204d416ce9be/src/cargo/core/resolver/mod.rs#L1144 ?
Kelly Thomas Kline
@kellytk
Sep 25 2018 05:28
More information at https://ghostbin.com/paste/5kexc
Denis Lisov
@tanriol
Sep 25 2018 07:33
@yurique Sounds strange to me. I'd expect enums to be no slower than traits.
Denis Lisov
@tanriol
Sep 25 2018 07:56
@kellytk How about linking to the same path, not different ones?
Kelly Thomas Kline
@kellytk
Sep 25 2018 07:58
@tanriol Where?
Denis Lisov
@tanriol
Sep 25 2018 08:02
Actually, does the /Users/user/project/lib/repoa/lib/repob path even exist?
Kelly Thomas Kline
@kellytk
Sep 25 2018 08:05
Yes, my convention for local private repos is syncing their source into $project/lib/$repo
That puts repob into repoa/lib/repob, and both into the binary project/lib/
Kelly Thomas Kline
@kellytk
Sep 25 2018 08:14
The concrete case I'm working on is the binary is an actix web server, it includes the two libs, repoa is the core logic of the service, repob is the communication message data types, and repoa includes repob as well directly. I would reexport the message lib from the core logic lib, but I use the message lib in the frontend code, (Yew) another binary project, which doesn't need to run the core logic code and probably can't to begin with as it assumes an actix actor environment
Denis Lisov
@tanriol
Sep 25 2018 09:05
@kellytk The problem here is that you most likely don't want to "disambiguate" (use two separate copies of crate B) as the types would be different.
How about actually depending on
repob = {path = "./lib/repoa/lib/repob"}
Kelly Thomas Kline
@kellytk
Sep 25 2018 09:06
@tanriol Why would the types be different if the two includes of the lib bring in the same code, if through different files?
I could change it to that but it feels like "reaching in", or a bad programming practice
Denis Lisov
@tanriol
Sep 25 2018 09:06
If they are compiled as two separate crates, the types in them are separate too.
Another option is for repoa to reexport the types (possibly the entire repob crate).
However, if I were you I'd change the convention probably.
Kelly Thomas Kline
@kellytk
Sep 25 2018 09:10
What seems strange is if I published repob to crates I think I could add it as a dependency to both repoa and binary with no problems
How would you alter the convention?
Iurii Malchenko
@yurique
Sep 25 2018 10:06

@tanriol
so this is a simplification of how it looked before:

enum Tree { 
  NodeA, NodeB(Box<Tree>), NodeC(Box<Tree>, Box<Tree>)
}

impl Tree {
  fn do_it(&self) -> Something {
    match self {
      NodeA => …,
      NodeB(box ref sn) => sn.do_it(),
      NodeC(box ref sn1, box ref sn2) => 
        let d1 = sn1.do_it();
        let d2 = sn2.do_it();
        //.. 
    }
  }
}

and new version:

trait Tree

struct NodeA;
struct NodeB(Rc<Tree>);
struct NodeC(Rc<Tree>, Rc<Tree>);

impl Tree for NodeA {
  fn do_it(&self) -> Something {
      …
  }
}

impl Tree for NodeB {
  fn do_it(&self) -> Something {
      self.0.do_it()
  }
}

impl Tree for NodeC {
  fn do_it(&self) -> Something {
        let d1 = self.0.do_it();
        let d2 = self.1.do_it();
        //.. 
  }
}

eventually a tree like this gets constructed (and it’s quite large, a couple of hundreds of nodes roughly) and then the do_it() (and similar functions) is called a LOT

and you’re right, the performance of the recursive di_it() is virtually the same

the place that got much better (1ms -> 0.05ms) is preparing the tree and manipulating it (before the do_it() thing), that’s because of Box vs Rc

so in the benchmarks locally I have a 14x improvement of the performance of the main function (which includes all of the above)

on lambda it results in ~20x performance improvement

so the bottom line is - I should have known to use Rc from the very beginning

Denis Lisov
@tanriol
Sep 25 2018 10:11
Did you say that you have lots of paths that are shared with Rc but were duplicated with Box?
Iurii Malchenko
@yurique
Sep 25 2018 10:14
yeah
and I have a lot of .clone() in both cases, but with Rc it’s basically free
at least that’s how I understand what’s going on :)
it’s a bit more complicated, but essentially there are functions on nodes that transform it (return an updated node), like replacing a child node, or returning a child node instead, etc
in my current version those functions return Rc<>, and .clone whenever needed
Denis Lisov
@tanriol
Sep 25 2018 10:17
Yeah, that sounds plausible... especially that with hundreds of nodes this may be the difference between fitting the CPU cache (possibly even L1) and getting cache misses.
Thiez
@Thiez
Sep 25 2018 11:30
I imagine you could do even better by making one big enum that contains all possible node types, changing all your Rc<Tree> instances to usize, and then use indexing into a big Vec<TheMagicEnum>, but ymmv :)
"First make it work, then make it fast" and all that
But in such a case you basically get a copying and compacting garbage collector "for free" when you (recursively) copy your root node to a new Vec, which is pretty cool ;)
Iurii Malchenko
@yurique
Sep 25 2018 11:42
Sounds like an interesting idea =)
though nodes (some enum varianta) might contain other nodes, so that might get problematic
right now I made it use all available cores on lambda, and I’m getting 4k “units of work” per second
Scala implementation gives me like 10, maybe 20, “units of work” per second
so I’m going to stop optimizing, for a while at least :)

Scala implementation gives me like 10, maybe 20, “units of work” per second

well, not exactly 10-20, but considering the overheads that’s the actual result for the time limit I give it (and jvm is not hot, of course)

Thiez
@Thiez
Sep 25 2018 11:45
Is the source of your work going to be publicly available somewhere, or is it a closed-source internal thing? :)
Iurii Malchenko
@yurique
Sep 25 2018 11:48
unfortunately it’s internal :)
but I might end up putting together a blog about it, without the specifics of the actual task I’m doing :)
well, given that’d be of an interest for someone
Thiez
@Thiez
Sep 25 2018 11:49
I'd be delighted to read it. And the people on /r/rust are always happy to hear Rust success stories.
Iurii Malchenko
@yurique
Sep 25 2018 11:50
cool! I’ll be looking for some spare time then :)
Thiez
@Thiez
Sep 25 2018 11:51
"200x performance increase when using Rust instead of Scala on AWS Lambda" will get you a couple of eyeballs
Iurii Malchenko
@yurique
Sep 25 2018 11:51
=)
Michal 'vorner' Vaner
@vorner
Sep 25 2018 12:13
Well, I guess once I manage to pull off the thing I have POC in my work right now, doing „from 15GB of RAM with Java to 80MB with Rust“ might be a good topic for a blog post eventually too 😇
Lyle Mantooth
@IslandUsurper
Sep 25 2018 12:36
I'd read that. :)
Iurii Malchenko
@yurique
Sep 25 2018 12:37

Well, I guess once I manage to pull off the thing I have POC in my work right now, doing „from 15GB of RAM with Java to 80MB with Rust“ might be a good topic for a blog post eventually too

impressive :)

Michal 'vorner' Vaner
@vorner
Sep 25 2018 12:39
To be fair to java, part of that is due to a different data structure. Rust makes it more viable to use the data structure, but I don't think it would be outright impossible to do it in Java too, just much more work. And part is probably the GC and such.
Iurii Malchenko
@yurique
Sep 25 2018 12:41
Yeah, jvm can be fast
But in lambda it seems to be quite a challenge to achieve good performance (computational)
In my case it takes jvm something like 3-5 sec to start from cold; rust implementation starts instantly
Iurii Malchenko
@yurique
Sep 25 2018 12:53
btw locally java implementation is only ~5 times slower in benchmarks (which do warm up)
oh, no, something’s wrong with that scala benchmark :)
Iurii Malchenko
@yurique
Sep 25 2018 13:07
so locally jvm is 90 UOW per second, rust implementation - 3000 UOW per second
[ although there are some small deviations in the implementation details ]
Iurii Malchenko
@yurique
Sep 25 2018 13:41
but I think something’s wrong somewhere
shouldn’t really be that much of a difference
yeah, the difference on local machine seems to be about 2x in favour of rust implementation :)
Denis Lisov
@tanriol
Sep 25 2018 13:47
Sounds like you can write a blog post on correctly benchmarking Scala and Rust now...
Iurii Malchenko
@yurique
Sep 25 2018 13:52

Haha, true =)
the benchmarks were correct, actually, but in my rust code I suspect there’s some kind of a logical bug that shortcuts the calculation early (incorrectly) in some cases; disabling that shortcut logic completely showed believable results in benches

(still like 10x improvement in lambda :) )

Sylwester Rąpała
@xoac
Sep 25 2018 17:12
I am looking for collection to store futures I am waiting to be done. So I want to obtain this future from collection if resulted with ready I want remove it. I thought I can have 2 vectors and move values between them because LinkedList don't allow remove values and that surprised me.
Nathaniel McCallum
@npmccallum
Sep 25 2018 17:13
Can someone explain CastInto<T> to me? I want to be able to cast in a generic context. Using CastInto<T> sounds perfect to me. But the docs say it will never be stabilized.
Sylwester Rąpała
@xoac
Sep 25 2018 17:31
@npmccallum Why cast do you lose sth?
Nathaniel McCallum
@npmccallum
Sep 25 2018 17:32
I want to accept any integer type (generic), but take the lowest seven bits as a u8.
Sylwester Rąpała
@xoac
Sep 25 2018 17:34
so why don't accept Integer and then call value as u8? https://rust-num.github.io/num/num/trait.Integer.html
Oh I am not sure as u8 will work
Sylwester Rąpała
@xoac
Sep 25 2018 17:39
The other way is to create own trait that will have function cast_to_u8() -> u8 and implement it for all base types.
Lyle Mantooth
@IslandUsurper
Sep 25 2018 17:45
@npmccallum, maybe impl BitAnd<u8> to output a u8 instead of Self.
Buuut, that would only work for your own types, not primitives
Sylwester Rąpała
@xoac
Sep 25 2018 17:52

I am looking for collection to store futures I am waiting to be done. So I want to obtain this future from collection if resulted with ready I want remove it. I thought I can have 2 vectors and move values between them because LinkedList don't allow remove values and that surprised me.

Now I am thinking about iter over it and create 2 collections. One with results from futures and second with futures that returned not ready. Any idea how to do that?

Nathaniel McCallum
@npmccallum
Sep 25 2018 19:35
@IslandUsurper rust-lang/rust#54442
Lyle Mantooth
@IslandUsurper
Sep 25 2018 19:38
I guess I can see the reasoning there.
Denis Lisov
@tanriol
Sep 25 2018 19:48
@xoac Are you implementing your own futures executor?
Josh
@joshlemer
Sep 25 2018 20:01
What is your general workflow for finding how to import something? For example, if I am in intellij and I want to import BTreeSet, I might to COMMAND + N to find the BTreeSet implementation. But however, this is at alloc::btree:BTreeSet but I cannot import that path. I need to somehow find out that the publicly available path is std::collections::BTreeSet but I don't know how people find the public path
Denis Lisov
@tanriol
Sep 25 2018 20:06
In my case, there's a tab with documentation open anyway, so I just use its search field :-)
Sylwester Rąpała
@xoac
Sep 25 2018 20:27
@tanriol no, I just have a lot of request to do. And I have a struct RequestMaker and I make this requests in poll() of RequestMaker. That return Vec<Result>
For now I move then between two vectors of Vec<Box<Futere<...>>> so I hope it's cheep
D'Silva
@evnix
Sep 25 2018 20:38
is there a way to get the total size of all the fields in a struct. mem::size_of seems to add padding and other values.
Denis Lisov
@tanriol
Sep 25 2018 21:05
@xoac Is there any special reason for this architecture? I'd probably make a stream of requests and some buffering on top. Alternatively, have you seen FuturesUnordered?
@evnix Not without special crates. Why do you want to know that value?
D'Silva
@evnix
Sep 25 2018 21:16
@tanriol I want to read structs written to a file (using bincode). It is not possible to read without knowing the exact size that I must be reading. currently I just hardcode the exact number which I don't feel is good practice.
Denis Lisov
@tanriol
Sep 25 2018 21:20
You likely want to use bincode::serialized_size
...if you need size at all. Why not bincode::deserialize_from?
D'Silva
@evnix
Sep 25 2018 21:22
That's even better. Thank you!