These are chat archives for rust-lang/rust

29th
Apr 2018
Brandon Ros
@brandonros
Apr 29 2018 02:37
I see Rust has ODBC drivers, but I don't see anything for IBM DB2. I am able to compile Go + node.js against existing IBM DB2 drivers. They pull in the IBM ODBC code. Could the same be done for Rust with an existing ODBC package?
Hans W. Uhlig
@huhlig
Apr 29 2018 02:38
you should be able to. Rust provides full ffi capabilities and bind generation
Brandon Ros
@brandonros
Apr 29 2018 02:38
Would I have to write the integration... or would it be "plug and play:?
Hans W. Uhlig
@huhlig
Apr 29 2018 02:38
I can check on monday at work to see if anyone is writing a native one.
Brandon Ros
@brandonros
Apr 29 2018 02:38
You guys use DB2?
Hans W. Uhlig
@huhlig
Apr 29 2018 02:39
You can use bindgen against the raw C driver
I work for IBM :D
Brandon Ros
@brandonros
Apr 29 2018 02:39
That'd be so sick. :)
Hans W. Uhlig
@huhlig
Apr 29 2018 02:39
actually the DB2 guys work downstairs from me
Brandon Ros
@brandonros
Apr 29 2018 02:39
How large is the team these days?
The DB2 team
Hans W. Uhlig
@huhlig
Apr 29 2018 02:39
but Rust hasn't yet gained much traction, a couple of the system Z guys are using it so far
not entirely sure, I know there are quite a few
Brandon Ros
@brandonros
Apr 29 2018 02:39
I've read IBM has been going through a lot of structural changes. No more working from home, lots of layoffs. Is that true?
Hans W. Uhlig
@huhlig
Apr 29 2018 02:40
Our public statement is that we are undergoing a digital transformation and focusing on key objectives including Cloud, Artificial Intellegence and our Core Competencies
Brandon Ros
@brandonros
Apr 29 2018 02:41
Understood. We use a ton of old IBM stuff at my job.
WebSphere message queues
Liberty
DB2
It feels like we just grabbed everything off of the shelf lol
Hans W. Uhlig
@huhlig
Apr 29 2018 02:41
eh, I wouldn't call them old... Just mature
Brandon Ros
@brandonros
Apr 29 2018 02:41
DB2 was #1 most hated DB in StackOverflow 2018 developer survey, right?
Hans W. Uhlig
@huhlig
Apr 29 2018 02:41
was it
Brandon Ros
@brandonros
Apr 29 2018 02:41
Yep, lol, by a landslide
The version I have to use at work doesn't have LIMIT/OFFSET/built in pagination
Hans W. Uhlig
@huhlig
Apr 29 2018 02:44
ahh, Honestly I came in through the opensource team and ended up in the cloud team so for me I just use postgres
for me it scales nearly as well as oracle, db2 or mssql. If I need bigger I go back to real bigdata tools
Brandon Ros
@brandonros
Apr 29 2018 02:45
yep. i kind of hope IBM pulls the plug on DB2 so that the world has one less databsae to support
lol
Hans W. Uhlig
@huhlig
Apr 29 2018 02:47
heh, honestly we've expanded our support in many ways for it
BigSQL which is actually a pretty powerful BigData Aggregate Query platform is DB2
So is DashDB and Netezza
Brandon Ros
@brandonros
Apr 29 2018 02:48
do enterprises gravitate to it for the reasons you said, it's just mature?
and support is available professionaly?
Hans W. Uhlig
@huhlig
Apr 29 2018 02:50
yeah, it's mature, it's well supported, it has lots of tools and a lot of enterprise apps use it
SAP uses DB2 unless you splurge for Hana for example
plus I still think it's best in class if you're using XML based document storage
Brandon Ros
@brandonros
Apr 29 2018 02:51
I used to write e-commerce inventory management systems and potential users were like "oh... you are competing with my existing product, SAP"
is that like... a part of SAP's overall business?
I can't really put a finger on how to explain what SAP does... other than, "everything"
Hans W. Uhlig
@huhlig
Apr 29 2018 02:52
yeah, it's like oracle enterprise crm, erp, peoplesoft, etc
they are basically enterprise configurable crud workflow apps
Brandon Ros
@brandonros
Apr 29 2018 02:53
pivot to a different topic, I kind of have this silly juvenile brain itch
I have a node app that loads 1,000,000 jobs into a redis queue, then I am trying to toy with what has the highest concurrency in pulling them off of the queue and processing a job
Just as a basis, I removed the "process the job" part from the benchmark, and basically just have "pull item off of the queue" then do nothing
For some reason... node.js beat the snot out of golang, and I tried a ton of stuff
buffered channels, multiple workers, multiple instances
Hans W. Uhlig
@huhlig
Apr 29 2018 02:55
hmm
Brandon Ros
@brandonros
Apr 29 2018 02:55
I'm curious what a highly concurrent basic redisClient.blpop would look like in rust
like you seem super knowledgable so I'd gladly take a psuedo code writeup if you could... I was going to take a crack at it now... but I think this is a slightly above beginner requirement to get right in terms of performance + concurrency
and I don't know if I'm quite there yet
Hans W. Uhlig
@huhlig
Apr 29 2018 02:55
I usually write this stuff in java when I have to. Rust should have very good performance as long as the redis client driver is performant
I assume you're reusing your client and pooling connections?
Brandon Ros
@brandonros
Apr 29 2018 02:57
I was able to get like 80k TPS on 4 cores with one connection in node.js
Hans W. Uhlig
@huhlig
Apr 29 2018 02:57
was redis on the same machine?
Brandon Ros
@brandonros
Apr 29 2018 02:57
Yes, which is pretty stupid, I know lol
Hans W. Uhlig
@huhlig
Apr 29 2018 02:57
no, it's important for the benchmark
otherwise I have to ask you a bunch of network questions ;)
Brandon Ros
@brandonros
Apr 29 2018 02:58
well, the processes start to fight for CPU, in my eyes
Hans W. Uhlig
@huhlig
Apr 29 2018 02:58
I assume you assigned both your app and redis ample memory
cpu is unlikely your limiting factor
io is much more likely the case
Brandon Ros
@brandonros
Apr 29 2018 02:59
ah, redis persistence?
I tried to code it to a point where CPU would be the limiting factor, for what it's worth
I tried to make sure CPU usage was near 100% for each process
Hans W. Uhlig
@huhlig
Apr 29 2018 02:59
have you looked at it in a profiler?
see where your hotspots are?
Brandon Ros
@brandonros
Apr 29 2018 03:00
Well... I was happy with the 80k TPS in node.js, so I used it as the standard for the benchmark, and thought Golang would beat it out of the water based on everything I saw online
But, the fact that I couldn't even get it close out of the box was kind of discouraging
Hans W. Uhlig
@huhlig
Apr 29 2018 03:02
go lang from my experiance requires some... special consideration around threading
the whole no main thread everything is green makes more traditional work models more challenging
it also makes cache thrashing a problem occasionally
since YOU don't have control over the thread model without doing some out of the norm things
What problem were you specifically seeing in go?
Brandon Ros
@brandonros
Apr 29 2018 03:04
Yeah, I was bruteforcing settings into a black box with
how many workers should I do
how large should the channel buffer be
should I spawn multiple instances (because GOMAXPROCS was not doing itself on its own)
Hans W. Uhlig
@huhlig
Apr 29 2018 03:05
usually, with properly designed worker threads on a hyperthread cpu you want 2 per core
double or quadruple it if you have io latency
if it's all compute bound work, stick to two per core
Brandon Ros
@brandonros
Apr 29 2018 03:07
do you mean, spawn 8 processes on a 4 core hyperthreaded CPU manually
Hans W. Uhlig
@huhlig
Apr 29 2018 03:07
no
Brandon Ros
@brandonros
Apr 29 2018 03:07
or set GOMAXPROCS, because top was only showing 1 instance running
despite what i set
Hans W. Uhlig
@huhlig
Apr 29 2018 03:08
are you using fork or threads
Brandon Ros
@brandonros
Apr 29 2018 03:08
neither, is that the issue?
Hans W. Uhlig
@huhlig
Apr 29 2018 03:08
you're using goroutines right?
Brandon Ros
@brandonros
Apr 29 2018 03:08
yes
Hans W. Uhlig
@huhlig
Apr 29 2018 03:09
those are green threads then
are you using a connection pool?
Brandon Ros
@brandonros
Apr 29 2018 03:09
no because i am trying to recreate the 1 connection performance from node.js
I tried like 100 different things with go and it was 4am so I think I ended on just a single for{} loop being the fastest... is that bad?
Hans W. Uhlig
@huhlig
Apr 29 2018 03:12
hmm... lemmie look
im not a go programmer normally so this might take me half a sec
Brandon Ros
@brandonros
Apr 29 2018 03:14
https://github.com/sebasmagri/rust-concurrency-patterns/blob/master/examples/05-workers.rs whenever you have a second, is that the rust "concurrency pattern" you'd implement first for this kind of task?
Hans W. Uhlig
@huhlig
Apr 29 2018 03:16
basically
you want a bunch of workers in an infinite loop, waiting for data to be available
Brandon Ros
@brandonros
Apr 29 2018 03:17
yeah, from what i understand in that rust code
it makes 5 workers... is that 5 real underlying OS threads?
like pthreads/
Hans W. Uhlig
@huhlig
Apr 29 2018 03:17
the one thing about how that's written that's different is they have a single threaded local producer
yeah basically
Brandon Ros
@brandonros
Apr 29 2018 03:18
yeah i'm removing the producer part of the equation out of this benchmark just for simplicityh
Hans W. Uhlig
@huhlig
Apr 29 2018 03:18
are you familier with what a thread is at a low level and the difference between threads and green threads?
Brandon Ros
@brandonros
Apr 29 2018 03:18
i am
green threads = event loop/fake
threads = real CPU processes managed by OS scheduler
is that about right?
Hans W. Uhlig
@huhlig
Apr 29 2018 03:18
yep... basically you don't reallocate a new stack
Brandon Ros
@brandonros
Apr 29 2018 03:19
but then now it is up to your app to schedule + fight for CPU timing/resources?
Hans W. Uhlig
@huhlig
Apr 29 2018 03:19
it always was
Brandon Ros
@brandonros
Apr 29 2018 03:19
well... compared to the kernel doing it
kernel/OS
Hans W. Uhlig
@huhlig
Apr 29 2018 03:19
your OS reschedules your app weather you want it to or not
Ben Kimock
@saethlin
Apr 29 2018 03:19
I think goroutines are stackful
Brandon Ros
@brandonros
Apr 29 2018 03:19
they are. they just lowered it to like 2048 bytes or something
Hans W. Uhlig
@huhlig
Apr 29 2018 03:19
goroutines are just a thread with a forloop under your function
so it picks a function off the queue and runs it
I believe it will also juggle these and insert internal routine breaks
not sure though, I havent studied the go compiler a ton
all threads are stackful
Brandon Ros
@brandonros
Apr 29 2018 03:21
what was the direction java went
Hans W. Uhlig
@huhlig
Apr 29 2018 03:21
green threads just don't deallocate the stack space when their child function ends
Brandon Ros
@brandonros
Apr 29 2018 03:21
had green threads, and left them?
Hans W. Uhlig
@huhlig
Apr 29 2018 03:22
nah, you've got forkjoin which is a form of green threads
as are executors
Brandon Ros
@brandonros
Apr 29 2018 03:22
goroutines are a "fake" thread in the sense that it will not show up under ps/top/OS level metrics are real threads
correct?
Hans W. Uhlig
@huhlig
Apr 29 2018 03:22
the threads stay allocated but idle until they have work assigned

On Java you can run 1000’s or tens of 1000’s threads. On Go you can run hundreds of thousands or millions of goroutines.

Java threads map directly to OS threads, and are relatively heavyweight. Part of the reason they are heavyweight is their rather large fixed stack size. This caps the number of them you can run in a single VM due to the increasing memory overhead.

Go OTOH has a segmented stack that grows as needed. They are “Green threads”, which means the Go runtime does the scheduling, not the OS. The runtime multiplexes the goroutines onto real OS threads, the number of which is controlled by GOMAXPROCS. Typically you’ll want to set this to the number of cores on your system, to maximize potential parellelism.

Brandon Ros
@brandonros
Apr 29 2018 03:23
The runtime multiplexes the goroutines onto real OS threads, the number of which is controlled by GOMAXPROCS.
well I'm dying to know why it didn't see my CPU starving process
and spawn some up lol, despite the value being set
Hans W. Uhlig
@huhlig
Apr 29 2018 03:24
yeah
apparently this control isn't deterministic
Brandon Ros
@brandonros
Apr 29 2018 03:29
if you had to gamble/guess
Hans W. Uhlig
@huhlig
Apr 29 2018 03:29
its possible the go redis client is badly optimized
Brandon Ros
@brandonros
Apr 29 2018 03:29
would you think rust would beat go in a benchmark like this?
(i tried two different ones just to be sure, but you could be right)
Hans W. Uhlig
@huhlig
Apr 29 2018 03:29
honestly the language is likely going to be irrellevant, it's going to be the quality of the redis client code
Id say give it a try, it's not that much code
also make sure you're not transcoding video or using your anti-virus in the background
it can skew your results
Brandon Ros
@brandonros
Apr 29 2018 03:31
so I'd agree that it's going to be the quality of the library, however
it should boil down to how well the language handles things like... sending/receiving network requests
serializing/deserializing strings/structures... kind of...
i guess not really structures, just strings?
Hans W. Uhlig
@huhlig
Apr 29 2018 03:32
eh, except the language isn't responsible for handling networking
your OS is
the language just uses existing system calls
rust didn't reimplement your entire network stack
it just layers a thin vaneer on top of it
even Java didn't reimplement it
Do Duy
@juchiast
Apr 29 2018 11:04
Does anyone has issue with color in rustc stable?
https://i.imgur.com/Q7Dhvx3.png
nighly is fine, but stable has no colors
Denis Lisov
@tanriol
Apr 29 2018 11:15
@juchiast Linux?
Do Duy
@juchiast
Apr 29 2018 11:16
yes, Fedora 28
no color in gnome-terminal and terminator
xterm is fine
Denis Lisov
@tanriol
Apr 29 2018 11:19
Rust 1.25 used to use (rust-lang/rust#48588) an old crate for terminal handling which does not work with the new terminfo format used by ncurses 6.1
The fix is in 1.26, so I'd just switch to beta for the two weeks till it becomes stable.
Do Duy
@juchiast
Apr 29 2018 11:20
thanks!!
Andy Grove
@andygrove
Apr 29 2018 14:59
Hi. I'm working on a refactor of the Rust version of Apache Arrow to use traits and generics instead of enums. It has been going well so far but now I'm struggling with downcasting to get a specific type of array.
here are the key structs/traits
pub trait ArrayData : ArrowType {
    fn len(&self) -> usize;
}

pub struct Array {
    data: Box<ArrayData>,
    validity_bitmap: Option<Bitmap>
}
so at runime, I have meta data and already know what type of ArrayData I have (there are a few different impls)
I saw a forum posting suggesting adding an as_any method on the trait and then using downcasting so I tried that
impl<T> ArrayData for BufferArrayData<T> where T: ArrowPrimitiveType {
    fn as_any(&self) -> &Any {
        self
    }
}
but I get the error the parameter typeTmay not live long enough
Hans W. Uhlig
@huhlig
Apr 29 2018 15:02
you forgot lifetime
Andy Grove
@andygrove
Apr 29 2018 15:04
so I should try this?
pub trait ArrayData<'a> : ArrowType {
    fn len(&self) -> usize;
    fn as_any(&'a self) -> &'a Any;
}
Hans W. Uhlig
@huhlig
Apr 29 2018 15:04
I want to say yes but im not sure
I've learn to recognise but not entirely fix lifetime errors
but yeah, basically you need to establish the lifetime of T within the system
Here's a simplified example to show the problem I am trying to solve
will also ask over on Rust IRC
This is my attempt at using Any, but I don't know how to solve the issue with the lifetime of the generic type T
Hans W. Uhlig
@huhlig
Apr 29 2018 15:38
one sec, let me give it a try quick
part of the problem is that you don't know how long ArrayData lives
so you have a dangling reference to it
Andy Grove
@andygrove
Apr 29 2018 15:44
someone on Rust IRC showed me how to solve this, one sec ... I will share here
the trick is to declare T with a static lifetime ... this is the lifetime of the types, not the values
Hans W. Uhlig
@huhlig
Apr 29 2018 15:46
yeah
I was thinking that but im not yet an expert on how to handle it
is arrow written in rust?
Andy Grove
@andygrove
Apr 29 2018 15:49
there are multiple implementations (Rust, Go, C, Javascript, Java, Ruby ...)
all interoperable
Hans W. Uhlig
@huhlig
Apr 29 2018 15:49
ahh, interesting
Andy Grove
@andygrove
Apr 29 2018 15:49
so you can share columnar data with zero-copy
Hans W. Uhlig
@huhlig
Apr 29 2018 15:49
hopefully immutable memory
heh
Andy Grove
@andygrove
Apr 29 2018 15:49
yes
I'm using it in https://datafusion.rs/
Hans W. Uhlig
@huhlig
Apr 29 2018 15:51
wow... I was actually starting to work on something similar
you're a lot futher along than I am
I will ask, did you do batch or stream design?
basically spark or flink model
Kelly Thomas Kline
@kellytk
Apr 29 2018 20:53
@andygrove That's a neat looking project
Kelly Thomas Kline
@kellytk
Apr 29 2018 21:11
Does Redox OS have a Gitter channel? I'm not finding one