These are chat archives for rust-lang/rust

17th
Jul 2017
Lance Carlson
@lancecarlson
Jul 17 2017 02:05
I'm having issues CSV parsing a file that has invalid UTF-8 characters in it. What is the best way to do go about doing the equivalent of iconv -f utf-8 -t utf-8 -c in rust? Should I read the original file and write to a temp file while parsing the invalid UTF-8 characters out? I was trying to do this and got as far as writing to temp file, but had issues with the line by line parsing.
stevensonmt
@stevensonmt
Jul 17 2017 03:41
Are you using the rust-csv crate?
Unless you're using Serde, the standard CSV record types are
StringRecord
and
ByteRecord.
StringRecord should be used when you know your data to be valid UTF-8.
For data that may be invalid UTF-8, ByteRecord is suitable.
Lance Carlson
@lancecarlson
Jul 17 2017 03:49
@stevensonmt yes. I was about to get it to not parse certain columns by specifying u8's and all the ones I didn't need. The problem was that the struct ended up being gigantic because the csv file has like 100 fields
This struct alone slowed my compile time down to 43s. Before that it was at the slowest 5s
I figure it's better to deal with the utf on the entire file first and pre process it, then deserialization through csv parser with a much smaller struct of only the fields I need
RuRu92
@RuRu92
Jul 17 2017 06:34
@vberset That explains it and it works! thanks for the help :)
lemonxah
@lemonxah
Jul 17 2017 07:57
good morning
just a quick one i have a function fn check_state(state: State) -> &'static str { ... } and it qorks great while just returning static strings but what i f want to return a string that i have formated first
i know you can use &format!(..) but then that String does not live long enough
Denis Lisov
@tanriol
Jul 17 2017 08:01
Then make your function return a String, not a &'static str
lemonxah
@lemonxah
Jul 17 2017 08:05
i am very new to rust :) as you can no doubt see :) when making the function return a String does that then move ownership to the calling scope?
Max Frai
@max-frai
Jul 17 2017 08:08
@lemonxah show the code of check_state and what you return
lemonxah
@lemonxah
Jul 17 2017 08:09
fn check_state(state: OrderState) -> String {
    match state {
        OrderState::New => "New Order".to_string(),
        OrderState::PaymentPending {system, reason} => format!("Payment pending for system: {}, reason: {}", system, reason),
        _ => "I dont care for these".to_string(),
    }
}
its a very stupid fn i am just playing around to start learning it
Denis Lisov
@tanriol
Jul 17 2017 08:10
Exactly, when you return a String, you pass the ownership to your caller and it's now their responsibility.
Max Frai
@max-frai
Jul 17 2017 08:10
@lemonxah there is also to_owned method and it better describes what you do with string :)
"New Order".to_owned() will do the same thing
But it describes the exact action you are doing
Denis Lisov
@tanriol
Jul 17 2017 08:11
If you frequently return static strings, you can return a Cow, but for now it would be premature :-)
lemonxah
@lemonxah
Jul 17 2017 08:12
i doubt it would be static stings like i said just playing around with the langauge
never really touched low level languages before
played a bit with c and c++ but i did high level, C# for 10 years and then scala for 5
dont want to go back to C# and the new company it would take too long to get them up to speed with scala so we got a middle ground around rust :)
Max Frai
@max-frai
Jul 17 2017 08:13
Nice choice :)
lemonxah
@lemonxah
Jul 17 2017 08:15
i just have to get all the things under control first its going to be a long road teaching an old dog new tricks
lemonxah
@lemonxah
Jul 17 2017 10:28
which rest frameworks do you guys prefer?
Max Frai
@max-frai
Jul 17 2017 10:47
@lemonxah What do you need as a result?
lemonxah
@lemonxah
Jul 17 2017 10:47
restfull api
building microservices
Max Frai
@max-frai
Jul 17 2017 10:47
Have you looked at https://rocket.rs/?
lemonxah
@lemonxah
Jul 17 2017 10:51
i have not looked at any yet i will be looking at that now thank you
rocket looks pretty cool
Max Frai
@max-frai
Jul 17 2017 10:54
@lemonxah yes, it has pretty api/interface, the only problem it requires nightly compiler
lemonxah
@lemonxah
Jul 17 2017 11:01
i dont mind the nightly for now
Lance Carlson
@lancecarlson
Jul 17 2017 13:16
Is there a max number of fields on a struct before your compile times get crazy long? I'm running into an issue with a large struct
stevensonmt
@stevensonmt
Jul 17 2017 13:43
@lancecarlson what does your code and source data actually look like?
stevensonmt
@stevensonmt
Jul 17 2017 14:10
To keep your struct to only the fields you are interested in, you might be able to make use of the xsv crate's index and select commands.
Lance Carlson
@lancecarlson
Jul 17 2017 14:50
@stevensonmt I was trying to avoid calling out to cli's
@stevensonmt if I went down that route, I could just use iconv -f utf-8 -t utf-8 -c <file>
that command strips invalid utf characters
stevensonmt
@stevensonmt
Jul 17 2017 16:22
okay, hopefully someone else will be along who can offer more help. I do think it might help if you could post an example of the source file and your current code though.
Lance Carlson
@lancecarlson
Jul 17 2017 16:22
@stevensonmt I think I'm going to go in the spawning of iconv subcommand route
avoid having a large struct :)
stevensonmt
@stevensonmt
Jul 17 2017 16:34
Have you looked at the xsv source code for the select command? If you know which columns you are going to be interested in you might be able to modify that to return something other than CliResult and keep everything in your app.