Hrmm. Could be memmap. Like I said it was the addition of clap that caused the release build to slow down. I have also confirmed that it's only the initial build.
Because it's only the initial build - I'm not to worried about it. Because the slowdown appears when my program is compiling and not clap - I doubt it's clap specifically and probably more of an interaction between some of the deps and my program.