At the moment I'm revising the recompression steps. Before, we had 4 steps (decompression -> recompression -> decompression -> recompression), the last 2 only used for partial matches. Looks like the last 2 can be removed, leading to a slightly worse compression ratio on these streams, but speeding up things a lot (for silesia.zip: 20% speedup, 0.2% worse compression). Next commit will address that, two temporary files less!
Sorry, I don't think I'll do such a switch, for 3 reasons: 1. Using the old code (with temporary files) would be a bad mix when everything is done without temporary files, 2. Adjusting the code to do the 2 stages in memory would be complex as all of the 4 stages are dependent, 3. When the new deflate handling ("difflate") is ready, there will be no more partial matches.
OK, another 15% speedup on silesia.zip (compression is exactly the same, nothing changed except removing the need for one of the temporary files). The next thing is to do zLib decompression in memory, after that issue #14 will be done and temporary files will only be used on GIF and some special files like multi-PNGs and JPG/MP3 files bigger than 64 MB.
Fixed two cases where the new versions with less temporary file usage had worse compression even for non-partial matches. Reason was Precomp thinking it had found the best compression/memory level combination although it hadn't. This was already a bug even before removing the 2 last recompression steps, but these steps prevented most of the impact, so the bug was hidden.
Quick status update: Just finished liblzma on-the-fly compression and decompression in sftt's branch, you can test it here: https://github.com/sftt/precomp-cpp/tree/schnaader - at the moment, you can use "-cl" (lzma), "-cx" (xz) and "-cm" (lzma2, multithreaded). I will try to clean up and merge soon. After that, the last temporary file will fall and 0.4.6 comes closer.
tested 3 times and everytime same result
here is the console output
https://drive.google.com/open?id=0BxQSCiRtuvAScEItaUowbjhBM3M
maybe helpful for u
https://drive.google.com/open?id=0BxQSCiRtuvAScEItaUowbjhBM3M
maybe helpful for u
The good news is, I recently found out that intense mode is slowed down massively by temporary files, so these two modes will be much faster when done.
As for Brute being for small files, at the time of building they were supposed to be run on small files, but whats the problem in speeding them up now ;)
Why don't ya instead of trying to decompress deflate at every position try to decode Huffman?
Its faster than deflate approach
On top of that, should we focussing on MT too?
Yeah, MT will dominate 0.4.7 work, can be used both in compression and restoring and is easy achievable Monear speedup.
*linear
Also, even after temps, you should reduce disk activity, Precomp seeks forward and backward way many times, leading to slower access
File: File:html book 20161029.zip
Precomp(schnaader/precomp-cpp@9ddc049): -intense0 -cn-
Time: 33 minute(s), 15 second(s)
Size: 162,947,142 bytes
PZLIB V3 (Hotfix): -m2 -x -s -b128m -t1
Time: 2.8745 minutes
Size: 170,417,418 bytes
Reflate(v1l1): c
Time: 15.781 seconds
size: 170,417,418 bytes
Don't you think precomp is way too slow?
Is it because very small zlib streams inside or any thing else?