Where communities thrive


  • Join over 1.5M+ people
  • Join over 100K+ communities
  • Free without limits
  • Create your own community
People
Repo info
Activity
  • Aug 26 22:57
    nicolas-comerci commented #140
  • Aug 24 07:04
    M-Gonzalo commented #140
  • Aug 23 20:56
    M-Gonzalo opened #141
  • Aug 23 20:28
    M-Gonzalo commented #140
  • Aug 23 15:14
    nicolas-comerci commented #140
  • Aug 23 14:09
    M-Gonzalo commented #140
  • Aug 21 21:14
    nicolas-comerci edited #140
  • Aug 21 21:13
    nicolas-comerci edited #140
  • Aug 21 21:11
    nicolas-comerci edited #140
  • Aug 21 09:35
    nicolas-comerci synchronize #140
  • Aug 20 11:48
    schnaader labeled #140
  • Aug 20 11:48
    schnaader assigned #140
  • Aug 20 11:47
    schnaader review_requested #140
  • Aug 20 09:06
    nicolas-comerci edited #140
  • Aug 20 09:06
    nicolas-comerci edited #140
  • Aug 20 09:06
    nicolas-comerci edited #140
  • Aug 20 09:03
    nicolas-comerci synchronize #140
  • Aug 19 22:32
    nicolas-comerci opened #140
  • Aug 14 08:38
    SophrosyneX edited #139
  • Aug 14 08:05
    SophrosyneX opened #139
Abhishek Sharma
@RamiroCruzo
As for Brute being for small files, at the time of building they were supposed to be run on small files, but whats the problem in speeding them up now ;)
Christian Schneider
@schnaader
Well, even after speeding up, it will be very slow and adding file format support for files that contain headerless deflate streams is much better than using brute mode which is slow and has many false positives that decrease compression ratio.
Abhishek Sharma
@RamiroCruzo
Well its true in a way too
Why don't ya instead of trying to decompress deflate at every position try to decode Huffman?
Its faster than deflate approach
Christian Schneider
@schnaader
At the end, this is the same thing that zLib does here if the stream is invalid - decoding huffman and returning an error code. Removing the zLib wrapper overhead shouldn't speed it up that much.
Abhishek Sharma
@RamiroCruzo
Hmmm Okie then :D Well, was only collecting even the smallest speed gain we can get
On top of that, should we focussing on MT too?
Christian Schneider
@schnaader
I haven't tested it, though, so it actually might help a bit. But my guess would be like 5-10% faster whereas removing temporary files will make things 5-10 times faster.
Yeah, MT will dominate 0.4.7 work, can be used both in compression and restoring and is easy achievable Monear speedup.
*linear
Abhishek Sharma
@RamiroCruzo
After MT, those 5-10% would be great XD
Also, even after temps, you should reduce disk activity, Precomp seeks forward and backward way many times, leading to slower access
Christian Schneider
@schnaader
Yes, especially the input file buffering could be done better.
Abhishek Sharma
@RamiroCruzo
Yep :D
Abhishek Sharma
@RamiroCruzo
Dead room eh...
Abhishek Sharma
@RamiroCruzo
There sir? Got a few interesting results with Precomp
Christian Schneider
@schnaader
Yes, I'm online.
Christian Schneider
@schnaader
Finally merged with the lzma2 branch. The new multithreaded lzma2 compression method is now used by default.
Abhishek Sharma
@RamiroCruzo
Nicee :D
Prince Gupta
@guptaprince

File: File:html book 20161029.zip

Precomp(schnaader/precomp-cpp@9ddc049): -intense0 -cn-
Time: 33 minute(s), 15 second(s)
Size: 162,947,142 bytes

PZLIB V3 (Hotfix): -m2 -x -s -b128m -t1
Time: 2.8745 minutes
Size: 170,417,418 bytes

Reflate(v1l1): c
Time: 15.781 seconds
size: 170,417,418 bytes

Don't you think precomp is way too slow?
Is it because very small zlib streams inside or any thing else?

Christian Schneider
@schnaader
Intense mode is very slow because of the temporary files that are created even if they aren't used, next version will fix that. After that, it should be comparable to the pzlib result. Note that reflate will still be faster because it only recompresses once, something that will be adressed by difflate.
Abhishek Sharma
@RamiroCruzo
Yahallo sir, long time no see, finally difflate is coming?
Prince Gupta
@guptaprince
what should be the blueprint to remove temps
Prince Gupta
@guptaprince
i m thinking of taking the lazy approach, fmemopen for Linux and for the windows using CreateFileMap and Then use_fdopen
for FILE*
for buffers have to write half of the precomp again i think
Christian Schneider
@schnaader
Look at the paq variants with zlib routines, they are what I'll use as blueprints. Basically, no need to keep the whole file in memory, only 64 KB portions since deflate uses 32 KB windows and recompression needs another 32 KB as “lookback“
Christian Schneider
@schnaader
E.g. have a look at paq8px_v75.cpp
Christian Schneider
@schnaader
The memory mapping way is possible for all the other streams but also has drawbacks - the stream has to fit into memory with lazy approaches which was a limit that lead me to using temporary files originally. Also, as you said, there is no portable mmap, so it's an ifdef and testing hell. But it's a way to go, e.g. I did similar with packJPG/MP3 (everything in memory for streams up to 64 MB, use temporary files else)
Things will get more complicated with stdin/stdout support which basically means "can't use fseek, have to use workarounds"
E.g. Issue #55
Abhishek Sharma
@RamiroCruzo
Instead of Creating a filemap, we can go around with the good old circular buffers, load a big affordable chunk into mem, that'll still allow us to use seek and will be one read and write only
jagannatharjun
@jagannatharjun
i was working on antiz and notice that it uses diff files, precomp uses them too?
Christian Schneider
@schnaader
Looking at the source, I guess you're talking about handling of partial matches and everything around the recomp-tresh/diffbytes variables.
Yes, Precomp has that, too. It calls bytes these "penalty bytes" and tolerates some of them in recompressed streams, e.g. see the function compare_file_mem_penalty.
Christian Schneider
@schnaader
I think both solutions aren't the best. I originally implemented them in Precomp because of PDF files which often have one differing byte in streams (Precomp stores them together with 4 byte offsets, so byte count is multiplied by 5 and it shows "Penalty bytes were used: 5 bytes"). See FlashMX.pdf (the ZIP is calles pdf-test.zip) from this site for example.
Christian Schneider
@schnaader
The best way to handle these differences would not be on byte level, but on bit level as Deflate uses bitstreams. To make things worse, Deflate packs those bitstreams to bytes in reverse, e.g. the bitstream "ABCDEFGH IJKLMNOP" (16 bits) is encoded as "HGFEDCBA PONMLKJI" so even if e.g. the part "HIJKLM" would differ, an approach that doesn't consider this would get two seperated diffs.
Christian Schneider
@schnaader
If you want to test something in this direction, have a look at Matt Mahoney's Silesia page. The file "silesia.zip" from there has a ZIP stream for each of the 12 files. I didn't test it with AntiZ yet, but Precomp processes 8 of the streams and most of them are "heavy" partial matches, like 1.4 MB of 6.0 MB match.
Using "precomp -cn" on silesia.zip gives 88 MB instead of 67 MB which doesn't look bad at first, but considering that uncompressing every stream completely leads to 211 MB, it is.
Ah, I have tested it with AntiZ before, see this post at encode.ru
jagannatharjun
@jagannatharjun
precomp stores full file hash of the original file?
jagannatharjun
@jagannatharjun
one more thing it looks like Antiz uses diff for every files rather then finding optimal zlib params, doesn't this is what we need for headerless aka deflate stream
jagannatharjun
@jagannatharjun
Even after no Temporary file precomp is twice fast then antiz :sparkles:
Christian Schneider
@schnaader
Precomp doesn't store a hash if -co is used, see schnaader/precomp-cpp#66
-cn, not -co
AntiZ has a slightly different approach on the params, uses some heuristics to make guesses about them. Also, IIRC, it doesn't remember parameters that were used before, which speeds up Pump for most files
Pump= Precomp damn autocorrect
jagannatharjun
@jagannatharjun
antiz tries to extract information from headers.. Precomp not?
Christian Schneider
@schnaader
Precomp does it too, but the headers are too small and unreliable. See RFC 1950: https://www.ietf.org/rfc/rfc1950.txt