@TobiasKnauss
- JPG is Lossy, FLIF is Lossless. You can apply a lossy transformation in your conversion to reduce file size, the output will be stored in a lossless FLIF format but will have been degraded in quality from the original. In the CLI you would use
-Q70to set quality to ~70% of original.-Q100is the default and is lossless, all other settings are lossy. - I cannot find any reference in the FLIF spec for a maximum height/width dimension. I think I recall someone saying there is no limit on dimensions. If that is the case then you like just found a bug and should report it with a means to reproduce it.
- Perhaps @jonsneyers or @sipa or @hrj will know more about this, I'm unfamiliar with it
- The current FLIF source code is one implementation of the FLIF spec. It has not been optimized at all. So it will be very slow in comparison to encoders for more mature specs that have existed for decades and had time to be heavliy optimized (GIF/PNG/JPG/Etc)
@TheJaredWilcurt
Thank you very much for your reply!
Regarding 1), I created the FLIF directly from the JPG. I just doublechecked on flif.info, it compares the compression ratio of FLIF to other lossless formats, but not to JPG. So I think I should not do that either, since it would be comparing apples with oranges, isn't it?
Regarding 2), I have tested the same file again, this time with "FLIF-GUI 0.3" Filemill.exe. It worked. Now I'm testing again with my own code, it's running now. Don't know why. The exception came directly at the call of the function, not during execution, so I think it will be fine now. It will just take some time: The test with a 4096x4096 took 2:16 min, so I expect the 10000x10000 to take about 10 min or more. The difficulty on the 4096^2 file is, that every pixel is different (it contains all 24bit colors). The 10000^2 file is similar, it contains at least some identical pixels and not all of the 24bit colors.
Thank you very much for your reply!
Regarding 1), I created the FLIF directly from the JPG. I just doublechecked on flif.info, it compares the compression ratio of FLIF to other lossless formats, but not to JPG. So I think I should not do that either, since it would be comparing apples with oranges, isn't it?
Regarding 2), I have tested the same file again, this time with "FLIF-GUI 0.3" Filemill.exe. It worked. Now I'm testing again with my own code, it's running now. Don't know why. The exception came directly at the call of the function, not during execution, so I think it will be fine now. It will just take some time: The test with a 4096x4096 took 2:16 min, so I expect the 10000x10000 to take about 10 min or more. The difficulty on the 4096^2 file is, that every pixel is different (it contains all 24bit colors). The 10000^2 file is similar, it contains at least some identical pixels and not all of the 24bit colors.
@TobiasKnauss There is a flif.exe file (along with all required Windows files for it to run on any Windows OS) here:
https://github.com/FLIF-hub/node-flif/tree/master/executables/win32
So you can run a pre-built version of the
https://github.com/FLIF-hub/node-flif/tree/master/executables/win32
So you can run a pre-built version of the
flif CLI, in case you think your build isn't working correctly
@TheJaredWilcurt I am using the DLL, not the EXE, because I want to build a C# interface around it. But that's a good idea for a doublecheck. Thanks!
The encoding of the 10000^2 file has finished, it took about 11 min. Worked as it is expected to.
Now waiting for replies of other users on the issue 3 with flif_import_image_RGBA().
Regarding 4), I will mostly encode 1280x960x8bpp images from industrial GigE Cameras. It takes 5 seconds for one image (slooow...), so I'll have to use one CPU thread for every image, because I usually get 4 images (or more) every 15 seconds.
The encoding of the 10000^2 file has finished, it took about 11 min. Worked as it is expected to.
Now waiting for replies of other users on the issue 3 with flif_import_image_RGBA().
Regarding 4), I will mostly encode 1280x960x8bpp images from industrial GigE Cameras. It takes 5 seconds for one image (slooow...), so I'll have to use one CPU thread for every image, because I usually get 4 images (or more) every 15 seconds.
@TheJaredWilcurt It's not like taking photos of a landscape. In industrial applications, the main purposes are precision, repeatability and reproducability. The photos are used for detecting geometric objects, so when I have to repeat the operation for failure analysis, I need to make sure that I get the same results, so I have to store the photos without loss of details.
FLIF is slow partially because the implementation is not optimized (the encoder in particular) and partially because it's inherently slow. JPEG XL will be faster, at least for lossless. For lossy in FUIF mode it's reasonably fast, for lossy in Pik mode it's currently quite slow because of the iterations of perceptual optimization.
oh!
ignore what i said then
i assumed jonsneyers was working on FUIF instead
JPEG XL = FUIF + PIK + improvements made in the last 10 months or so
@jonsneyers Are you still working on basic things or are you already optimizing performance? I'm asking because I'd like to know whether you could already give some outlook on the performance compared to FLIF.
FUIF = FLIF + Squeeze transform + quantization + MANIAC tweaks - weird scan order that allows you to predict from many neighbors but is really bad for memory locality hence speed - ColorBucket transform. For lossless it's roughly equivalent to flif -N, for progressive and lossy it's better than FLIF, and it's also faster.
We are currently polishing the committee draft, which will go to ballot to the national bodies of ISO (ANSI etc) on August 5th. If all goes well we have a draft international standard in November and a published standard in April 2020.
So it's pretty stable but we can still have bitstream breaking changes pretty much until November.
:-/
2 questions:
2 questions:
- For my tests I mainly copied the code from test.c, including all encoder settings. From flif.exe command line help I see that -N means "no interlacing". How can I achieve that with flif_encoder_set_interlaced() ? This function currently takes "1", so I thought it was a numeric level, but based on the command line args -N and -I, it looks like its a flag, so should I use "0"?
- What are your plans about optimizing encoder performance? Decoding actually is quite unimportant for me, because it will be done only occasionally, but encoding will be done frequently.
Overall this sounds like I should expect a working-for-me solution by mid to end of 2020.
to speed flif encoding up, you can do fewer MANIAC tree learning iterations, setting -R 1 or if you want something quick -R 0 (and then later do a transcode with -R 2)
in fuif / jpeg xl you have more fine-grained control over encode speed, in flif you need an integer number of learning iterations and the default is -R 2, in fuif you can do a fractional number of learning iterations and the default is basically -R 0.5
I just got the AccessViolationException again in flif_encoder_encode_file() when encoding the 10000x10000x32bpp PNG. The RAM usage is about 900MB in the moment of the exception, but it's 1400MB if no exception occurs. You may want to check whether the memory allocation is done correctly on large files; maybe not on the FLIF encoder since you stopped working on it, but on the FUIF encoder.
well, I should stop mentioning a file type. I load every file into a .Net Bitmap object, so the original file type is pretty irrelevant.
@jonsneyers Yes, on Windows it's BGRA (https://stackoverflow.com/a/8104553). Is it different on other systems? Is there a way to make FLIF "compatible" to the Windows byte order? This would speed up the copying of image data a lot prior to the actual encoding.
@TheJaredWilcurt : I compiled the flif library using the "dl_make_vs.bat"
@sipa "the overhead of calling an exe" should be some tens of milliseconds, which is nothing even on my medium-sized images that take 5 seconds. It would only be relevant at encoding of tiny PNG images that take about 100ms.
@TheJaredWilcurt : I compiled the flif library using the "dl_make_vs.bat"
@sipa "the overhead of calling an exe" should be some tens of milliseconds, which is nothing even on my medium-sized images that take 5 seconds. It would only be relevant at encoding of tiny PNG images that take about 100ms.
Tests with different values of interlace and learn-repeat finished. Results on various kinds of image data are too long to post it here, but I can send it to anybody who's interested.
Summary for the 1280x960x8bpp:
Summary for the 1280x960x8bpp:
image.bmp 1280x960xFormat8bppIndexed : image=1228800 bytes, sourcefile=1229878 bytes, sourcefile/image=100,087728%
Interlaced = 1, LearnRepeat = 3, encoding: 00:00:04.4502546, file=430089 bytes, file/image=35,000732% file/sourcefile=34,970%
Interlaced = 0, LearnRepeat = 3, encoding: 00:00:04.2122409, file=433344 bytes, file/image=35,265625% file/sourcefile=35,235%
Interlaced = 0, LearnRepeat = 1, encoding: 00:00:01.5380880, file=435377 bytes, file/image=35,431071% file/sourcefile=35,400%
I'm having some trouble getting good results out of FLIF. I have a set of BMP images at 128x128, so every image is 12kb. If I convert these images to WEBP they all compress and have filesizes lower than 12kb. But if I convert them to FLIF they are all larger than 12kb. On bigger files FLIF always seems to beat WEBP but on this imageset FLIF is doing worse than the uncompressed BMP files.
with my WEBP conversion I use -z 9 (highest lossy compression level) and with FLIF I use -E=100