ChibiOS Free Embedded RTOS

Hi all,

I was asked today about implementing zip file compression on a micro, most likely using fatfs on top of an SDCard. Just wondering if anyone has blazed this trail before and could offer any advice on existing software (preferably free licensed, but commercial licenses wouldn't necessarily be a deal-breaker). Or more importantly, what software should be avoided.

Yes, I am aware that there are tons of other compression libraries, but I think for interoperability, it must be zip.

My initial glance was at the free version that is the default on just about any Linux box http://infozip.sourceforge.net/. It seems to have a liberal license, has been used by millions of people, and isn't a hugely daunting codebase. However, I haven't looked deep enough to see if it requires the faculties of larger OSs. Nor have I looked into it's memory requirements.

Any pointers or advice is greatly appreciated, if you have any relevant experiences.

Thanks,
Dave

Hi,

Have you looked into 7z license? the code you need is probably in there.

Giovanni

Hi,

Just currious, what kind of application ?

Just my 2 cents

What is the real need of a zip file? Compression to limit size ? Compatibility with other software ?

If I am not wrong, FATFS in FAT32 could handle up to 2TB disk size, and max 4GB file size. Unless sd card price is important, is it relevant to make compression at microcontroller level ?

On my desk PC (WIN7, i5-6500, flash disk) , it compress a 553MBytes PDF file in 25 seconds, about 22MBytes/s, with 7-zip. The limitation is probably the disk IO speed.
What numbers on a STM32 at 168Mhz, "some" MHz on SPI for SD card transfers ?

Angelo

The zlib library, which seems to be the basis of a lot of the compression algorithms, is freely available, and doesn't seem too difficult to use.

The big problem on a micro is the requirement for a minimum 64K buffer for compression, and 32K for decompression, which is a significant demand even on the bigger micros if there's no external memory.

Thanks guys for the helpful insight:

@Giovanni:

Yep, I did look briefly at p7zip, but it seems it is LGPL which will not be compatible with static linkage. Also, it is written predominantly as C++ code which I don't think would fly for this project.

@Polux
Yes, this is a case of the end-user reqs don't necessarily match any technical reasoning. It is a situation of "this is how we expect the data". I think it is essentially being used as a least common denominator (all windows since XP can open ZIP natively), as an archiver (funnier, the file contents are already compressed). Sometimes we're asked to do silly things.

@steved
Whoa, I never actually looked under the zlib hood. I always assumed it was its own thing, but it is another implementation of DEFLATE (in fact written by two of the guys who did heavy lifting on the InfoZip (de)compressors). They even have an example 'minizip' application included as an example of creating PKZIP compatible archives.

Thanks very much for shining the light here. I agree that this might not be embeddable in some cases, but for this project I think 64k isn't that tall an order (the system does have a large-ish external SRAM).


I really appreciate the discussion folks!

dflogeras wrote:Thanks very much for shining the light here. I agree that this might not be embeddable in some cases, but for this project I think 64k isn't that tall an order (the system does have a large-ish external SRAM).

Be interested to hear how you implement this in the end; you've probably gathered that I'd researched enough to decide a likely way to go, but not actually implemented anything. I only need decompression, so 32K RAM (plus some overhead) might be manageable on a 32F767.

Do you need zip or something to decompress? LZ4?

https://community.arm.com/developer/ip- ... -and-later

wurstnase wrote:Do you need zip or something to decompress? LZ4?

https://community.arm.com/developer/ip- ... -and-later

Thanks Nico; that looks interesting, since I don't need a 'public' compression format; nor is the compression level critical. Must have looked past it; this is a useful benchmark test.
Memory requirements are certainly configurable for compression, and look to be for decompression - constrained memory environments are covered in the documentation (including sharing a buffer for input and output). (Although one comment suggests needing a 64K buffer for decompression, but that might just be for a specific function).
There's also a useful commentary here

Incidentally, re-reading my notes it looks as if DEFLATE might need 128K buffer for compression.