Zip compression on a micro

ChibiOS public support forum for all topics not covered by a specific support forum.

Moderators: utzig, lbednarz, tfAteba, barthess, RoccoMarco

dflogeras
Posts: 190
Joined: Tue Sep 03, 2013 8:16 pm
Has thanked: 2 times
Been thanked: 13 times

Zip compression on a micro

Postby dflogeras » Wed Jan 22, 2020 11:50 pm

Hi all,

I was asked today about implementing zip file compression on a micro, most likely using fatfs on top of an SDCard. Just wondering if anyone has blazed this trail before and could offer any advice on existing software (preferably free licensed, but commercial licenses wouldn't necessarily be a deal-breaker). Or more importantly, what software should be avoided.

Yes, I am aware that there are tons of other compression libraries, but I think for interoperability, it must be zip.

My initial glance was at the free version that is the default on just about any Linux box http://infozip.sourceforge.net/. It seems to have a liberal license, has been used by millions of people, and isn't a hugely daunting codebase. However, I haven't looked deep enough to see if it requires the faculties of larger OSs. Nor have I looked into it's memory requirements.

Any pointers or advice is greatly appreciated, if you have any relevant experiences.

Thanks,
Dave

User avatar
Giovanni
Site Admin
Posts: 13088
Joined: Wed May 27, 2009 8:48 am
Location: Salerno, Italy
Has thanked: 759 times
Been thanked: 638 times
Contact:

Re: Zip compression on a micro

Postby Giovanni » Thu Jan 23, 2020 8:20 am

Hi,

Have you looked into 7z license? the code you need is probably in there.

Giovanni

Polux
Posts: 23
Joined: Thu Apr 28, 2016 11:52 am
Been thanked: 6 times

Re: Zip compression on a micro

Postby Polux » Thu Jan 23, 2020 8:31 am

Hi,

Just currious, what kind of application ?

Just my 2 cents :)

What is the real need of a zip file? Compression to limit size ? Compatibility with other software ?

If I am not wrong, FATFS in FAT32 could handle up to 2TB disk size, and max 4GB file size. Unless sd card price is important, is it relevant to make compression at microcontroller level ?

On my desk PC (WIN7, i5-6500, flash disk) , it compress a 553MBytes PDF file in 25 seconds, about 22MBytes/s, with 7-zip. The limitation is probably the disk IO speed.
What numbers on a STM32 at 168Mhz, "some" MHz on SPI for SD card transfers ?

Angelo

steved
Posts: 733
Joined: Fri Nov 09, 2012 2:22 pm
Has thanked: 10 times
Been thanked: 108 times

Re: Zip compression on a micro

Postby steved » Thu Jan 23, 2020 9:25 am

The zlib library, which seems to be the basis of a lot of the compression algorithms, is freely available, and doesn't seem too difficult to use.

The big problem on a micro is the requirement for a minimum 64K buffer for compression, and 32K for decompression, which is a significant demand even on the bigger micros if there's no external memory.

dflogeras
Posts: 190
Joined: Tue Sep 03, 2013 8:16 pm
Has thanked: 2 times
Been thanked: 13 times

Re: Zip compression on a micro

Postby dflogeras » Thu Jan 23, 2020 1:07 pm

Thanks guys for the helpful insight:

@Giovanni:

Yep, I did look briefly at p7zip, but it seems it is LGPL which will not be compatible with static linkage. Also, it is written predominantly as C++ code which I don't think would fly for this project.

@Polux
Yes, this is a case of the end-user reqs don't necessarily match any technical reasoning. It is a situation of "this is how we expect the data". I think it is essentially being used as a least common denominator (all windows since XP can open ZIP natively), as an archiver (funnier, the file contents are already compressed). Sometimes we're asked to do silly things.

@steved
Whoa, I never actually looked under the zlib hood. I always assumed it was its own thing, but it is another implementation of DEFLATE (in fact written by two of the guys who did heavy lifting on the InfoZip (de)compressors). They even have an example 'minizip' application included as an example of creating PKZIP compatible archives.

Thanks very much for shining the light here. I agree that this might not be embeddable in some cases, but for this project I think 64k isn't that tall an order (the system does have a large-ish external SRAM).


I really appreciate the discussion folks!

steved
Posts: 733
Joined: Fri Nov 09, 2012 2:22 pm
Has thanked: 10 times
Been thanked: 108 times

Re: Zip compression on a micro

Postby steved » Thu Jan 23, 2020 6:11 pm

dflogeras wrote:Thanks very much for shining the light here. I agree that this might not be embeddable in some cases, but for this project I think 64k isn't that tall an order (the system does have a large-ish external SRAM).

Be interested to hear how you implement this in the end; you've probably gathered that I'd researched enough to decide a likely way to go, but not actually implemented anything. I only need decompression, so 32K RAM (plus some overhead) might be manageable on a 32F767.

User avatar
wurstnase
Posts: 121
Joined: Tue Oct 17, 2017 2:24 pm
Has thanked: 43 times
Been thanked: 30 times
Contact:

Re: Zip compression on a micro

Postby wurstnase » Thu Jan 23, 2020 7:43 pm

Do you need zip or something to decompress? LZ4?

https://community.arm.com/developer/ip- ... -and-later
\o/ Nico

steved
Posts: 733
Joined: Fri Nov 09, 2012 2:22 pm
Has thanked: 10 times
Been thanked: 108 times

Re: Zip compression on a micro

Postby steved » Fri Jan 24, 2020 10:21 am

wurstnase wrote:Do you need zip or something to decompress? LZ4?

https://community.arm.com/developer/ip- ... -and-later

Thanks Nico; that looks interesting, since I don't need a 'public' compression format; nor is the compression level critical. Must have looked past it; this is a useful benchmark test.
Memory requirements are certainly configurable for compression, and look to be for decompression - constrained memory environments are covered in the documentation (including sharing a buffer for input and output). (Although one comment suggests needing a 64K buffer for decompression, but that might just be for a specific function).
There's also a useful commentary here

Incidentally, re-reading my notes it looks as if DEFLATE might need 128K buffer for compression.


Return to “General Support”

Who is online

Users browsing this forum: No registered users and 5 guests