LwIP pbuf/memp corruption Topic is solved

[iggarpe]

I think there is a bug in ChibiOS lwipthread / demos that causes pbuf /memp corruption.

First, low_level_input() allocates a pbuf from the pool (PBUF_POOL).
Second, the thread loop calls pbuf_free(), although admitedly it does only for packets of unknown type.

Neither pbuf_alloc() nor pbuf_free() can be safely called from other context than the LwIP worker thread unless in lwipopts.h SYS_LIGHTWEIGHT_PROT is set and/or (not sure about this) LWIP_ALLOW_MEM_FREE_FROM_OTHER_CONTEXT is also set, and none of these two options is set in lwipopts.h files in demos shipped with ChibiOS.

Either lwipopts.h needs to be modified or pbuf_alloc_callback() / pbuf_free_callback() must be used instead.



Slightly related I filed this bug some time ago:

https://sourceforge.net/p/chibios/bugs/729/

TL;DR lwipthread calls netif_add() which cannot be called from other context than LwIP worker thread. netconn_netif_add() must be used instead or netif_add() called from the callback that can be passed to tcpip_init().

[Giovanni]

Hi,

Moving in "bug reports".

Giovanni

[iggarpe]

Just reporting that after extensive stress testing LwIP with changes to lwipopts.h I can confirm that my occasional LwIP pbuf assertion triggers are gone. No more pbufs with invalid ref count, no more memp sanity checks panic.

[Giovanni]

Thanks, I will make changes, do you have a patch ready?

Giovanni

[iggarpe]

Thanks, I will make changes, do you have a patch ready?

Nope, sorry. Change is trivial, just enable SYS_LIGHTWEIGHT_PROT in lwipopts.h

However first I was hoping for someone more familiary with LwIP mutual exclusion intricacies to clarify whether enabling LWIP_ALLOW_MEM_FREE_FROM_OTHER_CONTEXT is also necessary.

I've enabled both and pbuf/memp corruption is gone for good, but havent (and can't afford at the moment) to do extensive testing with just SYS_LIGHTWEIGHT_PROT enabled. The point is that according to LwIP documentation enabling LWIP_ALLOW_MEM_FREE_FROM_OTHER_CONTEXT can increase interrupt latency significantly, so it is certainly worth to find out whether it is necessary.

Also, not that I provided an alternative fix: use pbuf_alloc_callback() and pbuf_free_callback() instead of pbuf_alloc() / pbuf_free(). That would work for sure and not require any changes to lwipopts.h, but again, I would appreciate if some LwIP expert could chip in and provide advice on which is the best fix.

[iggarpe]

No clues about this?

Please note that it is out of the question whether this is a bug or not. ChibiOS demos, as shipped, will cause pbuf corruption due to pbuf alloc/free being done from the LwIP data pump thread (lwipthread.c) with SYS_LIGHTWEIGHT_PROT disabled.

We need someone knowledgeable about the LwIP internals to tell whether SYS_LIGHTWEIGHT_PROT enabled will suffice or LWIP_ALLOW_MEM_FREE_FROM_OTHER_CONTEXT enabled is also necessary.

My guess is that just SYS_LIGHTWEIGHT_PROT is necessary so thee is proper mutual exclusion when allocating/freeing pbuf. LWIP_ALLOW_MEM_FREE_FROM_OTHER_CONTEXT is only necessary if alloc/free is done from interrupt context or something.

[Giovanni]

Hi,

The problem is that I don't know for sure, SYS_LIGHTWEIGHT_PROT should use critical zones instead of mutexes for synchronization so it should work either way because the arch layer supplies both mechanisms.

Giovanni

[steved]

My guess is that just SYS_LIGHTWEIGHT_PROT is necessary so thee is proper mutual exclusion when allocating/freeing pbuf. LWIP_ALLOW_MEM_FREE_FROM_OTHER_CONTEXT is only necessary if alloc/free is done from interrupt context or something.

I don't claim to have any great knowledge in this area; however having been experimenting with lwIP (V2.0.2) I agree with your interpretation. I've been testing with this combination of options, mostly on a 32F429, using the sockets interface, and found it works OK. I'll try and remember to test without SYS_LIGHTWEIGHT_PROT set to see if it makes a difference.

[steved]

Have just tested with SYS_LIGHTWEIGHT_PROT set to zero, and the system did indeed crash (although it took rather longer than I expected, given that the network interface was running close to flat out). So can confirm SYS_LIGHTWEIGHT_PROT set to 1

[Giovanni]

bump

not sure what to do with this

Giovanni