System Deadlock

ChibiOS public support forum for all topics not covered by a specific support forum.

Moderators: RoccoMarco, lbednarz, utzig, tfAteba, barthess

Tabulous
Posts: 509
Joined: Fri May 03, 2013 12:02 pm
Has thanked: 7 times
Been thanked: 17 times

System Deadlock

Postby Tabulous » Fri Aug 04, 2017 10:13 am

Hi All
got an issue by where the system gets into a deadlock state.

In the system there are numerous threads doing different things.

One of the threads(SEND) on completion of sending some data posts a message back(MSG#1) to the main system event thread. It does this using

chMBPost( &mbox, ( msg_t )msg, TIME_INFINITE );


Other threads also post messages to the main system event thread too. The main system event thread can block for small amount of time when processing certain events, but my understanding is this should not effect the mailbox accepting new messages, they will just be queued. The only time a posting thread will be blocked is if the mailbox is full, which should never happen as I've got them set quite large.

What i'm seeing is Thread(SEND) gets locked on the chMBPost, now if i change the timeout to TIME_IMMEDIATE, then the system does not lock, but that message is not posted to the main system event thread, and thus i loose the MSG#1.


So how can a find out what the route cause of the issue is ?
worth noting the scheduler and other threads are still running, it just seems there is dealock between sending thread and main system event thread.

steved
Posts: 823
Joined: Fri Nov 09, 2012 2:22 pm
Has thanked: 12 times
Been thanked: 135 times

Re: System Deadlock

Postby steved » Fri Aug 04, 2017 1:30 pm

Have you tried timeouts in between TIME_IMMEDIATE and TIME_INFINITE? Maybe your main thread is blocking for longer than you expect.

It should be possible to check that the queue is genuinely full when a message post times out.

I had a problem recently which looked as if it might be related to the mailbox code; fixed it by increasing the stack (on an unrelated thread, IIRC).

Which version of Chibi are you using? I think Giovanni reworked the mailbox code relatively recently.

Tabulous
Posts: 509
Joined: Fri May 03, 2013 12:02 pm
Has thanked: 7 times
Been thanked: 17 times

Re: System Deadlock

Postby Tabulous » Fri Aug 04, 2017 2:16 pm

Hi Steve,
seems i under estimated the mailbox size, and this caused the issue.

Thanks for the response through.

Tabulous
Posts: 509
Joined: Fri May 03, 2013 12:02 pm
Has thanked: 7 times
Been thanked: 17 times

Re: System Deadlock

Postby Tabulous » Fri Aug 04, 2017 3:36 pm

One thing this did show is my WWDG implantation is rubbish.

To be fair i knew it was not the best, running the WWDG update from IDLE thread is not good, so looks like i'll have to go the full hog.

User avatar
Giovanni
Site Admin
Posts: 14444
Joined: Wed May 27, 2009 8:48 am
Location: Salerno, Italy
Has thanked: 1074 times
Been thanked: 921 times
Contact:

Re: System Deadlock

Postby Giovanni » Fri Aug 04, 2017 3:40 pm

If you designed with mailboxes sufficiently large to never block then I would use TIME_IMMEDIATE and set an halt in the timeout case (queue full).

This would help detection of problems or regressions.

About the watchdog, a common solution is to reset it from an high priority thread AND-waiting for event flags from all other threads, once the mask is complete it wakes up and resets the watchdog. This ensures that 1) the system is alive as a whole 2) all thread are running, no logical locks.

Giovanni

Tabulous
Posts: 509
Joined: Fri May 03, 2013 12:02 pm
Has thanked: 7 times
Been thanked: 17 times

Re: System Deadlock

Postby Tabulous » Fri Aug 04, 2017 4:48 pm

Giovanni wrote:If you designed with mailboxes sufficiently large to never block then I would use TIME_IMMEDIATE and set an halt in the timeout case (queue full).

This would help detection of problems or regressions.

About the watchdog, a common solution is to reset it from an high priority thread AND-waiting for event flags from all other threads, once the mask is complete it wakes up and resets the watchdog. This ensures that 1) the system is alive as a whole 2) all thread are running, no logical locks.

Giovanni


Yep that's the full hog :-) Bit more tricky with using the WWDG through.


Return to “General Support”

Who is online

Users browsing this forum: No registered users and 11 guests