STM32F4 "lockup" due to continuous I2C interrupts

[mingthemad]

Hi all,

I'm seeing problems with the Chibios I2C driver on the STM32F4 when there is noise
on the I2C data line. If the noise reaches a certain level, the processor "locks up" because
it is continuously servicing an I2C interrupt that never get cleared.

I was able to reduce the likelihood of lockups by adding the following code to
the I2C interrupt handler in os/hal/platforms/STM32/i2c_lld.c as follows:

Code: Select all

static void i2c_lld_serve_event_interrupt(I2CDriver *i2cp) {
  I2C_TypeDef *dp = i2cp->i2c;
  uint32_t regSR2 = dp->SR2;
  uint32_t event = dp->SR1;
  size_t t;

  // START OF NEW CODE
  if (dp->SR1 & I2C_SR1_BERR) {                     /* Bus error.           */
    dp->SR1 &= ~I2C_SR1_BERR;
    i2cp->errors = I2CD_BUS_ERROR;
    wakeup_isr(i2cp, RDY_RESET);
    return;
  }
 // END OF NEW CODE

But this has not solved the problem completely.

Now, the interrupt routine repeatedly follows a different
path through the case statement in the interrupt handler,
as shown below.

Code: Select all

   case I2C_EV8_2_MASTER_BYTE_TRANSMITTED:
     /* Catches BTF event after the end of transmission.*/
    if (dmaStreamGetTransactionSize(i2cp->dmarx) > 0) {
       /* Starts "read after write" operation, LSB = 1 -> receive.*/
       i2cp->addr |= 0x01;
       dp->CR1 |= I2C_CR1_START | I2C_CR1_ACK;
       dp->CR1 |= I2C_CR1_STOP;
       wakeup_isr(i2cp, RDY_OK);

I've checked the values of SR1 and SR2 and they look fine (no error bits set).

I'm using Chibios 2.4.1, but I have copied the I2C driver from 2.5.1
as there seemed to be a few differences that might have influenced this
issue. However, this made no difference to the problem.

I can reproduce this problem reliably using a signal generator connected
to the I2C data line via a diode. I'm injecting a 100ns low pulse every
1ms. The I2C bit rate is set to 27kHz.

I've been tearing my hair out on this one. My only solution so far is to replace
the Chibios I2C driver with a fully software-driven implementation that
bit-bashes GPIO pins directly.

I expect the noise to stop the I2C from achieving successful comms, but
it should not be possible to lock up the processor in this way regardless of
how much noise is present.

Does anyone have any ideas on how to fix this? I'm happy to try any suggestions
as I can reproduce the problem at will within seconds.

Craig

[iggarpe]

Could you post a bare minimal example showing the problem ? (I can inject the same pulse you do).

I'm willing to give it a go, since those lockups, no matter how rare, are sort of unacceptable in my design.

Cheers.

[Giovanni]

Thanks for helping with this, I am unable to do any serious testing with I2C, I am so tempted to just make a SW implementation and skip all the problems.

Giovanni

[mingthemad]

My first priority was to get the product working, so I have implemented software I2C in order to avoid the lockup problem.

I will try and get a small sample program for the STM32F4 working this week, but you should be able to reproduce the condition version easily. Just initialise an any I2C port for 27kHz and attempt to send out small commands to an I2C peripheral.

If you inject noise onto the SDA line via a diode (to simulate an open drain output), you will eventually lock up the processor. I injected 100ns low going pulses at 1kHz

Craig

[Prof. Dr. YoMan]

I am so tempted to just make a SW implementation and skip all the problems.

Yes!

[tdwebste]

If you inject noise onto the SDA line via a diode (to simulate an open drain output), you will eventually lock up the processor. I injected 100ns low going pulses at 1kHz

Craig

Do you know if this only a problem with the stm32f4 or might it also be a problem with the stm32f3?

[Giovanni]

The F3 has a new I2C peripheral, so probably not.

Giovanni

[Edoardo1982]

This behavior is due to a bug in the I2Cv1 driver: the event mask must be 0x00FF00FF

Code: Select all

#define I2C_EV_MASK 0x00FF00FF

Moreover, continuously unplugging and plugging the slave device, I see a new combination of flags (perhaps it's due to hw problem).
To manage this situation I add a new define

Code: Select all

#define I2C_EV5_MASTER_MODE_INVALID                                          \
  ((uint32_t)(((I2C_SR2_MSL | I2C_SR2_BUSY) << 16) | (I2C_SR1_SB | I2C_SR1_STOPF)))

My solution

Code: Select all

  case I2C_EV5_MASTER_MODE_INVALID:
    i2c_lld_abort_operation(i2cp);
    dp->CR2 &= ~I2C_CR2_ITEVTEN;
    break;

So, when a timeout is used, thread never blocks and the device always works after hot unplugging and plugging.

This code is no more needed

Code: Select all

    /* Errata 2.4.6 for STM32F40x, Spurious Bus Error detection in
       Master mode.*/
    i2cp->i2c->SR1 &= ~I2C_SR1_BERR;

Commit is in trunk

[karlchansen]

@Edoardo1982

EXCELLENT FIX!!!!

We are using an stm32f446re device and have been having intermittent i2c failures that required power-cycle. A recent feature-addition more than doubled the i2c traffic, and the failures began to occur about 1-in-10.

I applied the recommended fix above and have completed a dozen runs of the instrument with no i2c lockups. Suspect you may have fixed our issue. I'll post more when the number of runs is about 50.

Thanks again!

[Giovanni]

Great, good to know it fixed your issue.

Will back-port to other branches too.

Giovanni