Non-ISO extended-ASCII text

[chibby]

Hi,

I was using grep to search for occurrences of SPI_CR2_DS_2 in the chibios source tree and noticed it was showing many .h files as being "binary" and just stating that it found a match rather than printing the line concerned.

If grep detects control characters in a file it does this , having concluded that it is not reading a text file.

It did this on a whole host of .h files : eg.
community/testhal/STM32/STM32F4xx/EICU/os/ext/CMSIS/ST/stm32f091xc.h

iconv utility reveals the line in the initial part of each file throwing the problem:

Code: Select all

/**
******************************************************************************
  * @file    stm32f091xc.h
  * @author  MCD Application Team
  * @version V2.2.2
  * @date    26-June-2015
  * @brief   CMSIS STM32F091xC devices Peripheral Access Layer Header File.
  *
  *          This file contains:
  *           - Data structures and the address mapping for all peripherals
  *           - Peripheral's registers declarations and bits definition
  *           - Macros to access peripheraliconv:[b] illegal input sequence at position 494[/b]


So just after "Macros to access peripherali" there is an odd Non-ISO character. Since this header section is common to all headers for each arch, this is the same everywhere.

If I edit the file I see " s " at this position , deleting the char preceding the S fixes the problem.

The files affected are:

Code: Select all

Binary file os/ext/CMSIS/ST/stm32f030x6.h matches
Binary file os/ext/CMSIS/ST/stm32f378xx.h matches
Binary file os/ext/CMSIS/ST/stm32f031x6.h matches
Binary file os/ext/CMSIS/ST/stm32l486xx.h matches
Binary file os/ext/CMSIS/ST/stm32f071xb.h matches
Binary file os/ext/CMSIS/ST/stm32f078xx.h matches
Binary file os/ext/CMSIS/ST/stm32f038xx.h matches
Binary file os/ext/CMSIS/ST/stm32f051x8.h matches
Binary file os/ext/CMSIS/ST/stm32l471xx.h matches
Binary file os/ext/CMSIS/ST/stm32f042x6.h matches
Binary file os/ext/CMSIS/ST/stm32f301x8.h matches
Binary file os/ext/CMSIS/ST/stm32f058xx.h matches
Binary file os/ext/CMSIS/ST/stm32f358xx.h matches
Binary file os/ext/CMSIS/ST/stm32l475xx.h matches
Binary file os/ext/CMSIS/ST/stm32f030x8.h matches
Binary file os/ext/CMSIS/ST/stm32f756xx.h matches
Binary file os/ext/CMSIS/ST/stm32f303x8.h matches
Binary file os/ext/CMSIS/ST/stm32f302xc.h matches
Binary file os/ext/CMSIS/ST/stm32f746xx.h matches
Binary file os/ext/CMSIS/ST/stm32f373xc.h matches
Binary file os/ext/CMSIS/ST/stm32f303xc.h matches
Binary file os/ext/CMSIS/ST/stm32f334x8.h matches
Binary file os/ext/CMSIS/ST/stm32l485xx.h matches
Binary file os/ext/CMSIS/ST/stm32l476xx.h matches
Binary file os/ext/CMSIS/ST/stm32f070xb.h matches
Binary file os/ext/CMSIS/ST/stm32f048xx.h matches
Binary file os/ext/CMSIS/ST/stm32f072xb.h matches
Binary file os/ext/CMSIS/ST/stm32f745xx.h matches
Binary file os/ext/CMSIS/ST/stm32f302x8.h matches
Binary file os/ext/CMSIS/ST/stm32f328xx.h matches
Binary file os/ext/CMSIS/ST/stm32f030xc.h matches
Binary file os/ext/CMSIS/ST/stm32f070x6.h matches
Binary file os/ext/CMSIS/ST/stm32f098xx.h matches
Binary file os/ext/CMSIS/ST/stm32f318xx.h matches


Also the same files in : community/testhal/STM32/STM32F4xx/EICU/os/ext/

It seems that some odd char has got into this header section had has been copied in a template to all versions.

It is the character before the "s" that is giving problems. I'm guessing it is an apostrophe in someone's locale / editor but it is non-standard and is causing basic linux / unix utilities to barf.

It looks like this may originate in ST's CMSIS code but it would be nice to fix it Chibios tree.

[Giovanni]

This is caused usually by pasting text taken/extracted from a PDF or similar.

Those files are generated automatically I believe, it should be reported to ST.

Giovanni

[chibby]

Yes, it appears to originate at ST and needs reporting there. However, I won't hold my breath for the time it will take global corporate monolith to respond and deal with it.

A single, recursive search and replace should be able to clean this up, so I posted the information here in the hope that a small efficient structure can fix it locally without waiting for ST to move their elephant sized butt.

[chibby]

The following *nix hieroglyphics, when run in the root directory of the Chibios tree, will replace all the defective lines with normally coded text:

[chibby]

[EDIT] Sorry , looks like the odd char is messing up most command line tools on linux, so it will need fixing by hand until the resp if fixed. Annoying.

OK I have found a way accounting for abnormal actions of sed when faced with this odd, non-ISO character code. It needs to be done as two separate commands. It should only be done once on a defective tree.

It splits the troublesome part into a separate line, then deletes it second time around.

Code: Select all

cd os/ext/CMSIS

find . -type f -exec sed -i  "s?\(^.*- Macros to access peripheral\)?\1's registers hardware\nERROR?" {} +
find . -type f -exec sed -i  "13d" {} +


Test case: should now only report binary file matches on true binary files and print out the matching lines in the header files as required.

Code: Select all

cd ~/chibios-svn
  grep -R  SPI_CR2_DS_2  .


[chibby]

Those files are generated automatically I believe, it should be reported to ST.

Giovanni

This bug is still in the code base, any chance the automatically could include the fix?

I did report this to ST but they have never even replied to the issue and have now deleted the old forum where it was reported. Don't expect a fix any time soon.

[Giovanni]

Hi,

I prefer do not fix vendor-provided files unless there are blocking issues, I would have to re-fix those again each time files are updated, I would be surprised if they fix them anytime soon or at all.

Best way is if you fix them on your local copy.

Giovanni

[chibby]

OK, I was hoping it could be automated, and thus not require intervention each time. For anyone who is using 'nix command line tools and hitting this problem, here is a more robust way to remove the offending characters. ( It could be done in one line with a bit of thought ).

Code: Select all

cd chibios-svn
find . -type f -exec sed -i  "s?\(^.*- Macros to access peripheral\)?\1's registers hardware\nERROR?" {} +
find . -type f -exec sed -i  "/^ERROR.*/d" {} +


now prints matching lines and sees files as text, not binary.

Code: Select all

grep -R  "Macros to access peripheral"  . 


[rew]

This should be easy with sed.
On each file do:

Code: Select all

sed -ie "s/peripheral.s/peripheral\'s/"

But... from the above I trust you tried that already.... When reporting the bug about "." not matching all characters, and reading the bug reporting guideline, as the very last "known bug, please don't report" they say: set LC_COLLATE and LC_CTYPE to "C" to prevent this behaviour. Sigh.
So if you use bash:

Code: Select all

export LC_COLLATE=C
exoirt LC_CTYPE=C
sed -ie "s/peripheral.s/peripheral\'s/" ./os/common/ext/CMSIS/ST/*/*.h

should do the trick....

P.S. tr -d '\222' also works. But tr doesn't have the handy "-i" option.