VPATH Bites Again...

[apmorton]

Hi All,

I really hate VPATH

I have just been bitten by VPATH again. On the plus side it only took me 2 hours to figure out what was going on this time.

VPATH is well known for causing problems, and in general I think once people are bitten by the most common case once they learn to work around it.

The most common case is something like this:

Code: Select all

CSRC += folder1/file.c
CSRC += folder2/file.c


Once you understand that VPATH is being used and what it does, its pretty obvious what the issue here is and how to avoid it going forward.
You just need to make sure all your object files will have unique filenames since they will all be put into the same folder during build, so only _one_ of these "file.c" source files will be built.

However, consider the following more subtle example that I just ran into today:

Code: Select all

CSRC += ../lib/mbed-crypto/library/bignum.c
CSRC += src/sha256.c


Can you see how this will fail? No? Neither did I for about 2 hours!
Maybe this gives you a hint?

I wanted to use just a few parts of a larger library (mbed-crypto), but because the source directory for that library is now in my VPATH every source file shipped with that library now potentially collides - not just the ones I use.

It happened that the directory containing mbed-crypto sorts before the directory containing my sha256.c file, so the sha256 shipped with mbed-crypto is now used instead of mine.

The workaround for me was to rename my sha256.c to something else, but this is not a solution. It just kicks the can down the road until the next time a library includes an ambiguous object file name. God forbid you need to use two libraries that have conflicting object file names.

The solution is to stop using VPATH and retain folder hierarchy for object files in the build output.
I don't know how this can be solved with plain Make - or even if it could be solved without making an absolute mess of the makefiles.

Alternative build systems exist, and many of them make this kind of problem easier to deal with.
I think it may be time to take a serious look at supporting one or two of the more popular ones.

I am not saying you should drop the plain Makefile's - for trivial projects it is sufficient.
For larger code bases with many external dependencies it starts to break down.

CMake is widely used and with modern versions (relatively) pain free to use
Meson is relatively new, but is gaining some wider industry adoption and is supposedly very nice to work with (although I have no personal experience on this front)


Thoughts?
If you aren't able/willing/have time to implement an alternative build system would you be open to community effort on this front?

Austin

[Giovanni]

The build system has a lot into it, remaking it using a different system would not be a trivial task.

Giovanni

[steved]

Would it be practicable to add an optional "check for duplicate files" step into the makefile immediately after setting VPATH? Either as an external program, or possibly MAKE itself is clever enough?

[Giovanni]

I think it would be possible, make a list of words (base names), sort it, scan it to find duplicates.

Giovanni

[steved]

Turns out its quite straightforward to dump the contents of VPATH to a file.

I added one line to {CHIBIOS}\os\common\startup\ARMCMx\compilers\GCC\mk\rules.mk:

Code: Select all

POST_MAKE_ALL_RULE_HOOK:
  $(file > $(BUILDDIR)/vpath.txt,$(VPATH))


This puts a space-separated list of paths into file vpath.txt in the build directory. That can easily be processed by a bit of Python or similar (seems little point in checking for duplicate file names on every make, unless its extremely quick).

Maybe the concept could be expanded to output other potentially useful information (and could help with debugging makefiles as well)