Re: Build errors when building git on MacOS 11 (x86-64) and for M1 macs

Re: Build errors when building git on MacOS 11 (x86-64) and for M1 macs

[Ævar Arnfjörð Bjarmason]

[CC-ing the main git@ list for the "bug in git" aspect of this]

On Thu, Jun 10 2021, Nikhil Gupta wrote:

>   | The following shell command exited with status 2:
>   |  
>   | $ CFLAGS=-I/opt/chef-workstation/embedded/include -O3 -D_FORTIFY_SOURCE=2 -fstack-protector CPPFLAGS=-I/opt/chef-workstation/embedded/include -O3
> -D_FORTIFY_SOURCE=2 -fstack-protector CXXFLAGS=-I/opt/chef-workstation/embedded/include -O3 -D_FORTIFY_SOURCE=2 -fstack-protector
> LDFLAGS=-Wl,-rpath,/opt/chef-workstation/embedded/lib -L/opt/chef-workstation/embedded/lib LD_RUN_PATH=/opt/chef-workstation/embedded/lib MAKE=gmake
> OMNIBUS_INSTALL_DIR=/opt/chef-workstation
> PATH=/opt/chef-workstation/bin:/opt/chef-workstation/embedded/bin:/Users/administrator/.buildkite-agent/builds/MM009-local-1/chef/chef-chef-workstation-master-omnibus-adhoc/omnibus/vendor/bundle/ruby/2.7.0/bin:/opt/angry-omnibus-toolchain/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin
> PKG_CONFIG_PATH=/opt/chef-workstation/embedded/lib/pkgconfig gmake prefix=/opt/chef-workstation/embedded bindir=/opt/chef-workstation/gitbin -j 10
> [...]
>   | * new script parameters
>   | builtin/archive.c:48:24: error: implicit declaration of function 'archive_format_from_filename' is invalid in C99
> [-Werror,-Wimplicit-function-declaration]
>   | const char *format = archive_format_from_filename(name_hint);
>   | ^
>   | builtin/archive.c:48:24: note: did you mean 'archive_read_open_filename'?
>   | /opt/chef-workstation/embedded/include/archive.h:527:15: note: 'archive_read_open_filename' declared here
>   | __LA_DECL int archive_read_open_filename(struct archive *,
>   | ^
>   | builtin/archive.c:48:15: warning: incompatible integer to pointer conversion initializing 'const char *' with an expression of type 'int'
> [-Wint-conversion]
>   | const char *format = archive_format_from_filename(name_hint);
>   | ^        ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
>   | builtin/archive.c:101:2: error: implicit declaration of function 'init_archivers' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
>   | init_archivers();
>   | ^
>   | builtin/archive.c:111:9: error: implicit declaration of function 'write_archive' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
>   | return write_archive(argc, argv, prefix, the_repository, output, 0);
>   | ^
>   | builtin/archive.c:111:9: note: did you mean 'write_or_die'?
>   | ./cache.h:1737:6: note: 'write_or_die' declared here
>   | void write_or_die(int fd, const void *buf, size_t count);
>   | ^
>   | 1 warning and 3 errors generated.
>   | gmake: *** [Makefile:2431: builtin/archive.o] Error 1
>   | gmake: *** Waiting for unfinished jobs....
>   |  
>   |  

I have no access to such a system, but I think think I see the problem
from what you've supplied here.

You supplied a CFLAGS=-I/opt/chef-workstation/embedded/include which has
an archive.h file that's unrelated to the archive.h file git expects.

Thus we include that and find things unrelated to git, and error when we
encounter a function we expected to have declared in our own archive.h.

The solution is something like defining a config.mak file where you add
flags with BASIC_CFLAGS +=, not =. See config.mak.uname for an
example. You'll then add new directories after our own -I.

This is arguably a bug in git's Makefile in that we should have that
"-I." in an ESSENTIAL_CFLAGS variable or something, I can't think of a
scenario where git would compile without it. That or things in builtin/
should include e.g. ../archive.h, perhaps such a thing isn't portable.

I think (but have not confirmed) that you probably got this far because
your compiler will stick -I. at the /end/ of the flags implicitly (or is
that standardized C behavior? I can't remember).

So it worked until we had a filename conflict, i.e. we'd find strbuf.h
in our own sources, but have an issue with a common name like archive.h.

Re: Build errors when building git on MacOS 11 (x86-64) and for M1 macs

[Felipe Contreras]

Ævar Arnfjörð Bjarmason wrote:

> This is arguably a bug in git's Makefile in that we should have that
> "-I." in an ESSENTIAL_CFLAGS variable or something, I can't think of a
> scenario where git would compile without it.

If they are so essential we probably don't even need a variable, just
put it directly into the $(CC) command.

-- 
Felipe Contreras

Re: Build errors when building git on MacOS 11 (x86-64) and for M1 macs

[Jeff King]

On Thu, Jun 10, 2021 at 03:52:09PM +0200, Ævar Arnfjörð Bjarmason wrote:

> I have no access to such a system, but I think think I see the problem
> from what you've supplied here.
> 
> You supplied a CFLAGS=-I/opt/chef-workstation/embedded/include which has
> an archive.h file that's unrelated to the archive.h file git expects.
> 
> Thus we include that and find things unrelated to git, and error when we
> encounter a function we expected to have declared in our own archive.h.

Yeah, I agree that is likely the problem.

> The solution is something like defining a config.mak file where you add
> flags with BASIC_CFLAGS +=, not =. See config.mak.uname for an
> example. You'll then add new directories after our own -I.

That would work, though I think in general we should consider
BASIC_CFLAGS to be an internal thing, and encourage people to use the
standard CFLAGS, etc.

> This is arguably a bug in git's Makefile in that we should have that
> "-I." in an ESSENTIAL_CFLAGS variable or something, I can't think of a
> scenario where git would compile without it. That or things in builtin/
> should include e.g. ../archive.h, perhaps such a thing isn't portable.

Yeah, we do say "-I." explicitly and unconditionally. I think we'd just
want to make sure it's before any user-provided flags. We could do
something like this:

diff --git a/Makefile b/Makefile
index 7d719ece46..b5794560e9 100644
--- a/Makefile
+++ b/Makefile
@@ -1241,7 +1241,7 @@ ALL_COMMANDS_TO_INSTALL += git-upload-archive$(X)
 ALL_COMMANDS_TO_INSTALL += git-upload-pack$(X)
 endif
 
-ALL_CFLAGS = $(DEVELOPER_CFLAGS) $(CPPFLAGS) $(CFLAGS)
+ALL_CFLAGS = $(DEVELOPER_CFLAGS) $(CPPFLAGS) $(BASIC_CFLAGS) $(CFLAGS)
 ALL_LDFLAGS = $(LDFLAGS)
 
 comma := ,
@@ -2106,7 +2106,6 @@ PAGER_ENV_CQ = "$(subst ",\",$(subst \,\\,$(PAGER_ENV)))"
 PAGER_ENV_CQ_SQ = $(subst ','\'',$(PAGER_ENV_CQ))
 BASIC_CFLAGS += -DPAGER_ENV='$(PAGER_ENV_CQ_SQ)'
 
-ALL_CFLAGS += $(BASIC_CFLAGS)
 ALL_LDFLAGS += $(BASIC_LDFLAGS)
 
 export DIFF TAR INSTALL DESTDIR SHELL_PATH

but I imagine that it is helpful for other BASIC_CFLAGS to be
overrideable by the user's CFLAGS (especially other system-level -I
directives we may include!). So yes, having an $(ESSENTIAL_CFLAGS) would
work. Or perhaps even just:

diff --git a/Makefile b/Makefile
index 7d719ece46..72517db31e 100644
--- a/Makefile
+++ b/Makefile
@@ -1202,7 +1202,7 @@ endif
 CFLAGS = -g -O2 -Wall
 LDFLAGS =
 CC_LD_DYNPATH = -Wl,-rpath,
-BASIC_CFLAGS = -I.
+BASIC_CFLAGS =
 BASIC_LDFLAGS =
 
 # library flags
@@ -1241,7 +1241,7 @@ ALL_COMMANDS_TO_INSTALL += git-upload-archive$(X)
 ALL_COMMANDS_TO_INSTALL += git-upload-pack$(X)
 endif
 
-ALL_CFLAGS = $(DEVELOPER_CFLAGS) $(CPPFLAGS) $(CFLAGS)
+ALL_CFLAGS = -I. $(DEVELOPER_CFLAGS) $(CPPFLAGS) $(CFLAGS)
 ALL_LDFLAGS = $(LDFLAGS)
 
 comma := ,

since this "-I." really is special.

> I think (but have not confirmed) that you probably got this far because
> your compiler will stick -I. at the /end/ of the flags implicitly (or is
> that standardized C behavior? I can't remember).

The standard just says (for quoted #include lines):

  The named source file is searched for in an implementation-defined
  manner.

Most compilers put the directory of the source file at the start of the
search path. E.g., gcc says under -I:

  1. For the quote form of the include directive, the directory of the
     current file is searched first.

Which is why we need "-I." at all; we are finding "archive.h" from
"builtin/*.c". I'm not sure what "current file" there means, or how
portable it is. Is it the source file being filed, or the file
containing the #include directive?

I'd hope it's consistently the latter. Otherwise "foo.h" which includes
"bar.h" cannot be included as "../foo.h" (from builtin/foo.c) and as
"foo.h" (from top-level foo.c).

If so, then yeah, using "../archive.h" (and dropping -I. entirely) would
be an option. Which is nice, because it makes things less magical and
more predictable (think what confusion we'd see if we introduced
"archive.h" into builtin/ ourselves).

We do use "../foo.h" in lots of places already. But the fact that we
_also_ have "-I." means I don't think we have any data on the "what does
current file mean" question above. My gut feeling is that it probably
does consistently work as we'd want it to, though.

-Peff

Re: Build errors when building git on MacOS 11 (x86-64) and for M1 macs

[Jeff King]

On Thu, Jun 10, 2021 at 10:36:27AM -0400, Jeff King wrote:

> Most compilers put the directory of the source file at the start of the
> search path. E.g., gcc says under -I:
> 
>   1. For the quote form of the include directive, the directory of the
>      current file is searched first.
> 
> Which is why we need "-I." at all; we are finding "archive.h" from
> "builtin/*.c". I'm not sure what "current file" there means, or how
> portable it is. Is it the source file being filed, or the file
> containing the #include directive?
> 
> I'd hope it's consistently the latter. Otherwise "foo.h" which includes
> "bar.h" cannot be included as "../foo.h" (from builtin/foo.c) and as
> "foo.h" (from top-level foo.c).
> 
> If so, then yeah, using "../archive.h" (and dropping -I. entirely) would
> be an option. Which is nice, because it makes things less magical and
> more predictable (think what confusion we'd see if we introduced
> "archive.h" into builtin/ ourselves).

There's a related case, which is that within trace2/foo.c, we'd include
trace2/foo.h. Without "-I.", the correct include there is just "foo.h".

So what was explicit actually becomes implicit. Here's a real example to
make this file work without -I:

  diff --git a/trace2/tr2_tgt_perf.c b/trace2/tr2_tgt_perf.c
  index a8018f18cc..592c194ab4 100644
  --- a/trace2/tr2_tgt_perf.c
  +++ b/trace2/tr2_tgt_perf.c
  @@ -1,15 +1,15 @@
  -#include "cache.h"
  -#include "config.h"
  -#include "run-command.h"
  -#include "quote.h"
  -#include "version.h"
  -#include "json-writer.h"
  -#include "trace2/tr2_dst.h"
  -#include "trace2/tr2_sid.h"
  -#include "trace2/tr2_sysenv.h"
  -#include "trace2/tr2_tbuf.h"
  -#include "trace2/tr2_tgt.h"
  -#include "trace2/tr2_tls.h"
  +#include "../cache.h"
  +#include "../config.h"
  +#include "../run-command.h"
  +#include "../quote.h"
  +#include "../version.h"
  +#include "../json-writer.h"
  +#include "tr2_dst.h"
  +#include "tr2_sid.h"
  +#include "tr2_sysenv.h"
  +#include "tr2_tbuf.h"
  +#include "tr2_tgt.h"
  +#include "tr2_tls.h"

So I dunno. I had hoped it would be a strict readability improvement,
but now I'm not so sure.

-Peff