Re: Is detecting endianness at compile-time unworkable?

[Michael Felt <aixtools@felt.demon.nl>]

A small step back...


On 7/30/2018 11:39 AM, Ævar Arnfjörð Bjarmason wrote:
> On Sun, Jul 29 2018, Michael wrote:
>
>> On 29/07/2018 22:06, brian m. carlson wrote:
>>> On Sun, Jul 29, 2018 at 09:48:43PM +0200, Michael wrote:
>>>> On 29/07/2018 21:27, brian m. carlson wrote:
>>>>> Well, that explains it.  I would recommend submitting a patch to
>>>>> https://github.com/cr-marcstevens/sha1collisiondetection, and the we can
>>>>> pull in the updated submodule with that fix.
>>>> Not sure I am smart enough to do that. I'll have to download, build, and see
>>>> what it says.
>>> The issue is that somewhere in lib/sha1.c, you need to cause
>>> SHA1DC_BIGENDIAN to be set.  That means you need to figure out what
>>> compiler macro might indicate that.
>> I remember - roughly - a few decades back - having an assignment to
>> write code to determine endianness. PDP and VAC were different iirc,
>> and many other micro-processors besides the 8088/8086/z85/68k/etc..
>>
>> If you are looking for a compiler macro as a way to determine this -
>> maybe you have one for gcc, but not for xlc. I do not know it - currently :)
> I'm not familiar with AIX, but from searching around I found this
> porting manual from IBM:
> http://www.redbooks.ibm.com/redbooks/pdfs/sg246034.pdf
This is from July 2001 - when AIX 5L, for Linux affinity, was new. AIX 
was (nearly) the #1 posix system, and linux was a minor player in the 
data center (in or out (now as IAAS)). IMHO, the recommendations made in 
2001 are probably no longer applicable (64-bit was fairly new, e.g., 
rather than common).
>
> There they suggest either defining your own macros, or testing the
> memory layout at runtime (see section "2.2.2.3 Technique 3: Testing
> memory layout" and surrounding sections).
>
> Perhaps it's worth taking a step back here and thinking about whether
> this whole thing is unworkable. It was hard enough to get this to work
> on the combination of Linux, *BSD and Solaris, but I suspect we'll run
> into increasingly obscure platforms where this is hard or impossible
> (AIX, HP/UX etc.)
>
> The reason we're in this hole is because we use this
> sha1collisiondetection library to do SHA-1, and the reason we have
> issues with it specifically (not OpenSSL et al) is because its only
> method of detecting endianness is at compile time.
Cannot speak for the "others", but as I have mentioned before - as AIX 
in only on POWER it is also only Big Endian - so a compiletime #if 
testing for _AIX will work fine
> This didn't use to be the case, it was changed in this commit:
> https://github.com/cr-marcstevens/sha1collisiondetection/commit/d597672
>
> Dan Shumow: Since the commit message doesn't say why, can you elaborate
> a bit on why this was done, i.e. is determining this at runtime harmful
> for performance? If not, perhaps it would be best to bring this back, at
> least as an option.
>
> And, as an aside, the reason we can't easily make it better ourselves is
> because the build process for git.git doesn't have a facility to run
> code to detect this type of stuff (the configure script is always
> optional). So we can't just run this test ourselves.
On AIX - I am required to run configure, and frankly, I am amazed that 
not everyone is running it. Among other things I an modifying the prefix 
(to /opt) and many of the others to different /var/git/* areas as I do 
not want to "polute" the BOS (base OS) and/or other packages/packagers. 
Officially, according the Linux FHS-3.0 I sould be using /opt/aixtools 
as prefix.

FYI: my current build process is:
wget git-{version}.tar.xz
xz -dc git-{version}.tar.xz
cd git-{version}.tar.xz
mv Makefile Makefile.git #my build scripts only run configure when 
Makefile does not exist
./configure ...
ln Makefile.git Makefile
make

I am amazed, as it rarely happens (maybe git is my first encounter) - 
that configure does not create a Makefile. This also complicates 
building git "out of tree".
>
> Junio: I've barked up that particular tree before in
> https://public-inbox.org/git/87a7x3kmh5.fsf@evledraar.gmail.com/ and I
> won't bore you all by repeating myself, except to say that this is yet
> another case where I wish we had a hard dependency on some way of doing
> checks via compiled code in our build system.
For AIX: again - the determination is simple. If _AIX is set to 1 then 
use BigEndian, or, use:
michael@x071:[/home/michael]uname
AIX
  i.e., something like:
$(uname) == "AIX" && BigEndian=1