Add flag to build compiler options with G++ on macOS/MacPorts for Tesseract 4.x. #1474

ic · 2018-04-13T01:23:27Z

Building with G++ on Darwin breaks with the MacPorts toolchain, when either AVX, AVX2, or SSE4.1 compiler option is set, unless G++ is actually CLANG.

This PR allows building with G++, by asking G++ to delegate assembly to the clang integrated assembler, instead of the GNU one.

Situation

GNU compilers choke when assembling with AS. After autogen.sh and configure, make ends quickly with such error trace:

# ... Cut the beginning of the trace
libtool: compile:  g++ -DHAVE_CONFIG_H -I. -I.. -O2 -DNDEBUG -I../ccstruct -I../ccutil -I../viewer -I/opt/local/include/leptonica -I/opt/local/include/pango-1.0 -I/opt/local/include/glib-2.0 -I/opt/local/lib/glib-2.0/include -I/opt/local/include -I/opt/local/include/cairo -I/opt/local/include/glib-2.0 -I/opt/local/lib/glib-2.0/include -I/opt/local/include -I/opt/local/include/pixman-1 -I/opt/local/include -I/opt/local/include/freetype2 -I/opt/local/include -I/opt/local/include/freetype2 -I/opt/local/include/libpng16 -I/opt/local/include -mavx -g -O2 -std=c++11 -MT libtesseract_avx_la-dotproductavx.lo -MD -MP -MF .deps/libtesseract_avx_la-dotproductavx.Tpo -c dotproductavx.cpp  -fno-common -DPIC -o .libs/libtesseract_avx_la-dotproductavx.o
/var/folders/dy/7c6v04gn4x96qz9w1xrl_6r00000gn/T//cc9mtjB1.s:46:no such instruction: `vmovapd (%rdi), %ymm0'
/var/folders/dy/7c6v04gn4x96qz9w1xrl_6r00000gn/T//cc9mtjB1.s:47:no such instruction: `vmulpd (%rsi), %ymm0,%ymm0'
/var/folders/dy/7c6v04gn4x96qz9w1xrl_6r00000gn/T//cc9mtjB1.s:61:no such instruction: `vmovapd (%rdi,%rax,8), %ymm1'
# ... several more missing instructions
make[3]: *** [libtesseract_avx_la-dotproductavx.lo] Error 1
make[2]: *** [all-recursive] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2

This is a common issue on macOS, where we need to ask the compiler to use the clang integrated assembler by passing -Wa,-q.

The problem happens at least from tag 4.0.0-beta.1 to the present master at 742a0875077e16f0010e99f1bf05823cd9bb725d.

Tests conducted

Configuration

Commit: 742a0875077e16f0010e99f1bf05823cd9bb725d (current master at this time)
macOS 10.12.6
MacPorts 2.4.2
Build with /opt/local/bin/g++, itself built with MacPorts (g++ (MacPorts gcc6 6.4.0_0) 6.4.0)
Leptonica 1.75.3
Machine with AVX, AVX2, SSE4.1 support.

Tests after the fix

Simple `make`

Builds fine until completion.
CLI trial, from the repository home:

export TEST_IMG=/tmp/tesseract_test.jpg
curl -o $TEST_IMG https://i.ytimg.com/vi/4KhEQKyByXs/maxresdefault.jpg
export TESSDATA_PREFIX=/path/to/4.0/tessdata
./api/.libs/tesseract $TEST_IMG stdout -l eng

Results in:

Warning. Invalid resolution 0 dpi. Using 70 instead.
Estimating resolution as 2122
SAMPLE TEXT

`make training`

Builds fine until completion.
Same CLI trial as above.

Extra Notes

I am aware of this issue/documentation where the build seems to work fine on macOS 10.13 with Homebrew. Here I build with MacPorts, so the problem may be due to this toolchain.

Building with G++ on Darwin breaks when either AVX, AVX2, or SSE4.1 compiler option is set, unless G++ is actually CLANG. This commit allows to build with G++, by asking G++ to delegate assembly to the clang integrated assembler, instead of the GNU one.

amitdo · 2018-04-13T08:28:24Z

Homebrew/legacy-homebrew#28984

stweil · 2018-04-20T19:54:21Z

This change introduced a regression: now building with the standard compiler (clang++) no longer works on macOS.

stweil · 2018-04-20T19:56:35Z

configure.ac

+if $avx -o $avx2 -o $sse41; then
+  case "${host_os}" in
+    *darwin* | *-macos10*)
+       if test -z "$CLANG"; then


$CLANG is always undefined here, so the condition is always true.

stweil · 2018-04-20T19:59:14Z

configure.ac

+         # When using AVX, AVX2, or SSE4.1:
+         # Must tell AS to use clang integrated assembler,
+         # instead of the GNU based system assembler.
+         CXXFLAGS="$CXXFLAGS -Wa,-q"


clang++ fails with those flags which are always set.

CLANG is defined here:
[CLANG=yes], [CLANG=no])

Your code is placed above it.

amitdo · 2018-04-20T20:30:38Z

.travis.yml:

os:
- linux
#- osx

Why?

stweil · 2018-04-20T20:39:32Z

It was removed in commit 63e1862.

stweil · 2018-04-20T20:44:30Z

As Travis uses cmake, it would not have found the regression which was only for autoconf.

amitdo · 2018-04-20T20:48:29Z

Do you know how to change travis.yml to use autotools on macOS?

stweil · 2018-04-20T20:54:38Z

Yes, I think so. At least for Linux it should be easy:‌ just replace the build commands for Leptonica and Tesseract.

amitdo · 2018-04-20T20:59:31Z

I didn't realize it uses CMake on Linux too.

ic · 2018-04-22T02:17:34Z

Thank you for the feedback. The test mimics what is done in several places of the scripts, to detect the active tool chain. I supposed it was working as expected. I can address this issue tomorrow. But is there a bigger issue here for clang/gcc support? The test is really doing the same check as a few other places in the scripts...

amitdo · 2018-04-22T06:10:30Z

The other places don't have this issue.

stweil · 2018-04-22T16:24:40Z

@ic, I was not able to successfully build git master with x86_64-apple-darwin17-g++ from MacPorts. It did not find a clang assember in /opt/local/bin. What about reverting your commit until there is a better solution? That would at least fix the default build with Apple's clang compiler.

stweil · 2018-04-22T16:30:26Z

I wonder whether we really have to support a g++ which cannot handle AVX instructions. Isn't that a buggy installation which has to be fixed in MacPorts?

And if we work around that problem, it should be sufficient to add -Wa,q for those files in arch which really use AVX (and maybe also SSE4.1) instructions.

ic · 2018-04-23T02:29:12Z

@stweil I have tried building from master now with x86_64-apple-darwin17-g++ from MacPorts (I was on darwin16 so far). It worked, but I already have clang installed with MacPorts too. Would that be a hidden requirement? I could not reproduce your error.

Fixing the default build is top priority, though. So I have followed the lead from @amitdo and removed the dependency to CLANG. The result for now:

It builds fine on the latest 4.x tag (40f4311) with g++ and clang++ (Autoconf).
~~It now builds fine on master~~ It does not work on master with clang++ under Autoconf, ending with a ld: symbol(s) not found for architecture x86_64 (clang version 6.0.0 (tags/RELEASE_600/final), x86_64-apple-darwin17.5.0).
It builds on master with g++ under Autoconf, and with clang++ under Cmake.

Build environments:

> g++ --version
g++ (MacPorts gcc6 6.4.0_0) 6.4.0
> clang++ --version
clang version 6.0.0 (tags/RELEASE_600/final)
Target: x86_64-apple-darwin17.5.0
Thread model: posix
InstalledDir: /opt/local/libexec/llvm-6.0/bin

The patch, before any tentative PR or rollback:

diff --git a/configure.ac b/configure.ac
index 6e5673ef..8f6bbce0 100644
--- a/configure.ac
+++ b/configure.ac
@@ -135,8 +135,8 @@ fi
 if $avx -o $avx2 -o $sse41; then
   case "${host_os}" in
     *darwin* | *-macos10*)
-       if test -z "$CLANG"; then
-         # When using AVX, AVX2, or SSE4.1:
+       if test g++ = "$CXX"; then
+         # When using AVX, AVX2, or SSE4.1 with g++:
          # Must tell AS to use clang integrated assembler,
          # instead of the GNU based system assembler.
          CXXFLAGS="$CXXFLAGS -Wa,-q"

On g++ from MacPorts (and HomeBrew):
The as documentation on Darwin provides a flag to switch between clang and GNU assemblers (see also @amitdo link to Homebrew/legacy-homebrew#28984). Is it a bug or just how Apple's stack has been designed? No idea, but the Apple ecosystem seems to lead to using that flag for AVX and the likes low-level optimizations. The baseline to me is it would be great the build script just deals with the different scenarios.

ic · 2018-04-23T02:33:26Z

For the record, if this PR is rolled back, here is how I was proceeding before:

./autogen.sh && ./configure CXXFLAGS=-Wa,-q

When building with g++.

Please note that the configure.ac commands default to g++:

AC_PROG_CXX([g++ clang++])

On line 13 at this time.

ic · 2018-04-23T02:39:25Z

For TravisCI to test building with Autotools on macOS, any interest in scripts like this one ? This is to build Tesseract with Autotools, and I have another one for Leptonica (both used to packaged for running locally on Darwin and remotely on AWS Lambda).

stweil · 2018-04-23T07:03:11Z

Should we change the preferred compiler order for macOS?

ic · 2018-04-23T07:43:16Z

@stweil Anyone deeper into the build system to ask to?

Simple opinion: Given the situation and cmake using clang anyway, it makes sense.

g++ builds require extra flags rejected by clang++. The bug is that the flags are actually added unconditionally. This commit fixes the condition. See tesseract-ocr#1474

ic · 2018-04-23T07:47:08Z

#1516 is a proposal based on the discussion here. It just addresses fixing the build with clang. The other discussion items should probably be different PRs.

g++ builds require extra flags rejected by clang++. The bug is that the flags are actually added unconditionally. This commit fixes the condition. See tesseract-ocr#1474

ic added 2 commits April 13, 2018 09:39

Remove useless echo statement.

1642d88

zdenop merged commit 1d4959c into tesseract-ocr:master Apr 13, 2018

amitdo mentioned this pull request Apr 20, 2018

[macos] unsupported argument '-q' to option 'Wa,' #1500

Closed

stweil reviewed Apr 20, 2018

View reviewed changes

ic mentioned this pull request Apr 23, 2018

Fix wrong environment test that breaks clang++ builds. #1516

Merged

ic mentioned this pull request Apr 23, 2018

Revert failed attempt to support MacPort's g++ #1522

Merged

Add flag to build compiler options with G++ on macOS/MacPorts for Tesseract 4.x. #1474

Add flag to build compiler options with G++ on macOS/MacPorts for Tesseract 4.x. #1474

Uh oh!

Conversation

ic commented Apr 13, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Situation

Tests conducted

Configuration

Tests after the fix

Simple make

make training

Extra Notes

Uh oh!

amitdo commented Apr 13, 2018

Uh oh!

stweil commented Apr 20, 2018

Uh oh!

stweil Apr 20, 2018

Choose a reason for hiding this comment

Uh oh!

stweil Apr 20, 2018

Choose a reason for hiding this comment

Uh oh!

amitdo Apr 22, 2018

Choose a reason for hiding this comment

Uh oh!

amitdo commented Apr 20, 2018

Uh oh!

stweil commented Apr 20, 2018

Uh oh!

stweil commented Apr 20, 2018

Uh oh!

amitdo commented Apr 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

stweil commented Apr 20, 2018

Uh oh!

amitdo commented Apr 20, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ic commented Apr 22, 2018 via email

Uh oh!

amitdo commented Apr 22, 2018

Uh oh!

stweil commented Apr 22, 2018

Uh oh!

stweil commented Apr 22, 2018

Uh oh!

ic commented Apr 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ic commented Apr 23, 2018

Uh oh!

ic commented Apr 23, 2018

Uh oh!

stweil commented Apr 23, 2018

Uh oh!

ic commented Apr 23, 2018

Uh oh!

ic commented Apr 23, 2018 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ic commented Apr 13, 2018 •

edited

Loading

Simple `make`

`make training`

amitdo commented Apr 20, 2018 •

edited

Loading

amitdo commented Apr 20, 2018 •

edited

Loading

ic commented Apr 23, 2018 •

edited

Loading

ic commented Apr 23, 2018 •

edited

Loading