Skip to content

Do not use _GNU_SOURCE gratuitously. #1027

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jun 25, 2023
Merged

Do not use _GNU_SOURCE gratuitously. #1027

merged 2 commits into from
Jun 25, 2023

Conversation

przemoc
Copy link
Contributor

@przemoc przemoc commented Jun 17, 2023

What is needed to build whisper.cpp and examples is availability of stuff defined in The Open Group Base Specifications Issue 6 (https://pubs.opengroup.org/onlinepubs/009695399/) known also as Single Unix Specification v3 (SUSv3) or POSIX.1-2001 + XSI extensions.

There is no need to penalize musl libc which simply follows standards.

Not having feature test macros in source code gives greater flexibility to those wanting to reuse it in 3rd party app, as they can build it with minimal FTM (_XOPEN_SOURCE=600) or other FTM depending on their needs.

It builds without issues in Alpine (musl libc), Ubuntu (glibc), MSYS2.

@ggerganov
Copy link
Member

The macOS-latest action seems to fail with this change - SDL2 is not happy for some reason

@przemoc
Copy link
Contributor Author

przemoc commented Jun 25, 2023

Thanks for running CI checks. I do not have access to any macOS platform, so debugging it may be a bit challenging.

I whisper.cpp build info: 
I UNAME_S:  Darwin
I UNAME_P:  i386
I UNAME_M:  x86_64
I CFLAGS:   -I.              -O3 -DNDEBUG -std=c11   -fPIC -D_XOPEN_SOURCE=600 -pthread -mf16c -mavx -DGGML_USE_ACCELERATE
I CXXFLAGS: -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -D_XOPEN_SOURCE=600 -pthread
I LDFLAGS:   -framework Accelerate
I CC:       Apple clang version 14.0.0 (clang-1400.0.29.202)
I CXX:      Apple clang version 14.0.0 (clang-1400.0.29.202)

c++ -I. -I./examples -O3 -DNDEBUG -std=c++11 -fPIC -D_XOPEN_SOURCE=600 -pthread examples/stream/stream.cpp examples/common.cpp examples/common-ggml.cpp examples/common-sdl.cpp ggml.o whisper.o -o stream `sdl2-config --cflags --libs`  -framework Accelerate
In file included from examples/stream/stream.cpp:7:
In file included from ./examples/common-sdl.h:3:
In file included from /usr/local/include/SDL2/SDL.h:32:
In file included from /usr/local/include/SDL2/SDL_main.h:25:
/usr/local/include/SDL2/SDL_stdinc.h:532:5: error: use of undeclared identifier 'memset_pattern4'
    memset_pattern4(dst, &val, dwords * 4);
    ^
1 error generated.

przemoc added 2 commits June 25, 2023 15:15
What is needed to build whisper.cpp and examples is availability of
stuff defined in The Open Group Base Specifications Issue 6
(https://pubs.opengroup.org/onlinepubs/009695399/) known also as
Single Unix Specification v3 (SUSv3) or POSIX.1-2001 + XSI extensions.

There is no need to penalize musl libc which simply follows standards.

Not having feature test macros in source code gives greater flexibility
to those wanting to reuse it in 3rd party app, as they can build it with
minimal FTM (_XOPEN_SOURCE=600) or other FTM depending on their needs.

It builds without issues in Alpine (musl libc), Ubuntu (glibc), MSYS2.
This is an attempt at fixing macOS build error coming from SDL2 relying
on Darwin extension memset_pattern4/8/16 coming from Apple's string.h.
@przemoc
Copy link
Contributor Author

przemoc commented Jun 25, 2023

Looks like my attempt at fixing macOS issue was successful, others checks will be most likely successful too, but let's wait for them to finish.

@ggerganov ggerganov merged commit 3f7a03e into ggml-org:master Jun 25, 2023
ggerganov added a commit to ggml-org/ggml that referenced this pull request Jun 25, 2023
@przemoc przemoc deleted the remove-gnu-source-ftm branch June 25, 2023 13:45
ggerganov added a commit that referenced this pull request Jul 2, 2023
@ggerganov
Copy link
Member

I synced the latest ggml from llama.cpp and the new NUMA-related changes fail to compile:

https://github.com/ggerganov/whisper.cpp/actions/runs/5438047714/jobs/9889005968

Temporary reverted the change until we resolve the build issues.
I think we should first fix ggml-org/llama.cpp#2035 and then I will propagate the changes back to whisper.cpp

@przemoc
Copy link
Contributor Author

przemoc commented Jul 2, 2023

Sorry, @ggerganov, it turned out I didn't have time this weekend to play with llama.cpp and cmake. :(
Adding selected lines in 1st commit from ggml-org/llama.cpp#2035 would most likely solve failing build in whisper.cpp, but of course only for Makefile-driven builds.

BTW What's cmake usage state in whisper.cpp? I don't think CI was doing any cmake-driven build last time I've seen those PR actions. Is it expected to be working? If yes, then I guess I may have broken it with my FTM sanitization patches that weren't touching CMakeLists.txt.

@ggerganov
Copy link
Member

The main purpose for having CMake build system is to support Windows.

I think the CI wasn't failing before because the NUMA changes were not present.
Yesterday I synced them from the llama.cpp repo and they started causing problems.

Don't worry - whenever you have the time. It's not a big problem

@przemoc
Copy link
Contributor Author

przemoc commented Jul 17, 2023

I looked at llama.cpp a bit on Sunday and went with Howard's approach for CMake changes to simply follow recent FTM improvements from Makefile - seemed like viable way way of doing that. I had to fix other things first, though, to make it build in MSYS2.

@ggerganov, when you'll have a spare minute, please trigger PR checks in ggml-org/llama.cpp#2035 and the other one mentioned there that should be merged first, so that I could see if there are still any issues that I should resolve.

And when that will be clean, I can create analogous remove-gnu-source-ftm for whisper.cpp. Not sure if GitHub allows merging second time from same PR (considering it was reverted), as I would prefer to have updated rebased version here and not create yet another PR.

@przemoc
Copy link
Contributor Author

przemoc commented Jul 21, 2023

Thanks for triggering PR checks in llama.cpp. Stuff seem to be building fine there. I ended up creating new PR in whisper.cpp, #1129, to avoid potential issues with GitHub merging UX.

jacobwu-b pushed a commit to jacobwu-b/Transcriptify-by-whisper.cpp that referenced this pull request Oct 24, 2023
* Do not use _GNU_SOURCE gratuitously.

What is needed to build whisper.cpp and examples is availability of
stuff defined in The Open Group Base Specifications Issue 6
(https://pubs.opengroup.org/onlinepubs/009695399/) known also as
Single Unix Specification v3 (SUSv3) or POSIX.1-2001 + XSI extensions.

There is no need to penalize musl libc which simply follows standards.

Not having feature test macros in source code gives greater flexibility
to those wanting to reuse it in 3rd party app, as they can build it with
minimal FTM (_XOPEN_SOURCE=600) or other FTM depending on their needs.

It builds without issues in Alpine (musl libc), Ubuntu (glibc), MSYS2.

* examples : include SDL headers before other headers

This is an attempt at fixing macOS build error coming from SDL2 relying
on Darwin extension memset_pattern4/8/16 coming from Apple's string.h.
jacobwu-b pushed a commit to jacobwu-b/Transcriptify-by-whisper.cpp that referenced this pull request Oct 24, 2023
jacobwu-b pushed a commit to jacobwu-b/Transcriptify-by-whisper.cpp that referenced this pull request Oct 24, 2023
* Do not use _GNU_SOURCE gratuitously.

What is needed to build whisper.cpp and examples is availability of
stuff defined in The Open Group Base Specifications Issue 6
(https://pubs.opengroup.org/onlinepubs/009695399/) known also as
Single Unix Specification v3 (SUSv3) or POSIX.1-2001 + XSI extensions.

There is no need to penalize musl libc which simply follows standards.

Not having feature test macros in source code gives greater flexibility
to those wanting to reuse it in 3rd party app, as they can build it with
minimal FTM (_XOPEN_SOURCE=600) or other FTM depending on their needs.

It builds without issues in Alpine (musl libc), Ubuntu (glibc), MSYS2.

* examples : include SDL headers before other headers

This is an attempt at fixing macOS build error coming from SDL2 relying
on Darwin extension memset_pattern4/8/16 coming from Apple's string.h.
jacobwu-b pushed a commit to jacobwu-b/Transcriptify-by-whisper.cpp that referenced this pull request Oct 24, 2023
landtanin pushed a commit to landtanin/whisper.cpp that referenced this pull request Dec 16, 2023
* Do not use _GNU_SOURCE gratuitously.

What is needed to build whisper.cpp and examples is availability of
stuff defined in The Open Group Base Specifications Issue 6
(https://pubs.opengroup.org/onlinepubs/009695399/) known also as
Single Unix Specification v3 (SUSv3) or POSIX.1-2001 + XSI extensions.

There is no need to penalize musl libc which simply follows standards.

Not having feature test macros in source code gives greater flexibility
to those wanting to reuse it in 3rd party app, as they can build it with
minimal FTM (_XOPEN_SOURCE=600) or other FTM depending on their needs.

It builds without issues in Alpine (musl libc), Ubuntu (glibc), MSYS2.

* examples : include SDL headers before other headers

This is an attempt at fixing macOS build error coming from SDL2 relying
on Darwin extension memset_pattern4/8/16 coming from Apple's string.h.
landtanin pushed a commit to landtanin/whisper.cpp that referenced this pull request Dec 16, 2023
iThalay pushed a commit to iThalay/whisper.cpp that referenced this pull request Sep 23, 2024
* Do not use _GNU_SOURCE gratuitously.

What is needed to build whisper.cpp and examples is availability of
stuff defined in The Open Group Base Specifications Issue 6
(https://pubs.opengroup.org/onlinepubs/009695399/) known also as
Single Unix Specification v3 (SUSv3) or POSIX.1-2001 + XSI extensions.

There is no need to penalize musl libc which simply follows standards.

Not having feature test macros in source code gives greater flexibility
to those wanting to reuse it in 3rd party app, as they can build it with
minimal FTM (_XOPEN_SOURCE=600) or other FTM depending on their needs.

It builds without issues in Alpine (musl libc), Ubuntu (glibc), MSYS2.

* examples : include SDL headers before other headers

This is an attempt at fixing macOS build error coming from SDL2 relying
on Darwin extension memset_pattern4/8/16 coming from Apple's string.h.
iThalay pushed a commit to iThalay/whisper.cpp that referenced this pull request Sep 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants