llama : add xcframework build script #11996
Conversation
Package.swift
Outdated
| .binaryTarget( | ||
| name: "llama", | ||
| path: "build-ios/llama.xcframework" | ||
| ), | ||
| //.systemLibrary(name: "llama", pkgConfig: "llama"), |
There was a problem hiding this comment.
Something that is not clear to me yet is now that we build a XCFramework, do we need this SPM package? Could we simply add the framework to the project and skip the package?
There was a problem hiding this comment.
The way we are using SPM where we expect llama.cpp to be installed on the local system, my understanding is that these are two ways of doing the same thing and perhaps we should just use framework which seems to be pretty simple.
The option of using SPM for source code distribution (not binary distribution or what is might be called when using a system library) would enable end users to include llama.cpp in their projects using something like this their projects Package.swift:
dependencies: [
.package(url: "https://github.com/ggerganov/llama.cpp.git", from: "1.0.0")
]
SPM would then download the source code, compile it, etc. But I'm not sure how much work it would require to get this working and maintain.
There was a problem hiding this comment.
Yes, so we want to move away from the workflow where the SPM package builds the llama from source for 2 reasons:
- We would need to maintain a second build system (i.e. the SPM package)
- It will not work when
ggmlbecomes a submodule in the future (SPM does not support submodules)
So it seems to me that we should probably remove the Package.swift all together and use the framework approach.
There was a problem hiding this comment.
So it seems to me that we should probably remove the
Package.swiftall together and use the framework approach.
This sounds like a good to me.
There was a problem hiding this comment.
@ggerganov I've removed Package.swift now as it was causing issues with xcodebuild and also produced an error in xcode (but it was still possible to build/run). I've updated the CI build to remove the install of llama.cpp as it should no longer be required, and also added a FRAMEWORKD_FOLDER_PATH for the xcodebuild commands.
There was a problem hiding this comment.
Both, there is also https://github.com/ggml-org/llama.cpp/tree/master/Sources
There was a problem hiding this comment.
I'll remove spm-headers and Sources shortly. (Package.swift has been removed and in llama : remove Package.swift)
There was a problem hiding this comment.
Package dependency management is not just an easy way to integrate dependencies.
It's also there to manage them. SPM actually makes use of the Git version tags, so, when using SPM, one can explicitly define a version they want (need) to use, and distribute that knowledge to others in a team.
If you remove Package.swift, that capability gets lost.
Also, if there ever will be support for executing build scripts on installation of SPM dependencies, this build script could be referenced there.
(cough In that regard, I would like to promote CocoaPods again, as that is capable of exactly such a feature…)
Currently, the nicest way would be to have a Github Action build the xcframework during a release and add that to a Github release page.
That could be referenced from a Package.swift and suddenly people would have a really easy time to integrate without needing a lot of knowledge on how to compile C++ things.
There was a problem hiding this comment.
Just to make sure I've understood this correctly, I as a user would be able to add the following to my project's Package.swift:
dependencies: [
.package(url: "https://github.com/ggml-org/llama.cpp.git", from: "b4777")
]And the Swift Package Manager would then clone the repository and look for a Package.swift file in the root of llama.cpp. This Package.swift could then looks something like the following:
import PackageDescription
let package = Package(
name: "llama",
platforms: [
.iOS(.v14),
.macOS(.v10_15),
.tvOS(.v14),
.visionOS(.v1)
],
products: [
.library(name: "llama", targets: ["llama"])
],
targets: [
.binaryTarget(
name: "llama",
url: "https://github.com/ggml-org/llama.cpp/releases/download/b4777/llama-b4777-xcframework.zip",
checksum: "the-sha256-checksum-here"
)
]
)Without Swift Package (Manual Integration):
- User has to manually go to our GitHub releases page
- Find the correct version of the xcframework.zip
- Download it
- Extract the zip file
- Drag the XCFramework into their Xcode project
- Configure build settings (embedding, signing, etc.)
- Repeat all these steps whenever they want to update to a newer version
With Swift Package (Automated Integration):
- User adds a single line to their dependencies
- Swift Package Manager automatically downloads the right binary
- Proper linking and integration happens automatically
- Updating is as simple as changing a version number
A very basic example can be found here
There was a problem hiding this comment.
Exactly.
To be completely honest, though, I should mention this one:
The checksum is mandatory and is only known after creating the xcframework.zip.
So, someone or something would need to update that in the Package.swift file after creating the new xcframework and, at best, then tag the release accordingly, or force move an already tagged release.
If that is too much hassle, the Package.swift part could be done in another repo which releases separately. Like https://github.com/srgtuszy/llama-cpp-swift.
github-actions
bot
added
the
devops
|
Tagging people who participated in recent iOS/Swift build discussions - please provide feedback about the proposed build changes in this PR. The new XCFramework approach proposed here should resolve all build issues for 3rd-party Swift projects depending on @jiabochao @nvoter @pgorzelany @Animaxx @MrMage @jhen0409 @hnipps @yhondri @tladesignz |
|
with xcframework I am able to build the swift example project, but when run it got error: when I trying to use in other project got error: |
|
@Animaxx Thanks for trying this out!
I'm trying to figure this out. Can you tell me if you are running this using the iphone simulator in xcode or some other way I can reproduce this?
I've been able to reproduce this and looking into it now. Thanks! Update: I've been looking at this issue and have updated the script to use static libraries which I should probably have done in the first place. With these changes, using the generated XCFramework I'm able to run the llama.swiftui example, and I also created a standalone example that imported the import SwiftUI
import llama
struct ContentView: View {
var body: some View {
VStack {
Image(systemName: "globe")
.imageScale(.large)
.foregroundStyle(.tint)
Text("Hello, world!")
}
.padding()
.onAppear {
// Test llama framework functionality
let params = llama_context_default_params()
print("Llama context params initialized:")
print("n_ctx: \(params.n_ctx)")
print("n_threads: \(params.n_threads)")
let start = ggml_time_us()
print("start: \(start)")
}
}
}
#Preview {
ContentView()
}I ran out of time today and I'll be out Mon-Tue, but I'll revisit the issue @Animaxx reported when I'm back on Wednesday. |
|
Hi @danbev thank you for the PR! The error I got is from debugging on the real device, not sure if that related to xcframework sign |
|
with the latest commit works on device now! trying to have validate archives got the error |
|
also when I use the xcframework to build for MacOS app, got error: with unsigned alert |
Ah, I missed building for macos, it currently only builds for ios. I'll try adding a bulid for macos. |
|
Thank you so much, @danbev, for starting this! I took your work for a test drive:
It ran on the following devices:
I didn't try out visionOS, since I don't have a device and I was too lazy to download the simulator. I stumbled over various minor things, however, so I suggest to take care of the following:
|
|
Im not sure what issue I'm running into on my m4 pro MacBook running latest version of Sequoia. I have all of the Xcode dependencies updated and I cannot get passed this. I ran into it the other day when I tried too but the same commit that had issues that day worked once today when I tried again and after pulling in the latest macOS changes from your repo I'm back to the same issues. |
|
@blaineam, that looks like your C compiler toolchain (LLVM) is broken. Run this: xcode-select --install |
|
🫶 |
yeah that didn't fix it, I already had that installed. needed to run The build made it farther, however it did end up failing on: |
Unfortunately, this log line doesn't tell how it failed. Anyway, this brings me to an idea:
Did you just upgrade from Sonoma to Sequoia? In that case, a lot of brew installed build tools are broken. You might want to run brew reinstall installed(That'll take time a lot of time, depending on the stuff you installed via brew. Maybe |
its not happy about something just not sure what. it seems cmake fails to find the c compiler and cxx compiler identifications sometimes and if either of those fail it cannot build. on rare occasion it finds them though ... had one almost successful run and then it failed too. then subsequent builds fail each time. no clue what is wrong on my MacBook to cause such intermittent issues but something is. I tried running the following: brew list | xargs brew reinstall
sudo rm -rf /Library/Developer/CommandLineTools;
brew install cmake
sudo xcodebuild -license accept
sudo xcodebuild -runFirstLaunch
sudo xcode-select --reset
sudo xcode-select --installCLI OutputAri/llama.cpp-temp/llama.cpp on xcframework-build-10747 took 8s Ari/llama.cpp-temp/llama.cpp on xcframework-build-10747
/Users/blainemiller/Documents/mine/Personal/Apps/Ari/llama.cpp-temp/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-quants.c:741:22: warning: implicit CompileC /Users/blainemiller/Documents/mine/Personal/Apps/Ari/llama.cpp-temp/llama.cpp/build-ios-device/build/ggml-cpu.build/Release-iphoneos/Objects-normal/arm64/ggml-cpu-traits.o /Users/blainemiller/Documents/mine/Personal/Apps/Ari/llama.cpp-temp/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-traits.cpp normal arm64 c++ com.apple.compilers.llvm.clang.1_0.compiler (in target 'ggml-cpu' from project 'llama.cpp') CompileC /Users/blainemiller/Documents/mine/Personal/Apps/Ari/llama.cpp-temp/llama.cpp/build-ios-device/build/ggml-cpu.build/Release-iphoneos/Objects-normal/arm64/ggml-cpu-hbm.o /Users/blainemiller/Documents/mine/Personal/Apps/Ari/llama.cpp-temp/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-hbm.cpp normal arm64 c++ com.apple.compilers.llvm.clang.1_0.compiler (in target 'ggml-cpu' from project 'llama.cpp') CompileC /Users/blainemiller/Documents/mine/Personal/Apps/Ari/llama.cpp-temp/llama.cpp/build-ios-device/build/ggml-cpu.build/Release-iphoneos/Objects-normal/arm64/ggml-cpu-c0b5ffc4095df3ac83da75871c8389dd.o /Users/blainemiller/Documents/mine/Personal/Apps/Ari/llama.cpp-temp/llama.cpp/ggml/src/ggml-cpu/ggml-cpu.c normal arm64 c com.apple.compilers.llvm.clang.1_0.compiler (in target 'ggml-cpu' from project 'llama.cpp') CompileC /Users/blainemiller/Documents/mine/Personal/Apps/Ari/llama.cpp-temp/llama.cpp/build-ios-device/build/ggml-cpu.build/Release-iphoneos/Objects-normal/arm64/ggml-cpu-aarch64.o /Users/blainemiller/Documents/mine/Personal/Apps/Ari/llama.cpp-temp/llama.cpp/ggml/src/ggml-cpu/ggml-cpu-aarch64.cpp normal arm64 c++ com.apple.compilers.llvm.clang.1_0.compiler (in target 'ggml-cpu' from project 'llama.cpp') note: Run script build phase 'Generate CMakeFiles/ZERO_CHECK' will be run during every build because the option to run the script phase "Based on dependency analysis" is unchecked. (in target 'ZERO_CHECK' from project 'llama.cpp') The following build commands failed: ... -- The C compiler identification is unknown |
|
I am getting an error when I try to compile this. Lots of lines that look like I also see these lines in the output I should note that I tested this before you added mac support (which I am very happy about) and it worked fine. I haven't tried reinstalling all my brew packages because it seems a bit heavy since it was working. |
|
so I noticed one of the errors was about Disk IO errors so I moved the llama.cpp repo to a non iCloud Drive synced folder and now builds are consistent and mostly working each time. It fails to build macOS with the same issues @schlu pointed out every time. iOS and iOS sim seem to build ok now that Im not using a icloud synced folder. |
|
@schlu Thanks for reporting this, I'm looking into this now. |
|
Much worse now. I stops right away. I rolled back to commit 14d48be to make sure it was still working and it was still compiling the iOS version fine. Here is the entire output |
|
Ran into the same error as well. Didn't have time to test a previous commit though. |
Sorry about this (I am running the build locally and checking that the builds work in various projects to try to avoid wasting your time). I've pushed another commit which I hope will address this 🤞 |
|
It built fully this time with no errors I could see. |
Don't worry, thank you for your time. I am getting this error now: I went and downloaded the visonOS stuff in Xcode but the error persisted:
|
|
I do see this adding my app with the llama.xcframework to App Store Connect. And they emailed me these issues: Here the link they reference too: |
|
@schlu Can you tell me the version of cmake you are using? I'm asking because I think CMake added official support for visionOS in version 3.28.0. Based on this in the error you are seeing, perhaps that is the issue: System is unknown to cmake, create:
Platform/visionOS to use this system, please post your config file on discourse.cmake.org so it can be added to cmake
CMake Error at CMakeLists.txt:2 (project):If this is the case I'll add a check to the script. |
I've not tried this but I'll take a closer look at this tomorrow 👍 |
This commit adds the ability to create a GitHub release with the xcframework build artifact.
This commit adds scripts that can validate the iOS, macOS, tvOS, and VisionOS applications. The scripts create a simple test app project, copy the llama.xcframework to the test project, build and archive the app, create an IPA from the archive, and validate the IPA using altool. The motivation for this is to provide some basic validation and hopefully avoid having to manually validate apps in Xcode.
This commit removes the Package.swift file, as we are now building an XCFramework for the project.
danbev
force-pushed
the
xcframework-build-10747
branch
from
da0ea36 to
69a6d36
Compare
|
I've rebased this to hopefully make it a little easier to review after all the iterations. I'm not sure how we should handle the Package.swift issue mentioned in this comment. At the moment Package.swift has been removed by a commit in this PR. For local testing I've been using this as a Package.swift (not checked in): // swift-tools-version: 5.10
import PackageDescription
let package = Package(
name: "llama",
platforms: [
.iOS(.v14),
.macOS(.v10_15),
.visionOS(.v1)
],
products: [
.library(name: "llama.cpp", targets: ["llama-framework"])
],
targets: [
.binaryTarget(
name: "llama-framework",
//url: "https://github.com/ggml-org/llama.cpp/releases/download/bXXXX/llama-bXXXX-xcframework.zip",
//checksum: "the-sha256-checksum-here"
// The following can be used for local testing. Run the build-xcframework.sh
// script first to generate the XCFramework.
path: "build-apple/llama.xcframework"
)
]
)This was only to verify that I could create a swift project and use the framework as a dependency from it. Perhaps we can investigate this further in a follow up commit to figure out what the best way is to handle the checksum issue? |
|
@blaineam @tladesignz @schlu @ggerganov @slaren Thanks for all the testing and discussions! If possible it would be nice to this merged so people can start using it. I'll take care of any issues that might crop up (I'm thinking about the removal of Package.swift which I'm not sure if it will effect other projects that might be using it). The next release should then generate a // swift-tools-version: 5.10
// The swift-tools-version declares the minimum version of Swift required to build this package.
import PackageDescription
let package = Package(
name: "MyLlamaPackage",
targets: [
.executableTarget(
name: "MyLlamaPackage",
dependencies: [
"LlamaFramework"
]),
.binaryTarget(
name: "LlamaFramework",
url: "https://github.com/ggml-org/llama.cpp/releases/download/bxxxx/llama-bxxxx-xcframework.zip",
checksum: "<checksum>"
)
]
) |
|
Thank you @danbev, this turned out to be some massive amount of work! Well done! |
* llama : add xcframework build script This commit adds a script to build an XCFramework for Apple ios, macos, visionos, and tvos platforms. The generated XCFramework can then be added to a project and used in the same way as a regular framework. The llama.swiftui example project has been updated to use the XCFramework and can be started using the following command: ```console $ open examples/llama.swiftui/llama.swiftui.xcodeproj/ ``` Refs: ggml-org#10747 * examples : remove llama.cpp (source dir ref) from project.pbxproj This commit removes the reference to llama.cpp from the project.pbxproj file since Package.swift has been removed. * ci : updated build.yml to use build-xcframework.sh * ci : add xcframework build to github releases This commit adds the ability to create a GitHub release with the xcframework build artifact. * scripts : add apple app validation scripts This commit adds scripts that can validate the iOS, macOS, tvOS, and VisionOS applications. The scripts create a simple test app project, copy the llama.xcframework to the test project, build and archive the app, create an IPA from the archive, and validate the IPA using altool. The motivation for this is to provide some basic validation and hopefully avoid having to manually validate apps in Xcode. * llama : remove Package.swift This commit removes the Package.swift file, as we are now building an XCFramework for the project. * llama : remove Sources and spm-headers directories * llama : use TargetConditionals.h for visionOS/tvOS
|
Hi @danbev thank you for the great work you have done here! I was able to wrap the xcframework in a Swift package and use it on iOS. Unfortunately when I try to build it on MacOS I get a build error: Did you encountered a similar error by any chance? |
I have not run into this, at least not yet. I've pushed a very basic project that I'm able to build and run here. This does not do anything useful apart from accessing the llama.cpp api using the xcframework so it might not be "advanced" enough to reproduce the error you are seeing. If you can figure out how I can reproduce this then please let me know and I'll look into it. |
I have an iOS and macOS app on the App Store already that uses the .xcframework from this PR and it is working nicely. I ran into a similar error to that when trying to use any SPM workaround for llama.cpp including even this .xcframework but only in a response from submitting it for notarization and the debug guide Apple links you to passes every time without issues. I couldn't ever figure out why Direct Distribution Notarization is failing now so just gave up on it. App Store is good enough for my app. |
I am also seeing this error when trying to build a macOS app. Additionally, Xcode gives me this warning: I've added the xcframework to my app as a url binary target. I think the issue is with the structure of the framework I see that building the xcframework locally creates the expected symlink, so I think zipping the release artifact with the |
|
@mchew Thanks for figuring this out! |
This commit adds the --symlinks option to the zip command used to create the xcframework zip file. This is necessary to create symlinks in the zip file. Without this option, the Versions symlink is stored as a regular directory entry in the zip file, rather than as a symlink in the zip which causes the followig error in xcode: ```console Couldn't resolve framework symlink for '/Users/danbev/work/ai/llama.cpp/tmp_1/build-apple/llama.xcframework/macos-arm64_x86_64/llama.framework/Versions/Current': readlink(/Users/danbev/work/ai/llama.cpp/tmp_1/build-apple/llama.xcframework/macos-arm64_x86_64/llama.framework/Versions/Current): Invalid argument (22) ``` Refs: ggml-org#11996 (comment)
This commit adds the --symlinks option to the zip command used to create the xcframework zip file. This is necessary to create symlinks in the zip file. Without this option, the Versions symlink is stored as a regular directory entry in the zip file, rather than as a symlink in the zip which causes the followig error in xcode: ```console Couldn't resolve framework symlink for '/Users/danbev/work/ai/llama.cpp/tmp_1/build-apple/llama.xcframework/macos-arm64_x86_64/llama.framework/Versions/Current': readlink(/Users/danbev/work/ai/llama.cpp/tmp_1/build-apple/llama.xcframework/macos-arm64_x86_64/llama.framework/Versions/Current): Invalid argument (22) ``` Refs: #11996 (comment)
* llama : add xcframework build script This commit adds a script to build an XCFramework for Apple ios, macos, visionos, and tvos platforms. The generated XCFramework can then be added to a project and used in the same way as a regular framework. The llama.swiftui example project has been updated to use the XCFramework and can be started using the following command: ```console $ open examples/llama.swiftui/llama.swiftui.xcodeproj/ ``` Refs: ggml-org#10747 * examples : remove llama.cpp (source dir ref) from project.pbxproj This commit removes the reference to llama.cpp from the project.pbxproj file since Package.swift has been removed. * ci : updated build.yml to use build-xcframework.sh * ci : add xcframework build to github releases This commit adds the ability to create a GitHub release with the xcframework build artifact. * scripts : add apple app validation scripts This commit adds scripts that can validate the iOS, macOS, tvOS, and VisionOS applications. The scripts create a simple test app project, copy the llama.xcframework to the test project, build and archive the app, create an IPA from the archive, and validate the IPA using altool. The motivation for this is to provide some basic validation and hopefully avoid having to manually validate apps in Xcode. * llama : remove Package.swift This commit removes the Package.swift file, as we are now building an XCFramework for the project. * llama : remove Sources and spm-headers directories * llama : use TargetConditionals.h for visionOS/tvOS
This commit adds the --symlinks option to the zip command used to create the xcframework zip file. This is necessary to create symlinks in the zip file. Without this option, the Versions symlink is stored as a regular directory entry in the zip file, rather than as a symlink in the zip which causes the followig error in xcode: ```console Couldn't resolve framework symlink for '/Users/danbev/work/ai/llama.cpp/tmp_1/build-apple/llama.xcframework/macos-arm64_x86_64/llama.framework/Versions/Current': readlink(/Users/danbev/work/ai/llama.cpp/tmp_1/build-apple/llama.xcframework/macos-arm64_x86_64/llama.framework/Versions/Current): Invalid argument (22) ``` Refs: ggml-org#11996 (comment)
|
Thank you! Can confirm it now works on MacOS. |
This commit adds the --symlinks option to the zip command used to create the xcframework zip file. This is necessary to create symlinks in the zip file. Without this option, the Versions symlink is stored as a regular directory entry in the zip file, rather than as a symlink in the zip which causes the followig error in xcode: ```console Couldn't resolve framework symlink for '/Users/danbev/work/ai/llama.cpp/tmp_1/build-apple/llama.xcframework/macos-arm64_x86_64/llama.framework/Versions/Current': readlink(/Users/danbev/work/ai/llama.cpp/tmp_1/build-apple/llama.xcframework/macos-arm64_x86_64/llama.framework/Versions/Current): Invalid argument (22) ``` Refs: ggml-org/llama.cpp#11996 (comment)
This commit adds a script to build an XCFramework for Apple
ios, macos, visionos, and tvos platforms.
The generated XCFramework can then be added to a project and used in
the same way as a regular framework. The llama.swiftui example project
has been updated to use the XCFramework and can be started using the
following command:
$ open examples/llama.swiftui/llama.swiftui.xcodeproj/Refs: #10747