HLS 2.3.0.0 release possibly borked on Windows - `libstdc++-6.dll` not found

[MorrowM]

I've had two people in the FP Discord haskell-tooling channel having issues with HLS 2.3.0.0 on Windows. After some troubleshooting we see there's an issue involving libstdc++-6.dll not being found.

Your environment

Which OS do you use?
Windows 11
Which version of GHC do you use and how did you install it?
9.4.7
How is your project built (alternative: link to the project)?
(Not relevant)
Which LSP client (editor/plugin) do you use?
VSCode+vscode-haskell
Which version of HLS do you use and how did you install it?
2.3.0.0 with ghcup install hls 2.3.0.0
Have you configured HLS in any way (especially: a hie.yaml file)?
No

Steps to reproduce

Simply run haskell-language-server-wrapper-2.3.0.0.exe in a cmd window (powershell crashes silently, cmd shows an actual error).

Expected behaviour

Debug output is displayed.

Actual behaviour

We get the following error window popping up:

Additional information

This manifests in VSCode by displaying "The Haskell (PF) server crashed 5 times in the last 3 minutes. The server will not be restarted." The output pane shows:

(Sorry for the screenshot, this is what I have from the conversation.)

I suspect #3820 might be related (possibly they had 2.2.0.0 set, which is why running haskell-language-server-wrapper didn't reveal the issue).

[Bodigrim]

(Just passing by)
Check which hls dependencies involve C++, which manifests in Cabal files as extra-libraries: stdc++. A proper way to depend on it is fairly arcane:

    if impl(ghc >= 9.4)
      build-depends: system-cxx-std-lib == 1.0
    elif os(darwin) || os(freebsd)
      extra-libraries: c++
    elif os(openbsd)
      extra-libraries: c++ c++abi pthread
    elif os(windows)
      -- GHC's Windows toolchain is based on clang/libc++ in GHC 9.4 and later
      if impl(ghc < 9.3)
        extra-libraries: stdc++
      else
        extra-libraries: c++ c++abi
    elif arch(wasm32)
      cxx-options: -fno-exceptions
      extra-libraries: c++ c++abi
    else
      extra-libraries: stdc++

If a package violates this (e. g., declares extra-libraries: stdc++ unconditionally), the bindist is likely to be broken, especially on Windows.

[michaelpj]

cc @wz1000

[KripkesBeard]

I have the exact same issue (Windows 10 but otherwise same environment) and have the same output logs, with the exception that haskell-language-server-wrapper-2.3.0.0.exe in cmd crashes with the text "The application was unable to start correctly (0xc000007b). Click OK to close the application." as an application error, not a systems error.

I had hls 2.2.0.0 set as default without 2.3.0.0 installed, and when I opened a Haskell project in VSCode it prompted me to install 2.3.0.0, which I did, and after installing it has been crashing on start up with the same output as the screenshot above.

Edit:

I uninstalled 2.3.0.0 and tried to reproduce the error to see what the HLS logs are if I said no to installing 2.3.0.0 and this is what I got

[client] INFO Finding haskell-language-server
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose upgrade' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose list -t hls -c installed -r' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose list -t cabal -c installed -r' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose list -t stack -c installed -r' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose whereis hls 2.2.0.0' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose whereis cabal 3.8.1.0' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose whereis stack 2.9.1' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Executing 'ghc.exe --numeric-version' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose run --hls 2.2.0.0 --cabal 3.8.1.0 --stack 2.9.1 --install' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Working out the project GHC version. This might take a while...
[client] INFO Executing 'haskell-language-server-wrapper --project-ghc-version' in cwd 'd:\GitHub\fragments'
[client] INFO The GHC version for the project or file: 9.4.7
[client] INFO Platform constants: Windows, A_64
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose list -t hls -c installed -r' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose whereis bindir' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose whereis hls 2.3.0.0' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] ERROR Error executing 'ghcup --no-verbose whereis hls 2.3.0.0' with error code 30
[client] ERROR stderr: esc[1m[ Error ]esc[0m [sec]8;;https://errors.haskell.org/messages/GHCup-00130esc\GHCup-00130esc]8;;sec] The version 2.3.0.0 of the tool hls is not installed.

which prompts me to install 2.3.0.0 and when I say no

[client] INFO Checking for ghcup installation
[client] INFO found ghcup at ghcup
[client] INFO Executing 'ghcup --no-verbose whereis ghc 9.4.7' in cwd 'C:\Users\Peter\AppData\Local\Programs\Microsoft VS Code'
[client] ERROR Error: Project requires HLS but it isn't installed
at c:\Users\Peter.vscode\extensions\haskell.haskell-2.4.1\dist\extension.js:1:61763
at Array.forEach (<anonymous>)
at c:\Users\Peter.vscode\extensions\haskell.haskell-2.4.1\dist\extension.js:1:61709
at Generator.next (<anonymous>)
at s (c:\Users\Peter.vscode\extensions\haskell.haskell-2.4.1\dist\extension.js:1:49145)

[angerman]

Even

-- GHC's Windows toolchain is based on clang/libc++ in GHC 9.4 and later

from @Bodigrim's comment is not necessarily true. The toolchain GHC bindists for windows ship, ship by default for UCRT, not MSVCRT from 9.4 onwards anymore. However this does not mean GHC 9.4+ can't be built against MSVCRT anymore. The whole assumption that GHC and the toolchain are bundled is deeply flawed imo. Yes, we do ship a specific toolchain with GHC in our windows binary distributions, but making the universal assumption that everyone depends on this is some tight coupling I'd rather we don't do.

Also you can use UCRT but with the Gnu toolchain. MSYS2 classifies those environments as UCRT64 (ucr64 + ghc) and CLANG64 (ucrt64 + llvm).

There is system-cxx-std-lib in recent GHCs trying to work around this to some degree.

[andreabedini]

I tried to find the misbeaving package that imports stdc++ unconditionally, as @Bodigrim suggests.

I downloaded the install plans for the release from https://github.com/haskell/haskell-language-server/releases/download/2.3.0.0/plan_json.tar and used a simple script to download all the cabal files involved.

$ mkdir x86_64-mingw64-ghc-9.4.7-plan; cat x86_64-mingw64-ghc-9.4.7-plan.json | jq -r '."install-plan"[] | select(."pkg-cabal-sha256") | [.id, ."pkg-cabal-sha256"] | @tsv' | parallel --bar --colsep '\t' 'curl --silent -o x86_64-mingw64-ghc-9.4.7-plan/{1}.cabal https://casa.stackage.org/{2}'

... but grepping for stdc++ does not return anything 🤔

PS: perhaps we should upload the build-infos too!

[angerman]

we could just ship the libstdc++6.dll alongside.

[GuenterHummel]

libgcc_s_seh-1. dll and libwinpthread-1.dll are also missing, I think. it was linked dynamically and not statically as usual

[Bodigrim]

@andreabedini text does depend on stdc++, so at least one package should have been found by grep.

Which binaries are affected? This can be checked with ldd.

[wz1000]

Here's a summary of the problem

• HLS 2.3 is the first version to ship with text-2.0 for GHC 9.2 and 9.0
• We link system libraries dynamically on windows
• It is currently completely non-deterministic which version of hls-wrapper we ship (i.e. whichever job finishes the last, we ship that binary)

On GHC 9.0 and 9.2, text links against stdc++ directly, on 9.4+ it depends on system-cxx-lib

As a result, the haskell-language-server-9.0.2, haskell-language-server-9.2.8 and haskell-language-server-wrapper executables are broken.

Possible sollutions are:

1. Link system libraries statically on windows, like we already do Haskell libraries (by setting static: True in the cabal.project
2. avoid c++ dependency by using text -simdutf
3. Ship stdc++.dll along with the binaries, but this might be complicated because we might want to ship different versions for each executable depending on what toolchain/ghc version they support

1 or 2 seem the easiest to do. We should also modify CI to ensure that we can control which version of haskell-language-server-wrapper (i.e. which GHC it is built with) we ship.

[andreabedini]

@andreabedini text does depend on stdc++, so at least one package should have been found by grep.

@Bodigrim Good point. It turns out that plan uses the pre-installed text, so text's cabal file is not specified in the plan.

On GHC 9.0 and 9.2, text links against stdc++ directly, on 9.4+ it depends on system-cxx-lib

Doesn't what I wrote above plus this imply that the problem could be with the ghc bindist?
We have evicence that hls-2.3.0.0 for x86_64-mingw64-ghc-9.4.7 was compiled using the pre-installed text library from the GHC binary distribution:

❯ cabal-plan-filter.jq pkg-name:text < x86_64-mingw64-ghc-9.4.7-plan.json
{
  "type": "pre-existing",
  "id": "text-2.0.2",
  "pkg-name": "text",
  "pkg-version": "2.0.2",
  "depends": [
    "array-0.5.4.0",
    "base-4.17.2.0",
    "binary-0.8.9.1",
    "bytestring-0.11.5.2",
    "deepseq-1.4.8.0",
    "ghc-prim-0.9.1",
    "template-haskell-2.19.0.0"
  ]
}

And, if I am not mistaken, text depends on system-cxx-std-lib on ghc-9.4

    if impl(ghc >= 9.4)
      build-depends: system-cxx-std-lib == 1.0

Therefore stdc++ should be part of the GHC bindist?

I am confused.

[wz1000]

@andreabedini the haskell-language-server-9.4.7 binary seems to work for me, the 9.2 and 9.0 binaries are the ones that seem broken. Is this not the case for you?

[andreabedini]

@wz1000 I am on linux :) I was only trying to troubleshoot. I picked x86_64-mingw64-ghc-9.4.7 from the information in the OP.