attempt to fix CI - continued

[deronnax]

Grmmm, I can't reopen #1658 because I force-pushed in between. I got a pretty good result, most job working.
Only test pypy-3.10 is failing, because a missing mdb.c file, at the installation of lmdb.
Since I noticed that zlmdb is a crossbar.io project, I thought you might be better informed than me on the problem.

Oh, btw, I took the liberty to change apt for apt-get, because there was warning: apt is a high level interface for humans, it's not for automation, and it's not meant to be stable in its interface as well.

Collecting lmdb>=1.4.0 (from zlmdb>=21.2.1->autobahn==24.4.2)
  Downloading lmdb-1.6.2.tar.gz (881 kB)
     ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 881.4/881.4 kB 83.8 MB/s eta 0:00:00
  Preparing metadata (setup.py): started
  Preparing metadata (setup.py): finished with status 'error'
  error: subprocess-exited-with-error
  
  × python setup.py egg_info did not run successfully.
  │ exit code: 1
  ╰─> [74 lines of output]
      patching file lmdb.h
      patching file mdb.c
      cc1: fatal error: build/lib/mdb.c: No such file or directory
      ```

[oberstet]

thanks a lot!

so the PR did run this time, and in fact succeeded for some of the workflows, CPython, both 3.9 and 3.11. great!

the pypy workflow failed unfortunately. welcome to the world of complex dependencies:(

cffi.VerificationError: CompileError: command '/usr/bin/gcc' failed with exit code 1
...

https://github.com/crossbario/autobahn-python/actions/runs/15168210434/job/42662700146?pr=1659#step:6:197

so sth is failing when building LMDB on PyPy on this setup

let me preface, in general, these kind of issues are "typical", tracking down and fixing is unfortunately necessary, and it is both annoying and time consuming - but can be fixed usually - at least my experience of spending countless hours historically with CI/CD and build stuff like this.

it is a consequence of Autobahn supporting both CPython and PyPy, and having non trivial dependencies.

I haven't looked in detail or tried to track down this one, but a casual glance caught my eye on 2 things:

the kind of C compiler might be relevant, this is gcc not llvm, the kind of how Python - in this case PyPy - builds extension modules from sources might be relevant. the recommended and best way (in my experience, and in general) with PyPy is:

make the build use CFFI (https://cffi.readthedocs.io/en/latest/) whenever possible, not CPyExt stuff

too long to explain here, but: while Python itself (the language and run-time) is fairly "standardized", the extension API is not! that is, on CPython, the "standard" is only defacto, and that means, simply whatever API the (one) implementation of CPython happened to use since it exists

CFFI is a modern effort, and also one that stems from PyPy, it is tricky to explain all the bells and whistles and details in just a paragraph, but here is an intro to the problem (which isn't related to Autobahn specifically)

https://pypy.org/posts/2018/09/inside-cpyext-why-emulating-cpython-c-8083064623681286567.html

I would recommend tracking it down by: install PyPy on your local dev PC (I assume Linux), and then build LMDB from sources there - with gcc (as this build unsuccessfully tried to do), or llvm, or ..

get LMDB from sources working on a local PyPy (one directly from upstream https://pypy.org/download.html, on Ubuntu)

sorry. I know it sucks. this is why CI/CD of Autobahn and Crossbar.io is non-trivial

on the plus side: the know how gained would be pretty useful in general for you, not only in relation to Autobahn, but Python in general! I'd say. and in any case: thanks a lot for helping! I (at this time) sadly don't have time, and those kind of issues is exactly the reason why it is "not done in a minute", always stuff like this comes up with sth looking "trivial" at first.

final side note: I still need to manually approve every workflow run, it will be gone once a first PR by you would be merged to main - so if you are tired of waiting for me, you might create another PR with some trivial change like just 1 char changed on the README or similar, which I could simply merge as is - I don't know how to configure GH manually/explicitly to allow before a first merged PR (lemme know if you do, I would do) ..

[meejah]

@oberstet I think you could just manually invite them as "contributor" in "settings -> collaborators" but also your proposed solution seems easy too ;)

[oberstet]

Since I noticed that zlmdb is a crossbar.io project, I thought you might be better informed than me on the problem.

yes, pls see above, try locally. LMDB is required. in Autobahn. also. and Crossbar.io. which builds on that in 2 stages:

https://github.com/crossbario/zlmdb
https://github.com/crossbario/cfxdb

too long to type, if you like, we could have a call and I can explain. this is advanced stuff, but necessary (for Crossbar.io absolutely, and it won't change/disappear).

LMDB is an in-memory (persistant) transactional database which is awesome and high performance, and Crossbar.io uses it for various advanced features, like eg WAMP event persistency etc ..

[oberstet]

ahh, hi @meejah ! long time;)

thanks for your note, if I could add @deronnax to be allowed to run workflows on AB even for the first PR manually - and only that permission - I would do!

can't find it though ..

[meejah]

I think you want "Add People" and then "invite" them, and give "Triage" role ... although GitHub is so inconsistent sometimes, it's entirely possible the org I'm looking at is different from this one 😮‍💨

[oberstet]

done. even though I'm also not entirely sure what it means .. but ... anyways, let's try it;)

[oberstet]

another route, which I wanted to try at some point: I forgot which AI, but I've recently read about being able to allow/make an AI directly work on issues/code in a GH repo - including adding new tests, docs and everything.

IF it would work, that would be cool/helpful!

as in: pls fix all open issues in Autobahn;)

or even only: create a new PR for AB with LMDB and PyPy working. I mean, it's crazy, but hey, if it can come up with a proper PR 100% green and we can look before merge, why not?

it's crazy, I think it was in the context of https://modelcontextprotocol.io/introduction ... do you wanna give AI full SSH and DBA access? maybe not. maybe only on confined systems. but the promise - of not having to spend hours and hours fiddling with deps and details .. sounds attractive?

[dimbleby]

You could try asking lmdb to publish a suitable wheel. Then neither you nor anyone else would have to figure out how to build that project from source.

They publish lots of wheels, it is very likely an accident that they've missed one for pypy+linux.

[deronnax]

I had the idea, but I went to check: they don't do pypy-linux build, that would be a bit of a strech to ask. Is it even a thing?

[dimbleby]

Is it even a thing?

yes, I really think it is likely only an accident that lmdb omits this