enable CPU HBM

[jikunshang]

this pr try to enable CPU HBM with memkind library to allocate hbm memory.

see #2602

[netrunnereve]

Are you seeing any performance improvements?

[jikunshang]

Are you seeing any performance improvements?

sorry for late reply, I just got test machine recently.
On Xeon CPU Max 9462, model use llama2-7b:
baseline inference performance is 124 ms/token,
with this enhancement and HBM enabled, inference performance is 87 ms/token, which can get about 40% perf gain.

[jikunshang]

hi @ggerganov, can you take a review at your convenience?

[hydroo]

Sorry for my ignorance.
Why doesn't the system allocate on HBM without this change?
Is this a system with DDR and HBM, and some BIOS setting (I vaguely remember caching vs other modes (For Optane and Knights* chips)) makes it such that the system either prioritizes DDR or even never touches HBM?
I'm asking, because it's unintuitive to me that a system wouldn't perhaps prioritize using HBM over DDR.

[jikunshang]

Sorry for my ignorance. Why doesn't the system allocate on HBM without this change? Is this a system with DDR and HBM, and some BIOS setting (I vaguely remember caching vs other modes (For Optane and Knights* chips)) makes it such that the system either prioritizes DDR or even never touches HBM? I'm asking, because it's unintuitive to me that a system wouldn't perhaps prioritize using HBM over DDR.

Yes, you are right.
Actually, there are 3 kinds memory mode for Xeon Max Cpu serious: HBM only memory mode, Flat memory mode(1LM), Cache memory mode(2LM). For cache memory mode, it will work as you describe, but it will lock of fine grained memory management since all HBM memory are transparent, like L4 cache.
This code change is target for Flat memory mode, HBM and DDR are exposed to software as separate address space. we can use HBM on demand.
More details about HBM configuration can be found here

[ggerganov]

Merge if CI passes

[jikunshang]

Hi @ggerganov can you approve again for workflows? thanks!

[kunger97]

Does HBM compile the default in LLAMA.CPP or do you need to specify during compilation?