Nvidia extension #2882

Realmbird · 2024-05-09T02:26:10Z

Describe Your Changes

Fixes Issues

Closes feat: support Nvidia NIMs remote API #2854
Closes #

Self Checklist

Added relevant comments, esp in complex areas
Updated docs (for bug fixes / features)
Created issues for follow-up changes or refactoring needed

…lder.ts

Realmbird · 2024-05-09T02:45:39Z

Currently has error with api so not 100% working
2024-05-08T20:29:43.223Z [SPECS]::OS Version: #21-Ubuntu SMP PREEMPT_DYNAMIC Wed Apr 3 14:54:32 UTC 2024
2024-05-08T20:29:43.223Z [SPECS]::OS Platform: linux
2024-05-08T20:29:43.224Z [SPECS]::OS Release: 6.5.0-1020-oem
2024-05-08T20:29:43.224Z [APP]::{"notify":true,"run_mode":"cpu","nvidia_driver":{"exist":false,"version":""},"cuda":{"exist":false,"version":""},"gpus":[],"gpu_highest_vram":"","gpus_in_use":[""],"is_initial":false,"vulkan":false}
2024-05-08T20:31:21.952Z [SPECS]::Version: 0.1.3
2024-05-08T20:31:21.954Z [SPECS]::Parallelism: 16
2024-05-08T20:31:21.954Z [SPECS]::Machine: x86_64
2024-05-08T20:31:21.955Z [SPECS]::Free Mem: 5712773120
2024-05-08T20:31:21.953Z [SPECS]::CPUs: [{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1554,"times":{"user":437700,"nice":210,"sys":144670,"idle":5604050,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1425,"times":{"user":442640,"nice":410,"sys":145430,"idle":5642180,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1393,"times":{"user":525600,"nice":440,"sys":174950,"idle":5498940,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1057,"times":{"user":344370,"nice":980,"sys":110540,"idle":5774290,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1572,"times":{"user":578500,"nice":270,"sys":212800,"idle":5373700,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1569,"times":{"user":349780,"nice":380,"sys":78640,"idle":5826690,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1106,"times":{"user":600050,"nice":2300,"sys":171860,"idle":5413480,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1565,"times":{"user":369180,"nice":80,"sys":77430,"idle":5806010,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1495,"times":{"user":381150,"nice":120,"sys":133730,"idle":5671520,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1491,"times":{"user":305630,"nice":230,"sys":101980,"idle":5820330,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1332,"times":{"user":253360,"nice":190,"sys":86110,"idle":5887660,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1610,"times":{"user":213020,"nice":140,"sys":78250,"idle":5804210,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1380,"times":{"user":183650,"nice":70,"sys":68400,"idle":5987400,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1303,"times":{"user":159840,"nice":110,"sys":60630,"idle":5985440,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":869,"times":{"user":135100,"nice":40,"sys":53880,"idle":6049800,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":400,"times":{"user":134800,"nice":50,"sys":82380,"idle":5989210,"irq":0}}]
2024-05-08T20:31:21.954Z [SPECS]::Endianness: LE
2024-05-08T20:31:21.955Z [SPECS]::Total Mem: 16453881856
2024-05-08T20:31:21.955Z [SPECS]::OS Version: #21-Ubuntu SMP PREEMPT_DYNAMIC Wed Apr 3 14:54:32 UTC 2024
2024-05-08T20:31:21.955Z [SPECS]::OS Platform: linux
2024-05-08T20:31:21.955Z [SPECS]::OS Release: 6.5.0-1020-oem
2024-05-08T20:31:21.956Z [APP]::{"notify":true,"run_mode":"cpu","nvidia_driver":{"exist":false,"version":""},"cuda":{"exist":false,"version":""},"gpus":[],"gpu_highest_vram":"","gpus_in_use":[""],"is_initial":false,"vulkan":false}
2024-05-08T20:33:18.601Z [SPECS]::Version: 0.1.3
2024-05-08T20:33:18.603Z [SPECS]::CPUs: [{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1200,"times":{"user":456710,"nice":210,"sys":147550,"idle":5685260,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1152,"times":{"user":457910,"nice":410,"sys":147460,"idle":5729540,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1043,"times":{"user":545950,"nice":440,"sys":177970,"idle":5579800,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":548,"times":{"user":361140,"nice":980,"sys":113890,"idle":5857830,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1199,"times":{"user":604740,"nice":280,"sys":216150,"idle":5446530,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1152,"times":{"user":363980,"nice":380,"sys":80460,"idle":5915900,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1152,"times":{"user":624850,"nice":2300,"sys":175120,"idle":5489580,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1191,"times":{"user":382580,"nice":80,"sys":79210,"idle":5895960,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1178,"times":{"user":398920,"nice":120,"sys":136300,"idle":5756060,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1152,"times":{"user":318760,"nice":230,"sys":104100,"idle":5910080,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1173,"times":{"user":264240,"nice":190,"sys":88090,"idle":5979450,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1300,"times":{"user":222250,"nice":140,"sys":79950,"idle":5894270,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1139,"times":{"user":191930,"nice":70,"sys":69950,"idle":6082250,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":981,"times":{"user":166800,"nice":110,"sys":62050,"idle":6081870,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1080,"times":{"user":140930,"nice":40,"sys":55320,"idle":6147460,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1200,"times":{"user":140130,"nice":50,"sys":84060,"idle":6086660,"irq":0}}]
2024-05-08T20:33:18.603Z [SPECS]::Machine: x86_64
2024-05-08T20:33:18.603Z [SPECS]::Endianness: LE
2024-05-08T20:33:18.604Z [SPECS]::Free Mem: 6521778176
2024-05-08T20:33:18.604Z [SPECS]::Parallelism: 16
2024-05-08T20:33:18.605Z [SPECS]::Total Mem: 16453881856
2024-05-08T20:33:18.605Z [SPECS]::OS Version: #21-Ubuntu SMP PREEMPT_DYNAMIC Wed Apr 3 14:54:32 UTC 2024
2024-05-08T20:33:18.605Z [SPECS]::OS Platform: linux
2024-05-08T20:33:18.605Z [SPECS]::OS Release: 6.5.0-1020-oem
2024-05-08T20:33:18.605Z [APP]::{"notify":true,"run_mode":"cpu","nvidia_driver":{"exist":false,"version":""},"cuda":{"exist":false,"version":""},"gpus":[],"gpu_highest_vram":"","gpus_in_use":[""],"is_initial":false,"vulkan":false}
2024-05-08T20:43:21.640Z [SPECS]::Version: 0.1.3
2024-05-08T20:43:21.643Z [SPECS]::Machine: x86_64
2024-05-08T20:43:21.642Z [SPECS]::CPUs: [{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1531,"times":{"user":584980,"nice":210,"sys":169520,"idle":6038470,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1516,"times":{"user":602250,"nice":410,"sys":164930,"idle":6075880,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1595,"times":{"user":683010,"nice":440,"sys":200950,"idle":5925530,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1585,"times":{"user":496600,"nice":980,"sys":133890,"idle":6210490,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1550,"times":{"user":753230,"nice":290,"sys":239340,"idle":5775130,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1591,"times":{"user":491780,"nice":380,"sys":96270,"idle":6283520,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1587,"times":{"user":780560,"nice":2300,"sys":198300,"idle":5815930,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":400,"times":{"user":523310,"nice":80,"sys":94390,"idle":6251480,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1502,"times":{"user":541080,"nice":120,"sys":160100,"idle":6097940,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1585,"times":{"user":449380,"nice":230,"sys":126300,"idle":6266100,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1537,"times":{"user":384940,"nice":190,"sys":109480,"idle":6345230,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1467,"times":{"user":335200,"nice":140,"sys":100300,"idle":6257740,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1437,"times":{"user":297710,"nice":90,"sys":89190,"idle":6467120,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1558,"times":{"user":267040,"nice":110,"sys":79440,"idle":6472750,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1213,"times":{"user":235600,"nice":60,"sys":72110,"idle":6545870,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":916,"times":{"user":228200,"nice":50,"sys":101510,"idle":6489390,"irq":0}}]
2024-05-08T20:43:21.643Z [SPECS]::Endianness: LE
2024-05-08T20:43:21.643Z [SPECS]::Parallelism: 16
2024-05-08T20:43:21.644Z [SPECS]::Total Mem: 16453881856
2024-05-08T20:43:21.644Z [SPECS]::Free Mem: 7386509312
2024-05-08T20:43:21.644Z [SPECS]::OS Version: #21-Ubuntu SMP PREEMPT_DYNAMIC Wed Apr 3 14:54:32 UTC 2024
2024-05-08T20:43:21.644Z [SPECS]::OS Platform: linux
2024-05-08T20:43:21.645Z [SPECS]::OS Release: 6.5.0-1020-oem
2024-05-08T20:43:21.645Z [APP]::{"notify":true,"run_mode":"cpu","nvidia_driver":{"exist":false,"version":""},"cuda":{"exist":false,"version":""},"gpus":[],"gpu_highest_vram":"","gpus_in_use":[""],"is_initial":false,"vulkan":false}
2024-05-08T20:52:45.693Z [NITRO]::Debug: Request to kill Nitro
2024-05-09T02:21:27.004Z [SPECS]::Version: 0.1.3
2024-05-09T02:21:27.007Z [SPECS]::Parallelism: 16
2024-05-09T02:21:27.007Z [SPECS]::Free Mem: 7490146304
2024-05-09T02:21:27.007Z [SPECS]::Total Mem: 16453873664
2024-05-09T02:21:27.007Z [SPECS]::OS Version: #21-Ubuntu SMP PREEMPT_DYNAMIC Wed Apr 3 14:54:32 UTC 2024
2024-05-09T02:21:27.005Z [SPECS]::CPUs: [{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1200,"times":{"user":220820,"nice":260,"sys":51320,"idle":1255730,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1200,"times":{"user":242720,"nice":160,"sys":44420,"idle":1250390,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1190,"times":{"user":246050,"nice":280,"sys":51440,"idle":1232990,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1200,"times":{"user":230600,"nice":190,"sys":39590,"idle":1270120,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1176,"times":{"user":227100,"nice":120,"sys":53070,"idle":1250470,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1195,"times":{"user":240450,"nice":160,"sys":40460,"idle":1260770,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1200,"times":{"user":257570,"nice":1690,"sys":70730,"idle":1198150,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":400,"times":{"user":190640,"nice":1830,"sys":40990,"idle":1309080,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1200,"times":{"user":241080,"nice":70,"sys":48160,"idle":1222510,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1200,"times":{"user":220180,"nice":90,"sys":43840,"idle":1275730,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1200,"times":{"user":201520,"nice":30,"sys":40660,"idle":1297770,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1200,"times":{"user":185680,"nice":90,"sys":40520,"idle":1262890,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1199,"times":{"user":172820,"nice":90,"sys":38320,"idle":1329700,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1215,"times":{"user":160550,"nice":50,"sys":36710,"idle":1341580,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1225,"times":{"user":146740,"nice":40,"sys":34590,"idle":1360470,"irq":0}},{"model":"13th Gen Intel(R) Core(TM) i5-1340P","speed":1181,"times":{"user":136090,"nice":50,"sys":37920,"idle":1362040,"irq":0}}]
2024-05-09T02:21:27.006Z [SPECS]::Endianness: LE
2024-05-09T02:21:27.006Z [SPECS]::Machine: x86_64
2024-05-09T02:21:27.008Z [SPECS]::OS Platform: linux
2024-05-09T02:21:27.008Z [SPECS]::OS Release: 6.5.0-1020-oem
2024-05-09T02:21:27.008Z [APP]::{"notify":true,"run_mode":"cpu","nvidia_driver":{"exist":false,"version":""},"cuda":{"exist":false,"version":""},"gpus":[],"gpu_highest_vram":"","gpus_in_use":[""],"is_initial":false,"vulkan":false}

hiro-v · 2024-05-09T02:49:53Z

Community user support thread in: https://discord.com/channels/1107178041848909847/1236435708881797211

namchuai · 2024-06-04T05:24:03Z

@Realmbird , I think this one can be closed since #2934 had been merged. Please reopen if you think otherwise.

Realmbird added 6 commits May 8, 2024 19:08

Update name of package.json

ad32e2c

copy of read me

6e2b7c5

inital config from copying mistral

39c3e79

Adding 2 nvidia models for testing

dc64367

Changing nvidia extension have comment and questions in file

af86ab9

updated setting.json

fd63ced

Realmbird marked this pull request as draft May 9, 2024 02:26

Bearer key not needed already in core/src/node/api/restful/helper/bui…

439daa2

…lder.ts

Realmbird mentioned this pull request May 9, 2024

feat: support Nvidia NIMs remote API #2854

Open

hiro-v requested a review from louis-jan May 9, 2024 02:41

hiro-v assigned Realmbird May 9, 2024

Realmbird added 3 commits May 8, 2024 23:45

Update extension name

e53af1c

Adding more fields to model run time parameters

7e3da1e

Changing folder creation to allow for / first attempt

400b015

namchuai closed this Jun 4, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nvidia extension #2882

Nvidia extension #2882

Realmbird commented May 9, 2024

Realmbird commented May 9, 2024

hiro-v commented May 9, 2024

namchuai commented Jun 4, 2024

Nvidia extension #2882

Nvidia extension #2882

Conversation

Realmbird commented May 9, 2024

Describe Your Changes

Fixes Issues

Self Checklist

Realmbird commented May 9, 2024

hiro-v commented May 9, 2024

namchuai commented Jun 4, 2024