Roadmap June 2023

[ggerganov]

New roadmap format as Github project: https://github.com/users/ggerganov/projects/7

Outdated below

Previous: Roadmap May 2023

News

The ggml project has been funded:

• Announcement: https://twitter.com/ggerganov/status/1666120568993730561
• Homepage: https://ggml.ai

Tasks

• Refactoring pass

  Didn't get to this in May - should do this in June
  "There is a lot of code duplication in ggml.c which probably can be simplified with a good set of macros. The goal is to keep the code size manageable, while we avoid reaching "macro hell""

• Integrate recent efforts for training

  Amazing work by @xaedes continues to impress: Train Text from scratch #1652
  Ultimately, with the ability to train mini models, I am interested in making a small prototype of the following idea for faster inference: Combine large LLM with small LLM for faster inference #630 (comment)

• Integrate recent efforts in improving the threading of ggml

  Some very good points and analysis in Fine tune MUL_MAT, new threading (spin+wait/notify), speedup q_f32 BLAS by splitting COMPUTE stage #1632
  Will look into integrating most of the stuff into ggml to try and improve the CPU performance further

• Extend Metal shaders to support other quantizations + optimize performance

  Currently, the Metal implementation supports just Q4_0 and F16. Also, the existing implementation is probably far from optimal. More info: llama : Metal inference #1642
  Very good field for contributions

• Implement inference of new models

  There are already some very interesting models that should be supported by ggml:

  • Segment Anything Model (SAM)
    Still working on the Encoder - progress is a bit slow due to several new operators involved, but I think it is slowly working out: examples : add sample SAM inference ggml#74

  • Falcon

  • Bark (text-to-speech)

• Advance the community effort for unified ggml model format

  This work has been recently initiated and aims to provide a future-proof file format for ggml models:
  ggml : unified file format ggml#220

• Add llama_state

  See past Roadmaps - have been postponing this for quite some time. See Roadmap May 2023 #1220 (reply in thread) if interested in giving it a try

• Add GPU backend prototypes following the Metal example

  For example, it would be interesting if we can add a WebGPU or Vulkan backends in a similar way as we did with Metal. I'm completely unfamiliar with the details of these frameworks, but I'm hoping that people might be interested in giving it a try

[AlphaAtlas]

For example, it would be interesting if we can add a WebGPU or Vulkan backends in a similar way as we did with metal.

Are there Metal-like zero copy mechanisms in either of these frameworks? It seems like a necessity for IGPs.

Integrate recent efforts for training

Maybe the recent MeZO (forward pass only) training paper is relevant to this effort? https://github.com/princeton-nlp/MeZO

[sandorkonya]

@niklaskorz i saw your comment here - any thoughts on the task "Add GPU backend prototypes following the Metal example" with Vulkan / WebGPU?

[bullno1]

For llama_state, is it safe to say that all the states touched in llama_copy_state_data should be isolated into a new struct?

What would be come of llama_copy_state_data and llama_set_state_data then? llama_serialize_state and llama_deserialize_state?

Context: I plan to put this behind a grpc service so per-client state is needed. Currently, state-switching is done via llama_set_state_data.

Edit: Actually there are also those metric variables like t_sample_us I guess those should be moved to llama_state too?

[bullno1]

Ok, so I have started refactoring into llama_state.

I think I will leave metrics inside llama_context.

They are mostly informational and has no bearings on the output.

I don't want to duplicate all the sampling functions.
They are basically context-free and currently, the context is only used to record metrics.

The duplication is not that dramatic: https://github.com/bullno1/llama.cpp/blob/llama_state/llama.h

[howard0su]

do we consider to switch ggml.c to ggml.cpp so that we can leverage template instead of macro to simplify the code?

[nightscape]

@philpax would switching to C++ break the current approach of https://github.com/rustformers/llm/ ?

[philpax]

It wouldn't break it (especially if the ABI remains C), but adding C++ means that a C++ toolchain is required, which increases the number of build dependencies. We personally encountered this with our use of a bindings generator which required C++ and caused problems for some Fedora users.

I'd suggest keeping it in C and offering a C++ wrapper; that's what most libraries in this situation do when they want a C++-friendly interface.

[wtarreau]

@hfassold

I would strongly advocate to go to C++, if only for the templates ...

On the opposite, C++ hinders contributions. Virtually every developer can understand and modify C as everything is explicit, there's no magic; but much less are able to even just parse C++ which is cryptic by nature. I would instead advocate for dropping the few bits of C++ from llama.cpp to make it a more portable and more accessible full-C project!

[clort81]

I would strongly advise to not

[DeveloperPaul123]

Switching to C++ would be great in my opinion. I don't do much work with C so switching to C++ would actually allow me to contribute. Right now, it's a bit more effort for me to reason about the code though I can't say I've spent too much time with it.

[nivibilla]

Hey could batch Inference be a task to add for next month maybe? I think it would really help with using this at scale.

[ggerganov]

Batch inference has been demonstrated as part of #1360:

llama.cpp/examples/baby-llama/baby-llama.cpp

Lines 768 to 777 in 16b9cd1

	struct ggml_tensor * forward_batch(
	struct llama_model * model,
	struct llama_kv_cache * cache,
	struct ggml_context * ctx0,
	struct ggml_cgraph * gf,
	struct ggml_tensor * tokens_input,
	const int n_tokens,
	const int n_past,
	const int n_batch) {

Although I haven't played with the code in too much detail, so not 100% sure how feasible for general application it is yet.
If you give it a try, definitely let us know your experience

[nivibilla]

Ah I see. Thanks !

[nivibilla]

Btw, I don't know any cpp. So I'm as good as llama coding itself lol

[ziwang-com]

https://github.com/ziwang-com/AGM/issues/155
llamacpp归一化与AGM阿格姆项目

[dennislysenko]

Hey @ggerganov, I'm having trouble finding references to text to speech in the new roadmap project. Is that still in the plan?

[ggerganov]

The latest update was @PABannier implementing Meta's Encodec codec with ggml which was an important step forward. I don't think I'll be able to work on bark before finishing the existing roadmap, but if someone else is already working on it, I can add it to the roadmap and reference the respective project.

[PABannier]

I'm working on bark.cpp.