How to Auto-Parse Prompt from Text File Without Reloading Model?

[mohammedalsayegh]

Hi there! I'm currently experimenting with llama.cpp to summarize large text files. My approach involves splitting a large text file into smaller chunks and adding chat format markers at the beginning and end of each chunk.

My current setup demands reloading the model for each promote, which is not efficient. I'm seeking a solution to automate parsing from a text file without the need for reloading the model each time. I haven't come across any command-line options that handle a single shot answer for multiple promote within a single file or allow passing multiple files to automate the parsing process.

If there is a way through the system command prompt or via llama.cpp, I would appreciate guidance.

@echo off

REM Check if directory argument is provided
if "%~1"=="" (
    echo Usage: %~nx0 directory
    exit /b
)

REM Path to the program
set PROGRAM="main.exe"

REM Desired parameters for the program
set PARAMETERS=-c 2048 --temp 0.0 --top_p 0.0 --top_k 1.0 -n -1 -m Phi-3-mini-128k-instruct.Q8_0.gguf

REM Directory containing the text files
set FILES_DIR=%~1

REM Navigate to the specified directory
cd /d FILES_DIR

REM Initialize response.txt
type nul > response.txt

REM Process each part file
for %%i in (*_part*.txt) do (
    echo Processing %%i...
    REM Call the program with the specified parameters and input file
    %PROGRAM% %PARAMETERS% -f "%%i" | findstr /R /C:"<|assistant|>" >> response.txt
)

echo All files processed.