CODEWITHSUNDEEP
bashgnome-terminalllamallamacpp
Fred Åberg
Fred Åberg

Reputation: 1

Is there a way to automate formatting terminal input when talking to an LLM model?

Currently when the user sends input to the LLM it has to be in the format of:

<|im_start|>user\
User input here.<|im_end|>

This is very cumbersome and makes prompt engineering very tedious. Right now i am using llama.cpp to run a llama2 model locally in the terminal - is there a way to automatically insert user input between the two tokens (im_start and im_end)?

I might have approached this problem in the wrong way, but the following is what I have tried:

My current (so far unsuccessful) method is to use a bash script to launch the LLM in another terminal and redirecting the output of the current terminal to the LLM-terminal. Later I was going to make a script that the first "launcher-terminal" would, which would format all user input and send it to the LLM-terminal.

Sofar I haven't been able to get the communication between the terminals working. The script would be trivial for me to make. It is moreso the bash-scripting that I'm struggling with.

I suspect my bash implementation is highly flawed, I'm not very experienced with it.

RUN_PATH="/llama.cpp/main"
MODEL_PATH="/llama.cpp/models/dolphin-2.9-llama3-8b-q4_K_M.gguf"
INIT_PATH="prompt.txt"

FLAGS="-n 512 --keep -1 --ctx-size 2048 --temp 0.0"

LAUNCH_CMD=".${RUN_PATH} -m .${MODEL_PATH} ${FLAGS} --color  --interactive-first --file .${INIT_PATH}"

# gnome-terminal -- "${LAUNCH_CMD}"
# Here im launching an impostor python program which just prints the input it receives.
gnome-terminal -- python3 impostor.py

LAUNCHER_PID="${$}"
LLM_PID="$PPID"

# It seems I don't get the PIDs i expect to get.
echo "Launcher PID: ${LAUNCHER_PID}"
echo "LLM PID:  ${LLM_PID}"

# Set STD-output of the launcher to LLM
ps -p $LLM_PID << ps -p $LAUNCHER_PID
/bin/bash

When typing echo hello world in the launcher terminal, it does not show up in the LLM-terminal, which is the desired effect.

Upvotes: 0

Views: 186

Answers (1)

Diego Torres Milano
Diego Torres Milano

Reputation: 69388

If you want to connect to the standard input and output of a running process you can use a coprocess.

Something like

coproc ./impostor.py

and then you can read queries, modify them, and send to the coprocess

while read -rp 'Prompt: ' p; do printf '<|im_start|>user\\\n%s<|im_end|>\n' "$p" >&"${COPROC[1]}"; done

To read the responses read from ${COPROC[0]}

Upvotes: 0

Related Questions