How to customise the LlamaIndex starter tuto to use the latest Llama model hosted on "akash.network"

Question

The akash chat API is supposed to be compatible with openai : https://chatapi.akash.network/documentation, it's compatible with the basic OpenAI SDK :

import openai
import textwrap
client = openai.OpenAI(
    api_key="sk-xxxxxxxx",
    base_url="https://chatapi.akash.network/api/v1"
)

response = client.chat.completions.create(
    model="Meta-Llama-3-1-8B-Instruct-FP8",
    messages = [
        {
            "role": "user",
            "content": "Who are you?"
        }
    ],
)

print(textwrap.fill(response.choices[0].message.content, 50))

If I customize the LlamaIndex starter tutorial like this :

from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
from llama_index.llms.openai import OpenAI

API_BASE_URL="https://chatapi.akash.network/api/v1"
EMBEDDING_MODEL="BAAI/bge-small-en-v1.5"
LLM_MODEL="Meta-Llama-3-3-70B-Instruct"

# Load your documents
documents = SimpleDirectoryReader("data").load_data()

# Pick an available OpenAI compatible model
custom_llm = OpenAI(api_base=API_BASE_URL, model=LLM_MODEL)

# Initialize the HuggingFace embedding model
embedding_model = HuggingFaceEmbedding(model_name=EMBEDDING_MODEL)

# Set the local embedding model
Settings.embed_model = embedding_model

# Build the index using the local embeddings
index = VectorStoreIndex.from_documents(documents, llm=custom_llm)

query_engine = index.as_query_engine(llm=custom_llm)
response = query_engine.query("What did the author do growing up?")
print(response)

I get this error from the client lib : "Please provide a valid OpenAI model name in: o1, o1-2024-12-17, o1-preview, o1-preview-2024-09-12, o1-mini, (...)"

  File "C:\Dev\projects\llama-index-starter-tuto\.venv\Lib\site-packages\llama_index\llms\openai\utils.py", line 236, in openai_modelname_to_contextsize
    raise ValueError(
ValueError: Unknown model 'Meta-Llama-3-3-70B-Instruct'. Please provide a valid OpenAI model name in: o1, o1-2024-12-17, o1-preview, o1-preview-2024-09-12, o1-mini, (...)

If I remove the "llm" parameter from "as_query_engine" and set a OPENAI_API_BASE=https://chatapi.akash.network/api/v1 env variable, I get this from the API : "Not allowed to call model=gpt-3.5-turbo. Allowed team models = ['llama3-8b', 'Meta-Llama-3-1-405B-Instruct-FP8', 'llama3-8b-instruct', 'Meta-Llama-3-1-8B-Instruct-FP8', (...)"

openai.AuthenticationError: Error code: 401 - {'error': {'message': "Authentication Error, Team=643a4183-7eb9-4c20-8e31-db45843bffbe not allowed to call model=gpt-3.5-turbo. Allowed team models = ['llama3-8b', 'Meta-Llama-3-1-405B-Instruct-FP8', 'llama3-8b-instruct', 'Meta-Llama-3-1-8B-Instruct-FP8', 'Meta-Llama-3-2-3B-Instruct', 'nvidia-Llama-3-1-Nemotron-70B-Instruct-HF', 'Meta-Llama-3-3-70B-Instruct', (...)]", 'type': 'auth_error', 'param': 'None', 'code': '401'}}

Does it mean there is no way to use llama_index.llms.openai to reach the Akash endpoint ? The client lib expect a set of models and the API expect an other set ?

UPDATE :

I've also tried to use LlamaAPI class :

from llama_index.llms.llama_api import LlamaAPI
#(...)
#custom_llm = OpenAI(api_base=API_BASE_URL, model=LLM_MODEL)
custom_llm = LlamaAPI(api_key=LLAMA_API_KEY)

but there is no way to customize the api base URL.

Answer 1

Here is the answer from informations found on LlamaIndex Discord (I hoped they had a real community forum with clear threads and a search engine) :

"Currently, llama_index prevents using custom models with their OpenAI class because they need to be able to infer some metadata from the model name."

"OpenAILike is a thin wrapper around the OpenAI model that makes it compatible with 3rd party tools that provide an openai-compatible api."

source : https://docs.llamaindex.ai/en/stable/api_reference/llms/openai_like/

from llama_index.embeddings.huggingface import HuggingFaceEmbedding
from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, Settings
#from llama_index.llms.openai import OpenAI
from llama_index.llms.openai_like import OpenAILike

API_BASE_URL="https://chatapi.akash.network/api/v1"
EMBEDDING_MODEL="BAAI/bge-small-en-v1.5"
LLM_MODEL="Meta-Llama-3-3-70B-Instruct"

# Load your documents
documents = SimpleDirectoryReader("data").load_data()

# Pick an available OpenAI compatible model
custom_llm = OpenAILike(api_base=API_BASE_URL, model=LLM_MODEL)

# Initialize the HuggingFace embedding model
embedding_model = HuggingFaceEmbedding(model_name=EMBEDDING_MODEL)

# Set the local embedding model
Settings.embed_model = embedding_model

# Build the index using the local embeddings
index = VectorStoreIndex.from_documents(documents, llm=custom_llm)

query_engine = index.as_query_engine(llm=custom_llm)
response = query_engine.query("What did the author do growing up?")
print(response)