use LLm hosted on sagemaker with LlamaIndex

Question

I have a LLM model hosted on sagemaker endpoint. I have a code which with langchain to use the sagemaker endpoint hosted LLM.

from langchain.embeddings import SagemakerEndpointEmbeddings
content_handler = ContentHandler()
embeddings = SagemakerEndpointEmbeddings(
    # credentials_profile_name="credentials-profile-name",
    endpoint_name="paraphrase-multilingual-mpnet-base-v2",
    region_name="me-central-1",
    content_handler=content_handler,
)

I need assistance in using LlamaIndex to deploy a pre-trained LLM model on Amazon SageMaker. Despite my research efforts, I couldn't find any useful information. Can someone provide me with a code snippet or guidance on how to achieve this?

Answer 1

Below is a code snippet that could help. You can read more here: Interacting with LLM deployed in Amazon SageMaker Endpoint with LlamaIndex
!pip install llama-index-llms-sagemaker-endpoint !pip install llama-index

from llama_index.llms.sagemaker_endpoint import SageMakerLLM
llm = SageMakerLLM(
    endpoint_name="paraphrase-multilingual-mpnet-base-v2",
    region_name="me-central-1"
)

# Call complete with a prompt
resp = llm.complete(
    "Paul Graham is", formatted=True
)  # formatted=True to avoid adding system prompt
print(resp)