Build a custom RAG agent with LangGraph4j¶

Port of the LangGraph Agentic RAG tutorial to LangGraph4j.

Build a retrieval agent with LangGraph4j that decides when to search a vector store versus answering the user directly.

LangChain4j provides the model, retrieval, embedding, vector-store, and tool APIs. LangGraph4j provides graph orchestration.

Setup¶

In [1]:

var userHomeDir = System.getProperty("user.home");
var localRespoUrl = "file://" + userHomeDir + "/.m2/repository/";
var langchain4jVersion = "1.18.1";
var langgraph4jVersion = "1.9.0-beta6";

var userHomeDir = System.getProperty("user.home"); var localRespoUrl = "file://" + userHomeDir + "/.m2/repository/"; var langchain4jVersion = "1.18.1"; var langgraph4jVersion = "1.9.0-beta6";

In [2]:

%dependency /add-repo local \{localRespoUrl} release|never snapshot|always
%dependency /add org.slf4j:slf4j-jdk14:2.0.9
%dependency /add org.bsc.langgraph4j:langgraph4j-langchain4j:\{langgraph4jVersion}
%dependency /add dev.langchain4j:langchain4j:\{langchain4jVersion}
%dependency /add dev.langchain4j:langchain4j-open-ai:\{langchain4jVersion}
%dependency /resolve

%dependency /add-repo local \{localRespoUrl} release|never snapshot|always %dependency /add org.slf4j:slf4j-jdk14:2.0.9 %dependency /add org.bsc.langgraph4j:langgraph4j-langchain4j:\{langgraph4jVersion} %dependency /add dev.langchain4j:langchain4j:\{langchain4jVersion} %dependency /add dev.langchain4j:langchain4j-open-ai:\{langchain4jVersion} %dependency /resolve

Repository local url: file:///Users/bsorrentino/.m2/repository/ added.
Adding dependency org.slf4j:slf4j-jdk14:2.0.9
Adding dependency org.bsc.langgraph4j:langgraph4j-langchain4j:1.9.0-beta6
Adding dependency dev.langchain4j:langchain4j:1.18.1
Adding dependency dev.langchain4j:langchain4j-open-ai:1.18.1
Solving dependencies
Resolved artifacts count: 19
Add to classpath: /Users/bsorrentino/Library/Jupyter/kernels/rapaio-jupyter-kernel/mima_cache/org/slf4j/slf4j-jdk14/2.0.9/slf4j-jdk14-2.0.9.jar
Add to classpath: /Users/bsorrentino/Library/Jupyter/kernels/rapaio-jupyter-kernel/mima_cache/org/slf4j/slf4j-api/2.0.9/slf4j-api-2.0.9.jar
Add to classpath: /Users/bsorrentino/Library/Jupyter/kernels/rapaio-jupyter-kernel/mima_cache/org/bsc/langgraph4j/langgraph4j-langchain4j/1.9.0-beta6/langgraph4j-langchain4j-1.9.0-beta6.jar
Add to classpath: /Users/bsorrentino/Library/Jupyter/kernels/rapaio-jupyter-kernel/mima_cache/dev/langchain4j/langchain4j-skills/1.19.0-beta29/langchain4j-skills-1.19.0-beta29.jar
Add to classpath: /Users/bsorrentino/Library/Jupyter/kernels/rapaio-jupyter-kernel/mima_cache/org/commonmark/commonmark/0.28.0/commonmark-0.28.0.jar
Add to classpath: /Users/bsorrentino/Library/Jupyter/kernels/rapaio-jupyter-kernel/mima_cache/org/commonmark/commonmark-ext-yaml-front-matter/0.28.0/commonmark-ext-yaml-front-matter-0.28.0.jar
Add to classpath: /Users/bsorrentino/Library/Jupyter/kernels/rapaio-jupyter-kernel/mima_cache/org/bsc/langgraph4j/langgraph4j-core/1.9.0-beta6/langgraph4j-core-1.9.0-beta6.jar
Add to classpath: /Users/bsorrentino/Library/Jupyter/kernels/rapaio-jupyter-kernel/mima_cache/org/bsc/async/async-generator/5.0.0/async-generator-5.0.0.jar
Add to classpath: /Users/bsorrentino/Library/Jupyter/kernels/rapaio-jupyter-kernel/mima_cache/dev/langchain4j/langchain4j/1.18.1/langchain4j-1.18.1.jar
Add to classpath: /Users/bsorrentino/Library/Jupyter/kernels/rapaio-jupyter-kernel/mima_cache/dev/langchain4j/langchain4j-core/1.18.1/langchain4j-core-1.18.1.jar
Add to classpath: /Users/bsorrentino/Library/Jupyter/kernels/rapaio-jupyter-kernel/mima_cache/org/jspecify/jspecify/1.0.0/jspecify-1.0.0.jar
Add to classpath: /Users/bsorrentino/Library/Jupyter/kernels/rapaio-jupyter-kernel/mima_cache/com/fasterxml/jackson/core/jackson-annotations/2.22/jackson-annotations-2.22.jar
Add to classpath: /Users/bsorrentino/Library/Jupyter/kernels/rapaio-jupyter-kernel/mima_cache/com/fasterxml/jackson/core/jackson-core/2.22.1/jackson-core-2.22.1.jar
Add to classpath: /Users/bsorrentino/Library/Jupyter/kernels/rapaio-jupyter-kernel/mima_cache/com/fasterxml/jackson/core/jackson-databind/2.22.1/jackson-databind-2.22.1.jar
Add to classpath: /Users/bsorrentino/Library/Jupyter/kernels/rapaio-jupyter-kernel/mima_cache/org/apache/opennlp/opennlp-tools/2.5.9/opennlp-tools-2.5.9.jar
Add to classpath: /Users/bsorrentino/Library/Jupyter/kernels/rapaio-jupyter-kernel/mima_cache/dev/langchain4j/langchain4j-open-ai/1.18.1/langchain4j-open-ai-1.18.1.jar
Add to classpath: /Users/bsorrentino/Library/Jupyter/kernels/rapaio-jupyter-kernel/mima_cache/dev/langchain4j/langchain4j-http-client/1.18.1/langchain4j-http-client-1.18.1.jar
Add to classpath: /Users/bsorrentino/Library/Jupyter/kernels/rapaio-jupyter-kernel/mima_cache/dev/langchain4j/langchain4j-http-client-jdk/1.18.1/langchain4j-http-client-jdk-1.18.1.jar
Add to classpath: /Users/bsorrentino/Library/Jupyter/kernels/rapaio-jupyter-kernel/mima_cache/com/knuddels/jtokkit/1.1.0/jtokkit-1.1.0.jar

In [3]:

import java.io.FileInputStream;
import java.util.logging.LogManager;
import org.slf4j.LoggerFactory;

try( var file = new FileInputStream("./logging.properties")) {
    LogManager.getLogManager().readConfiguration( file );
}

var log = LoggerFactory.getLogger("AgenticRag");

import java.io.FileInputStream; import java.util.logging.LogManager; import org.slf4j.LoggerFactory; try( var file = new FileInputStream("./logging.properties")) { LogManager.getLogManager().readConfiguration( file ); } var log = LoggerFactory.getLogger("AgenticRag");

Preprocess documents¶

Use three posts from Lilian Weng's blog. Fetch page content with LangChain4j UrlDocumentLoader and TextDocumentParser.

In [4]:

import dev.langchain4j.data.document.loader.UrlDocumentLoader;
import dev.langchain4j.data.document.parser.TextDocumentParser;
import java.util.List;

var urls = List.of(
        "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/",
        "https://lilianweng.github.io/posts/2024-07-07-hallucination/",
        "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/"
);

var docs = urls.stream()
        .map(url -> UrlDocumentLoader.load(url, new TextDocumentParser()))
        .toList();

import dev.langchain4j.data.document.loader.UrlDocumentLoader; import dev.langchain4j.data.document.parser.TextDocumentParser; import java.util.List; var urls = List.of( "https://lilianweng.github.io/posts/2024-11-28-reward-hacking/", "https://lilianweng.github.io/posts/2024-07-07-hallucination/", "https://lilianweng.github.io/posts/2024-04-12-diffusion-video/" ); var docs = urls.stream() .map(url -> UrlDocumentLoader.load(url, new TextDocumentParser())) .toList();

In [5]:

import dev.langchain4j.data.document.splitter.DocumentSplitters;

var textSplitter = DocumentSplitters.recursive(100, 50);

import dev.langchain4j.data.document.splitter.DocumentSplitters; var textSplitter = DocumentSplitters.recursive(100, 50);

Create a retriever tool¶

In [6]:

import dev.langchain4j.data.segment.TextSegment;
import dev.langchain4j.model.openai.OpenAiEmbeddingModel;
import dev.langchain4j.rag.content.retriever.EmbeddingStoreContentRetriever;
import dev.langchain4j.store.embedding.EmbeddingStoreIngestor;
import dev.langchain4j.store.embedding.inmemory.InMemoryEmbeddingStore;

var embeddingModel = OpenAiEmbeddingModel.builder()
        .apiKey(System.getenv("OPENAI_API_KEY"))
        .modelName("text-embedding-3-small")
        .build();

var embeddingStore = new InMemoryEmbeddingStore<TextSegment>();

var ingestor = EmbeddingStoreIngestor.builder()
        .documentSplitter(textSplitter)
        .embeddingModel(embeddingModel)
        .embeddingStore(embeddingStore)
        .build();

ingestor.ingest(docs);

var retriever = EmbeddingStoreContentRetriever.builder()
        .embeddingStore(embeddingStore)
        .embeddingModel(embeddingModel)
        .maxResults(3)
        .build();

import dev.langchain4j.data.segment.TextSegment; import dev.langchain4j.model.openai.OpenAiEmbeddingModel; import dev.langchain4j.rag.content.retriever.EmbeddingStoreContentRetriever; import dev.langchain4j.store.embedding.EmbeddingStoreIngestor; import dev.langchain4j.store.embedding.inmemory.InMemoryEmbeddingStore; var embeddingModel = OpenAiEmbeddingModel.builder() .apiKey(System.getenv("OPENAI_API_KEY")) .modelName("text-embedding-3-small") .build(); var embeddingStore = new InMemoryEmbeddingStore<TextSegment>(); var ingestor = EmbeddingStoreIngestor.builder() .documentSplitter(textSplitter) .embeddingModel(embeddingModel) .embeddingStore(embeddingStore) .build(); ingestor.ingest(docs); var retriever = EmbeddingStoreContentRetriever.builder() .embeddingStore(embeddingStore) .embeddingModel(embeddingModel) .maxResults(3) .build();

In [7]:

import dev.langchain4j.agent.tool.P;
import dev.langchain4j.agent.tool.Tool;
import dev.langchain4j.rag.content.retriever.ContentRetriever;
import dev.langchain4j.rag.query.Query;
import org.bsc.langgraph4j.langchain4j.tool.LC4jToolService;
import java.util.stream.Collectors;

class BlogTools {
    private final ContentRetriever retriever;

    BlogTools(ContentRetriever retriever) {
        this.retriever = retriever;
    }

    @Tool("Search and return information about Lilian Weng blog posts.")
    String retrieveBlogPosts(@P("search query") String query) {
        return retriever.retrieve(new Query(query)).stream()
                .map(content -> content.textSegment().text())
                .collect(Collectors.joining("\n\n"));
    }
}

var toolService = LC4jToolService.builder()
        .toolsFromObject(new BlogTools(retriever))
        .build();

import dev.langchain4j.agent.tool.P; import dev.langchain4j.agent.tool.Tool; import dev.langchain4j.rag.content.retriever.ContentRetriever; import dev.langchain4j.rag.query.Query; import org.bsc.langgraph4j.langchain4j.tool.LC4jToolService; import java.util.stream.Collectors; class BlogTools { private final ContentRetriever retriever; BlogTools(ContentRetriever retriever) { this.retriever = retriever; } @Tool("Search and return information about Lilian Weng blog posts.") String retrieveBlogPosts(@P("search query") String query) { return retriever.retrieve(new Query(query)).stream() .map(content -> content.textSegment().text()) .collect(Collectors.joining("\n\n")); } } var toolService = LC4jToolService.builder() .toolsFromObject(new BlogTools(retriever)) .build();

Generate a query or respond¶

In [8]:

import dev.langchain4j.data.message.ChatMessage;
import dev.langchain4j.data.message.UserMessage;
import dev.langchain4j.model.chat.request.ChatRequest;
import dev.langchain4j.model.chat.request.ChatRequestParameters;
import dev.langchain4j.model.openai.OpenAiChatModel;
import org.bsc.langgraph4j.action.NodeAction;
import org.bsc.langgraph4j.langchain4j.serializer.std.LC4jStateSerializer;
import org.bsc.langgraph4j.prebuilt.MessagesState;

var responseModel = OpenAiChatModel.builder()
        .apiKey(System.getenv("OPENAI_API_KEY"))
        .modelName("gpt-5.4-mini")
        .temperature(0.0)
        .build();

var stateSerializer = new LC4jStateSerializer<MessagesState<ChatMessage>>(MessagesState::new);

NodeAction<MessagesState<ChatMessage>> generateQueryOrRespond = state -> {
    var params = ChatRequestParameters.builder()
            .toolSpecifications(toolService.toolSpecifications())
            .build();
    var request = ChatRequest.builder()
            .parameters(params)
            .messages(state.messages())
            .build();
    var response = responseModel.chat(request);
    return Map.of("messages", response.aiMessage());
};

import dev.langchain4j.data.message.ChatMessage; import dev.langchain4j.data.message.UserMessage; import dev.langchain4j.model.chat.request.ChatRequest; import dev.langchain4j.model.chat.request.ChatRequestParameters; import dev.langchain4j.model.openai.OpenAiChatModel; import org.bsc.langgraph4j.action.NodeAction; import org.bsc.langgraph4j.langchain4j.serializer.std.LC4jStateSerializer; import org.bsc.langgraph4j.prebuilt.MessagesState; var responseModel = OpenAiChatModel.builder() .apiKey(System.getenv("OPENAI_API_KEY")) .modelName("gpt-5.4-mini") .temperature(0.0) .build(); var stateSerializer = new LC4jStateSerializer<MessagesState<ChatMessage>>(MessagesState::new); NodeAction<MessagesState<ChatMessage>> generateQueryOrRespond = state -> { var params = ChatRequestParameters.builder() .toolSpecifications(toolService.toolSpecifications()) .build(); var request = ChatRequest.builder() .parameters(params) .messages(state.messages()) .build(); var response = responseModel.chat(request); return Map.of("messages", response.aiMessage()); };

Grade documents¶

In [9]:

import dev.langchain4j.data.message.ToolExecutionResultMessage;
import dev.langchain4j.model.output.structured.Description;
import dev.langchain4j.service.AiServices;
import dev.langchain4j.service.SystemMessage;
import org.bsc.langgraph4j.action.EdgeAction;

class GradeDocuments {
    @Description("Relevance score: 'yes' if relevant, or 'no' if not relevant")
    public String binaryScore;
}

interface DocumentGrader {
    @SystemMessage("Grade whether the retrieved document is relevant to the user question. Return yes or no.")
    GradeDocuments grade(@dev.langchain4j.service.UserMessage String prompt);
}

var documentGrader = AiServices.create(DocumentGrader.class, responseModel);

EdgeAction<MessagesState<ChatMessage>> gradeDocuments = state -> {
    var question = state.messages().stream()
            .filter(UserMessage.class::isInstance)
            .map(UserMessage.class::cast)
            .findFirst()
            .map(UserMessage::singleText)
            .orElse("");
    var context = state.lastMessage()
            .filter(ToolExecutionResultMessage.class::isInstance)
            .map(ToolExecutionResultMessage.class::cast)
            .map(ToolExecutionResultMessage::text)
            .orElse("");
    var score = documentGrader.grade("Retrieved document:\n" + context + "\n\nUser question: " + question);
    return "yes".equalsIgnoreCase(score.binaryScore) ? "generate_answer" : "rewrite_question";
};

import dev.langchain4j.data.message.ToolExecutionResultMessage; import dev.langchain4j.model.output.structured.Description; import dev.langchain4j.service.AiServices; import dev.langchain4j.service.SystemMessage; import org.bsc.langgraph4j.action.EdgeAction; class GradeDocuments { @Description("Relevance score: 'yes' if relevant, or 'no' if not relevant") public String binaryScore; } interface DocumentGrader { @SystemMessage("Grade whether the retrieved document is relevant to the user question. Return yes or no.") GradeDocuments grade(@dev.langchain4j.service.UserMessage String prompt); } var documentGrader = AiServices.create(DocumentGrader.class, responseModel); EdgeAction<MessagesState<ChatMessage>> gradeDocuments = state -> { var question = state.messages().stream() .filter(UserMessage.class::isInstance) .map(UserMessage.class::cast) .findFirst() .map(UserMessage::singleText) .orElse(""); var context = state.lastMessage() .filter(ToolExecutionResultMessage.class::isInstance) .map(ToolExecutionResultMessage.class::cast) .map(ToolExecutionResultMessage::text) .orElse(""); var score = documentGrader.grade("Retrieved document:\n" + context + "\n\nUser question: " + question); return "yes".equalsIgnoreCase(score.binaryScore) ? "generate_answer" : "rewrite_question"; };

Rewrite the question¶

In [10]:

NodeAction<MessagesState<ChatMessage>> rewriteQuestion = state -> {
    var question = state.messages().stream()
            .filter(UserMessage.class::isInstance)
            .map(UserMessage.class::cast)
            .findFirst()
            .map(UserMessage::singleText)
            .orElse("");
    var response = responseModel.chat(UserMessage.from("Formulate an improved search question: " + question));
    return Map.of("messages", UserMessage.from(response.aiMessage().text()));
};

NodeAction<MessagesState<ChatMessage>> rewriteQuestion = state -> { var question = state.messages().stream() .filter(UserMessage.class::isInstance) .map(UserMessage.class::cast) .findFirst() .map(UserMessage::singleText) .orElse(""); var response = responseModel.chat(UserMessage.from("Formulate an improved search question: " + question)); return Map.of("messages", UserMessage.from(response.aiMessage().text())); };

Generate an answer¶

In [11]:

NodeAction<MessagesState<ChatMessage>> generateAnswer = state -> {
    var question = state.messages().stream()
            .filter(UserMessage.class::isInstance)
            .map(UserMessage.class::cast)
            .findFirst()
            .map(UserMessage::singleText)
            .orElse("");
    var context = state.messages().stream()
            .filter(ToolExecutionResultMessage.class::isInstance)
            .map(ToolExecutionResultMessage.class::cast)
            .map(ToolExecutionResultMessage::text)
            .collect(Collectors.joining("\n\n"));
    var prompt = "Question: %s\n<context>\n%s\n</context>".formatted(question, context);
    var response = responseModel.chat(UserMessage.from(prompt));
    return Map.of("messages", response.aiMessage());
};

NodeAction<MessagesState<ChatMessage>> generateAnswer = state -> { var question = state.messages().stream() .filter(UserMessage.class::isInstance) .map(UserMessage.class::cast) .findFirst() .map(UserMessage::singleText) .orElse(""); var context = state.messages().stream() .filter(ToolExecutionResultMessage.class::isInstance) .map(ToolExecutionResultMessage.class::cast) .map(ToolExecutionResultMessage::text) .collect(Collectors.joining("\n\n")); var prompt = "Question: %s\n<context>\n%s\n</context>".formatted(question, context); var response = responseModel.chat(UserMessage.from(prompt)); return Map.of("messages", response.aiMessage()); };

Assemble the graph¶

In [12]:

import dev.langchain4j.data.message.AiMessage;
import dev.langchain4j.invocation.InvocationContext;
import dev.langchain4j.invocation.InvocationParameters;
import org.bsc.langgraph4j.StateGraph;
import org.bsc.langgraph4j.action.Command;

import static org.bsc.langgraph4j.StateGraph.START;
import static org.bsc.langgraph4j.StateGraph.END;
import static org.bsc.langgraph4j.action.AsyncEdgeAction.edge_async;
import static org.bsc.langgraph4j.action.AsyncNodeAction.node_async;

EdgeAction<MessagesState<ChatMessage>> routeOnToolCalls = state -> {
    var last = state.lastMessage().orElse(null);
    if (last instanceof AiMessage aiMessage && aiMessage.hasToolExecutionRequests()) {
        return "retrieve";
    }
    return "respond";
};

NodeAction<MessagesState<ChatMessage>> retrieve = state -> {
    var last = state.lastMessage().orElseThrow();
    if (last instanceof AiMessage aiMessage && aiMessage.hasToolExecutionRequests()) {
        return toolService.execute(
                        aiMessage.toolExecutionRequests(),
                        InvocationContext.builder()
                                .invocationParameters(InvocationParameters.from(state.data()))
                                .build(),
                        "messages")
                .thenApply(Command::update)
                .join();
    }
    return Map.of();
};

var workflow = new StateGraph<>(MessagesState.SCHEMA, stateSerializer)
        .addNode("generate_query_or_respond", node_async(generateQueryOrRespond))
        .addNode("retrieve", node_async(retrieve))
        .addNode("rewrite_question", node_async(rewriteQuestion))
        .addNode("generate_answer", node_async(generateAnswer))
        .addEdge(START, "generate_query_or_respond")
        .addConditionalEdges("generate_query_or_respond", edge_async(routeOnToolCalls), Map.of(
                "retrieve", "retrieve",
                "respond", END
        ))
        .addConditionalEdges("retrieve", edge_async(gradeDocuments), Map.of(
                "generate_answer", "generate_answer",
                "rewrite_question", "rewrite_question"
        ))
        .addEdge("rewrite_question", "generate_query_or_respond")
        .addEdge("generate_answer", END);

var graph = workflow.compile();

import dev.langchain4j.data.message.AiMessage; import dev.langchain4j.invocation.InvocationContext; import dev.langchain4j.invocation.InvocationParameters; import org.bsc.langgraph4j.StateGraph; import org.bsc.langgraph4j.action.Command; import static org.bsc.langgraph4j.StateGraph.START; import static org.bsc.langgraph4j.StateGraph.END; import static org.bsc.langgraph4j.action.AsyncEdgeAction.edge_async; import static org.bsc.langgraph4j.action.AsyncNodeAction.node_async; EdgeAction<MessagesState<ChatMessage>> routeOnToolCalls = state -> { var last = state.lastMessage().orElse(null); if (last instanceof AiMessage aiMessage && aiMessage.hasToolExecutionRequests()) { return "retrieve"; } return "respond"; }; NodeAction<MessagesState<ChatMessage>> retrieve = state -> { var last = state.lastMessage().orElseThrow(); if (last instanceof AiMessage aiMessage && aiMessage.hasToolExecutionRequests()) { return toolService.execute( aiMessage.toolExecutionRequests(), InvocationContext.builder() .invocationParameters(InvocationParameters.from(state.data())) .build(), "messages") .thenApply(Command::update) .join(); } return Map.of(); }; var workflow = new StateGraph<>(MessagesState.SCHEMA, stateSerializer) .addNode("generate_query_or_respond", node_async(generateQueryOrRespond)) .addNode("retrieve", node_async(retrieve)) .addNode("rewrite_question", node_async(rewriteQuestion)) .addNode("generate_answer", node_async(generateAnswer)) .addEdge(START, "generate_query_or_respond") .addConditionalEdges("generate_query_or_respond", edge_async(routeOnToolCalls), Map.of( "retrieve", "retrieve", "respond", END )) .addConditionalEdges("retrieve", edge_async(gradeDocuments), Map.of( "generate_answer", "generate_answer", "rewrite_question", "rewrite_question" )) .addEdge("rewrite_question", "generate_query_or_respond") .addEdge("generate_answer", END); var graph = workflow.compile();

Run the agentic RAG¶

In [13]:

for (var step : graph.stream(Map.of(
        "messages",
        UserMessage.from("What does Lilian Weng say about types of reward hacking?")
))) {
    step.state().lastMessage()
            .filter(AiMessage.class::isInstance)
            .map(AiMessage.class::cast)
            .map(AiMessage::text)
            .ifPresent(System.out::println);

}

for (var step : graph.stream(Map.of( "messages", UserMessage.from("What does Lilian Weng say about types of reward hacking?") ))) { step.state().lastMessage() .filter(AiMessage.class::isInstance) .map(AiMessage.class::cast) .map(AiMessage::text) .ifPresent(System.out::println); }

START 
execute: [retrieveBlogPosts] 

Lilian Weng describes **reward hacking** as the agent finding ways to **maximize the reward signal without actually accomplishing the intended task**.

She groups reward hacking into a few broad types:

1. **Specification gaming / loophole exploitation**  
   The agent exploits an imperfectly designed reward function or environment rule.  
   - Example: getting points by repeatedly triggering a scoring event rather than solving the task.

2. **Proxy over-optimization**  
   The reward is only a proxy for the real goal, and the agent learns to optimize the proxy in a way that diverges from the intended objective.  
   - Example: maximizing “engagement” by producing sensational but low-quality content.

3. **Reward model exploitation / adversarial behavior**  
   In learned reward settings, the policy finds inputs or behaviors that fool the reward model into giving high scores, even though humans would judge them poorly.

4. **Distribution shift / out-of-distribution exploitation**  
   The agent behaves in ways that were not well covered during training, so the reward function or model fails in unexpected ways.

Her main point is that reward hacking is not just “cheating,” but a general consequence of **misalignment between the true goal and the reward signal**.

If you want, I can also summarize the specific examples she gives in the post.
Lilian Weng describes **reward hacking** as the agent finding ways to **maximize the reward signal without actually accomplishing the intended task**.

She groups reward hacking into a few broad types:

1. **Specification gaming / loophole exploitation**  
   The agent exploits an imperfectly designed reward function or environment rule.  
   - Example: getting points by repeatedly triggering a scoring event rather than solving the task.

2. **Proxy over-optimization**  
   The reward is only a proxy for the real goal, and the agent learns to optimize the proxy in a way that diverges from the intended objective.  
   - Example: maximizing “engagement” by producing sensational but low-quality content.

3. **Reward model exploitation / adversarial behavior**  
   In learned reward settings, the policy finds inputs or behaviors that fool the reward model into giving high scores, even though humans would judge them poorly.

4. **Distribution shift / out-of-distribution exploitation**  
   The agent behaves in ways that were not well covered during training, so the reward function or model fails in unexpected ways.

Her main point is that reward hacking is not just “cheating,” but a general consequence of **misalignment between the true goal and the reward signal**.

If you want, I can also summarize the specific examples she gives in the post.

In [ ]: