Chat Client APIs and Streaming

 Chat Client APIs and Streaming

call(). content() (Most Common)

Returns only the generated text.

@GetMapping("/content")
public String content(@RequestParam String question) {

    return chatClient.prompt()
            .user(question)
            .call()
            .content();
}

Flow:

User
   │
   ▼
ChatClient
   │
   ▼
LLM
   │
   ▼
String


Use when you only need the answer.

call().chatResponse()

Returns the complete response object, not just the text.

@GetMapping("/response")
public ChatResponse response(@RequestParam String question) {

    return chatClient.prompt()
            .user(question)
            .call()
            .chatResponse();
}


Chat Response contains much more than the content.

http://localhost:8080/api/response?question=Explain Kafka

output are in json format .


 Streaming

Instead of waiting for the complete answer, stream it token by token.

@GetMapping(value="/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<String> stream(@RequestParam String question) {

    return chatClient.prompt()
            .user(question)
            .stream()
            .content();
}

Now the browser receives:

data:Certainly

data:!

data: Here's

data: an

data: explanation


instead of waiting for the full response.

Why "data:"?


SSE (Server-Sent Events) has a standard format.


One message looks like
data: Kafka
Notice the blank line afterward.
Another message
data: is
Another
data: a



The browser understands this format automatically.

Streaming ChatResponse


Instead of streaming plain text:


Flux<String>


you can stream complete response objects.


@GetMapping(value="/stream-response",
        produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<ChatResponse> streamResponse(@RequestParam String question){

    return chatClient.prompt()
            .user(question)
            .stream()
            .chatResponse();
}
Useful when you want metadata while streaming.


Mapping to a Java Object

Suppose you ask:

Give employee details.

Create a record:

public record Employee(
        String name,
        int age,
        String city
) {}
Then:

Employee employee =
        chatClient.prompt()
                .user("Generate employee details")
                .call()
                .entity(Employee.class);


Spring AI asks the model for structured output and maps it into your Java type.

List Mapping

public record Employee(
        String name,
        int age
) {}
Then:

List<Employee> employees =
        chatClient.prompt()
                .user("Generate 5 employees")
                .call()
                .entity(new ParameterizedTypeReference<List<Employee>>() {});

Bean Mapping

Bean Mapping


public class Student {

    private String name;
    private int age;

    // getters/setters
}

Then:

Student student =
        chatClient.prompt()
                .user("Generate student")
                .call()
                .entity(Student.class);

JavaScript Frontend (Production)

<!DOCTYPE html>
<html>
<body>

<button onclick="start()">Ask AI</button>

<pre id="output"></pre>

<script>

function start(){

    const eventSource = new EventSource(
            "http://localhost:8080/stream?question=Explain Kafka"
    );

    eventSource.onmessage = function(event){

        document.getElementById("output").textContent += event.data;

    };

}

</script>

</body>
</html>

This is how many AI chat applications display responses in real time.


What Happens Internally

User
   │
   ▼
ChatClient
   │
   ▼
OpenAI Streaming API
   │
   ▼
Token 1
   │
   ▼
Flux<String>
   │
   ▼
Spring WebFlux
   │
   ▼
Browser / Postman / curl

Flux<String> is a Publisher from Project Reactor. Instead of waiting for the complete response, it emits each chunk as soon as it arrives, which is why streaming feels much more responsive in AI chat applications.


Production Flow

User
   │
   ▼
ChatClient
   │
   ▼
OpenAI Streaming API
   │
   ▼
Chunk 1 → "Certainly"
   │
Chunk 2 → "!"
   │
Chunk 3 → "Here"
   │
Chunk 4 → "'s"
   │
Chunk 5 → "an"
   │
Chunk 6 → "explanation"
   │
Chunk 7 → "of"
   │
Chunk 8 → "Kafka"

Spring AI wraps these chunks in a Flux<String> and Spring WebFlux sends each one as an SSE event.


Interview Question

Q: Why does stream().content() return Flux<String> instead of String?

Answer:

  • String waits until the entire response is generated.
  • Flux<String> is a reactive stream from Project Reactor.
  • As soon as the LLM generates a chunk/token, Spring AI emits it.
  • This reduces perceived latency and enables ChatGPT-like typing effects.


Spring AI receives each chunk.

Instead of storing everything,

it immediately sends each chunk into a Flux.


Think of Flux as a pipe.


What is Flux?

Imagine a water pipe.

Water Tank

↓

Pipe

↓

Tap

Water comes continuously.

Not all at once.

Flux is exactly the same idea.

Instead of water,

it carries data.


Flux means:

"I don't have one value."

"I have many values coming over time."


Why not List<String>?

Suppose the AI generates:

Kafka

is

a

distributed

event

streaming

platform

List would wait until everything is finished.

Wait...

Wait...

Wait...

Return List

Flux returns immediately.

Kafka

↓

is

↓

a

↓

distributed

↓

...

What does Spring WebFlux do?


Spring AI only creates the Flux.


Someone has to send it to the browser.


That is Spring WebFlux's job.
OpenAI

↓

Spring AI

↓

Flux<String>

↓

Spring WebFlux

↓

Browser
Why do you see


data: Kafka

data: is

data: a


Because Spring WebFlux converts every Flux element into an SSE message.


Complete Flow


User

↓

Controller

↓

ChatClient

↓

OpenAI



"Kafka"

↓

Flux emits

↓

Spring WebFlux

↓

data: Kafka

↓

Browser

What is Flux<String>?




Flux<String> is a reactive stream from Project Reactor that emits multiple values over time. 

In Spring AI, each value is typically a chunk of the LLM's generated response, allowing the client to receive text progressively instead of waiting for the entire response.



What Spring WebFlux Does


Spring WebFlux takes each emitted chunk from the Flux<String> and sends it to the client using Server-Sent Events (SSE). 

Q: Why do we use stream() in Spring AI?


Answer:stream() enables real-time AI responses. 
Instead of waiting for the complete response from the LLM, 
Spring AI receives generated chunks/tokens continuously and exposes them as a Flux<String>. 
Spring WebFlux then streams these chunks to the client using Server-Sent Events (SSE), 
improving responsiveness and user experience for AI chat applications.



Comments

Post a Comment

Popular posts from this blog

Async/await

  Async/await An async function is a function declared with the async keyword, and the await keyword is permitted within them. The async and await keywords enable asynchronous, promise-based behavior to be written in a cleaner style, avoiding the need to explicitly configure promise chains. "async and await make promises easier to write" async  makes a function return a Promise await  makes a function wait for a Promise  Async functions Let’s start with the async keyword. It can be placed before a function, like this: async function f() {   return 1; } The word “async” before a function means one simple thing: a function always returns a promise. Other values are wrapped in a resolved promise automatically. For instance, this function returns a resolved promise with the result of 1; let’s test it: async function f() {   return 1; } f().then(alert); // 1 …We could explicitly return a promise, which would be the same: async function f() {   return Promis...
Read more

Two Sum II - Input Array Is Sorted

 Given a 1-indexed array of integers numbers that is already sorted in non-decreasing order , find two numbers such that they add up to a specific target number. Let these two numbers be numbers[index1] and numbers[index2] where 1 <= index1 < index2 <= numbers.length. Return the indices of the two numbers, index1 and index2, added by one as an integer array [index1, index2] of length 2. The tests are generated such that there is exactly one solution. You may not use the same element twice. Your solution must use only constant extra space .   Example 1: Input: numbers = [2,7,11,15], target = 9 Output: [1,2] Explanation: The sum of 2 and 7 is 9. Therefore, index1 = 1, index2 = 2. We return [1, 2]. Example 2: Input: numbers = [2,3,4], target = 6 Output: [1,3] Explanation: The sum of 2 and 4 is 6. Therefore index1 = 1, index2 = 3. We return [1, 3]. Example 3: Input: numbers = [-1,0], target = -1 Output: [1,2] Explanation: The sum of -1 and 0 is -1. Therefore index1 = 1,...
Read more

Comparable Vs. Comparator in Java

  In Java , Comparable and Comparator both are interfaces used to sort objects. Comparable It defines  natural ordering  within a class. It is implemented  inside  the class being sorted. It uses the compareTo() method. It allows sorting  by one one . Comparable provides a  single sorting sequence . In other words, we can sort the collection on the basis of a single element such as id, name, and price. Comparable  affects the original class , i.e., the actual class is modified. Comparable is present in  java.lang  package. We can sort the list elements of Comparable type by  Collections.sort(List)  method. Only one compareTo() method can be implemented, restricting flexibility. Comparator It defines  custom sorting logic It is implemented  in a separate class  or using lambda expressions . It uses the compare() method. It allows sorting  by multiple criteria . The Comparator provides  multiple sorting...
Read more