Dataflow

 

1. What is Dataflow?

Google Cloud Dataflow

Dataflow is a fully managed service used to process and transform large amounts of data.

In simple words:

Dataflow takes data from one place, processes it, transforms it, and sends it to another place.

Imagine a banking application.

Every second, thousands of transactions happen.

You receive data like this:

Transaction

Amount

Status

T101

500

SUCCESS

T102

1000

FAILED

T103

2000

SUCCESS

T104

300

SUCCESS

You want to:

  • Filter failed transactions.

  • Calculate total successful payments.

  • Group transactions by city.

  • Send processed data to BigQuery.

Dataflow can perform these operations automatically as part of a data pipeline.

When people say "process data", they simply mean doing some work on the data before storing or analyzing it.

Raw Data

Imagine Kafka receives:

{
  "txnId": 101,
  "amount": 5000,
  "merchant": "amazon",
  "status": "success"

}

This is raw data.

Processing = Modifying the Data

1. Filtering

Remove unwanted records.

Before:

txn1 SUCCESS
txn2 FAILED

txn3 SUCCESS

After:
txn1 SUCCESS
txn3 SUCCESS

2. Transforming

Change the format.

Before:

{
  "merchant": "amazon"

}
After:
{
  "merchant": "AMAZON"
}

3. Enriching

Add extra information.

Before:

{
  "merchantId": 101

}
After:
{
  "merchantId": 101,
  "merchantName": "Amazon",
  "category": "E-commerce"
}

4. Aggregating

Calculate totals.

Before:

Txn1 ₹100
Txn2 ₹200

Txn3 ₹300
After:
Total Revenue = ₹600

5. Validating

Remove bad data.

Before:

{
  "amount": -500

}
After:
Record Rejected

In Your Spring Boot Experience

This code is "processing":

Transaction txn = kafkaMessage;
if(txn.isValid()) {
    txn.setFee(txn.getAmount() * 0.01);
    txn.setMerchantName(fetchMerchant());
    save(txn);

}

You're:

  • validating 
  • calculating fee 
  • enriching merchant data 

That's data processing.

Then Where Does BigQuery Come?

After processing:

Kafka
  |
  v
Dataflow
(Filter + Validate + Enrich)
  |
  v

BigQuery Table

Stored Data:

txnIdamountmerchantfee
1015000Amazon50
1022000Flipkart20

Now BigQuery is used to ask questions:

SELECT SUM(amount)

FROM transactions;

Simple Formula

Raw Data
   ↓
Process Data
(Filter, Validate, Enrich, Transform)
   ↓
Store Data
(BigQuery)
   ↓
Analyze Data
(SQL Queries)

Dataflow = Chef cooks the food 🍳
Big Query = Customer eats and analyzes the food 📊

Dataflow changes the data. Big Query asks questions on the data. That's the core difference. 

2. How does Dataflow work?

Think of a water-filter machine.

  • Input: Dirty water.

  • Processing: Filter the water.

  • Output: Clean water.

Dataflow works similarly.

  • Input: Raw data.

  • Processing: Filter, transform and enrich data.

  • Output: Clean, processed data.



Example:
Raw Transaction Data
        |
        v
     Dataflow
        |
        v
  Filter Transactions
        |
        v
 Calculate Total Amount
        |
        v
    BigQuery
        |
        v
   Dashboard

3. Two important processing types

This is very important for your GCP certification.

Dataflow supports two processing modes.

A. Batch Processing

Batch means processing a collection of data together.

Example:

A bank has 10 lakh transactions from yesterday.

You want to generate a daily report.

Instead of processing each record immediately, you process the collected records together.






Real-world examples:

  • Daily transaction reports.

  • Monthly salary processing.

  • Historical customer data analysis.

  • Daily sales reports.

Remember: Batch = Process collected data together.

B. Stream Processing

Stream means processing data continuously as it arrives.

Example:

A customer makes a payment of ₹500.

The transaction event arrives in the system, and Dataflow can process it immediately without waiting for the end of the day.



Real-world examples:

  • Real-time payment monitoring.

  • Fraud detection pipelines.

  • IoT sensor monitoring.

  • Live transaction dashboards.

Remember: Stream = Process incoming data continuously.

Batch vs Stream

Batch

Stream

Processes collected data

Processes continuous incoming data

Scheduled or triggered

Continuously running

Daily reports

Real-time monitoring

Historical analysis

Live analytics

4. How does Dataflow work with other Google Cloud services?

Your course mentions several services. Let's understand their roles.




Service

What it does

Pub/Sub

Receives and distributes messages

Apache Kafka

Event streaming and messaging

Datastore

Stores application data

Dataflow

Processes and transforms data

BigQuery

Analyses processed data

Bigtable

Stores large-scale, low-latency data

Vertex AI

Builds and runs AI/ML applications

Looker Studio

Creates dashboards and reports

Note: Your lecture says Agent Platform and Data Studio; these refer to Google's AI platform and dashboard ecosystem.

5. What is Apache Beam?

This is an important concept.

You heard this in your lecture:

Dataflow supports Java and Python APIs through the Apache Beam SDK.

Let's understand.

Apache Beam is a programming model used to create data processing pipelines.

Google Dataflow is one of the services that can execute those pipelines.

Think of it like this:

  • Apache Beam = Instructions or recipe.

  • Dataflow = Machine that executes the recipe.

Apache Beam SDK

Write pipeline using Java / Python

Google Cloud Dataflow

Executes and manages the pipeline

Processed Data

As a Java developer

You can write Dataflow pipelines using Java.

For example:

Pipeline pipeline = Pipeline.create(options);

pipeline
    .apply(TextIO.read().from("input.txt"))
    .apply(Filter.by(line -> line.contains("SUCCESS")))
    .apply(TextIO.write().to("output"));

pipeline.run();

What does this code do?

  1. Creates a pipeline.

  2. Reads data from a text file.

  3. Filters records containing SUCCESS.

  4. Writes the filtered records to an output file.

  5. Runs the pipeline.

This is an introductory Apache Beam example using Java.

6. What is autoscaling in Dataflow?

Imagine your pipeline normally receives 1,000 records per second.

Suddenly, it receives 1 lakh records per second.

Without sufficient processing capacity, the pipeline may become slow.

Dataflow can automatically increase or decrease worker resources based on workload and configuration.

  • Low workload → Fewer workers.

  • High workload → More workers.

  • Workload decreases → Workers can scale down.

This reduces the need to manually manage infrastructure.

7. What is monitoring in Dataflow?

Dataflow integrates with Google Cloud Observability.

You can monitor:

  • Pipeline errors.

  • Processing delays.

  • Worker utilisation.

  • Data throughput.

  • Failed records and job status.

You can configure alerts to notify you when something goes wrong.

8. BigQuery vs Dataflow – Important difference

Since we just studied BigQuery, let's compare them.

BigQuery

Dataflow

Data warehouse

Data processing service

Stores and analyses data

Transforms and processes data

Executes SQL queries

Executes data pipelines

Analytical reports

Batch and streaming pipelines

Simple example:

Dataflow cleans and transforms transaction data.

BigQuery stores the processed data and helps you analyse it.

They work together.

Comments

Post a Comment

Popular posts from this blog

Async/await

  Async/await An async function is a function declared with the async keyword, and the await keyword is permitted within them. The async and await keywords enable asynchronous, promise-based behavior to be written in a cleaner style, avoiding the need to explicitly configure promise chains. "async and await make promises easier to write" async  makes a function return a Promise await  makes a function wait for a Promise  Async functions Let’s start with the async keyword. It can be placed before a function, like this: async function f() {   return 1; } The word “async” before a function means one simple thing: a function always returns a promise. Other values are wrapped in a resolved promise automatically. For instance, this function returns a resolved promise with the result of 1; let’s test it: async function f() {   return 1; } f().then(alert); // 1 …We could explicitly return a promise, which would be the same: async function f() {   return Promis...
Read more

Valid Parentheses

Given a string s containing just the characters '(', ')', '{', '}', '[' and ']', determine if the input string is valid. An input string is valid if: Open brackets must be closed by the same type of brackets. Open brackets must be closed in the correct order. Every close bracket has a corresponding open bracket of the same type.   Example 1: Input: s = "()" Output: true Example 2: Input: s = "()[]{}" Output: true Example 3: Input: s = "(]" Output: false Example 4: Input: s = "([])" Output: true   Stack-Based Logic: Traverse each character in the string. If it's an opening bracket ( '(' , '{' , '[' ), push it onto the stack. If it's a closing bracket ( ')' , '}' , ']' ): If the stack is empty , return false (nothing to match). If the top of the stack is the corresponding opening bracket , pop it. Otherwise, return false . ...
Read more

kafka

  ✅ spring.kafka.bootstrap-servers=localhost:9092 🧠 Meaning: The address of the Kafka broker (the server that stores and processes messages). ✅ Why: This tells Spring Boot where to send/receive Kafka messages. 🛠 If you’re running Kafka locally, use localhost:9092 . In production, you may have multiple brokers: kafka1:9092,kafka2:9092,... ✅ spring.kafka.consumer.group-id=my-group 🧠 Meaning: Assigns the consumer group ID to your consumer. ✅ Why: Kafka uses this to track which consumer read which message. If two consumers share the same group ID, Kafka will load-balance messages between them. 🔄 Helps in parallel processing and message tracking ✅ spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer ✅ spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer 🧠 Meaning: Kafka sends binary data . These settings tell Kafka how to convert data back into Java strings (deserialize). ✅ W...
Read more