What is BigQuery?

1. What is BigQuery?

Google BigQuery

A cloud-based data warehouse provided by Google Cloud.

In simple words:

BigQuery is a service used to store, manage, and analyse huge amounts of data very quickly.

Imagine Amazon, Flipkart or PhonePe.

Every day, they generate millions of records:

  • Customer details

  • Transactions

  • Payments

  • Orders

  • Products

  • Sales reports

They need a place where they can store this data and analyse it.

BigQuery helps them do that.

2. Real-world example

Imagine you work at PhonePe.

Every day, 10 lakh people make payments.

Your database contains records like these:

Customer

Amount

City

Rahul

₹500

Pune

Amit

₹200

Mumbai

Priya

₹1,000

Bangalore

Rahul

₹300

Pune

Amit

₹700

Mumbai

Now your manager asks:

How much total payment was made in Pune?

You can write an SQL query:

SELECT SUM(amount)

FROM payments

WHERE city = 'Pune';


Result: ₹800.

Now imagine you have 10 crore or 100 crore transaction records.

BigQuery can analyse this huge amount of data without you managing the underlying servers.

3. Why do we need BigQuery?

Traditional databases like MySQL and PostgreSQL are commonly used for application operations.

For example:

  • Login and registration

  • Saving customer details

  • Updating account information

  • Processing transactions

But when you need to analyse billions of records, a data warehouse like BigQuery is very useful.

Traditional database

BigQuery

Mainly application transactions

Mainly data analysis

Insert, update, delete

Large-scale analytical queries

Often requires infrastructure management

Serverless

OLTP workloads

OLAP workloads

Remember:

  • OLTP = Handling daily transactions.

  • OLAP = Analysing large amounts of data.

4. Important features of BigQuery

Let's understand the terms from your course.

1. Serverless

You don't need to create or manage servers.

Google manages the underlying infrastructure.

2. Highly scalable

Suppose today you have 1 lakh records and tomorrow you have 100 crore records.

BigQuery can scale to handle large data workloads.

3. Cost-effective

You generally pay based on storage and query processing, depending on your pricing model.

4. Fast queries

BigQuery uses Google's distributed infrastructure to process large datasets quickly.

5. SQL support

You can use SQL to analyse data.

If you know MySQL or PostgreSQL SQL, you already have a good starting point.


5. How does BigQuery work?


Understand this simple flow:

  1. Data: You collect data from applications, files, databases, etc.

  2. BigQuery: You load that data into BigQuery.

  3. Dataset: You organise your tables into datasets.

  4. Tables: Your actual records are stored in tables.

  5. SQL Query: You write SQL to analyse the data.

  6. Result: BigQuery returns the result.

6. Important BigQuery terminology

Term

Easy meaning

Project

Main Google Cloud container

Dataset

Collection of related tables

Table

Stores data in rows and columns

Query

SQL command to retrieve or analyse data

Schema

Structure of a table (columns and data types)

API

Allows applications to communicate with BigQuery

Example structure:

Google Cloud Project

       |

       v

    Dataset

       |

       v

     Tables

       |

       v

    Rows & Columns


7. How can Java developers use BigQuery?

Imagine your Spring Boot application wants to analyse payment transactions.

Spring Boot Application
          |
          v
     BigQuery API
          |
          v
     BigQuery Service
          |
          v
      SQL Query
          |
          v
       Results

You can use Google's Java Client Library to connect your Java application to BigQuery.

You can also use Python, .NET and other client libraries.

5. How BigQuery executes a query

Suppose you execute:

SELECT city, SUM(amount)
FROM `banking-project.banking.transactions`
GROUP BY city;

What happens?

  1. You submit the SQL query.

    BigQuery receives your SQL command.

  2. Query processing begins.

    BigQuery understands what data is needed.

  3. Compute resources process the data.

    The work can be distributed across multiple resources.

  4. Results are combined.

    BigQuery calculates the total amount for each city.

  5. You receive the result.

    The output is displayed in the BigQuery console or returned through an API.


How does a Java developer use Google BigQuery?

Let's understand this from your Java Spring Boot developer perspective.

Imagine you are working on a banking application at XYZ Bank. Your application processes thousands or millions of transactions.

You want to analyse:

  • Total transactions per day.

  • Total money transferred by customers.

  • Failed transactions.

  • Transactions by country.

  • Monthly transaction reports.

BigQuery helps your Java application analyse this huge amount of data.

1. How Java and BigQuery communicate

Suppose you have a Spring Boot application.

Java Spring Boot Application

Controller → Service → Repository

Google BigQuery Java Client Library

Connects Java with BigQuery

Google BigQuery

Stores data and executes SQL queries

Query Results

Returned to Java application

Java sends an SQL query to BigQuery. BigQuery executes it and sends the result back to Java.

2. Real-world example: Banking transaction analysis

Suppose BigQuery has a table named transactions.

transaction_id

customer

amount

status

T101

Rahul

500

SUCCESS

T102

Amit

1000

FAILED

T103

Priya

2000

SUCCESS

T104

Rahul

300

SUCCESS

Your manager wants the total successful transaction amount.

First, write SQL:

SELECT SUM(amount) AS total_amount
FROM `banking-project.banking.transactions`
WHERE status = 'SUCCESS';

Result: 2800

Now let's do the same thing using Java.

3. Step-by-step Java implementation

Step 1: Add BigQuery dependency

In your Spring Boot project's pom.xml:

<dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud-bigquery</artifactId>
    <version>2.54.0</version>
</dependency>

This library allows Java to communicate with BigQuery.

Step 2: Configure authentication

Your Java application needs permission to access Google Cloud.

For local development, you can use:

gcloud auth application-default login

For production, use an appropriately permissioned service account or Workload Identity. Never commit service account keys to Git.

Step 3: Create a Java service

Create BigQueryService.java:

@Service
public class BigQueryService {

    private final BigQuery bigQuery;

    public BigQueryService() {
        this.bigQuery =
            BigQueryOptions.getDefaultInstance().getService();
    }

    public long getTotalTransactions() throws Exception {

        String sql = """
            SELECT SUM(amount) AS total_amount
            FROM `banking-project.banking.transactions`
            WHERE status = 'SUCCESS'
            """;

        QueryJobConfiguration queryConfig =
            QueryJobConfiguration.newBuilder(sql).build();

        TableResult result = bigQuery.query(queryConfig);

        FieldValueList row = result.iterateAll()
                                   .iterator().next();

        return row.get("total_amount").getLongValue();
    }
}

What is happening here?

  1. BigQueryOptions – Creates a BigQuery client.

  2. QueryJobConfiguration – Prepares the SQL query.

  3. bigQuery.query() – Executes the query.

  4. TableResult – Receives the query result.

  5. getLongValue() – Extracts the total amount.

Note: This example assumes the query returns a non-null total. Production code should handle empty/null results and exceptions.

Step 4: Create a REST API

@RestController
@RequestMapping("/api/bigquery")
public class BigQueryController {

    @Autowired
    private BigQueryService service;

    @GetMapping("/total")
    public long getTotal() throws Exception {
        return service.getTotalTransactions();
    }
}

Now call:

GET http://localhost:8080/api/bigquery/total

Expected response:

2800
Java → BigQuery → SQL → Result

You have now connected a Java Spring Boot application to Google BigQuery!

4. How does actual banking data reach BigQuery?

This is very important for real-world projects.

BigQuery is generally not used as the main database for processing every customer payment.

A common architecture looks like this:

Java Spring Boot

Processes customer payments

PostgreSQL / MySQL

Stores actual transaction records

Kafka / Data Pipeline

Transfers transaction data

Google BigQuery

Stores and analyses large-scale data

Reports / Analytics Dashboard

For example:

  • PostgreSQL handles payment transactions.

  • Kafka transfers transaction events.

  • BigQuery analyses millions of transactions.

  • Managers view daily and monthly reports.

5. What should you remember as a Java developer?

Technology

Responsibility

Java / Spring Boot

Application development

PostgreSQL

Transaction database

Kafka

Event/data transfer

BigQuery

Large-scale data analytics

SQL

Querying and analysing data

My recommendation: Learn BigQuery in this order:

  1. BigQuery fundamentals

  2. Projects, datasets and tables

  3. Loading data into BigQuery

  4. SQL queries

  5. Java integration

  6. BigQuery with Kafka

  7. Real-world banking analytics project

Comments

Post a Comment

Popular posts from this blog

Async/await

  Async/await An async function is a function declared with the async keyword, and the await keyword is permitted within them. The async and await keywords enable asynchronous, promise-based behavior to be written in a cleaner style, avoiding the need to explicitly configure promise chains. "async and await make promises easier to write" async  makes a function return a Promise await  makes a function wait for a Promise  Async functions Let’s start with the async keyword. It can be placed before a function, like this: async function f() {   return 1; } The word “async” before a function means one simple thing: a function always returns a promise. Other values are wrapped in a resolved promise automatically. For instance, this function returns a resolved promise with the result of 1; let’s test it: async function f() {   return 1; } f().then(alert); // 1 …We could explicitly return a promise, which would be the same: async function f() {   return Promis...
Read more

Valid Parentheses

Given a string s containing just the characters '(', ')', '{', '}', '[' and ']', determine if the input string is valid. An input string is valid if: Open brackets must be closed by the same type of brackets. Open brackets must be closed in the correct order. Every close bracket has a corresponding open bracket of the same type.   Example 1: Input: s = "()" Output: true Example 2: Input: s = "()[]{}" Output: true Example 3: Input: s = "(]" Output: false Example 4: Input: s = "([])" Output: true   Stack-Based Logic: Traverse each character in the string. If it's an opening bracket ( '(' , '{' , '[' ), push it onto the stack. If it's a closing bracket ( ')' , '}' , ']' ): If the stack is empty , return false (nothing to match). If the top of the stack is the corresponding opening bracket , pop it. Otherwise, return false . ...
Read more

kafka

  ✅ spring.kafka.bootstrap-servers=localhost:9092 🧠 Meaning: The address of the Kafka broker (the server that stores and processes messages). ✅ Why: This tells Spring Boot where to send/receive Kafka messages. 🛠 If you’re running Kafka locally, use localhost:9092 . In production, you may have multiple brokers: kafka1:9092,kafka2:9092,... ✅ spring.kafka.consumer.group-id=my-group 🧠 Meaning: Assigns the consumer group ID to your consumer. ✅ Why: Kafka uses this to track which consumer read which message. If two consumers share the same group ID, Kafka will load-balance messages between them. 🔄 Helps in parallel processing and message tracking ✅ spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer ✅ spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer 🧠 Meaning: Kafka sends binary data . These settings tell Kafka how to convert data back into Java strings (deserialize). ✅ W...
Read more