Dataprep

 

1. What is Dataprep?

Raw Data
   |
   v
Dataflow
(Filter, Transform, Enrich)
   |
   v
BigQuery Tables
(Store Processed Data)
   |
   v
SQL Queries
(Dashboard, Reports, Analytics)

Dataprep is a data preparation tool used to clean, organise, and transform raw data before analysing it.

In simple words:

Imagine someone gives you an Excel file containing 10 lakh customer records.

But the data has problems:

  • Some names are missing.

  • Some phone numbers are invalid.

  • Some dates have different formats.

  • Some records are duplicated.

  • Some amounts contain incorrect values.

Before analysing this data, you need to clean it.

Dataprep helps you clean and prepare this data visually, without writing code.

2. Real-world example

Imagine you work on a banking application.

You receive this raw transaction data:

Customer

Amount

City

Rahul

500

Pune

Amit

NULL

Mumbai

Priya

2000

pune

Rahul

500

Pune

Sneha

Invalid

Bangalore

There are some problems:

  1. Amit's amount is missing.

  2. Pune and pune have different capitalisation.

  3. Rahul's transaction is duplicated.

  4. Sneha's amount is invalid.

You need to clean this data before generating reports.

Dataprep helps you:

  • Remove duplicate records.

  • Handle missing values.

  • Correct data formats.

  • Standardise city names.

  • Identify incorrect values.


3. How does Dataprep work?

Think of it as a data-cleaning pipeline.

Raw Data

Excel, CSV, BigQuery, Cloud Storage

Dataprep

Clean, format, remove duplicates, validate

Clean Data

Structured and ready for analysis

BigQuery / Cloud Storage

Analytics and machine learning

4. Important features of Dataprep

Let's understand the terms from your lecture.

Feature

Easy explanation

Visual data preparation

Clean data using a graphical interface

Automatic schema detection

Identifies column names and structures

Data type detection

Identifies numbers, dates, strings, etc.

Join suggestions

Suggests which datasets can be combined

Anomaly detection

Identifies unusual or incorrect data

Serverless

No infrastructure management

Auto-suggestions

Recommends transformations based on your actions

What is schema detection?

Suppose your uploaded file contains:

Customer_Name, Amount, Transaction_Date
Rahul, 500, 2026-11-01
Amit, 1000, 2026-11-02

Dataprep can identify:

  • Customer_Name → String

  • Amount → Number

  • Transaction_Date → Date

You don't need to manually define everything.

What is anomaly detection?

Suppose normal transaction amounts are between ₹100 and ₹10,000.

Suddenly, one record contains ₹10,00,000.

Dataprep can help identify this unusual value for investigation.

It does not automatically mean the transaction is fraudulent.

5. How Dataprep works with other Google Cloud services

Your lecture mentions BigQuery, Cloud Storage and Dataflow.

Let's understand their responsibilities.


Example workflow:

Step 1: Input data

Data comes from:

  • BigQuery

  • Cloud Storage

  • Uploaded CSV or Excel files

Step 2: Dataprep

Clean and prepare the raw data.

Step 3: Dataflow

Process the prepared data through a transformation pipeline when required.

Step 4: Output

Send the refined data to:

  • BigQuery for analytics.

  • Cloud Storage for storage.

  • Other systems for further processing.

6. Dataprep vs Dataflow vs BigQuery

This is particularly important for GCP certification.

Service

Main responsibility

Simple meaning

Dataprep

Data preparation

Clean the data

Dataflow

Data processing

Process the data

BigQuery

Data analytics

Analyse the data

Remember with this example:

Imagine you receive 10 lakh banking transactions.

  • Dataprep: Removes duplicate and incorrect records.

  • Dataflow: Processes and transforms incoming transaction data.

  • BigQuery: Analyses transactions and generates reports.


Raw Data
   |
   v
Dataflow
(Filter, Transform, Enrich)
   |
   v
BigQuery Tables
(Store Processed Data)
   |
   v
SQL Queries
(Dashboard, Reports, Analytics)


Dataprep is a low-code visual tool for data cleaning and preparation, whereas Dataflow is a fully managed distributed processing engine used for building large-scale batch and streaming data pipelines.

Comments

Post a Comment

Popular posts from this blog

Async/await

  Async/await An async function is a function declared with the async keyword, and the await keyword is permitted within them. The async and await keywords enable asynchronous, promise-based behavior to be written in a cleaner style, avoiding the need to explicitly configure promise chains. "async and await make promises easier to write" async  makes a function return a Promise await  makes a function wait for a Promise  Async functions Let’s start with the async keyword. It can be placed before a function, like this: async function f() {   return 1; } The word “async” before a function means one simple thing: a function always returns a promise. Other values are wrapped in a resolved promise automatically. For instance, this function returns a resolved promise with the result of 1; let’s test it: async function f() {   return 1; } f().then(alert); // 1 …We could explicitly return a promise, which would be the same: async function f() {   return Promis...
Read more

Valid Parentheses

Given a string s containing just the characters '(', ')', '{', '}', '[' and ']', determine if the input string is valid. An input string is valid if: Open brackets must be closed by the same type of brackets. Open brackets must be closed in the correct order. Every close bracket has a corresponding open bracket of the same type.   Example 1: Input: s = "()" Output: true Example 2: Input: s = "()[]{}" Output: true Example 3: Input: s = "(]" Output: false Example 4: Input: s = "([])" Output: true   Stack-Based Logic: Traverse each character in the string. If it's an opening bracket ( '(' , '{' , '[' ), push it onto the stack. If it's a closing bracket ( ')' , '}' , ']' ): If the stack is empty , return false (nothing to match). If the top of the stack is the corresponding opening bracket , pop it. Otherwise, return false . ...
Read more

kafka

  ✅ spring.kafka.bootstrap-servers=localhost:9092 🧠 Meaning: The address of the Kafka broker (the server that stores and processes messages). ✅ Why: This tells Spring Boot where to send/receive Kafka messages. 🛠 If you’re running Kafka locally, use localhost:9092 . In production, you may have multiple brokers: kafka1:9092,kafka2:9092,... ✅ spring.kafka.consumer.group-id=my-group 🧠 Meaning: Assigns the consumer group ID to your consumer. ✅ Why: Kafka uses this to track which consumer read which message. If two consumers share the same group ID, Kafka will load-balance messages between them. 🔄 Helps in parallel processing and message tracking ✅ spring.kafka.consumer.key-deserializer=org.apache.kafka.common.serialization.StringDeserializer ✅ spring.kafka.consumer.value-deserializer=org.apache.kafka.common.serialization.StringDeserializer 🧠 Meaning: Kafka sends binary data . These settings tell Kafka how to convert data back into Java strings (deserialize). ✅ W...
Read more