CodeWithClarity

  1. What is Dataprep? Raw Data    |    v Dataflow (Filter, Transform, Enrich)    |    v BigQuery Tables (Store Processed Data)    |    v SQL Queries (Dashboard, Reports, Analytics) Dataprep is a data preparation tool used to clean, organise, and transform raw data before analysing it. In simple words: Imagine someone gives you an Excel file containing 10 lakh customer records. But the data has problems: Some names are missing. Some phone numbers are invalid. Some dates have different formats. Some records are duplicated. Some amounts contain incorrect values. Before analysing this data, you need to clean it. Dataprep helps you clean and prepare this data visually, without writing code. 2. Real-world example Imagine you work on a banking application. You receive this raw transaction data: Customer Amount City Rahul 500 Pune Amit NULL Mumbai Priya 2000 pune Rahul 500 Pune Sneha Invalid Bangalore There are some problems: Ami...

  1. What is Dataflow? Google Cloud Dataflow Dataflow is a fully managed service used to process and transform large amounts of data. In simple words: Dataflow takes data from one place, processes it, transforms it, and sends it to another place. Imagine a banking application. Every second, thousands of transactions happen. You receive data like this: Transaction Amount Status T101 500 SUCCESS T102 1000 FAILED T103 2000 SUCCESS T104 300 SUCCESS You want to: Filter failed transactions. Calculate total successful payments. Group transactions by city. Send processed data to BigQuery. Dataflow can perform these operations automatically as part of a data pipeline. When people say "process data" , they simply mean doing some work on the data before storing or analyzing it . Raw Data Imagine Kafka receives: {   "txnId": 101,   "amount": 5000,   "merchant": "amazon",   "status": "success" } This is raw data. Processing = Mod...

1. What is BigQuery? Google BigQuery A cloud-based data warehouse provided by Google Cloud. In simple words: BigQuery is a service used to store, manage, and analyse huge amounts of data very quickly. Imagine Amazon, Flipkart or PhonePe. Every day, they generate millions of records: Customer details Transactions Payments Orders Products Sales reports They need a place where they can store this data and analyse it. BigQuery helps them do that. 2. Real-world example Imagine you work at PhonePe. Every day, 10 lakh people make payments. Your database contains records like these: Customer Amount City Rahul ₹500 Pune Amit ₹200 Mumbai Priya ₹1,000 Bangalore Rahul ₹300 Pune Amit ₹700 Mumbai Now your manager asks: How much total payment was made in Pune? You can write an SQL query: SELECT SUM(amount) FROM payments WHERE city = 'Pune'; Result: ₹800. Now imagine you have 10 crore or 100 crore transaction records. BigQuery can analyse this huge amount of data without you managing the under...

  What is Google Cloud Marketplace? Google Cloud Marketplace is a platform that allows you to quickly deploy pre-configured, production-ready software solutions on Google Cloud. Think of it as an app store for cloud infrastructure and enterprise software . Instead of manually installing and configuring software, you can deploy ready-made solutions with a few clicks. Key Benefits 1. Fast Deployment Marketplace provides pre-built deployment packages from third-party vendors. You can quickly deploy: Databases Monitoring tools Security solutions CI/CD tools Business applications without configuring everything from scratch. 2. Production-Grade Solutions Solutions available in Marketplace are created and maintained by software vendors. Many deployments are already configured using: Terraform Deployment templates Best practices This reduces setup effort and deployment errors. 3. Unified Billing All charges appear on your Google Cloud bill. Benefits: Single invoice Simplified accounting Ce...

1. What is Infrastructure as Code (IaC)? First, imagine you want to create infrastructure in Google Cloud. For example, you need: 2 Virtual Machines (VMs). 1 VPC network. 1 Firewall rule. 1 Load Balancer. There are three ways to create these resources. 1. Google Cloud Console (GUI) You manually click buttons and fill in details to create resources. 2. Cloud Shell (CLI) You execute commands to create resources. 3. Terraform (Infrastructure as Code) You write configuration files that describe the infrastructure you need. Terraform creates it automatically. Simple definition: Infrastructure as Code (IaC) means creating and managing cloud infrastructure using code instead of manually creating resources. For example, instead of manually creating 10 VMs, you write a Terraform configuration and Terraform creates them for you. 2. Why do we need Infrastructure as Code? Suppose your company has three environments: Development environment Testing environment Production environment You need almos...

  Cloud CDN Overview Cloud CDN (Content Delivery Network) uses Google's globally distributed edge locations (Points of Presence) to cache content closer to users, improving performance and reducing load on backend servers. Key Benefits Faster content delivery through edge caching. Reduced latency for end users. Lower backend serving costs. Easy to enable on an Application Load Balancer backend service. How Cloud CDN Works Components Application Load Balancer Backend services: Managed Instance Groups (VMs) Cloud Storage Buckets Cloud CDN Edge Cache Locations Request Flow 1. First Request (Cache Miss) User requests content. Request reaches nearest CDN edge location. CDN checks its cache. Content is not found → Cache Miss . CDN may check nearby cache locations. If still unavailable: Request is sent to the Application Load Balancer. Load Balancer forwards it to the appropriate backend. Backend returns the content. If cacheable, CDN stores a copy at the edge location. 2. Subsequent Re...