GCP Networking — Logging & Traffic Analysis
- Get link
- X
- Other Apps
GCP Networking — Logging & Traffic Analysis
1. What is Cloud Logging?
Cloud Logging collects and stores logs generated by Google Cloud resources.
In this section, the main focus is on:
- VPC Flow Logs
- Flow Analyzer
- Packet Mirroring
- Cloud NAT Logging
- Exporting logs to BigQuery
- Visualizing logs using Data Studio / Looker Studio
Simple flow
VM / VPC ↓ Traffic & Logs ↓ Cloud Logging ↓ BigQuery ↓ Data Studio / Looker Studio ↓ Reports & Dashboards
2. VPC Flow Logs
What is it?
VPC Flow Logs record sampled information about network traffic going to and from VM instances.
Think of it as a network traffic diary.
For example:
VM-A 10.10.0.2 | | TCP traffic ↓ VM-B 10.50.0.2
VPC Flow Logs can tell you things such as:
Who communicated with whom?
Which port was used?
Which protocol?
How much data was transferred?
When did the communication happen?
Important
VPC Flow Logs are enabled per subnet.
If you enable Flow Logs on:
Subnet-A
then the VM instances inside that subnet have their traffic logged.
You don't enable it individually for each VM.
3. Why use VPC Flow Logs?
VPC Flow Logs are useful for:
🔍 Network monitoring
Understand network communication.
🛡️ Security analysis
Identify suspicious or unexpected traffic.
🕵️ Forensics
Investigate what happened during a security incident.
💰 Cost optimization
Understand network usage and potentially reduce networking costs.
🐛 Troubleshooting
Understand communication between VMs.
4. Near-real-time visibility
Google Cloud provides near-real-time visibility into flow logs.
The course mentions log updates approximately every 5 seconds.
So you can monitor network activity without waiting for a long period.
Also, logging the traffic doesn't introduce an additional routing delay for the actual packets.
Important distinction
Actual network traffic ↓ Continues normally Flow logging ↓ Records information about the traffic
The log recording is separate from forwarding the actual packet.
5. What information does VPC Flow Logs contain?
One of the most important concepts is the 5-tuple.
5-Tuple
A network connection can be identified using:
| Field | Meaning |
|---|---|
| Source IP | Where traffic came from |
| Destination IP | Where traffic is going |
| Source Port | Port on the sender |
| Destination Port | Port on the receiver |
| Protocol | TCP, UDP, etc. |
Example:
Source IP : 10.10.0.2 Source Port : 5000 Destination IP : 10.50.0.2 Destination Port: 443 Protocol : TCP
Together, these are called the 5-tuple.
Other information
VPC Flow Logs can also contain:
- Start time
- End time
- Number of packets
- Number of bytes
- VM/instance information
- VPC information
- Geographic information
6. VM-to-VM traffic
Suppose:
VM-A 10.10.0.2 | | 1,224 bytes ↓ VM-B 10.50.0.2
VM-B sends a response:
VM-B 10.50.0.2 | | 5,342 bytes ↓ VM-A 10.10.0.2
If Flow Logs are enabled on both subnets, traffic can be recorded from both sides.
Important
For VM-to-VM traffic where you want both sides represented:
VM-A subnet → Flow Logs ON VM-B subnet → Flow Logs ON
7. Flow Analyzer
Normally, you could analyze VPC Flow Logs by writing SQL queries.
But Google Cloud provides Flow Analyzer to make this easier.
Flow Analyzer
It helps you understand:
- Inbound traffic
- Outbound traffic
- Source/destination IPs
- Ports
- Protocols
- Traffic patterns
It uses 5-tuple granularity.
Simple idea
Instead of:
Write complex SQL ↓ Analyze logs ↓ Understand traffic
you can use:
Flow Analyzer ↓ Analyze VPC traffic ↓ Understand traffic
It is built using Log Analytics and powered by BigQuery.
8. Flow Logs vs Flow Analyzer
Don't confuse these two.
| VPC Flow Logs | Flow Analyzer |
|---|---|
| Collects traffic information | Analyzes traffic information |
| Produces log data | Helps understand the data |
| Enabled at subnet level | Used for analysis |
| Records sampled network flows | Provides traffic analysis |
| Data can be sent to BigQuery | Uses BigQuery/Log Analytics capabilities |
Easy memory trick
Flow Logs = Collect
Flow Analyzer = Analyze
9. Packet Mirroring
Packet Mirroring is different from VPC Flow Logs.
VPC Flow Logs
Records metadata about network flows.
Example:
Source IP → Destination IP Port Protocol Bytes Packets Time
It does not give you the complete packet payload.
Packet Mirroring
Packet Mirroring copies network traffic from selected VM instances and sends the copies to another system for inspection.
Original traffic ↓ VM ↓ Normal destination + ↓ Mirrored copy ↓ Security / inspection system
10. What does Packet Mirroring capture?
Packet Mirroring can capture:
- Ingress traffic
- Egress traffic
- Packet data
- Headers
- Payloads
So it provides much deeper visibility than Flow Logs.
Example
You could use security software to inspect mirrored traffic:
VM ↓ Packet Mirroring ↓ Security appliance ↓ Threat detection
This can help detect:
- Security threats
- Anomalies
- Application performance problems
11. Important Packet Mirroring difference
VPC Flow Logs
Sampling
Traffic ↓ Some traffic information recorded
Packet Mirroring
Traffic is copied for inspection
Traffic ↓ Complete mirrored traffic ↓ Inspection system
The course emphasizes that Packet Mirroring exports all traffic rather than only traffic observed during sampling periods.
Trade-off
Packet Mirroring consumes additional bandwidth because traffic is being copied.
12. Cloud NAT Logging
Cloud NAT allows private VMs to access the internet without requiring public IP addresses.
Cloud NAT can also generate logs.
Cloud NAT Logging records things like:
1. NAT connection created
Private VM 10.0.0.5 ↓ Cloud NAT ↓ Internet
A log can be generated when the NAT connection is created.
2. NAT port unavailable
If Cloud NAT doesn't have an available port for a connection:
VM ↓ Cloud NAT ↓ ❌ No NAT port available
A log can be generated for the dropped packet.
You can choose:
Log connections Log dropped packets Both
The logs are sent to Cloud Logging.
13. Cloud NAT Logging limitation
Cloud NAT Logging handles:
- TCP
- UDP
For dropped packets, the logging described here applies to outbound/egress TCP and UDP traffic.
It does not log dropped incoming packets in the same way.
14. Cloud Logging → BigQuery
Cloud Logging is useful for viewing individual logs.
But when you have large amounts of network data, analyzing everything directly in Cloud Logging can become difficult.
That's where BigQuery comes in.
Architecture
VPC Flow Logs ↓ Cloud Logging ↓ BigQuery ↓ SQL analysis
BigQuery is designed to run SQL queries over very large datasets.
15. Why export logs to BigQuery?
You can use BigQuery to analyze:
📈 Traffic growth
Understand how network traffic is increasing.
Month 1 → 100 GB Month 2 → 150 GB Month 3 → 250 GB
This can help with capacity planning.
💰 Cost optimization
Find where network traffic is going and determine whether architecture changes could reduce network costs.
🛡️ Security / forensics
Investigate:
Which IPs communicated with my server? When? How much traffic? Which ports?
16. Example: Find top IP addresses
Suppose your web server receives traffic from:
10.1.1.5 10.1.1.10 10.1.1.15 203.x.x.x
You can query BigQuery to find:
Top IP addresses ↓ Traffic volume ↓ Communication patterns
Then you can investigate those IPs further.
For example, depending on your security and architecture requirements, you might decide to:
- Change infrastructure location
- Optimize network architecture
- Investigate suspicious traffic
- Restrict unwanted access
The decision itself depends on your organization's requirements; BigQuery provides the data for analysis.
17. BigQuery + Data Studio / Looker Studio
BigQuery is excellent for data analysis, but raw SQL results aren't always easy for everyone to understand.
So you can connect BigQuery to Data Studio (now Looker Studio).
VPC Flow Logs ↓ Cloud Logging ↓ BigQuery ↓ Looker Studio ↓ Dashboard
Looker Studio can turn raw data into:
- Charts
- Graphs
- Metrics
- Tables
- Dashboards
- Reports
18. BigQuery vs Looker Studio
Remember this distinction:
BigQuery
Analyze the data
SQL ↓ Large datasets ↓ Results
Looker Studio
Visualize the data
BigQuery data ↓ Charts ↓ Graphs ↓ Dashboard
Easy memory trick
BigQuery = Analyze
Looker Studio = Visualize
19. Complete real-world example
Imagine you have an application:
Internet ↓ Load Balancer ↓ Web VM ↓ Application VM ↓ Database
You want to understand network traffic.
Step 1 — Enable VPC Flow Logs
Enable Flow Logs on the relevant subnets.
Web subnet → Flow Logs ON App subnet → Flow Logs ON DB subnet → Flow Logs ON
Step 2 — Logs are generated
You get information such as:
Source IP Destination IP Port Protocol Bytes Packets Time
Step 3 — Send logs to BigQuery
Cloud Logging ↓ BigQuery
Step 4 — Analyze
Use SQL to answer:
Which IP sends the most traffic? Which VM receives the most traffic? Which ports are most used? How much traffic is generated? How is traffic changing over time?
Step 5 — Visualize
Connect BigQuery to Looker Studio:
BigQuery ↓ Looker Studio ↓ Network Dashboard
You can create charts such as:
Traffic by VM Traffic by IP Traffic by region Traffic over time Top source IPs Top destination IPs
20. Most important comparison
| Feature | VPC Flow Logs | Packet Mirroring | Cloud NAT Logging |
|---|---|---|---|
| Main purpose | Network-flow visibility | Deep packet inspection | NAT monitoring |
| Data | Flow metadata | Packet data + headers/payload | NAT connections/errors |
| Sampling | Yes | Captures mirrored traffic | Event-based |
| Useful for | Monitoring, troubleshooting, forensics | Security inspection, anomaly detection | NAT troubleshooting |
| Stored in | Cloud Logging | Forwarded for inspection | Cloud Logging |
| BigQuery analysis | Yes | Can be analyzed depending on destination/setup | Yes, after export |
21. Exam / interview points ⭐
Remember these:
- VPC Flow Logs are enabled per subnet.
- They record sampled network flows.
- The 5-tuple = source IP, destination IP, source port, destination port, protocol.
- Flow Logs can contain bytes, packets, timestamps, VM/VPC and geographic information.
- Flow Analyzer helps analyze VPC traffic without manually writing complex SQL.
- Packet Mirroring copies traffic for deeper inspection.
- Packet Mirroring can capture headers and payloads.
- Packet Mirroring consumes additional bandwidth.
- Cloud NAT Logging records NAT connections and certain NAT errors.
- Cloud NAT Logging supports TCP and UDP.
- BigQuery = large-scale SQL analysis.
- Looker Studio/Data Studio = visualization and dashboards.
- Typical architecture:
VPC ↓ Flow Logs ↓ Cloud Logging ↓ BigQuery ↓ Looker Studio ↓ Dashboard
One-line summary
VPC Flow Logs collect network-flow information, Flow Analyzer helps analyze it, Packet Mirroring provides deeper packet-level inspection, Cloud NAT Logging monitors NAT events, BigQuery performs large-scale analysis, and Looker Studio turns the results into dashboards.
- Get link
- X
- Other Apps
Comments
Post a Comment