Hybrid Enviroments And Migration
What is BGP in AWS?
BGP (Border Gateway Protocol) is like a GPS for internet traffic. It helps AWS and your network find the best path to send data between your on-premises data center and AWS cloud (or between AWS regions).
In AWS, BGP is mainly used in Direct Connect and VPN connections to exchange routes (network paths).
Key Components of BGP in AWS
Autonomous System Number (ASN)
Think of it as your network’s ID card in BGP.
AWS has its own ASN (default is
64512), and you (the customer) can use your ASN.When two networks talk via BGP, they exchange their ASNs to know who they are.
Customer Gateway (CGW)
Your on-premises router/firewall that connects to AWS.
It speaks BGP to advertise your local network routes.
Virtual Private Gateway (VGW) or Transit Gateway (TGW)
AWS’s router at the cloud side.
It listens to BGP announcements from your Customer Gateway.
VGW = older/simple option.
TGW = newer, central hub for connecting multiple VPCs and on-premises sites.
BGP Peering Session
The “handshake” between your Customer Gateway and AWS gateway.
Over this session, they exchange route information dynamically (so no manual route updates needed).
Route Advertisements
AWS tells you: “Hey, here are the IP ranges for my VPCs.”
You tell AWS: “Hey, here are the IP ranges for my on-prem network.”
BGP makes sure both sides know how to reach each other.
Prefixes (IP Ranges)
These are the actual routes (like 10.0.0.0/16 for AWS VPC, 192.168.0.0/24 for on-prem).
BGP exchanges these prefixes.
Redundancy & Failover
BGP supports multiple paths (e.g., two VPN tunnels or two Direct Connect links).
If one fails, traffic automatically reroutes via the other.
Where You Use BGP in AWS
Site-to-Site VPN → BGP helps exchange routes between AWS and your data center.
Direct Connect → BGP is used over the dedicated fiber link for routing.
Transit Gateway → BGP can connect multiple networks and simplify management.
In super simple words: BGP in AWS is like an automatic map updater between your network and AWS. Instead of you manually writing “to reach this AWS network, go here,” BGP exchanges those directions for you, handles multiple routes, and ensures traffic always finds the best way.
IPSec and VPN
IPsec (Internet Protocol Security) is a set of security protocols. Think of it like the lock + encryption system for data traveling across the internet.
It makes sure your data is:
Encrypted (no one can read it)
Authenticated (you know it’s from the right source)
Integrity-protected (data wasn’t changed in transit)
👉 But IPsec is just the technology — it doesn’t create the tunnel by itself. It’s the “security engine.”
VPN (Virtual Private Network) is the tunnel that connects two networks (or devices) securely over the internet.
Imagine a private tunnel inside a public road (the internet).
Your traffic goes through this tunnel, hidden from outsiders.
A VPN uses IPsec (or other protocols) to keep the tunnel safe.
👉 So, VPN is the tunnel and IPsec is the lock + encryption system protecting it.
IPsec + VPN in AWS
When you set up a Site-to-Site VPN in AWS:
The VPN tunnel = secure connection between your data center and AWS VPC.
IPsec = provides encryption and authentication for the traffic in that tunnel.
BGP (optional, from our last chat) = automatically exchanges routing information inside that tunnel.
Simple Analogy
VPN = A private road between your office and AWS.
IPsec = The armored walls + guards that make the road safe.
The Two Phases of IPsec
🔑 Phase 1: IKE Security Association (IKE SA)
Goal: Build a secure channel to negotiate things.
This is like two people agreeing on a language and secret code before starting a conversation.
What happens:
Devices authenticate each other (using shared key or certificates).
They agree on encryption algorithms (AES, 3DES, etc.).
They exchange keys securely (using Diffie-Hellman).
They establish the IKE SA (a control channel).
👉 Think of Phase 1 as setting up a safe meeting room.
👉 This phase is also about agreeing on how to talk securely.
🔒 Security Elements in Phase 1:
Encryption algorithms → How data is scrambled.
Examples: AES-128, AES-256, 3DES
AWS usually supports AES-128 or AES-256 (AES is the modern choice).
Authentication algorithms → How to check data integrity.
Examples: SHA-1, SHA-2 (SHA-256, SHA-384, SHA-512)
AWS recommends SHA-2 (SHA-256 or higher).
Authentication method → How devices trust each other.
Options:
Pre-Shared Key (PSK) – AWS uses this in VPN by default.
Certificates (less common in AWS Site-to-Site).
Diffie-Hellman (DH) group → Used for key exchange strength.
Examples: Group 2, 14, 19, 20
Higher group = stronger security.
AWS supports groups like 14 (2048-bit), 19/20 (elliptic curve).
Lifetime → How long the Phase 1 tunnel lasts before renegotiation.
AWS default: 28,800 seconds (8 hours)
👉 End result: A secure negotiation channel.
📦 Phase 2: IPsec Security Association (IPsec SA)
Goal: Actually protect the user traffic.
Once the meeting room is secure, they now start the real business.
What happens:
Devices agree on what traffic to protect (e.g., 10.0.0.0/16 ↔ 192.168.0.0/24).
They agree on encryption/authentication for data packets.
They establish the IPsec SA, which encrypts the actual data.
👉 Think of Phase 2 as securely exchanging the documents inside the meeting room.
👉 This is where your actual user traffic is encrypted.
🔒 Security Elements in Phase 2:
Encryption algorithms → Scrambles real traffic.
Examples: AES-128, AES-256, AES-GCM
AWS supports AES-128, AES-256, AES-GCM-128, AES-GCM-256
Authentication algorithms → Verifies data wasn’t altered.
Examples: SHA-1, SHA-2 family (SHA-256, SHA-384, SHA-512)
AWS supports SHA-2 (preferred)
Perfect Forward Secrecy (PFS) → Adds extra Diffie-Hellman exchange for every Phase 2 session.
Prevents future key compromise from exposing past traffic.
AWS supports DH groups like 2, 14, 19, 20.
Lifetime → How long the Phase 2 tunnel lasts before renegotiation.
AWS default: 3,600 seconds (1 hour)
👉 End result: A secure data tunnel.
AWS Site to site
What it is?
AWS Site-to-Site VPN is a secure private tunnel between your office/data center network and your AWS VPC over the internet. It uses IPsec encryption to make sure your traffic is safe.
Why it is used
To connect your on-premises network to AWS securely.
For hybrid cloud setups (part of your apps run in AWS, part still on-prem).
As a backup connection to AWS Direct Connect.
To extend your corporate network into AWS so servers in AWS feel like they’re on your local LAN.
How it is used
You create a Customer Gateway (CGW) in AWS that represents your on-prem router/firewall.
You create a Virtual Private Gateway (VGW) (or Transit Gateway for bigger setups) and attach it to your VPC.
You create a VPN Connection in AWS linking the CGW and VGW.
AWS automatically builds two IPsec tunnels for redundancy.
You configure your on-prem device using the AWS VPN configuration file (AWS gives you settings for Cisco, Fortinet, Palo Alto, etc.).
You set up routing (static or dynamic with BGP) so both sides know how to reach each other’s networks.
Key Components / Functionalities
Customer Gateway (CGW)
Your on-prem router/firewall.
Builds the tunnel from your side.
Virtual Private Gateway (VGW) or Transit Gateway (TGW)
AWS’s router on the cloud side.
VGW = for one VPC.
TGW = for multiple VPCs / complex networks.
VPN Tunnels (2 per connection)
Always two IPsec tunnels (for high availability).
If one fails, traffic shifts to the other.
IPsec Encryption
Keeps traffic secure (Phase 1 = negotiation, Phase 2 = actual data).
Routing
Static: You manually tell AWS which network prefixes exist.
Dynamic (BGP): Routes are exchanged automatically.
BGP (optional but important)
Dynamically exchanges routes.
Helps with failover, scaling, and ease of management.
Super Simple Summary
What: AWS Site-to-Site VPN = a secure tunnel over the internet.
Why: To connect your office/data center with AWS securely and privately.
How: You set up CGW (your side) ↔ VGW/TGW (AWS side) with 2 IPsec tunnels, routing, and optionally BGP.
Key Pieces: Customer Gateway, Virtual/Transit Gateway, VPN tunnels, IPsec, Routing (Static/BGP).
Not Fully Highly Available Diagram
Highly Available Diagram
Static and Dynamic VPN
AWS Direct Connect
AWS Direct Connect is a dedicated physical network connection from your data center/office directly into AWS.
👉 Instead of sending traffic over the public internet (like VPN), Direct Connect gives you a private, high-speed, low-latency link straight to AWS.
Why it is used
Faster + more reliable: Lower latency and consistent performance compared to internet VPN.
Cheaper at scale: Data transfer costs are often lower than over the internet.
Security: It’s a private link (though you can still add IPsec for extra encryption if needed).
Hybrid cloud setups: For companies running big workloads across on-prem + AWS.
Compliance: Some industries (finance, healthcare) require private connections.
How it is used
You order a Direct Connect port at an AWS Direct Connect location (AWS data centers/partners).
Your network provider (or you, if colocated) connects your data center to that port.
You create a Virtual Interface (VIF) to connect Direct Connect to your AWS environment:
Private VIF → Connects to a VPC (through VGW or TGW).
Public VIF → Connects to AWS public services (like S3, DynamoDB).
Transit VIF → Connects to multiple VPCs via a Transit Gateway.
You configure BGP peering between your router and AWS to exchange routes.
Your on-prem traffic now flows over the private link instead of the internet.
Key Components / Functionalities
Direct Connect Location
AWS or partner data center where DX is available.
You physically connect here.
Direct Connect Port
The physical connection (1 Gbps, 10 Gbps, 100 Gbps options).
Cross Connect
The physical cable between your router and AWS’s router inside the data center.
Customer Router
Your on-prem router that connects to AWS DX port.
Virtual Interface (VIF)
The logical connection inside DX.
Private VIF, Public VIF, Transit VIF.
Gateway (VGW/TGW)
How AWS routes traffic into your VPC.
BGP (Border Gateway Protocol)
Exchanges routes between your network and AWS.
Non-Resilience Diagram
Resilience But Ok Diagram
Resilience Better Diagram
Resilience Great Diagram
Public VIF and VPN
Neither public or private VIFS offer any form of encryption.
Public VIFs+IPSec VPN is a way to provide access to private VPC resources, using an encrypted IPSEC tunnel for transit.
AWS Transit gateway
The AWS Transit gateway is a network gateway which can be used to significantly simplify networking between VPC's, VPN and Direct Connect.
It can be used to peer VPCs in the same account, different account, same or different region and supports transitive routing between networks.
Without Transit gateway
With Transit gateway
AWS Storage Gateway
AWS Storage Gateway is a hybrid cloud storage service that connects your on-premises environment (your office servers) with AWS cloud storage.
It allows you to store data in AWS without changing your existing applications or workflows. Think of it as a bridge between your local storage and the cloud.
Why it is used
Hybrid storage: Keep some data on-prem and move the rest to the cloud.
Backup & disaster recovery: Automatically store copies in AWS.
Archiving: Move old data to cheaper storage like S3 Glacier.
Seamless integration: Your apps see the storage like a local drive, but it’s in AWS.
Cost efficiency: Pay only for the cloud storage you use.
How it works
Install the Storage Gateway: On your on-prem server, virtual machine, or EC2.
Connect to AWS: Storage Gateway communicates with AWS services (S3, Glacier, EBS).
Choose a gateway type (3 main types).
Data flows transparently between your local apps and AWS.
Gateway Types
Volume Gateway
What it does: Provides block-level storage (like a hard drive).
Modes:
Cached Volumes: Keep frequently accessed data on-prem, rest in AWS.
Stored Volumes: Keep all data on-prem and backup snapshots in AWS.
Use case: Block storage, disaster recovery.
Analogy: Your local hard drive is mirrored to a secure cloud drive.
Volume Stored Mode
Volume Cached Mode
Tape Gateway VTL
What it does: Provides a virtual tape library (VTL) compatible with backup apps.
Where it goes: Stores tapes in S3/Glacier.
Use case: Replace physical backup tapes with cloud tapes.
Analogy: Virtual storage vault replacing old tape reels.
Traditional Tape Backup
Storage Gateway Tape VTL
File Gateway
What it does: Provides file-level access via NFS or SMB.
Where it goes: Stores files as objects in S3.
Use case: Backup files, migrate file shares to cloud.
Analogy: A shared office folder that automatically copies files to a remote warehouse.
Security & Features
Encryption: Data in transit (TLS) and at rest (S3 encryption).
IAM Integration: Control who can access your cloud data.
Compression & Deduplication: Reduce storage costs.
Snapshots: Incremental backups for disaster recovery.
Simple Analogy Wrap-up
File Gateway: Office folder automatically synced to AWS warehouse.
Volume Gateway: Local hard drive mirrored to cloud.
Tape Gateway: Replace physical backup tapes with cloud tapes.
AWS Snow Family
AWS Snow products are physical devices that let you move large amounts of data to/from AWS without relying solely on the internet.
AWS Snowball
What it is:
A physical, rugged storage device you can ship to your site, copy your data onto it, and then ship it back to AWS.
Why it is used:
Move terabytes to petabytes of data quickly.
Useful when internet transfer is slow or expensive.
Ideal for backups, disaster recovery, or migrating large datasets.
Analogy:
Imagine sending a locked suitcase full of files to the cloud instead of emailing thousands of emails.
Key features:
Encrypted storage (256-bit encryption).
Durable & rugged for shipping.
Can handle offline data transfer.
AWS Snowball Edge
What it is:
A Snowball device with compute capabilities.
It can store data AND run local processing (EC2 instances, Lambda functions).
Why it is used:
Move data AND process it at the edge before sending it to AWS.
Useful in remote locations, IoT sites, or field operations.
Reduces the need to send all raw data to AWS.
Analogy:
A portable server + storage box that can do calculations on-site before shipping the results to the cloud.
Key features:
Storage: Up to 100 TB per device.
Compute: Run EC2 instances and Lambda functions locally.
Secure: Encrypted, tamper-resistant.
AWS Snowmobile
What it is:
A truck-sized shipping container for exabyte-scale data transfer.
Literally a semi-truck full of storage that AWS brings to your site.
Why it is used:
Transfer huge amounts of data (up to 100 PB per Snowmobile).
Used by enterprises or data centers migrating entire on-premises datasets to AWS.
Analogy:
Think of it as a moving truck for your entire company’s data.
Key features:
Extremely high capacity (exabytes).
Secured with encryption & GPS tracking.
Used when Snowball is too small for the data volume.
Storage Capacity
Up to 80 TB
Up to 100 TB
Up to 100 PB+
Compute
No
Yes (EC2/Lambda)
No
Transport
Shipped
Shipped
Truck delivery
Use Case
Data migration
Edge processing
Massive data center migration
Analogy
Suitcase
Portable server
Moving truck
Summary:
Snowball → Move large data offline.
Snowball Edge → Move + process data at the edge.
Snowmobile → Move entire data centers to AWS.
AWS Directory service
AWS Directory Service is a managed service that allows you to set up and run directories in the AWS Cloud.
A directory is like a database of users, computers, and permissions.
It helps you centrally manage identities, authentication, and access for your AWS resources and applications.
Why it is used
User management: Control who can log in and what they can access.
Single Sign-On (SSO): Users can access multiple apps with one login.
Integration: Connect AWS resources with on-prem Active Directory.
Security: Centralized authentication reduces risks of weak passwords or mismanaged accounts.
How it works
Set up a directory (managed by AWS).
Add users and groups.
Connect applications, EC2 instances, and other AWS services.
Authenticate users against the directory when they log in.
Directory Service Types
Simple AD
Lightweight, cheaper directory powered by Samba.
Limited features compared to Microsoft AD.
Analogy: Small office directory for basic employee login.
AWS Managed Microsoft AD
Fully managed Microsoft Active Directory in AWS.
Can trust your on-prem AD for hybrid environments.
Analogy: Hiring a cloud-based IT admin who manages your AD.
AD Connector
Proxy to connect AWS with your on-prem Active Directory.
No data stored in AWS; it just forwards authentication requests.
Analogy: A receptionist forwarding login requests to your main office.
Amazon Cognito (related, for apps)
Manage user sign-up/sign-in for web/mobile apps.
Can integrate with social logins like Google/Facebook.
Analogy: Club membership system with multiple ways to register.
Which one to pick
Security & Features
Centralized authentication → All logins go through the directory.
Group-based permissions → Manage access per team/role.
Encryption in transit → Protect login credentials.
Single Sign-On (SSO) → One login for multiple apps.
Simple Analogy Wrap-up
Directory Service = Cloud-based phonebook + security gate.
Managed Microsoft AD = Professional admin managing your directory.
Simple AD = Small office directory.
AD Connector = Receptionist forwarding requests to main office.
Cognito = Club membership system for apps.
AWS DataSync
AWS DataSync is a fully managed service that automates and accelerates moving data between:
On-premises storage → AWS (S3, EFS, FSx)
AWS → AWS
AWS → On-premises
It’s designed to move large amounts of data quickly, securely, and reliably without building complex scripts or manual processes.
Why it is used
Data migration: Move files from your office servers to AWS.
Replication & backups: Keep on-prem and cloud storage synchronized.
Disaster recovery: Copy critical data to AWS for safety.
Large datasets: Handles TBs to PBs of data efficiently.
How it works
Deploy the DataSync agent (on-premises VM or appliance).
Configure source and destination locations (NFS, SMB, S3, EFS, FSx).
Start the transfer task. DataSync handles:
Efficient transfer
Validation and retries
Compression & encryption
Components
DataSync Agent
Connects your on-prem storage to AWS.
Analogy: A “delivery truck” that carries your data to the cloud.
Source & Destination Locations
Where the data comes from and goes to.
Analogy: Home (source) → Warehouse (AWS cloud).
Tasks
Define how and when data is moved.
Analogy: A scheduled delivery plan for your data.
Monitoring & Logging
Tracks success/failure, speed, and errors.
Analogy: GPS tracking for the delivery truck.
Security & Features
Encryption in transit → TLS-secured transfer.
Data validation → Ensures source & destination match.
Automatic retry & fault tolerance → Handles network failures.
Scalable & fast → Optimized for large datasets.
Simple Analogy Wrap-up
DataSync = Automated, secure “data delivery truck” moving files to the cloud.
Agent = The truck itself.
Source & Destination = Pickup and drop-off points.
Task = Delivery schedule.
Monitoring = GPS tracking + proof of delivery.
Amazon FSx
Amazon FSx provides fully managed, high-performance file storage in the cloud. You can access it like a regular network file share, but AWS manages the infrastructure, backups, and scalability.
There are different FSx types for different workloads:
FSx for Windows File Server
What it is:
Managed Windows-compatible file system.
Supports SMB protocol, Active Directory integration, and Windows ACLs.
Why it is used:
Move Windows file shares to AWS without changing applications.
Centralize user permissions via Active Directory.
Backup Windows-based files reliably.
Analogy:
A cloud-based network drive for your Windows users, just like the shared drive in your office.
Key Features:
Full Windows compatibility (SMB, NTFS permissions).
Active Directory integration.
Automatic backups and scaling.
FSx for Lustre
What it is:
High-performance file system optimized for compute-intensive workloads.
Integrates seamlessly with S3 for fast access to cloud data.
Why it is used:
HPC (High-Performance Computing), ML, analytics, or big data workloads.
Process large datasets quickly.
Can link to S3 bucket for fast temporary storage.
Analogy:
A super-fast SSD-based scratchpad for scientists or engineers working on large datasets.
Key Features:
Extremely high throughput and low latency.
Scale storage and compute independently.
Temporary or persistent storage.
Quick Comparison Table
Protocol
SMB (Windows)
Lustre (Linux/HPC)
Workload
Windows apps
HPC, ML, Analytics
Performance
Moderate-High
Very High
AD Integration
Yes
No
S3 Integration
No
Yes
Analogy
Cloud shared drive
Super-fast scratchpad
Simple Analogy Wrap-up
FSx for Windows → Your office shared network drive in the cloud.
FSx for Lustre → A high-speed workspace for heavy data processing.
AWS Transfer Family
AWS Transfer Family is a managed service for transferring files into and out of AWS using standard protocols like:
SFTP (Secure File Transfer Protocol)
FTPS (FTP over SSL/TLS)
FTP
It allows you to move files to Amazon S3 or EFS without building your own transfer servers.
Why it is used
Legacy systems integration: Many companies still use SFTP/FTP/FTPS for file transfers.
Secure file transfers: Fully managed with encryption and audit logging.
Automated workflow: Files can be automatically stored in S3 or EFS.
Compliance: Meets regulatory requirements for secure data transfer.
How it works
Create a Transfer Server → AWS-managed endpoint for SFTP/FTP/FTPS.
Create Users → Define who can access which files or folders.
Connect Storage → Map to S3 bucket or EFS file system.
Transfer Files → Users upload/download files securely.
Monitor & Audit → Use CloudWatch and CloudTrail for logging.
Components
Transfer Server
Endpoint where clients connect via SFTP/FTP/FTPS.
Analogy: The front desk where people hand in or pick up packages.
Users
Define access and permissions for each client.
Analogy: Membership cards with different access rights.
Storage Back-End
Files go into Amazon S3 or EFS automatically.
Analogy: Warehouse where packages are stored safely.
Monitoring & Logging
Tracks transfers, successes/failures, and audit logs.
Analogy: CCTV cameras and logs for every package movement.
Endpoint Types
Security & Features
Encryption in transit → All protocols are secure (SFTP/FTPS).
IAM integration → Control who can create users or access data.
CloudTrail logging → Auditable record of file access.
VPC endpoints → Can restrict traffic to your private network.
Simple Analogy Wrap-up
Transfer Family = Managed “file delivery service” into AWS.
Transfer Server = Front desk for file handoff.
Users = People with membership cards controlling access.
Storage = Secure warehouse (S3/EFS) for your files.
Monitoring = CCTV & logs for every file movement.
Last updated