System Design Course

1 What do we offer?
2 What are Databases?
3 Storage and Retrieval
4 What is a NoSQL database?
5 Types of Databases: Graph, TimeSeries and Object
6 What database should you choose?
7 What is Data Consistency?
8 Linearizable Consistency
9 Eventual Consistency
10 Causal Consistency
11 Quorum
12 Data Consistency Levels Tradeoffs
13 Transaction Isolation Levels – Read Uncommitted Data
14 Read Committed
15 Repeatable Reads
16 Serializable Isolation Level
17 Transaction Level Implementations
18 Conclusion – Transaction Isolation
19 Caching: Basics
20 Caching: Global vs. Local Cache
21 Global Cache Internals
22 Write Policies: Write Back Policy
23 Write Through Policy
24 Write Around Policy
25 Replacement Policies: LRU, LFU and Segmented LRU
26 Breakdown: The physical layer
27 Breakdown: The Routing Layer
28 Breakdown: The behavioral layer
29 Connecting to the internet: ISPs, DNS and everything in between
30 Internal routing: MAC addresses and NAT
31 HTTP, WebSockets, TCP and UDP
32 Communication Standards: REST, GraphQL and GRPC
33 Head of line blocking
34 Video transmission: WebRTC and HTTP-DASH
35 Primary Replica Architectures
36 WAL and Change Data Capture
37 Write Amplification and Split Brain
38 Database Migrations
39 Migrating the Database
40 Migration Across Regions
41 The Oracle and the Timer Wheel
42 Partitioning and Real-life Optimisations
43 Security aspects in distributed systems
44 Token Based Auth
45 SSO and OAuth
46 Access Control Lists and Rule Engines
47 Attack Vectors – Hackers, Developers and Code
48 How videos are protected inside CDNs
49 Design an algorithm for an Audio Search Engine
50 Introduction to design tradeoffs
51 Pull vs. Push Architectures
52 Memory vs. Latency
53 Throughput vs. Latency
54 Consistency vs. Availability
55 Latency vs. Accuracy
56 SQL vs. NoSQL databases
57 Relations Between Tradeoffs
58 The Problem of Distributed Consensus
59 Basic Algorithm
60 Improving the algorithm
61 Termination Conditions
62 Practical Considerations
63 Introduction to Observability
64 Logging – Best Practices
65 Monitoring Metrics
66 Anomaly Detection: Holt-Winters Algorithm
67 Root Cause Analysis
68 Conclusion to Observability
69 Scope and Requirement setting
70 Chapter #1: Service Registration and Proxies
71 Chapter #2: Authentication & Global Caching
72 Chapter #3: API contracts & Versioning
73 Chapter #4: Sending, Tagging & Searching Emails
74 Chapter #5: Contacts & Groups
75 Requirements of a chess website
76 Handling connections at scale
77 Consistent Hashing vs. Sharding
78 Connection related thundering herds
79 Request Batching and Conclusion
80 Live Streaming Requirement Breakdown
81 Video Ingestion and Transformation
82 Transferring processed videos to end users
83 Virtual Advertising for live sports events
84 Google Docs Requirements
85 Document Schema
86 Storing Documents
87 Version History
88 Avoiding Thundering Herds in Crons
89 Compression and Caching
90 Concurrent Writes with Locks
91 Operational Transform Overview
92 Permission Management
93 Requirements of a cab aggregator
94 Static Pricing of Rides
95 Ride Matching
96 Calculating Estimated Time of Arrival
97 Matching Drivers with Live Rides
98 Requirements of a Map application
99 Routing Challenges in Maps
100 Partitioning Algorithm: Splitting the graph into regions
101 Finding the shortest path – A* search
102 GeoHash and other proximity filters
103 Detecting traffic jams and broadcasting updates
104 Calculating an accurate and efficient estimated time of arrival
105 Calling App Design
106 Concept #1: Breaking calls into dialogs
107 Concept #2: The state machine
108 Concept #3: Charging Users
109 Concept #4: Consistent Hashing for caching call state
110 Converting data sources to a lake
111 The Map Reduce Architecture
112 Fault tolerance and engineering optimizations
113 Real time analytics
114 Ranking restaurants in real time and batch
115 Introduction to Low Level Design Series
116 The need for good coding practices
117 Setting Requirements and Objects
118 API and Package Structuring
119 Parent vs. Base class functions
120 Setting the main method
121 Debugging the code
122 Defining responsibilities of AI and Game classes
123 Unit Tests
124 Benefits of Single Responsibility Principle
125 Looking for a Smart AI
126 The need for stateless move exploration
127 Prototype Design Pattern
128 DRY: Dont Repeat Yourself
129 Iterator Design Pattern
130 DRY: Extracting Common Pieces
131 New feature request: Find Forks!
132 Building a Rule Engine
133 Builder Design Pattern
134 Summary: Open-Closed Principle
135 New feature request: Timers!
136 Encapsulating responsibilities in objects
137 Extracting Nested Objects
138 Having Multiple AI personalities
139 Singleton and Null Object Design Pattern
140 Chain of Responsibility Pattern
141 Interface Segregation Principle
142 Dependency Inversion Principle
143 New feature request: Undo Moves with Momento!
144 Proxy and Flyweight Design Pattern
145 The Labyrinth: Five Level Deep For-Loops!
146 State and Strategy Design Pattern
147 Writing Tests for Timed Games
148 Factory Design Pattern
149 Unit Tests and Object Pools
150 Setting Timed Test Cases
151 Fixing bugs in Game Logic and Factories
152 Strategy Design Pattern: Implementation and Benefits
153 New Feature: Email Notifications
154 Inheritance vs. Composition
155 The Publisher Subscriber Model
156 Problem Statement
157 Defining objects states and behaviors
158 Simplified Balances algorithm
159 APIs and coding requirements
160 Low Level Design
161 Coding: Building the service skeleton and defining objects
162 Coding: Algorithm
163 Coding: Test cases
164 Conclusion and Tips
165 #1: Caching: Engineering requirements
166 #2: Read your own writes
167 #3: Defining the core requirements
168 #4: Asynchronous processing: Keeping threads available
169 #5: View Consistency through thread assignment
170 Code #1: Code skeleton for ‘Get’ and ‘Set’
171 Code #2: Setting up fast evictions
172 Code #3: Thread Assignment and updating GET + SET
173 Code #4: Hot loading and Request Collapsing
174 What is an event bus?
175 Breaking down engineering requirements
176 The mechanics of the bus
177 Why use threads?
178 Code Skeleton and using an extensible Retry Algorithm
179 The Keyed Executor
180 Pull and Push Subscribers
181 Defining the data structures in the bus
182 Wrapper Classes
183 Coding the tests
184 Conclusion
185 What will you learn
186 Rate Limiter Overview
187 Implementation Details
188 API Breakdown
189 Models, Controllers, Packages and ThreadPools
190 Reading and Writing Data
191 RateLimiter Client Setup
192 Sliding Window Algorithm Setup
193 Dependency Inversion
194 Sliding Window with Backpressure
195 Clearing Stale Entries in Sliding Window
196 January 2022: Datastores and their Tradeoffs
197 March 2022: Threads and Request Collapsing
198 April 2022: Chess Low Level Design
199 May 2022: Message Queues
200 June – Design a Payment System
201 July – Caching, Paxos and Zookeeper
202 Data Corruption and Alerts Monitoring
203 Caching: Doubts and Clarifications
204 August – Database Doubts
205 Design a subscription manager
206 Rule Engines
207 October – Kafka Internals
208 October 2022 – Implementing a File System
209 Pessimistic and Optimistic Locking
210 Producers and Consumers: November Meet
211 IRCTC Algorithm: December
212 Google Docs and Chess Design Queries
213 Jan 2023: DNS, CDNs, and everything in between
214 Jan 2023: API design and other discussions
215 Feb 2023: Permission Management System
216 March 2023: Eventual Consistency Levels
217 March 2023: Fetch top K hits in a distributed system
218 April 2023: Distributed Consensus and Load Balancing
219 April 2023: Graph Databases with Facebook TAO
220 May 2023: The SideCar Pattern
221 May 2023: Permission Management
222 June 2023: Clients, Comments and APIs
223 July 2023: Design an Event Handler
224 July 2023: Highly Scalable Concurrent Booking Systems
225 August 2023 – Google Calendar Low-Level Design
226 August 2023 – Designing IRCTC
227 September – Design Judge Launch
228 September – Tech in a startup
229 October – Live Watch Party Design
230 November – Memcached Reading
231 November – Memcached Revision
232 December: Memcached Paper Reading – 2
233 Monoliths vs Microservices
234 Horizontal vs Vertical Scaling
235 Load Balancing
236 An introduction to distributed systems
237 Single Point of Failure
238 Sharding
239 Service discovery and Heartbeats
240 Capacity Planning and Estimation: How much data does YouTube store daily?
241 Content Delivery Networks
242 API Design Goals
243 API Design in Practice
244 The Message Queue: Problem Statement
245 Asynchronous processing: benefits
246 Publisher Subscriber Models
247 Event Driven Architectures
248 Event Driven Architectures – Advantages
249 Event Driven Architectures – Disadvantages
250 Distributed data consistency
251 Data Consistency across Continents
252 Leader-follower architectures
253 Tradeoffs – 2 phase commit protocol
254 Difference between SQL and NoSQL
255 Cassandra Internals
256 Quorom
257 Write Heavy DB Design
258 Write Heavy DB Design – Merging Sorted String Tables
259 Write Heavy DB Design – Query optimisations
260 Location Representation
261 Data Structure – Quad Tree
262 Range Queries and Hilbert Curve
263 Motivation
264 Infrastructure Requirements
265 Practical considerations
266 Containers: A friendly Introduction
267 Containers: Usecases
268 System Requirements
269 One to one messaging
270 Last Seen Timestamps
271 Group Messaging
272 Requirements
273 Image Storage: Files vs. Database BLOB
274 Profile Creation and Authentication
275 One to One chat messaging
276 Matching right-swiped users
277 Serving recommendations to users
278 Requirements
279 Designing the Data Schema
280 Following Users
281 User Feed Service
282 Scaling the Instagram news feed
283 Anomaly Detection
284 Video Processing
285 ISP Synergy
286 System Design Tips
287 Dos and Don’ts
288 Online Judge