Learning path

Parallel and distributed computing

Understand concurrency versus parallelism, thread pools, processes, speed-up limits, MapReduce, logical clocks, and replication with quorums.

About this subject

Parallel computing uses several processors to finish one job sooner. Distributed computing coordinates several machines that can fail independently and have no shared clock. The two share a theme, which is that doing things at the same time makes correctness and performance harder to reason about.

This path covers thread pools for waiting work, processes for CPU-bound work, Amdahl's and Gustafson's laws, the MapReduce model, Lamport logical clocks, and quorum replication. Each lesson uses a small, complete program that runs on its own. Python has the full sequence. The other languages listed below add lessons that use that language's own idioms and standard library, such as Java threads, C++ smart pointers, C# LINQ, or Node.js streams, and appear only for the topics the language supports. The programs are simplified models chosen to make an idea visible, so they are a foundation for further study rather than a replacement for a full course or for production-grade software.

Before you start: Functions, classes, and the operating-systems lesson on synchronization.

1
Python
Readable code for automation, data, and first programs.
Concurrency with thread pools for waiting work
Overlap tasks that mostly wait, such as network or disk requests, and keep results in order.
Parallelism with processes for CPU-bound work
Split heavy computation across worker processes that run on separate cores.
Speed-up limits: Amdahl's and Gustafson's laws
Compute how much faster a program can get when only part of it runs in parallel.
MapReduce: word count in three phases
Express a computation as map, shuffle, and reduce steps that can run on many machines.
Logical time: Lamport clocks
Order events across machines without a shared clock by counting events and message receipts.
Replication and quorums
Read and write a majority of replicas so that every read overlaps the latest write.
2
C
Learn memory, pointers, and efficient systems programming.
C: splitting a sum across threads without sharing
Give each thread its own range and result slot, then combine the results after joining.
C: the speed-up limit of Amdahl's law
Compute the best possible speed-up for a program in which only part of the work can run in parallel.
3
C++
Build fast applications with modern types and classes.
C++: splitting a sum across std::async tasks
Run two halves of a computation concurrently and combine the results with futures.
C++: lock-free counting with std::atomic
Increment a shared counter from four threads without a mutex.
4
Java
Understand object oriented programs and the JVM.
Java: splitting work across a thread pool
Divide a sum into chunks, run them on an ExecutorService, and combine the Future results.
Java: parallel streams and CompletableFuture
Count items with a parallel stream and combine two asynchronous results.
5
C#
Build typed applications on .NET.
C#: running tasks in parallel with Task.WhenAll
Start independent computations as tasks and await all of their results.
C#: Parallel.For with Interlocked
Spread loop iterations over cores and combine results safely with an atomic add.
6
Kotlin
Write concise, typed code for JVM applications.
Kotlin threads, AtomicInteger, and an executor
Start threads that update a shared counter safely and split a sum across a thread pool.
7
JavaScript
Add behavior to the web and build applications.
JavaScript: concurrency without parallelism
See how async functions interleave on a single thread and finish together under Promise.all.
JavaScript: real parallelism with worker threads
Split an array between two worker threads and add their partial results.
8
TypeScript
JavaScript with static types for dependable code.
TypeScript: running async tasks with a concurrency limit
Start at most N tasks at once, collect results in input order, and measure the peak concurrency.

About this subject

Python

Concurrency with thread pools for waiting work

Parallelism with processes for CPU-bound work

Speed-up limits: Amdahl's and Gustafson's laws

MapReduce: word count in three phases

Logical time: Lamport clocks

Replication and quorums

C

C: splitting a sum across threads without sharing

C: the speed-up limit of Amdahl's law

C++

C++: splitting a sum across std::async tasks

C++: lock-free counting with std::atomic

Java

Java: splitting work across a thread pool

Java: parallel streams and CompletableFuture

C#

C#: running tasks in parallel with Task.WhenAll

C#: Parallel.For with Interlocked

Kotlin

Kotlin threads, AtomicInteger, and an executor

JavaScript

JavaScript: concurrency without parallelism

JavaScript: real parallelism with worker threads

TypeScript

TypeScript: running async tasks with a concurrency limit