Python Advanced From Intermediate to Pro – Master Modern Python in 2026

If you're seeing this book's cover or link pointing to Amazon.com (USA marketplace)

Table of Contents

Advanced Python Mastery – From Intermediate to Pro in 2026 Complete Notes for Serious Learners & Developers

1. Python Intermediate Recap & Advanced Setup 1.1 Quick Review: Lists, Dicts, Functions, Modules 1.2 Virtual Environments & pip (venv, requirements.txt) 1.3 Code Formatting & Linting (Black, Flake8, isort) 1.4 Type Hints & Static Typing (typing module, mypy) 1.5 Debugging Techniques (pdb, logging, VS Code debugger)

2. Object-Oriented Programming (OOP) in Depth 2.1 Classes & Objects – Advanced Features 2.2 init, self, str, repr 2.3 Inheritance & super() 2.4 Method Overriding & Polymorphism 2.5 Encapsulation: Private & Protected Members 2.6 Properties (@property, @setter, @deleter) 2.7 Class Methods, Static Methods, @classmethod, @staticmethod 2.8 Multiple Inheritance & Method Resolution Order (MRO) 2.9 Abstract Base Classes (abc module) 2.10 Composition vs Inheritance – When to Use What

3. Advanced Data Structures & Collections 3.1 collections module: namedtuple, deque, Counter, defaultdict, OrderedDict 3.2 dataclasses (Python 3.7+) – Cleaner Classes 3.3 Heapq – Priority Queues 3.4 Bisect – Binary Search & Insertion

4. Functional Programming Tools 4.1 Lambda Functions – Advanced Uses 4.2 map(), filter(), reduce() (functools) 4.3 List, Dict & Set Comprehensions – Advanced Patterns 4.4 Generator Expressions vs List Comprehensions 4.5 Generators & yield – Memory Efficient Iteration 4.6 Generator Functions, Generator Expressions 4.7 yield from & Sub-generators 4.8 itertools module – Powerful Iterators

5. Decorators & Higher-Order Functions 5.1 What are Decorators? 5.2 Writing Simple Decorators 5.3 Decorators with Arguments 5.4 @property, @classmethod, @staticmethod as Decorators 5.5 @lru_cache (functools) – Memoization 5.6 Chaining Decorators 5.7 Class Decorators

6. Context Managers & with Statement 6.1 Understanding Context Managers 6.2 Writing Custom Context Managers (enter, exit) 6.3 @contextmanager Decorator 6.4 Common Use Cases: File Handling, Database Connections, Timing

7. Exception Handling – Advanced 7.1 try-except-else-finally Deep Dive 7.2 Raising Custom Exceptions 7.3 Creating Custom Exception Classes 7.4 Exception Chaining (cause, context) 7.5 Logging vs print() for Production

8. File Handling & Data Formats 8.1 Reading/Writing Text & Binary Files 8.2 with Statement Best Practices 8.3 CSV – csv module 8.4 JSON – json module & serialization 8.5 Pickle – Serializing Python Objects 8.6 Working with Large Files (chunk reading)

9. Concurrency & Parallelism 9.1 Threading vs Multiprocessing vs Asyncio 9.2 threading module – Basics & Locks 9.3 multiprocessing – Process Pools 9.4 asyncio – Async/Await Syntax 9.5 aiohttp, async file I/O 9.6 When to Use What (GIL Explanation)

10. Metaclasses & Advanced OOP 10.1 What are Metaclasses? 10.2 type() as a Metaclass 10.3 Creating Custom Metaclasses 10.4 new vs init 10.5 Practical Use Cases (Frameworks, ORMs)

11. Design Patterns in Python 11.1 Singleton, Factory, Abstract Factory 11.2 Observer, Strategy, Decorator Pattern 11.3 Pythonic Alternatives to Classic Patterns

12. Performance Optimization 12.1 Time & Space Complexity Basics 12.2 Profiling (cProfile, timeit) 12.3 Efficient Data Structures 12.4 Caching & Memoization 12.5 NumPy & Pandas for Speed

13. Testing in Python 13.1 unittest vs pytest 13.2 Writing Unit Tests 13.3 Mocking (unittest.mock) 13.4 Test-Driven Development (TDD) Basics

14. Popular Libraries & Real-World Tools 14.1 requests – HTTP & APIs 14.2 BeautifulSoup & Scrapy – Web Scraping 14.3 pandas & NumPy – Data Manipulation 14.4 Flask / FastAPI – Web Development 14.5 SQLAlchemy / Django ORM – Databases

15. Mini Advanced Projects & Best Practices 15.1 CLI Tool with argparse / click 15.2 Async Web Scraper 15.3 Custom Decorator-based Logger 15.4 Thread-Safe Counter Class 15.5 Data Pipeline with Generators 15.6 PEP 8, PEP 257, Documentation & Git Workflow

16. Next Level Roadmap (2026+) 16.1 Web Development (FastAPI, Django) 16.2 Data Science / Machine Learning 16.3 DevOps & Automation 16.4 Contributing to Open Source 16.5 Recommended Books & Courses

1. Python Intermediate Recap & Advanced Setup

Before diving into advanced Python, let’s quickly refresh the core concepts and set up a professional development environment (2026 standard).

1.1 Quick Review: Lists, Dicts, Functions, Modules

Lists (mutable, ordered)

Python

fruits = ["apple", "banana", "mango"] fruits.append("kiwi") # add fruits[1] = "cherry" # update print(fruits[::2]) # ['apple', 'mango']

Dictionaries (mutable, key-value)

Python

person = {"name": "Anshuman", "age": 25, "city": "Muzaffarpur"} person["skills"] = ["Python", "FastAPI"] print(person.get("phone", "Not found")) # Not found for k, v in person.items(): print(f"{k}: {v}")

Functions (with args, *kwargs, defaults, return)

Python

def greet(name="Guest", hobbies, *extra): msg = f"Hello {name}!" if hobbies: msg += f" Hobbies: {', '.join(hobbies)}" if extra: msg += f" | {extra}" return msg print(greet("Rahul", "coding", "cricket", city="Patna")) # Hello Rahul! Hobbies: coding, cricket | {'city': 'Patna'}

Modules (import styles)

Python

import math as m from datetime import datetime from random import randint, choice print(m.sqrt(16)) # 4.0 print(datetime.now().strftime("%d-%b-%Y")) # 05-Mar-2026 print(randint(1, 100))

Quick self-check question: What’s the difference between list.append() and list.extend()? (Answer: append adds one item, extend adds multiple from iterable)

1.2 Virtual Environments & pip (venv, requirements.txt)

Never install packages globally — always use virtual environments.

Create & activate venv (2026 best way):

Bash

# Create python -m venv venv # Windows venv\Scripts\activate # Mac/Linux source venv/bin/activate # You’ll see (venv) in terminal

Install packages

Bash

pip install requests pandas fastapi uvicorn

Create requirements.txt

Bash

pip freeze > requirements.txt

Install from requirements.txt (on new machine / sharing project)

Bash

pip install -r requirements.txt

Deactivate

Bash

deactivate

Pro Tip: Add venv/ to .gitignore so it’s not pushed to GitHub.

1.3 Code Formatting & Linting (Black, Flake8, isort)

Write beautiful, consistent code automatically.

Install the most popular tools (2026 standard stack):

Bash

pip install black flake8 isort mypy

Recommended VS Code settings (add to settings.json):

JSON

{ "editor.formatOnSave": true, "python.formatting.provider": "black", "isort.args": ["--profile", "black"], "python.linting.flake8Enabled": true, "python.linting.mypyEnabled": true }

Common commands

Bash

# Format all files black . # Sort imports isort . # Check style violations flake8 . # Fix auto-fixable issues (isort + black) isort . && black .

Before vs After (Black + isort)

Python

# Before def my_function(a,b,c=10,*args,**kwargs):pass # After def my_function(a, b, c=10, args, *kwargs): pass

1.4 Type Hints & Static Typing (typing module, mypy)

Add types to make code self-documenting and catch bugs early.

Basic type hints

Python

def add(a: int, b: int) -> int: return a + b def greet(name: str = "Guest") -> str: return f"Hello, {name}!" scores: list[int] = [85, 92, 78] person: dict[str, str | int] = {"name": "Anshuman", "age": 25}

Advanced hints (Python 3.9+)

Python

from typing import List, Dict, Optional, Union, Any def process_data(data: List[Dict[str, Union[int, str]]]) -> Optional[float]: # ... return None

Using mypy (static type checker)

Bash

mypy your_script.py # Or check whole project mypy .

Install mypy

Bash

pip install mypy

Benefits:

• VS Code shows errors instantly

• Reduces runtime bugs

• Makes large projects maintainable

1.5 Debugging Techniques (pdb, logging, VS Code debugger)

1. print() debugging (quick but limited)

2. logging module (production standard)

Python

import logging logging.basicConfig( level=logging.DEBUG, format="%(asctime)s - %(levelname)s - %(message)s", filename="app.log" ) logging.debug("This is debug") logging.info("User logged in: Anshuman") logging.warning("Low disk space") logging.error("Failed to connect to API")

3. pdb – Built-in debugger

Python

import pdb def divide(x, y): pdb.set_trace() # breakpoint return x / y divide(10, 0)

Common pdb commands:

• n (next line)

• s (step into function)

• c (continue)

• p variable (print value)

• q (quit)

4. VS Code Debugger (most recommended in 2026)

1. Open your .py file

2. Click left sidebar → Run & Debug icon

3. Click "create a launch.json file" → Python

4. Set breakpoints (click left of line number)

5. Press F5 or green play button

You can:

• Step over/into/out

• Watch variables

• See call stack

• Debug async code too

Mini Setup Checklist (run once per project):

python -m venv venv source venv/bin/activate # or venv\Scripts\activate on Windows pip install black flake8 isort mypy requests pandas fastapi uvicorn

This completes the full Python Intermediate Recap & Advanced Setup section — now you're ready for real advanced topics!

2. Object-Oriented Programming (OOP) in Depth

OOP lets you model real-world entities as objects with data (attributes) and behavior (methods). Python’s OOP is flexible, powerful, and very “Pythonic”.

2.1 Classes & Objects – Advanced Features

Class → Blueprint Object → Instance of the class

Basic syntax (recap + advanced)

Python

class Person: # Class attribute (shared by all instances) species = "Homo sapiens" def init(self, name, age): # Instance attributes self.name = name self.age = age def introduce(self): return f"Hi, I'm {self.name}, {self.age} years old." # Creating objects (instances) p1 = Person("Anshuman", 25) p2 = Person("Rahul", 24) print(p1.introduce()) # Hi, I'm Anshuman, 25 years old. print(p1.species) # Homo sapiens print(p2.species) # Homo sapiens (shared)

Advanced features already in use:

• self → reference to the current instance

• Class attributes vs instance attributes

• Methods are functions attached to the class

2.2 init, self, str, repr

Special (magic/dunder) methods:

• init → constructor (called when object is created)

• self → explicit reference to instance (Python passes it automatically)

str → human-readable string (used by print()) repr → unambiguous, developer-friendly string (used in REPL/debugger)

Python

class Book: def init(self, title, author, year): self.title = title self.author = author self.year = year def str(self): return f"{self.title} by {self.author} ({self.year})" def repr(self): return f"Book(title='{self.title}', author='{self.author}', year={self.year})" b = Book("Python Mastery", "Anshuman", 2026) print(b) # Python Mastery by Anshuman (2026) ← str print(repr(b)) # Book(title='Python Mastery', author='Anshuman', year=2026)

Tip: Always implement both — str for users, repr for debugging.

2.3 Inheritance & super()

Inheritance lets a child class reuse code from a parent class.

Python

class Employee: def init(self, name, salary): self.name = name self.salary = salary def work(self): return f"{self.name} is working." class Developer(Employee): def init(self, name, salary, language): # super() calls parent's init super().__init__(name, salary) self.language = language def work(self): return f"{self.name} is coding in {self.language}." dev = Developer("Anshuman", 120000, "Python") print(dev.work()) # Anshuman is coding in Python. print(dev.salary) # 120000 (inherited)

super() is better than hardcoding parent class name — supports multiple inheritance.

2.4 Method Overriding & Polymorphism

Overriding → Child redefines parent method Polymorphism → Same method name, different behavior

Python

class Animal: def speak(self): return "Some sound..." class Dog(Animal): def speak(self): return "Woof!" class Cat(Animal): def speak(self): return "Meow!" animals = [Dog(), Cat(), Animal()] for animal in animals: print(animal.speak()) # Woof! / Meow! / Some sound...

Duck typing in Python: "If it walks like a duck and quacks like a duck, it’s a duck."

2.5 Encapsulation: Private & Protected Members

Python uses naming conventions (no strict private):

• singleunderscore → protected (by convention, don’t access directly)

• __double_underscore → name mangling (pseudo-private)

Python

class BankAccount: def init(self, owner, balance=0): self.owner = owner self._balance = balance # protected self.__secret_pin = 1234 # name mangled def deposit(self, amount): if amount > 0: self._balance += amount def get_balance(self): return self._balance acc = BankAccount("Anshuman", 10000) print(acc._balance) # 10000 (works but discouraged) # print(acc.__secret_pin) # AttributeError print(acc._BankAccount__secret_pin) # 1234 (name mangled)

Best practice: Use _ for "don't touch this" and properties for controlled access.

2.6 Properties (@property, @setter, @deleter)

Make attributes look like variables but control access.

Python

class Circle: def init(self, radius): self._radius = radius @property def radius(self): return self._radius @radius.setter def radius(self, value): if value < 0: raise ValueError("Radius cannot be negative") self._radius = value @property def area(self): return 3.14159 self._radius * 2 @area.deleter def area(self): print("Area cache cleared (if any)") c = Circle(5) print(c.radius) # 5 ← looks like attribute print(c.area) # 78.53975 ← computed property c.radius = 10 # setter called # c.radius = -3 # ValueError del c.area # deleter called

Benefits: Validation, computed values, lazy evaluation, clean API.

2.7 Class Methods, Static Methods, @classmethod, @staticmethod

• @classmethod → receives class (cls) instead of instance (self)

• @staticmethod → no self or cls — regular function in class namespace

Python

class Employee: raise_amount = 1.10 # class attribute def init(self, name, salary): self.name = name self.salary = salary @classmethod def set_raise_amount(cls, amount): cls.raise_amount = amount @staticmethod def is_workday(day): # Monday=0 ... Sunday=6 return day.weekday() < 5 def apply_raise(self): self.salary *= Employee.raise_amount Employee.set_raise_amount(1.15) # class method import datetime today = datetime.date(2026, 3, 5) # Thursday print(Employee.is_workday(today)) # True

Use @classmethod for alternative constructors, factory methods. Use @staticmethod for utility functions related to class.

2.8 Multiple Inheritance & Method Resolution Order (MRO)

Python supports multiple inheritance.

Python

class A: def show(self): print("A") class B(A): def show(self): print("B") class C(A): def show(self): print("C") class D(B, C): pass d = D() d.show() # B print(D.mro()) # [<class '__main__.D'>, <class '__main__.B'>, <class '__main__.C'>, <class '__main__.A'>, ...]

MRO (Method Resolution Order) → C3 linearization Use ClassName.mro() or ClassName.__mro__ to see order.

super() in multiple inheritance works beautifully with MRO.

2.9 Abstract Base Classes (abc module)

Force subclasses to implement methods.

Python

from abc import ABC, abstractmethod class Shape(ABC): @abstractmethod def area(self): pass @abstractmethod def perimeter(self): pass class Rectangle(Shape): def init(self, length, width): self.length = length self.width = width def area(self): return self.length self.width def perimeter(self): return 2 (self.length + self.width) # s = Shape() # TypeError: Can't instantiate abstract class r = Rectangle(10, 5) print(r.area()) # 50

Use when: Defining interfaces or templates that must be followed.

2.10 Composition vs Inheritance – When to Use What

Inheritance ("is-a" relationship) → Dog is a Animal

Composition ("has-a" relationship) → Car has a Engine

Python

# Inheritance (is-a) class Car(Vehicle): ... # Composition (has-a) – usually preferred class Car: def init(self): self.engine = Engine() self.wheels = [Wheel() for _ in range(4)] def start(self): self.engine.start()

Rule of thumb (2026 best practice):

• Prefer composition over inheritance (composition gives more flexibility)

• Use inheritance only when there is a clear is-a relationship and you want to reuse behavior

• "Favor composition over inheritance" – Gang of Four Design Patterns

Mini Project – Bank System (OOP in action)

Python

from abc import ABC, abstractmethod class Account(ABC): def init(self, account_number, balance=0): self.account_number = account_number self._balance = balance @property def balance(self): return self._balance @abstractmethod def withdraw(self, amount): pass def deposit(self, amount): if amount > 0: self._balance += amount return True return False class SavingsAccount(Account): INTEREST_RATE = 0.04 def withdraw(self, amount): if amount <= self._balance: self._balance -= amount return True return False def add_interest(self): self._balance += self._balance * SavingsAccount.INTEREST_RATE

This completes the full OOP in Depth section — now you can build clean, scalable, professional object-oriented systems!

3. Advanced Data Structures & Collections

Python’s standard library provides specialized data structures that go beyond basic list, dict, set, and tuple. These save time and improve performance.

3.1 collections module: namedtuple, deque, Counter, defaultdict, OrderedDict

The collections module offers high-performance alternatives.

1. namedtuple – Tuple with named fields (readable like objects, lightweight)

Python

from collections import namedtuple # Define Point = namedtuple('Point', ['x', 'y']) p1 = Point(10, 20) p2 = Point(x=5, y=15) # keyword args also work print(p1.x, p1.y) # 10 20 print(p1) # Point(x=10, y=20) print(p1._asdict()) # {'x': 10, 'y': 20}

Benefits: More readable than plain tuples, immutable, no memory overhead like classes.

2. deque – Double-ended queue (fast append/pop from both ends)

Python

from collections import deque dq = deque([1, 2, 3]) dq.append(4) # right dq.appendleft(0) # left print(dq) # deque([0, 1, 2, 3, 4]) print(dq.pop()) # 4 (from right) print(dq.popleft()) # 0 (from left) dq.extend([5, 6]) # right extend dq.extendleft([ -1, -2]) # left extend (reverses order!) print(dq) # deque([-2, -1, 1, 2, 3, 5, 6])

Use cases: Queues, stacks, sliding windows, BFS, undo/redo.

3. Counter – Counts hashable objects (like frequency map)

Python

from collections import Counter words = ["apple", "banana", "apple", "cherry", "banana", "apple"] cnt = Counter(words) print(cnt) # Counter({'apple': 3, 'banana': 2, 'cherry': 1}) print(cnt['apple']) # 3 print(cnt.most_common(2)) # [('apple', 3), ('banana', 2)] # Arithmetic operations c1 = Counter(a=3, b=1) c2 = Counter(a=1, b=2) print(c1 + c2) # Counter({'a': 4, 'b': 3}) print(c1 - c2) # Counter({'a': 2})

Use cases: Word frequency, voting systems, finding duplicates/most common items.

4. defaultdict – Dictionary that provides default value for missing keys

Python

from collections import defaultdict # Normal dict → KeyError on missing key d = {} # d['key'] += 1 # Error # defaultdict dd = defaultdict(int) # default = 0 dd['a'] += 1 dd['b'] += 5 print(dd) # defaultdict(<class 'int'>, {'a': 1, 'b': 5}) # Other defaults dd_list = defaultdict(list) dd_list['fruits'].append("apple") dd_list['fruits'].append("banana") print(dd_list['fruits']) # ['apple', 'banana']

Use cases: Grouping, counting without checking if key in dict.

5. OrderedDict – Dictionary that remembers insertion order (Note: Since Python 3.7, regular dict also preserves order — OrderedDict is mostly for explicit clarity or older code.)

Python

from collections import OrderedDict od = OrderedDict() od['a'] = 1 od['b'] = 2 od['c'] = 3 print(od) # OrderedDict([('a', 1), ('b', 2), ('c', 3)])

When to use OrderedDict today: When you need .popitem(last=False) (FIFO behavior) or want to clearly signal order matters.

3.2 dataclasses (Python 3.7+) – Cleaner Classes

dataclasses reduce boilerplate when creating classes mainly for storing data.

Python

from dataclasses import dataclass, field @dataclass class Person: name: str age: int = 0 city: str = "Unknown" hobbies: list[str] = field(default_factory=list) # mutable default safe def introduce(self): return f"Hi, I'm {self.name} from {self.city}, {self.age} years old." p = Person("Anshuman", 25, "Muzaffarpur") print(p) # Person(name='Anshuman', age=25, city='Muzaffarpur', hobbies=[]) p.hobbies.append("coding") print(p.hobbies) # ['coding'] p2 = Person("Rahul") # age=0, city="Unknown" print(p2) # Person(name='Rahul', age=0, city='Unknown', hobbies=[])

Advantages over regular class:

• Auto init, repr, eq, ne

• No need to write init manually

• Safe mutable defaults with field(default_factory=...)

• Type hints are enforced by tools like mypy

Options: @dataclass(frozen=True) → immutable, @dataclass(order=True) → adds comparison methods.

3.3 Heapq – Priority Queues

heapq provides min-heap (smallest element first) — very efficient for priority queues.

Python

import heapq # List as heap (in-place) tasks = [] # Push (priority, task) heapq.heappush(tasks, (3, "Write report")) heapq.heappush(tasks, (1, "Fix bug")) heapq.heappush(tasks, (2, "Review code")) print(heapq.heappop(tasks)) # (1, 'Fix bug') ← smallest priority first # Peek without pop print(tasks[0]) # (2, 'Review code')

Real example – Task scheduler

Python

import heapq from datetime import datetime queue = [] heapq.heappush(queue, (datetime(2026, 3, 10), "Submit project")) heapq.heappush(queue, (datetime(2026, 3, 6), "Exam revision")) while queue: deadline, task = heapq.heappop(queue) print(f"{deadline.date()}: {task}")

Tip: Use tuples (priority, item) — heap compares first element, then second if tie.

3.4 Bisect – Binary Search & Insertion

bisect maintains sorted lists efficiently (O(log n) search/insert).

Python

import bisect sorted_list = [10, 20, 30, 40, 50] # Find insertion point pos = bisect.bisect_left(sorted_list, 25) print(pos) # 2 # Insert while keeping sorted bisect.insort(sorted_list, 25) print(sorted_list) # [10, 20, 25, 30, 40, 50] # Right insertion (for duplicates) bisect.insort_right(sorted_list, 25) print(sorted_list) # [10, 20, 25, 25, 30, 40, 50]

Use cases: Maintaining sorted data, finding rank/position, interval problems.

Mini Project – Leaderboard with heapq & bisect

Python

import heapq # Min-heap for top 5 scores (negative for max-heap effect) leaderboard = [] def add_score(name, score): heapq.heappush(leaderboard, (-score, name)) # negative → largest first if len(leaderboard) > 5: heapq.heappop(leaderboard) # remove lowest add_score("Anshuman", 950) add_score("Rahul", 880) add_score("Priya", 990) # Show top scores print("Top 5:") for score, name in sorted(leaderboard): # sort for display print(f"{name}: {-score}")

This completes the full Advanced Data Structures & Collections section — now you can write more efficient, Pythonic code!

4. Functional Programming Tools

4.1 Lambda Functions – Advanced Uses

Lambda functions are anonymous (nameless) one-line functions — ideal for short operations passed to higher-order functions.

Basic syntax lambda arguments: expression

Advanced & practical uses

1. Sorting with custom key

Python

students = [ {"name": "Anshuman", "marks": 92}, {"name": "Rahul", "marks": 85}, {"name": "Priya", "marks": 98} ] # Sort by marks descending sorted_students = sorted(students, key=lambda x: x["marks"], reverse=True) print([s["name"] for s in sorted_students]) # ['Priya', 'Anshuman', 'Rahul']

2. Immediate function call (IIFE-like)

Python

result = (lambda x, y: x * y + 10)(5, 3) print(result) # 25

3. With map/filter in one-liners

Python

numbers = [1, 2, 3, 4, 5] squared_even = list(map(lambda x: x**2, filter(lambda x: x % 2 == 0, numbers))) print(squared_even) # [4, 16]

4. As default value in dict (strategy pattern)

Python

operations = { "+": lambda x, y: x + y, "-": lambda x, y: x - y, "*": lambda x, y: x y } print(operations[""](10, 4)) # 40

Tip: Lambdas are great for short callbacks — but for complex logic (> 1–2 lines), use def for readability.

4.2 map(), filter(), reduce() (functools)

Higher-order functions that apply functions to iterables.

map(function, iterable, ...) → applies function to each item

Python

numbers = [1, 2, 3, 4] squares = map(lambda x: x**2, numbers) print(list(squares)) # [1, 4, 9, 16] # Multiple iterables names = ["anshuman", "rahul"] cities = ["Muzaffarpur", "Patna"] result = map(lambda n, c: f"{n.title()} from {c}", names, cities) print(list(result)) # ['Anshuman from Muzaffarpur', 'Rahul from Patna']

filter(function, iterable) → keeps items where function returns True

Python

numbers = [10, 15, 20, 25, 30] evens = filter(lambda x: x % 2 == 0, numbers) print(list(evens)) # [10, 20, 30]

reduce(function, iterable, initial) → from functools — reduces iterable to single value

Python

from functools import reduce numbers = [1, 2, 3, 4, 5] product = reduce(lambda x, y: x * y, numbers) print(product) # 120 # With initial value total_with_bonus = reduce(lambda acc, x: acc + x, numbers, 100) print(total_with_bonus) # 115

Modern note (2026): map & filter are still useful, but list comprehensions + reduce are more common. reduce is powerful for accumulations.

4.3 List, Dict & Set Comprehensions – Advanced Patterns

Comprehensions are Python’s most “functional” and readable way to create collections.

Advanced List Comprehension

Python

# Nested + condition matrix = [[1, 2], [3, 4], [5, 6]] flattened = [num for row in matrix for num in row if num % 2 == 0] print(flattened) # [2, 4, 6]

Dict Comprehension

Python

names = ["Anshuman", "Rahul", "Priya"] scores = [92, 85, 98] result = {name: score for name, score in zip(names, scores) if score >= 90} print(result) # {'Anshuman': 92, 'Priya': 98}

Set Comprehension

Python

words = ["apple", "banana", "cherry", "date"] unique_lengths = {len(word) for word in words} print(unique_lengths) # {5, 6, 4}

Conditional expression inside

Python

numbers = range(10) parity = ["even" if n % 2 == 0 else "odd" for n in numbers] print(parity[:5]) # ['even', 'odd', 'even', 'odd', 'even']

4.4 Generator Expressions vs List Comprehensions

List comprehension → creates full list in memory Generator expression → lazy, memory-efficient (yields one item at a time)

Python

# List comp → full list now squares_list = [x**2 for x in range(1000000)] # uses lots of memory # Generator expression → yields on demand squares_gen = (x**2 for x in range(1000000)) # almost no memory print(next(squares_gen)) # 0 print(next(squares_gen)) # 1

When to use generator expressions:

• Large datasets

• One-time iteration

• Passing to functions like sum(), max(), list(), tuple()

Python

total = sum(x**2 for x in range(1000000)) # efficient! print(total)

4.5 Generators & yield – Memory Efficient Iteration

Generators are functions that use yield instead of return — they pause and resume execution.

Python

def countdown(n): while n > 0: yield n n -= 1 for num in countdown(5): print(num) # 5 4 3 2 1

Generator function vs normal function

Python

def normal(): return [1, 2, 3] # computes everything, returns list def gen(): yield 1 yield 2 yield 3 # pauses after each yield

4.6 Generator Functions, Generator Expressions

Generator expression (tuple-like syntax)

Python

gen_exp = (x**2 for x in range(10) if x % 2 == 0) print(list(gen_exp)) # [0, 4, 16, 36, 64]

Infinite generator example

Python

def infinite_fibonacci(): a, b = 0, 1 while True: yield a a, b = b, a + b fib = infinite_fibonacci() print(next(fib)) # 0 print(next(fib)) # 1 print(next(fib)) # 1 print(next(fib)) # 2

4.7 yield from & Sub-generators

yield from delegates iteration to another iterable/generator — cleaner for chaining.

Python

def chain_generators(): yield from range(3) # 0,1,2 yield from "abc" # a,b,c yield from countdown(3) # 3,2,1 for val in chain_generators(): print(val) # 0 1 2 a b c 3 2 1

Flatten nested lists

Python

def flatten(nested): for sublist in nested: yield from sublist data = [[1,2], [3,4], [5]] print(list(flatten(data))) # [1, 2, 3, 4, 5]

4.8 itertools module – Powerful Iterators

itertools provides fast, memory-efficient iterators.

Common useful functions

1. count, cycle, repeat

Python

from itertools import count, cycle, repeat for num in count(10, 5): # 10, 15, 20, ... print(num) if num > 30: break for color in cycle(["red", "green", "blue"]): # repeats forever print(color) # break when needed

2. chain, zip_longest

Python

from itertools import chain, zip_longest print(list(chain([1,2], "abc", range(3)))) # [1, 2, 'a', 'b', 'c', 0, 1, 2] a = [1, 2] b = [10, 20, 30] print(list(zip_longest(a, b, fillvalue=0))) # [(1, 10), (2, 20), (0, 30)]

3. groupby (needs sorted input)

Python

from itertools import groupby from operator import itemgetter data = [("apple", 5), ("banana", 3), ("apple", 2), ("cherry", 8)] sorted_data = sorted(data, key=itemgetter(0)) for fruit, group in groupby(sorted_data, key=itemgetter(0)): print(fruit, list(group)) # apple [('apple', 5), ('apple', 2)] # banana [('banana', 3)] # cherry [('cherry', 8)]

Mini Project – Infinite Prime Generator with itertools

Python

from itertools import count, islice def is_prime(n): if n < 2: return False for i in range(2, int(n**0.5) + 1): if n % i == 0: return False return True primes = (n for n in count(2) if is_prime(n)) first_10_primes = list(islice(primes, 10)) print(first_10_primes) # [2, 3, 5, 7, 11, 13, 17, 19, 23, 29]

This completes the full Functional Programming Tools section — now you can write concise, powerful, memory-efficient Python code!

5. Decorators & Higher-Order Functions

5.1 What are Decorators?

A decorator is a function that takes another function (or class) and returns a modified version of it — usually adding behavior before/after the original function runs.

Key points:

• Decorators are higher-order functions (functions that take or return functions).

• They use @ syntax for clean application.

• Common uses: logging, timing, authentication, caching, validation.

Basic mental model:

Python

@decorator def my_function(): pass # is equivalent to: my_function = decorator(my_function)

5.2 Writing Simple Decorators

Step-by-step: Simple logging decorator

Python

def logger(func): def wrapper(*args, **kwargs): print(f"Calling {func.__name__} with args={args}, kwargs={kwargs}") result = func(*args, **kwargs) print(f"{func.__name__} returned: {result}") return result return wrapper @logger def add(a, b): return a + b print(add(5, 3))

Output:

text

Calling add with args=(5, 3), kwargs={} add returned: 8 8

Key parts:

• wrapper is the inner function that adds behavior

• args, *kwargs → accepts any arguments

• Call original func and return its result

Preserve original function metadata (name, docstring, etc.)

Python

from functools import wraps def logger(func): @wraps(func) # Important! def wrapper(*args, **kwargs): print(f"→ {func.__name__} called") return func(*args, **kwargs) return wrapper

5.3 Decorators with Arguments

Sometimes you want to pass parameters to the decorator itself.

Example: Repeat decorator with count

Python

from functools import wraps def repeat(times): def decorator(func): @wraps(func) def wrapper(*args, **kwargs): for in range(times): result = func(*args, **kwargs) return result return wrapper return decorator @repeat(3) def sayhello(name): print(f"Hello, {name}!") say_hello("Anshuman")

Output:

text

Hello, Anshuman! Hello, Anshuman! Hello, Anshuman!

Structure: @repeat(3) → returns decorator → which returns wrapper

5.4 @property, @classmethod, @staticmethod as Decorators

These are built-in decorators that change how methods behave.

Python

class Circle: def init(self, radius): self._radius = radius @property # getter def radius(self): return self._radius @radius.setter # setter def radius(self, value): if value < 0: raise ValueError("Radius cannot be negative") self._radius = value @property def area(self): # computed property return 3.14159 self._radius * 2 @classmethod def from_diameter(cls, diameter): return cls(diameter / 2) # alternative constructor @staticmethod def is_valid_radius(r): return r >= 0 c = Circle(5) print(c.radius) # 5 ← looks like attribute print(c.area) # 78.53975 c.radius = 10 # setter works print(c.area) # 314.159 c2 = Circle.from_diameter(20) # class method print(c2.radius) # 10.0 print(Circle.is_valid_radius(-5)) # False ← static method

5.5 @lru_cache (functools) – Memoization

@lru_cache caches function results — huge performance boost for expensive recursive/ repeated calls.

Python

from functools import lru_cache @lru_cache(maxsize=128) # maxsize=None → unlimited def fibonacci(n): if n < 2: return n return fibonacci(n-1) + fibonacci(n-2) print(fibonacci(35)) # Very fast now! # Without cache: extremely slow

Use cases:

• Recursive algorithms (Fibonacci, tree traversal)

• API calls with same parameters

• Expensive computations

Clear cache: fibonacci.cache_clear()

5.6 Chaining Decorators

You can stack multiple decorators — they apply from bottom to top.

Python

def uppercase(func): @wraps(func) def wrapper(*args, **kwargs): result = func(*args, **kwargs) return result.upper() return wrapper def add_exclamation(func): @wraps(func) def wrapper(*args, **kwargs): return func(*args, **kwargs) + "!" return wrapper @uppercase @add_exclamation def greet(name): return f"Hello {name}" print(greet("Anshuman")) # HELLO ANSHUMAN!

Order matters: @uppercase applied last → final output is uppercase.

5.7 Class Decorators

Decorators can also modify classes (less common but powerful).

Example: Add timestamp attribute to class

Python

from datetime import datetime from functools import wraps def add_timestamp(cls): original_init = cls.__init__ @wraps(original_init) def new_init(self, args, *kwargs): original_init(self, args, *kwargs) self.created_at = datetime.now().strftime("%Y-%m-%d %H:%M:%S") cls.__init__ = new_init return cls @add_timestamp class User: def init(self, name): self.name = name u = User("Anshuman") print(u.created_at) # e.g. 2026-03-05 16:45:22

Common real-world class decorators:

• @dataclass

• @singleton (ensure only one instance)

• @register (auto-register classes in a factory)

Mini Project – Timing & Logging Decorator Combo

Python

import time from functools import wraps def timer(func): @wraps(func) def wrapper(*args, **kwargs): start = time.perf_counter() result = func(*args, **kwargs) end = time.perf_counter() print(f"{func.__name__} took {end - start:.4f} seconds") return result return wrapper def logger(func): @wraps(func) def wrapper(*args, **kwargs): print(f"→ Running {func.__name__}...") result = func(*args, **kwargs) print(f"← {func.__name__} done.") return result return wrapper @timer @logger def slow_task(n): time.sleep(n) return f"Task completed after {n} seconds" print(slow_task(2))

Output:

text

→ Running slow_task... ← slow_task done. slow_task took 2.0012 seconds Task completed after 2 seconds

This completes the full Decorators & Higher-Order Functions section — now you can write elegant, reusable, and professional Python code using one of its most loved features!

6. Context Managers & with Statement

6.1 Understanding Context Managers

A context manager is an object that defines setup and teardown actions for a block of code. It is used with the with statement to handle resources like files, database connections, locks, etc.

Why use context managers?

• Automatically close/release resources (even on exceptions)

• Cleaner code (no manual try-finally)

• Prevents resource leaks

Most common example – file handling

Python

# Old way (error-prone) f = open("data.txt", "w") try: f.write("Hello") finally: f.close() # Modern & safe way with open("data.txt", "w") as f: f.write("Hello") # auto-closed when block ends (even if error)

What happens behind the scenes:

1. open() returns a context manager object

2. enter() is called → returns the object (f)

3. Code inside with block runs

4. exit() is called → closes the file (even if exception occurs)

6.2 Writing Custom Context Managers (__enter__, exit)

You can create your own context manager by defining a class with enter and exit methods.

Example – Timing context manager

Python

import time class Timer: def enter(self): self.start = time.perf_counter() return self # returned value assigned to 'as' variable def exit(self, exc_type, exc_value, traceback): self.end = time.perf_counter() self.elapsed = self.end - self.start print(f"Execution time: {self.elapsed:.4f} seconds") # Return True to suppress exception, False to let it propagate return False # Usage with Timer() as t: time.sleep(1.5) print("Doing some work...") # Output: # Doing some work... # Execution time: 1.5012 seconds

Another example – Temporary directory change

Python

import os from pathlib import Path class ChangeDirectory: def init(self, path): self.new_path = path self.old_path = os.getcwd() def enter(self): os.chdir(self.new_path) return self def exit(self, *args): os.chdir(self.old_path) with ChangeDirectory(Path.home() / "Downloads"): print(os.getcwd()) # now in Downloads print(os.getcwd()) # back to original

Tip: exit receives exception info (exc_type, exc_value, traceback). Return True to suppress exception, False (default) to re-raise it.

6.3 @contextmanager Decorator

Writing full classes can be verbose. The @contextmanager decorator from contextlib lets you write context managers using a generator function — much simpler.

Syntax

Python

from contextlib import contextmanager @contextmanager def my_context(): # setup code try: yield value # value assigned to 'as' variable finally: # teardown code (always runs)

Example – Temporary value change

Python

from contextlib import contextmanager @contextmanager def temp_setting(obj, attr, new_value): old_value = getattr(obj, attr) setattr(obj, attr, new_value) try: yield finally: setattr(obj, attr, old_value) class Config: debug = False cfg = Config() with temp_setting(cfg, "debug", True): print(cfg.debug) # True print(cfg.debug) # False (restored)

Real-world example – Database transaction

Python

@contextmanager def transaction(db): db.begin() try: yield db db.commit() except Exception: db.rollback() raise finally: db.close() # Usage with transaction(my_db) as db: db.execute("INSERT INTO users ...")

6.4 Common Use Cases: File Handling, Database Connections, Timing

1. File Handling (built-in)

Python

with open("log.txt", "a") as f, open("error.txt", "a") as err: f.write("Info log\n") err.write("Error occurred\n") # Both files auto-closed

2. Database Connections (using contextlib + library)

Python

from contextlib import contextmanager import sqlite3 @contextmanager def sqlite_conn(db_path): conn = sqlite3.connect(db_path) try: yield conn conn.commit() except Exception: conn.rollback() raise finally: conn.close() with sqlite_conn("mydb.db") as conn: conn.execute("INSERT INTO users (name) VALUES (?)", ("Anshuman",))

3. Timing / Performance Measurement

Python

from contextlib import contextmanager import time @contextmanager def timer(description=""): start = time.perf_counter() yield elapsed = time.perf_counter() - start print(f"{description} took {elapsed:.4f} seconds") with timer("Data processing"): time.sleep(1.2) # heavy computation here # Output: Data processing took 1.2008 seconds

4. Thread Lock / Resource Management

Python

from threading import Lock from contextlib import contextmanager lock = Lock() @contextmanager def locked(): lock.acquire() try: yield finally: lock.release() with locked(): # critical section print("Thread-safe code here")

Mini Project – Suppressing Specific Exceptions

Python

from contextlib import contextmanager, suppress @contextmanager def ignore_errors(*exceptions): try: yield except exceptions: pass with ignore_errors(ZeroDivisionError, ValueError): print(10 / 0) # no crash print("This won't run") print("Program continues...")

This completes the full Context Managers & with Statement section — now you can manage resources safely and elegantly in any Python program!

7. Exception Handling – Advanced

7.1 try-except-else-finally Deep Dive

The full structure of exception handling in Python:

Python

try: # Code that might raise an exception risky_operation() except ExceptionType1 as e1: # Handle specific exception type 1 print(f"Type1 error: {e1}") except (Type2, Type3) as e23: # Handle multiple types print(f"Type2/3 error: {e23}") except Exception as e: # Catch-all (broad, use carefully) print(f"Unexpected error: {e}") # Optionally re-raise: raise else: # Runs only if NO exception occurred print("Success! No exceptions raised.") finally: # Always runs (cleanup), even on return/break/raise print("Cleanup: closing files, connections, etc.")

Key points to remember:

• else → only executes if no exception was raised in try (great for code that should run only on success)

• finally → always executes (cleanup, close files/connections, release locks)

• Order matters: except blocks are checked from top to bottom → specific → general

• Avoid bare except: (catches everything, including KeyboardInterrupt, SystemExit)

Real example – File processing with proper cleanup

Python

def process_file(filename): try: f = open(filename, "r") data = f.read() number = int(data.strip()) # might raise ValueError except FileNotFoundError: print("File not found!") return None except ValueError as ve: print(f"Invalid number format: {ve}") return None else: print("File read successfully!") return number * 2 finally: if 'f' in locals(): f.close() print("File closed in finally.") print(process_file("numbers.txt"))

7.2 Raising Custom Exceptions

Use raise to signal errors explicitly.

Basic raise

Python

if age < 0: raise ValueError("Age cannot be negative!")

Raise with custom message

Python

def divide(a, b): if b == 0: raise ZeroDivisionError("Cannot divide by zero – check your input!") return a / b

Re-raise (preserve original traceback)

Python

try: risky_code() except Exception as e: print("Logging error...") raise # re-raises the original exception with full traceback

Chaining with raise ... from (see 7.4)

7.3 Creating Custom Exception Classes

Custom exceptions make error handling clearer and more semantic.

Basic custom exception

Python

class InvalidAgeError(ValueError): """Raised when age is invalid (negative or unrealistic).""" pass def set_age(age): if age < 0: raise InvalidAgeError("Age cannot be negative") if age > 150: raise InvalidAgeError("Age seems unrealistic") print(f"Age set to {age}")

Advanced custom exception with attributes

Python

class PaymentFailedError(Exception): def init(self, amount, reason, transaction_id=None): self.amount = amount self.reason = reason self.transaction_id = transaction_id super().__init__(f"Payment of ₹{amount} failed: {reason}") def str(self): msg = f"Payment failed: {self.reason} (₹{self.amount})" if self.transaction_id: msg += f" - Transaction ID: {self.transaction_id}" return msg try: # Simulate payment gateway failure raise PaymentFailedError(5000, "Card declined", "TXN987654") except PaymentFailedError as e: print(e) # Payment failed: Card declined (₹5000) - Transaction ID: TXN987654 print(f"Amount lost: ₹{e.amount}")

Best practice:

• Inherit from built-in exceptions (ValueError, TypeError, RuntimeError, etc.)

• Add custom attributes for debugging/info

• Write docstrings for clarity

7.4 Exception Chaining (__cause__, context)

Python automatically tracks exception chains:

• context → the exception that was being handled when this one was raised

• cause → explicit cause (set with raise ... from)

Automatic chaining (implicit)

Python

try: 1 / 0 except ZeroDivisionError: raise ValueError("Division failed") # ValueError.__context__ = ZeroDivisionError

Explicit chaining with from

Python

def load_config(): try: with open("config.json") as f: return json.load(f) except FileNotFoundError as fnf: raise RuntimeError("Configuration loading failed") from fnf try: load_config() except RuntimeError as re: print(re) # Configuration loading failed print(re.__cause__) # shows FileNotFoundError

Use from None to suppress context

Python

raise ValueError("Bad value") from None # hides previous context

When to use:

• from → when new exception is a direct consequence (cleaner traceback)

• Automatic context → when handling one error leads to another

7.5 Logging vs print() for Production

print() → good for debugging, bad for production

logging → standard for real applications

Basic logging setup (recommended)

Python

import logging import sys # Configure once at app startup logging.basicConfig( level=logging.DEBUG, # DEBUG / INFO / WARNING / ERROR / CRITICAL format="%(asctime)s [%(levelname)s] %(name)s: %(message)s", datefmt="%Y-%m-%d %H:%M:%S", handlers=[ logging.FileHandler("app.log"), logging.StreamHandler(sys.stdout) ] ) logger = logging.getLogger(__name__) # name = current module name def process_payment(amount): try: if amount <= 0: raise ValueError("Amount must be positive") logger.info(f"Processing payment of ₹{amount}") # simulate success logger.debug("Payment gateway response: success") return True except ValueError as ve: logger.error("Payment validation failed", exc_info=True) # includes traceback return False except Exception as e: logger.critical("Unexpected error in payment", exc_info=True) raise

Levels quick reference:

• debug → detailed info (disabled in production)

• info → normal operation messages

• warning → something unexpected but not fatal

• error → handled error (operation failed)

• critical → severe error (app may crash)

Advantages over print():

• Levels (turn off debug in production)

• File + console output

• Timestamps, module names, tracebacks

• Configurable via logging.config (dictConfig, fileConfig)

• Thread-safe

Pro tip: Use logger.exception("Message") instead of logger.error(..., exc_info=True) — shorter and includes traceback automatically.

Mini Project – Robust File Processor with Logging & Custom Exception

Python

import logging from contextlib import contextmanager logging.basicConfig(level=logging.INFO, format="%(asctime)s [%(levelname)s] %(message)s") class FileProcessingError(Exception): pass @contextmanager def safe_file_read(filename): try: with open(filename, "r") as f: yield f.read() except FileNotFoundError: raise FileProcessingError(f"File {filename} not found") except PermissionError: raise FileProcessingError(f"No permission to read {filename}") except Exception as e: raise FileProcessingError(f"Unexpected error reading {filename}: {e}") try: with safe_file_read("config.txt") as content: logging.info("File read successfully") print(content) except FileProcessingError as fpe: logging.error(str(fpe))

This completes the full Exception Handling – Advanced section — now you can build robust, production-ready Python applications that handle errors gracefully!

8. File Handling & Data Formats

8.1 Reading/Writing Text & Binary Files

Python provides the built-in open() function to work with files.

Modes you will use most often:

• 'r' → read text (default)

• 'w' → write text (overwrites if exists)

• 'a' → append text

• 'rb' → read binary

• 'wb' → write binary

• 'r+' → read + write (file must exist)

Best practice: Always use with statement (automatically closes file)

Text file examples

Python

# Reading entire file with open("notes.txt", "r", encoding="utf-8") as file: content = file.read() # → one big string print(content) # Reading line by line (memory efficient) with open("log.txt", "r", encoding="utf-8") as file: for line in file: print(line.strip()) # process each line # Writing text with open("output.txt", "w", encoding="utf-8") as file: file.write("Hello, Anshuman!\n") file.write("This is line 2.\n") # Appending with open("log.txt", "a", encoding="utf-8") as file: file.write(f"New entry at {datetime.now()}\n")

Binary file example (copy image/video)

Python

with open("photo.jpg", "rb") as src: data = src.read() with open("backup.jpg", "wb") as dest: dest.write(data)

Important flags:

• encoding="utf-8" → almost always use for text files (handles Hindi, emojis, etc.)

• newline="" → use when writing CSV on Windows to avoid extra blank lines

8.2 with Statement Best Practices

The with statement is the safest and cleanest way to handle files (and other resources).

Correct & safe

Python

with open("data.txt", "r", encoding="utf-8") as f: content = f.read() # file is automatically closed here – even if exception occurs

Multiple files in one with

Python

with open("input.txt", "r") as src, open("copy.txt", "w") as dest: dest.write(src.read())

Nested with (when needed)

Python

with open("config.json") as cfg: with open("backup.log", "a") as log: log.write("Config loaded successfully\n")

Never do this (risk of file not closing)

Python

f = open("file.txt") try: data = f.read() finally: f.close() # easy to forget finally

8.3 CSV – csv module

The csv module handles commas, quotes, delimiters, and newlines correctly — never use split(',') for real CSV.

Reading CSV

Python

import csv # Simple reader with open("students.csv", "r", encoding="utf-8") as f: reader = csv.reader(f) header = next(reader) # ['name', 'age', 'city'] for row in reader: print(row) # ['Anshuman', '25', 'Muzaffarpur'] # DictReader – most useful with open("students.csv", "r", encoding="utf-8") as f: reader = csv.DictReader(f) for row in reader: print(row["name"], row["age"]) # Anshuman 25

Writing CSV

Python

import csv data = [ {"name": "Rahul", "age": 24, "city": "Patna"}, {"name": "Priya", "age": 23, "city": "Delhi"} ] with open("output.csv", "w", encoding="utf-8", newline="") as f: writer = csv.DictWriter(f, fieldnames=["name", "age", "city"]) writer.writeheader() # writes header row writer.writerows(data)

Tip: newline="" prevents extra blank lines on Windows.

8.4 JSON – json module & serialization

JSON is the most common format for APIs, config files, web data.

Reading JSON

Python

import json with open("config.json", "r", encoding="utf-8") as f: config = json.load(f) # directly gets dict/list print(config["api"]["key"]) # your-api-key-here

Writing JSON (pretty print)

Python

import json person = { "name": "Anshuman", "age": 25, "skills": ["Python", "FastAPI", "SQL"], "address": {"city": "Muzaffarpur", "state": "Bihar"}, "active": True } with open("person.json", "w", encoding="utf-8") as f: json.dump(person, f, indent=4, ensure_ascii=False) # indent=4 → beautiful formatting # ensure_ascii=False → allows Hindi/Unicode without escaping

String conversion (very common in APIs)

Python

json_string = json.dumps(person, indent=2, ensure_ascii=False) print(json_string) back_to_dict = json.loads(json_string)

8.5 Pickle – Serializing Python Objects

pickle can save almost any Python object (lists, dicts, classes, functions, models, etc.) — but only use it for trusted data.

Important warning: Never load pickle files from untrusted sources → security risk (can execute arbitrary code)

Basic usage

Python

import pickle data = { "model_weights": some_large_array, "training_history": [0.92, 0.85, 0.89], "timestamp": datetime.now() } # Save with open("model.pkl", "wb") as f: pickle.dump(data, f) # Load with open("model.pkl", "rb") as f: loaded = pickle.load(f)

Use cases:

• Save ML models (scikit-learn, PyTorch state_dict)

• Cache expensive computations

• Save game state, user sessions internally

Safer alternatives for sharing data: JSON, CSV, Parquet, HDF5

8.6 Working with Large Files (chunk reading)

Never load huge files (GBs) into memory at once.

Line-by-line (best for text/CSV)

Python

with open("very_large_log.txt", "r", encoding="utf-8") as f: for line in f: # process each line if "ERROR" in line: print(line.strip())

Chunk reading (binary or text)

Python

def process_in_chunks(filename, chunk_size=1024*1024): # 1 MB chunks with open(filename, "rb") as f: while chunk := f.read(chunk_size): # process chunk (e.g., hash, search bytes, upload) print(f"Processed {len(chunk)} bytes") process_in_chunks("big_video.mp4")

Memory-efficient CSV processing

Python

import csv with open("million_rows.csv", "r", encoding="utf-8") as f: reader = csv.DictReader(f) total = 0 for row in reader: total += float(row["sales"]) # no need to store all rows print(f"Total sales: ₹{total:,.2f}")

Mini Project – Simple Log Analyzer (JSON Lines + chunks)

Python

import json def analyze_logs(filename): error_count = 0 with open(filename, "r", encoding="utf-8") as f: for line in f: try: log = json.loads(line.strip()) if log.get("level") == "ERROR": error_count += 1 print(f"Error: {log['message']}") except json.JSONDecodeError: print("Skipping invalid JSON line") print(f"Total errors: {error_count}") analyze_logs("server_logs.jsonl")

This completes the full File Handling & Data Formats section — now you can confidently handle any kind of file, from small configs to massive logs and datasets!

9. Concurrency & Parallelism

9.1 Threading vs Multiprocessing vs Asyncio – Quick Comparison

ApproachUses threads?Uses processes?Best forGIL effectOverheadTypical use casethreadingYesNoI/O-bound (waiting: network, files, DB)Limited (GIL blocks CPU)LowSimple background tasks, many file readsmultiprocessingNoYesCPU-bound (heavy math, image processing)No GIL per processHigh (memory)Parallel computation, ML trainingasyncioNo (co-op)NoHigh-volume I/O (thousands of requests)No issueVery lowWeb servers, API clients, scraping

GIL (Global Interpreter Lock) short explanation: CPython has one GIL → only one thread can execute Python bytecode at a time. → Threads are great for waiting (I/O), but almost no speedup for CPU work. → Multiprocessing creates separate processes → each has its own GIL → true multi-core usage.

2026 decision guide:

• Need to wait for 100+ network calls? → asyncio (best) or threading

• Need to run heavy math on all CPU cores? → multiprocessing

• Simple script with few background tasks? → threading

• Modern web/API/scraping? → asyncio

9.2 threading module – Basics & Locks

threading is good when your bottleneck is waiting (network, disk, database), not CPU.

Basic example – concurrent downloads (simulated)

Python

import threading import time def download_file(file_id, delay): print(f"Starting download {file_id}") time.sleep(delay) # simulate network delay print(f"Finished download {file_id}") threads = [] for i in range(1, 6): t = threading.Thread(target=download_file, args=(i, 2.0)) threads.append(t) t.start() for t in threads: t.join() # wait for all threads to finish print("All downloads completed")

Race condition problem (shared variable)

Python

counter = 0 def increment(): global counter for in range(100000): temp = counter time.sleep(0.000001) # simulate tiny delay counter = temp + 1 threads = [threading.Thread(target=increment) for in range(10)] for t in threads: t.start() for t in threads: t.join() print(counter) # Expected 1000_000, but usually much less!

Solution: Lock (Mutex)

Python

counter = 0 lock = threading.Lock() def safe_increment(): global counter for in range(100000): with lock: # automatically acquire & release counter += 1 # Now counter == 1_000_000 every time

Other tools in threading:

• threading.RLock → reentrant lock (same thread can acquire multiple times)

• threading.Event, threading.Condition, threading.Semaphore

• queue.Queue → thread-safe queue (very useful)

Modern recommendation: Use concurrent.futures.ThreadPoolExecutor for simple thread pools.

9.3 multiprocessing – Process Pools

multiprocessing runs real OS processes → bypasses GIL → true parallelism on multiple CPU cores.

Important: Always protect main code with if name == "__main__": (especially on Windows)

Basic Process example

Python

from multiprocessing import Process def square(n): print(f"{n}² = {n*n}") if name == "__main__": processes = [Process(target=square, args=(i,)) for i in range(10)] for p in processes: p.start() for p in processes: p.join()

Best & easiest way: ProcessPoolExecutor

Python

from concurrent.futures import ProcessPoolExecutor import time def heavy_math(n): time.sleep(1) # simulate CPU work return n * n if name == "__main__": with ProcessPoolExecutor(max_workers=4) as executor: results = executor.map(heavy_math, range(10)) print(list(results)) # [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Shared data (be careful – slow & complex): Use multiprocessing.Value, Array, Manager, Queue, Pipe

Tip: For CPU-bound tasks → multiprocessing gives real speedup on multi-core machines.

9.4 asyncio – Async/Await Syntax

asyncio uses cooperative multitasking in a single thread — extremely efficient for I/O-heavy workloads.

Basic async example

Python

import asyncio async def fetch_data(source_id): print(f"Fetching from {source_id}...") await asyncio.sleep(1.5) # simulate network delay print(f"Done with {source_id}") return f"Data from {source_id}" async def main(): tasks = [fetch_data(i) for i in range(1, 6)] results = await asyncio.gather(*tasks) # run all concurrently print("All results:", results) if name == "__main__": asyncio.run(main()) # Python 3.7+ standard way

Output (finishes in ~1.5 seconds, not 7.5):

text

Fetching from 1... Fetching from 2... Fetching from 3... Fetching from 4... Fetching from 5... Done with 1 Done with 2 Done with 3 Done with 4 Done with 5 All results: ['Data from 1', 'Data from 2', 'Data from 3', 'Data from 4', 'Data from 5']

Key concepts:

• async def → defines a coroutine

• await → pauses coroutine until awaited task completes

• asyncio.gather() → wait for multiple tasks concurrently

• asyncio.create_task() → schedule without immediate await

9.5 aiohttp, async file I/O

aiohttp – asynchronous HTTP client/server (replaces requests for async code)

Install: pip install aiohttp

Concurrent HTTP requests

Python

import asyncio import aiohttp async def fetch(url): async with aiohttp.ClientSession() as session: async with session.get(url) as response: return await response.text() async def main(): urls = [ "https://python.org", "https://fastapi.tiangolo.com", "https://realpython.com" ] tasks = [fetch(url) for url in urls] results = await asyncio.gather(*tasks) print([len(html) for html in results]) # lengths of pages asyncio.run(main())

Async file reading/writing (aiofiles)

Python

# pip install aiofiles import aiofiles async def read_large_file(): async with aiofiles.open("big_log.txt", mode="r", encoding="utf-8") as f: async for line in f: # process line if "ERROR" in line: print(line.strip()) asyncio.run(read_large_file())

9.6 When to Use What (GIL Explanation)

GIL reminder: CPython has one Global Interpreter Lock → only one thread executes Python bytecode at a time. → I/O operations (network, disk) release GIL → threads work well → CPU-bound Python code keeps GIL → threads give almost no speedup

Practical 2026 decision table:

ScenarioRecommendedWhy / Alternative100+ API calls / web scrapingasyncio + aiohttpThousands concurrent, low CPU usageReading/writing many files or DB queriesasyncio + aiofilesNon-blocking I/OHeavy math, image processing, ML inferencemultiprocessingUses all CPU cores, bypasses GILSimple background tasks (email, logging)threadingEasy, low overheadWeb server (high connections)asyncio (FastAPI, aiohttp)Best scalabilityMixed I/O + CPU workasyncio + ProcessPoolExecutorAsync I/O + CPU offloaded to processes

Mini Project – Concurrent API Caller with asyncio

Python

import asyncio import aiohttp from aiohttp import ClientTimeout async def get_user(user_id): url = f"https://jsonplaceholder.typicode.com/users/{user_id}" timeout = ClientTimeout(total=5) try: async with aiohttp.ClientSession() as session: async with session.get(url, timeout=timeout) as resp: return await resp.json() except Exception as e: return {"error": str(e)} async def main(): tasks = [get_user(i) for i in range(1, 11)] results = await asyncio.gather(*tasks, return_exceptions=True) for user in results: if isinstance(user, dict) and "name" in user: print(f"{user['id']}: {user['name']}") else: print("Failed:", user) asyncio.run(main())

This completes the full Concurrency & Parallelism section — now you know how to choose and implement the right concurrency model for any task!

10. Metaclasses & Advanced OOP

10.1 What are Metaclasses?

In Python, everything is an object — including classes. Just like a class creates instances (objects), a metaclass creates classes.

Normal flow (you already know):

Python

class Person: pass p = Person() # Person is the class (blueprint), p is the instance

Metaclass flow:

• A metaclass is responsible for creating the Person class itself.

• The default metaclass in Python is type.

Python

print(type(Person)) # <class 'type'> print(type(int)) # <class 'type'> print(type(str)) # <class 'type'>

So type is the built-in metaclass that creates almost all classes.

When do you need custom metaclasses?

• You want to automatically add methods/attributes to classes

• Enforce coding standards (e.g., all methods must have docstrings)

• Auto-register classes (used in ORMs like SQLAlchemy, Django models, plugins)

• Change class creation behavior (very rare in everyday code)

10.2 type() as a Metaclass

type can be used in three ways:

1. As a function to check type: type(obj)

2. As a metaclass (default)

3. As a dynamic class factory

Dynamic class creation with type()

Python

# type(name, bases, dict) → creates a class dynamically def speak(self): return f"Hello, I'm {self.name}" Person = type( "Person", # class name (object,), # base classes { "name": "Anshuman", # class attribute "speak": speak, # method "age": 25 # another attribute } ) p = Person() print(p.speak()) # Hello, I'm Anshuman print(p.age) # 25

This is exactly how Python creates classes under the hood when you write class Person: ...

10.3 Creating Custom Metaclasses

To create a custom metaclass, you inherit from type and override special methods (usually new or init).

Most common pattern: Override new

Python

class MetaLogger(type): def new(cls, name, bases, attrs): # Add logging to every method automatically for attr_name, attr_value in attrs.items(): if callable(attr_value) and not attr_name.startswith("_"): def wrapper(*args, **kwargs): print(f"Calling {name}.{attr_name}()") result = attr_value(*args, **kwargs) print(f"{name}.{attr_name}() finished") return result attrs[attr_name] = wrapper return super().__new__(cls, name, bases, attrs) # Use the metaclass class MyService(metaclass=MetaLogger): def process_data(self, data): print(f"Processing: {data}") return len(data) s = MyService() s.process_data("Hello world")

Output:

text

Calling MyService.process_data() Processing: Hello world MyService.process_data() finished

10.4 new vs init

Both are special methods, but they serve different purposes:

MethodCalled onPurposeReturnsMost used in metaclasses?__new__Class creation (before instance)Creates & returns the object/class itselfThe new object/classYes (control class creation)__init__Instance creation (after new)Initializes the object (sets attributes)None (implicit)Rarely in metaclasses

Key difference in metaclasses:

• Override new when you want to modify the class dictionary (methods, attributes) before the class is created.

• Override init when you want to do something with the already-created class (less common).

Example: Enforce docstring on all methods

Python

class RequireDocstringMeta(type): def new(cls, name, bases, attrs): for attr_name, value in attrs.items(): if callable(value) and not attr_name.startswith("_"): if not value.__doc__: raise TypeError(f"Method {name}.{attr_name} must have a docstring!") return super().__new__(cls, name, bases, attrs) class MyClass(metaclass=RequireDocstringMeta): def good_method(self): """This has a docstring.""" pass # def bad_method(self): # This would raise TypeError # pass

10.5 Practical Use Cases (Frameworks, ORMs)

Metaclasses are rarely needed in everyday code — but they power some of the most famous Python libraries.

1. Django Models (ORM) Django uses metaclasses to turn a simple class into a full database model:

Python

class Book(models.Model): # metaclass=ModelBase title = models.CharField(max_length=200) author = models.ForeignKey(Author, on_delete=models.CASCADE)

→ Metaclass automatically adds manager (objects), table name, fields, etc.

2. SQLAlchemy Declarative Base

Python

class Base(DeclarativeBase): pass class User(Base): tablename = "users" id = Column(Integer, primary_key=True)

→ Metaclass inspects attributes and builds the mapping to database tables.

3. Plugin / Registry Systems Auto-register classes:

Python

class PluginRegistryMeta(type): registry = {} def new(cls, name, bases, attrs): new_cls = super().__new__(cls, name, bases, attrs) if name != "PluginBase": cls.registry[name] = new_cls return new_cls class PluginBase(metaclass=PluginRegistryMeta): pass class ImagePlugin(PluginBase): pass class VideoPlugin(PluginBase): pass print(PluginRegistryMeta.registry) # {'ImagePlugin': <class '__main__.ImagePlugin'>, 'VideoPlugin': <class '__main__.VideoPlugin'>}

4. Enforcing Design Patterns

• Singleton pattern via metaclass

• Auto-add logging, timing, validation to methods

• ABC-like enforcement without abc module

Important advice (2026):

• Use metaclasses only when you really need them — they make code harder to understand.

• Prefer decorators, class decorators, or __init_subclass__ (Python 3.6+) for most cases.

• If you're building a framework/library → metaclasses are powerful tools.

• For application code → avoid unless there's no simpler solution.

Mini Project – Auto-Register Commands (CLI style)

Python

class CommandRegistryMeta(type): commands = {} def new(cls, name, bases, attrs): new_cls = super().__new__(cls, name, bases, attrs) if name != "Command": cls.commands[name.lower()] = new_cls return new_cls class Command(metaclass=CommandRegistryMeta): @classmethod def execute(cls): raise NotImplementedError class HelpCommand(Command): @classmethod def execute(cls): print("Available commands:", list(CommandRegistryMeta.commands.keys())) class ClearCommand(Command): @classmethod def execute(cls): print("Screen cleared!") # Simulate CLI command = "help" if command in CommandRegistryMeta.commands: CommandRegistryMeta.commands[command].execute()

This completes the full Metaclasses & Advanced OOP section — now you understand one of Python's most advanced concepts and when (and when not) to use it!

11. Design Patterns in Python

11.1 Singleton, Factory, Abstract Factory

Singleton Pattern Ensures a class has only one instance and provides a global point of access.

Classic way (using metaclass)

Python

class SingletonMeta(type): instances = {} def _call__(cls, args, *kwargs): if cls not in cls._instances: instance = super().__call__(*args, **kwargs) cls._instances[cls] = instance return cls._instances[cls] class DatabaseConnection(metaclass=SingletonMeta): def init(self): self.connected = False def connect(self): if not self.connected: print("Connecting to database...") self.connected = True db1 = DatabaseConnection() db2 = DatabaseConnection() print(db1 is db2) # True – same instance

Pythonic alternative (module-level singleton) Most common in Python — just use a module!

Python

# database.py connection = None def getconnection(): global connection if connection is None: print("Creating new connection...") connection = "DB Connection Object" return connection

Factory Pattern Creates objects without specifying the exact class.

Python

class Button: def render(self): pass class WindowsButton(Button): def render(self): return "Render Windows-style button" class MacButton(Button): def render(self): return "Render Mac-style button" class GUIFactory: def create_button(self): pass class WindowsFactory(GUIFactory): def create_button(self): return WindowsButton() class MacFactory(GUIFactory): def create_button(self): return MacButton() # Usage def create_ui(factory: GUIFactory): button = factory.create_button() print(button.render()) factory = WindowsFactory() # or MacFactory() create_ui(factory)

Abstract Factory Creates families of related objects.

Pythonic: Use factories returning multiple related objects (often dicts or tuples).

Python

def get_theme_factory(theme): if theme == "dark": return { "button": "DarkButton", "checkbox": "DarkCheckbox", "background": "#1e1e1e" } else: return { "button": "LightButton", "checkbox": "LightCheckbox", "background": "#ffffff" } theme = get_theme_factory("dark") print(theme["background"]) # #1e1e1e

11.2 Observer, Strategy, Decorator Pattern

Observer Pattern (Publish-Subscribe) One object notifies many dependents when its state changes.

Pythonic way: Use callbacks / event system

Python

class Subject: def init(self): self._observers = [] def attach(self, observer): self._observers.append(observer) def detach(self, observer): self._observers.remove(observer) def notify(self, message): for observer in self._observers: observer.update(message) class NewsPublisher(Subject): def publish(self, headline): print(f"New headline: {headline}") self.notify(headline) class Subscriber: def init(self, name): self.name = name def update(self, message): print(f"{self.name} received: {message}") pub = NewsPublisher() sub1 = Subscriber("Anshuman") sub2 = Subscriber("Rahul") pub.attach(sub1) pub.attach(sub2) pub.publish("Python 3.14 released!") # Output: # New headline: Python 3.14 released! # Anshuman received: Python 3.14 released! # Rahul received: Python 3.14 released!

Modern Pythonic alternatives:

• Use blinker library (signals)

• Use asyncio events / queues

• Use property + callbacks

Strategy Pattern Define a family of algorithms, encapsulate each, make them interchangeable.

Pythonic way: Pass functions / lambdas

Python

def discount_strategy(order_total): return order_total * 0.9 # 10% off def premium_strategy(order_total): return order_total - 500 if order_total > 5000 else order_total def checkout(total, discount_func): final = discount_func(total) print(f"Original: ₹{total} → Final: ₹{final}") checkout(6000, discount_strategy) # Final: ₹5400.0 checkout(6000, premium_strategy) # Final: ₹5500

Decorator Pattern Dynamically add responsibilities to objects.

Python already has decorators — use @ syntax!

Python

def add_tax(func): def wrapper(price): return func(price) * 1.18 # 18% GST return wrapper @add_tax def get_book_price(): return 500 print(get_book_price()) # 590.0