Python: Advanced Concepts

Type Hints and Annotations

Type hints let you annotate variables, function parameters, and return values with their expected types. Python does not enforce these at runtime; they are metadata stored in__annotations__ and used by static analysis tools like mypy and Pyright, by IDEs for autocompletion and error highlighting, and by libraries like Pydantic for data validation. The governing module is typing, which provides generic aliases, union types, optional types, and protocol definitions.

dataclasses Module

The @dataclass decorator inspects a class's field annotations and automatically generates __init__, __repr__, and __eq__. This eliminates the repetitive boilerplate of writing these methods by hand. Passing order=True also generates comparison methods; frozen=True makes the class immutable and enables__hash__.

Fields with default values must come after fields without. For mutable defaults (lists, dicts), use field(default_factory=list) rather than assigning a list literal directly, for the same reason you avoid mutable defaults in regular function signatures. The __post_init__ method runs after the generated __init__ and is the place for custom validation or derived field computation.

Descriptor Protocol

A descriptor is any object that defines __get__, __set__, or__delete__. When a descriptor instance is stored as a class attribute, Python invokes these methods instead of performing a normal attribute access. This is how Python implements properties, class methods, static methods, and slots at a low level. Understanding descriptors gives you the ability to build reusable attribute behaviour that can be shared across any class.

A descriptor that defines both __get__ and __set__ is called a data descriptor and takes priority over the instance's __dict__. A descriptor that only defines __get__ is a non-data descriptor; the instance's __dict__takes priority over it.

__getattr__, __setattr__, and __delattr__

These three dunder methods let you intercept attribute access at the instance level.__getattr__(self, name) is called only when the normal attribute lookup fails, meaning the attribute was not found in the instance's __dict__ or its class hierarchy. It is the right hook for lazy attribute computation and proxy objects.

__setattr__(self, name, value) is called for every attribute assignment, not just missing ones. Because of this, you must be careful to callobject.__setattr__(self, name, value) inside your override to actually store the value; otherwise, your method calls itself recursively. __delattr__ mirrors this for deletion.

Metaclasses

In Python, everything is an object, including classes. A class is an instance of its metaclass. The default metaclass is type. When Python processes a class definition, it calls the metaclass to create the class object. By defining a custom metaclass, you can intercept this creation and modify the resulting class: enforce naming conventions, validate that certain methods are present, automatically register subclasses, or inject methods.

Metaclasses are a powerful tool, but they should be used sparingly. Class decorators can handle many of the same use cases with less complexity. Reach for a metaclass when you need to intercept the class creation process itself, particularly when you need logic to run at class definition time rather than at instantiation time.

__new__ vs __init__

__new__(cls, *args, **kwargs) is the static method responsible for creating a new instance. It receives the class itself as its first argument and must return the new object. __init__(self, *args, **kwargs) then receives that already-created object and sets up its state. For most classes you never need to override __new__, but it is essential for immutable types (where __init__ cannot change the value after creation), for implementing Singletons, and for customising instance creation in metaclasses.

Context Managers and the contextlib Module

A context manager is any object that implements __enter__ and __exit__. The with statement calls __enter__ on entry and __exit__ on exit, even if an exception occurs. __exit__ receives the exception type, value, and traceback as arguments; returning a truthy value suppresses the exception.

The contextlib module provides utilities that make context managers easier to write and compose. contextlib.contextmanager turns a generator function with a single yield into a context manager, with code before the yield acting as__enter__ and code after acting as __exit__. Other useful tools includecontextlib.suppress for silently ignoring specific exceptions,contextlib.redirect_stdout for capturing output, andcontextlib.ExitStack for dynamically composing multiple context managers.

Abstract Base Classes (ABCs) in Depth

The abc module provides ABC and abstractmethod. A class that inherits from ABC and declares methods with @abstractmethod cannot be instantiated directly; any concrete subclass must implement all abstract methods, or it too becomes abstract. Python raises a TypeError at instantiation time rather than at method-call time, catching the error early.

ABCs also support virtual subclasses via register(). Registering a class tells Python to consider it a subclass for isinstance() and issubclass() checks, without requiring actual inheritance. This lets existing classes participate in type-checking hierarchies retroactively. The collections.abc module uses this extensively to make built-in types like list passisinstance([], collections.abc.Sequence).

Protocols and Structural Subtyping

A typing.Protocol defines an interface by specifying which methods and attributes a conforming object must have. Unlike ABCs, a class does not need to inherit from or register with a Protocol to satisfy it. If it has the required methods with compatible signatures, it satisfies the Protocol structurally. This is sometimes called "duck typing with type checker support."

Protocols are especially valuable when you want to accept any object with a certain shape rather than requiring a specific base class. You can also use@runtime_checkable to enable isinstance() checks against the protocol at runtime, though only method and attribute presence (not signatures) is verified.

Mixin Classes

A mixin is a class designed to be mixed into other classes via multiple inheritance, adding a specific slice of behaviour without being a meaningful standalone class itself. Mixins usually have no __init__ and do not hold significant state of their own. They depend on the class they are mixed into having certain attributes or methods, which they document via type annotations or abstract methods.

Python's standard library uses this extensively: socketserver.ThreadingMixInand socketserver.ForkingMixIn add concurrency handling to server classes. Django uses mixins throughout its class-based views. The key discipline is to keep mixins small and focused on a single responsibility.

Memory Management and the gc Module

CPython manages memory primarily through reference counting. Every object tracks how many references point to it. When the count drops to zero, the object is immediately deallocated. This handles the vast majority of objects automatically. The problem is cyclic references: if object A holds a reference to B, and B holds a reference to A, neither count ever reaches zero even when both are unreachable from the program.

The gc module provides a cyclic garbage collector that periodically detects and breaks reference cycles. It organises objects into three generations based on how long they have survived. New objects start in generation 0; objects that survive a collection cycle are promoted. You can trigger collection manually, inspect tracked objects, and disable the collector if your program guarantees no cycles.

Weak References

A weak reference to an object does not prevent it from being garbage collected. When the only remaining references to an object are weak references, the object is collected and the weak references are automatically invalidated. This is useful for caches, where you want to cache objects but not prevent memory from being freed when the cache is the only thing keeping them alive.

weakref.ref(obj) creates a weak reference. Calling the reference object returns the original object if it is still alive, or None if it has been collected.weakref.WeakValueDictionary is a dict that holds weak references to its values, automatically removing entries when values are collected.

copy Module: copy() and deepcopy()

Assignment in Python creates another reference to the same object, not a new object. To create an independent copy, you use the copy module. copy.copy(obj)makes a shallow copy: it creates a new top-level container but the nested objects inside it are still shared with the original. copy.deepcopy(obj) recursively duplicates every object in the structure, producing a fully independent copy.

You can customise how your class is copied by implementing __copy__ and__deepcopy__. The __deepcopy__ method receives a memo dictionary that tracks already-copied objects to handle cyclic structures correctly.

Pickling and Serialisation

The pickle module serialises Python objects to a byte stream and deserialises them back. Unlike JSON, pickle can handle almost any Python object: custom class instances, functions, lambdas (with some restrictions), numpy arrays, and more. The protocol version controls the format; protocol 5 (Python 3.8+) is the most efficient for large data.

The critical security rule: never unpickle data from an untrusted source. The deserialisation process can execute arbitrary code. If you need to exchange data with external systems or untrusted parties, use JSON, MessagePack, or another format that does not allow code execution.

Quiz - Test Your Knowledge

Ten questions covering type hints, dataclasses, the descriptor protocol, attribute access hooks, metaclasses, object creation with __new__, context managers, abstract base classes, protocols versus ABCs, and the copy module. Several questions require understanding subtle distinctions, so read each option carefully.