Python Best Practices for Production Applications

Python Best Practices for Production Applications

Introduction

Python's simplicity attracts beginners, but production applications require discipline. This guide covers virtual environments for dependency isolation, type hints for code safety, comprehensive error handling, unit testing with pytest, logging strategies, dependency management with Poetry, and packaging best practices for enterprise Python development.

Virtual Environments

Why Virtual Environments?

Problems without virtual environments:

Global Python installation (C:\Python312):
├── requests==2.28.0 (ProjectA needs this)
├── requests==2.31.0 (ProjectB needs this) ❌ CONFLICT
├── django==4.2 (ProjectA)
├── flask==2.3 (ProjectB)
└── 127 other packages from various projects

Result: Dependency hell, broken projects, "works on my machine"

Solution with virtual environments:

ProjectA/
└── venv/
    └── Lib/site-packages/
        ├── requests==2.28.0 ✅
        └── django==4.2 ✅

ProjectB/
└── venv/
    └── Lib/site-packages/
        ├── requests==2.31.0 ✅
        └── flask==2.3 ✅

Creating Virtual Environments

Using venv (built-in):

# Create virtual environment
python -m venv venv

# Activate (Windows PowerShell)
.\venv\Scripts\Activate.ps1

# Activate (Linux/Mac)
source venv/bin/activate

# Verify activation (should show venv path)
where python
# Output: C:\Projects\MyApp\venv\Scripts\python.exe

# Install packages (isolated to this venv)
pip install requests flask pytest

# Deactivate
deactivate

Using virtualenv (more features):

# Install virtualenv
pip install virtualenv

# Create with specific Python version
virtualenv -p python3.11 venv

# Create without pip (install later)
virtualenv --no-pip venv

Using conda (data science):

# Create environment with specific Python version
conda create -n myapp python=3.11

# Activate
conda activate myapp

# Install packages
conda install numpy pandas scikit-learn

# List environments
conda env list

# Export environment
conda env export > environment.yml

# Recreate environment from file
conda env create -f environment.yml

Requirements Files

requirements.txt (pinned versions):

# requirements.txt
requests==2.31.0
flask==3.0.0
sqlalchemy==2.0.23
pytest==7.4.3
pytest-cov==4.1.0
black==23.12.0
mypy==1.7.1

# Generate from current environment
pip freeze > requirements.txt

# Install from requirements
pip install -r requirements.txt

requirements-dev.txt (development dependencies):

# requirements-dev.txt
-r requirements.txt  # Include production dependencies

pytest==7.4.3
pytest-cov==4.1.0
black==23.12.0
flake8==6.1.0
mypy==1.7.1
sphinx==7.2.6

Using Poetry (modern dependency management):

# Install Poetry
pip install poetry

# Initialize project
poetry init

# Add dependencies
poetry add requests flask sqlalchemy
poetry add --group dev pytest black mypy

# Install dependencies
poetry install

# Run command in virtual environment
poetry run python app.py
poetry run pytest

# Update dependencies
poetry update

# Export requirements.txt
poetry export -f requirements.txt --output requirements.txt

pyproject.toml (Poetry configuration):

[tool.poetry]
name = "myapp"
version = "1.0.0"
description = "Production Python application"
authors = ["Your Name <email@example.com>"]

[tool.poetry.dependencies]
python = "^3.11"
requests = "^2.31.0"
flask = "^3.0.0"
sqlalchemy = "^2.0.23"

[tool.poetry.group.dev.dependencies]
pytest = "^7.4.3"
black = "^23.12.0"
mypy = "^1.7.1"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Type Hints

Basic Type Annotations

Function signatures:

def calculate_total(price: float, quantity: int, discount: float = 0.0) -> float:
    """Calculate total price with optional discount.
    
    Args:
        price: Unit price
        quantity: Number of items
        discount: Discount percentage (0.0 to 1.0)
        
    Returns:
        Total price after discount
    """
    subtotal = price * quantity
    return subtotal * (1 - discount)

# Type checker validates calls
total = calculate_total(19.99, 3, 0.1)  # ✅ Valid
total = calculate_total("19.99", 3, 0.1)  # ❌ Type error: str not compatible with float

Variable annotations:

# Explicit type hints
name: str = "John Doe"
age: int = 30
salary: float = 75000.50
is_active: bool = True

# Type inference (type checker infers from assignment)
count = 10  # Inferred as int
message = "Hello"  # Inferred as str

# Multiple types with Union
from typing import Union

def process_id(user_id: Union[int, str]) -> str:
    """Accept int or string ID, return string."""
    return str(user_id)

# Modern syntax (Python 3.10+)
def process_id(user_id: int | str) -> str:
    return str(user_id)

Advanced Type Hints

Collections:

from typing import List, Dict, Set, Tuple, Optional

# List of strings
names: List[str] = ["Alice", "Bob", "Charlie"]
numbers: List[int] = [1, 2, 3, 4, 5]

# Dictionary mapping string keys to int values
scores: Dict[str, int] = {"Alice": 95, "Bob": 87, "Charlie": 92}

# Set of unique emails
emails: Set[str] = {"alice@example.com", "bob@example.com"}

# Tuple with fixed types
coordinates: Tuple[float, float] = (40.7128, -74.0060)
person: Tuple[str, int, bool] = ("Alice", 30, True)

# Optional (can be None)
middle_name: Optional[str] = None  # Same as Union[str, None]

def find_user(user_id: int) -> Optional[Dict[str, str]]:
    """Return user dict or None if not found."""
    if user_id in users:
        return users[user_id]
    return None

Generic types:

from typing import TypeVar, Generic, List

T = TypeVar('T')

class Stack(Generic[T]):
    """Generic stack that works with any type."""
    
    def __init__(self) -> None:
        self._items: List[T] = []
    
    def push(self, item: T) -> None:
        self._items.append(item)
    
    def pop(self) -> T:
        return self._items.pop()
    
    def peek(self) -> T:
        return self._items[-1]
    
    def is_empty(self) -> bool:
        return len(self._items) == 0

# Type-safe usage
int_stack: Stack[int] = Stack()
int_stack.push(10)
int_stack.push(20)
value: int = int_stack.pop()  # ✅ Type checker knows this is int

str_stack: Stack[str] = Stack()
str_stack.push("hello")
str_stack.push(42)  # ❌ Type error: int not compatible with str

Callable types:

from typing import Callable

# Function that takes int and returns str
transformer: Callable[[int], str] = str

# Function with multiple parameters
def apply_operation(x: int, y: int, operation: Callable[[int, int], int]) -> int:
    return operation(x, y)

def add(a: int, b: int) -> int:
    return a + b

def multiply(a: int, b: int) -> int:
    return a * b

result = apply_operation(5, 3, add)  # 8
result = apply_operation(5, 3, multiply)  # 15

Protocol (structural typing):

from typing import Protocol

class Drawable(Protocol):
    """Protocol for objects that can be drawn."""
    def draw(self) -> None: ...

class Circle:
    def draw(self) -> None:
        print("Drawing circle")

class Square:
    def draw(self) -> None:
        print("Drawing square")

def render(shape: Drawable) -> None:
    """Accepts any object with draw() method."""
    shape.draw()

# Both work without explicit inheritance
render(Circle())  # ✅
render(Square())  # ✅

Type Checking with mypy

mypy configuration (.mypy.ini):

[mypy]
python_version = 3.11
warn_return_any = True
warn_unused_configs = True
disallow_untyped_defs = True
disallow_incomplete_defs = True
check_untyped_defs = True
disallow_untyped_decorators = True
no_implicit_optional = True
warn_redundant_casts = True
warn_unused_ignores = True
warn_no_return = True

Running mypy:

# Check single file
mypy app.py

# Check entire project
mypy src/

# Strict mode (maximum type safety)
mypy --strict src/

# Generate HTML report
mypy src/ --html-report ./mypy-report

Error Handling

Exception Best Practices

Specific exceptions:

# ❌ Bad: Catch all exceptions
try:
    value = int(user_input)
except:
    print("Error occurred")

# ✅ Good: Catch specific exceptions
try:
    value = int(user_input)
except ValueError as e:
    print(f"Invalid number format: {e}")
except TypeError as e:
    print(f"Wrong type provided: {e}")

Custom exceptions:

class ApplicationError(Exception):
    """Base exception for application errors."""
    pass

class ValidationError(ApplicationError):
    """Raised when validation fails."""
    def __init__(self, field: str, message: str) -> None:
        self.field = field
        self.message = message
        super().__init__(f"{field}: {message}")

class DatabaseError(ApplicationError):
    """Raised when database operations fail."""
    pass

class AuthenticationError(ApplicationError):
    """Raised when authentication fails."""
    pass

# Usage
def create_user(email: str, password: str) -> None:
    if not email or "@" not in email:
        raise ValidationError("email", "Invalid email address")
    
    if len(password) < 8:
        raise ValidationError("password", "Password must be at least 8 characters")
    
    try:
        db.save_user(email, password)
    except Exception as e:
        raise DatabaseError(f"Failed to save user: {e}") from e

Context managers for cleanup:

from contextlib import contextmanager
from typing import Generator

@contextmanager
def database_connection(connection_string: str) -> Generator:
    """Context manager for database connections."""
    conn = None
    try:
        conn = connect(connection_string)
        yield conn
    except DatabaseError as e:
        if conn:
            conn.rollback()
        raise
    else:
        if conn:
            conn.commit()
    finally:
        if conn:
            conn.close()

# Usage
with database_connection("postgresql://localhost/mydb") as conn:
    cursor = conn.cursor()
    cursor.execute("INSERT INTO users (name) VALUES (?)", ("Alice",))
    # Automatically commits on success, rolls back on error, always closes

Retry decorator:

import time
from functools import wraps
from typing import Callable, TypeVar, Any

T = TypeVar('T')

def retry(max_attempts: int = 3, delay: float = 1.0):
    """Retry decorator for flaky operations."""
    def decorator(func: Callable[..., T]) -> Callable[..., T]:
        @wraps(func)
        def wrapper(*args: Any, **kwargs: Any) -> T:
            for attempt in range(1, max_attempts + 1):
                try:
                    return func(*args, **kwargs)
                except Exception as e:
                    if attempt == max_attempts:
                        raise
                    print(f"Attempt {attempt} failed: {e}. Retrying in {delay}s...")
                    time.sleep(delay)
            raise RuntimeError("Retry logic failed")  # Should never reach here
        return wrapper
    return decorator

@retry(max_attempts=3, delay=2.0)
def fetch_data_from_api(url: str) -> dict:
    """Fetch data with automatic retry on failure."""
    response = requests.get(url, timeout=10)
    response.raise_for_status()
    return response.json()

Testing with pytest

Test Structure

Basic tests:

# test_calculator.py
import pytest
from calculator import Calculator

def test_addition():
    """Test addition operation."""
    calc = Calculator()
    assert calc.add(2, 3) == 5
    assert calc.add(-1, 1) == 0
    assert calc.add(0, 0) == 0

def test_division():
    """Test division operation."""
    calc = Calculator()
    assert calc.divide(10, 2) == 5
    assert calc.divide(7, 2) == 3.5

def test_division_by_zero():
    """Test division by zero raises exception."""
    calc = Calculator()
    with pytest.raises(ValueError, match="Cannot divide by zero"):
        calc.divide(10, 0)

Fixtures:

# conftest.py (shared fixtures)
import pytest
from myapp.database import Database
from myapp.models import User

@pytest.fixture
def db():
    """Create test database."""
    database = Database(":memory:")
    database.create_tables()
    yield database
    database.close()

@pytest.fixture
def sample_user():
    """Create sample user for testing."""
    return User(
        id=1,
        email="test@example.com",
        name="Test User",
        is_active=True
    )

@pytest.fixture
def authenticated_client(client, sample_user):
    """Return authenticated test client."""
    client.login(sample_user)
    return client

# test_user.py
def test_create_user(db):
    """Test user creation with database fixture."""
    user = User(email="alice@example.com", name="Alice")
    db.save(user)
    
    retrieved = db.get_user_by_email("alice@example.com")
    assert retrieved.name == "Alice"

def test_user_serialization(sample_user):
    """Test user serialization with fixture."""
    json_data = sample_user.to_json()
    assert json_data["email"] == "test@example.com"
    assert json_data["is_active"] is True

Parametrized tests:

import pytest

@pytest.mark.parametrize("input_value,expected", [
    (2, 4),
    (3, 9),
    (4, 16),
    (5, 25),
    (-2, 4),
])
def test_square(input_value, expected):
    """Test square function with multiple inputs."""
    assert square(input_value) == expected

@pytest.mark.parametrize("email,is_valid", [
    ("alice@example.com", True),
    ("bob@test.org", True),
    ("invalid", False),
    ("@example.com", False),
    ("user@", False),
    ("", False),
])
def test_email_validation(email, is_valid):
    """Test email validation with various inputs."""
    assert validate_email(email) == is_valid

Mocking:

from unittest.mock import Mock, patch, MagicMock
import pytest

def test_api_call_with_mock():
    """Test function that calls external API."""
    with patch('requests.get') as mock_get:
        # Configure mock response
        mock_response = Mock()
        mock_response.status_code = 200
        mock_response.json.return_value = {"id": 1, "name": "Test"}
        mock_get.return_value = mock_response
        
        # Call function under test
        result = fetch_user_data(1)
        
        # Verify
        assert result["name"] == "Test"
        mock_get.assert_called_once_with("https://api.example.com/users/1", timeout=10)

def test_database_operation_with_mock(mocker):
    """Test database operation with pytest-mock."""
    mock_db = mocker.patch('myapp.database.Database')
    mock_db.return_value.get_user.return_value = User(id=1, name="Alice")
    
    service = UserService()
    user = service.get_user_by_id(1)
    
    assert user.name == "Alice"
    mock_db.return_value.get_user.assert_called_once_with(1)

Coverage

pytest-cov configuration:

# pytest.ini
[tool:pytest]
testpaths = tests
python_files = test_*.py
python_classes = Test*
python_functions = test_*
addopts = 
    --cov=src
    --cov-report=html
    --cov-report=term-missing
    --cov-fail-under=80

Running with coverage:

# Run tests with coverage
pytest --cov=src --cov-report=html

# View coverage report
# Open htmlcov/index.html in browser

# Generate terminal report
pytest --cov=src --cov-report=term-missing

# Fail if coverage below 80%
pytest --cov=src --cov-fail-under=80

Logging

Structured Logging

Basic configuration:

import logging
from logging.handlers import RotatingFileHandler
import sys

def setup_logging(log_level: str = "INFO") -> None:
    """Configure application logging."""
    
    # Create logger
    logger = logging.getLogger("myapp")
    logger.setLevel(getattr(logging, log_level.upper()))
    
    # Console handler
    console_handler = logging.StreamHandler(sys.stdout)
    console_handler.setLevel(logging.INFO)
    console_formatter = logging.Formatter(
        '%(asctime)s - %(name)s - %(levelname)s - %(message)s',
        datefmt='%Y-%m-%d %H:%M:%S'
    )
    console_handler.setFormatter(console_formatter)
    
    # File handler with rotation
    file_handler = RotatingFileHandler(
        'logs/app.log',
        maxBytes=10_000_000,  # 10 MB
        backupCount=5
    )
    file_handler.setLevel(logging.DEBUG)
    file_formatter = logging.Formatter(
        '%(asctime)s - %(name)s - %(levelname)s - %(pathname)s:%(lineno)d - %(message)s'
    )
    file_handler.setFormatter(file_formatter)
    
    # Add handlers
    logger.addHandler(console_handler)
    logger.addHandler(file_handler)

# Usage
setup_logging("DEBUG")
logger = logging.getLogger("myapp")

logger.debug("Detailed debug information")
logger.info("Application started")
logger.warning("Configuration file not found, using defaults")
logger.error("Failed to connect to database", exc_info=True)
logger.critical("System shutting down due to critical error")

Structured logging with contextvars:

import logging
from contextvars import ContextVar
from typing import Optional
import uuid

# Context variables for request tracking
request_id_var: ContextVar[Optional[str]] = ContextVar('request_id', default=None)
user_id_var: ContextVar[Optional[int]] = ContextVar('user_id', default=None)

class ContextFilter(logging.Filter):
    """Add context variables to log records."""
    
    def filter(self, record: logging.LogRecord) -> bool:
        record.request_id = request_id_var.get() or "N/A"
        record.user_id = user_id_var.get() or "N/A"
        return True

# Configure logger with context
logger = logging.getLogger("myapp")
handler = logging.StreamHandler()
handler.addFilter(ContextFilter())
formatter = logging.Formatter(
    '%(asctime)s - [%(request_id)s] [User:%(user_id)s] - %(levelname)s - %(message)s'
)
handler.setFormatter(formatter)
logger.addHandler(handler)

# Middleware to set context
def process_request(request):
    request_id_var.set(str(uuid.uuid4()))
    user_id_var.set(request.user.id if request.user else None)
    
    logger.info(f"Processing request: {request.path}")
    # All logs in this context will include request_id and user_id

Packaging

Project Structure

myapp/
├── src/
│   └── myapp/
│       ├── __init__.py
│       ├── main.py
│       ├── models.py
│       └── utils.py
├── tests/
│   ├── __init__.py
│   ├── test_main.py
│   └── test_models.py
├── docs/
│   └── README.md
├── pyproject.toml
├── setup.py (legacy)
├── requirements.txt
├── requirements-dev.txt
├── .gitignore
└── README.md

pyproject.toml (modern standard):

[build-system]
requires = ["setuptools>=61.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "myapp"
version = "1.0.0"
description = "Production Python application"
readme = "README.md"
authors = [
    {name = "Your Name", email = "email@example.com"}
]
license = {text = "MIT"}
classifiers = [
    "Development Status :: 5 - Production/Stable",
    "Intended Audience :: Developers",
    "Programming Language :: Python :: 3",
    "Programming Language :: Python :: 3.11",
]
keywords = ["example", "production", "application"]
dependencies = [
    "requests>=2.31.0",
    "flask>=3.0.0",
    "sqlalchemy>=2.0.0",
]
requires-python = ">=3.11"

[project.optional-dependencies]
dev = [
    "pytest>=7.4.0",
    "black>=23.0.0",
    "mypy>=1.7.0",
]

[project.scripts]
myapp = "myapp.main:main"

[project.urls]
Homepage = "https://github.com/username/myapp"
Documentation = "https://myapp.readthedocs.io"
Repository = "https://github.com/username/myapp"

Building and distributing:

# Build package
python -m build

# Creates:
# dist/myapp-1.0.0.tar.gz (source distribution)
# dist/myapp-1.0.0-py3-none-any.whl (wheel)

# Install locally
pip install -e .  # Editable install for development

# Upload to PyPI
pip install twine
twine upload dist/*

# Install from PyPI
pip install myapp

Best Practices Summary

1. Virtual Environments: Always use venv/virtualenv, never install globally
2. Type Hints: Add type annotations for all functions and class methods
3. Error Handling: Use specific exceptions, create custom exception hierarchy
4. Testing: Aim for 80%+ coverage, use fixtures and parametrization
5. Logging: Use structured logging with context, rotate log files
6. Dependencies: Pin versions in requirements.txt, use Poetry for complex projects
7. Code Quality: Run black (formatting), mypy (type checking), pytest (testing)

Key Takeaways

• Virtual environments isolate dependencies and prevent conflicts
• Type hints catch errors before runtime with mypy static analysis
• Custom exceptions provide clear error handling and debugging
• pytest fixtures and parametrization reduce test boilerplate
• Structured logging with context enables production troubleshooting
• Modern packaging with pyproject.toml simplifies distribution

Next Steps

• Implement CI/CD pipeline with GitHub Actions running tests and type checks
• Add pre-commit hooks to enforce code quality standards
• Configure Dependabot for automatic dependency updates
• Set up code coverage reporting with Codecov

Additional Resources

• Python Packaging Guide
• Type Hints Documentation
• pytest Documentation

---

Write Python like you mean it.