Regression Testing

Regression tests ensure that new changes, bug fixes, or refactorings don’t break existing functionality. They’re a core part of any quality assurance strategy, especially in agile teams working with short release cycles. Without regression tests, the fear of making changes grows, and code stagnates.

In a Nutshell

• Regression tests verify that previously working features continue to function.
• They run after every relevant change.
• Automation is essential to keep regression testing scalable.
• Unit, integration, and E2E tests can all serve as regression tests.
• A solid regression testing strategy prioritizes critical user paths.

What Regression Testing Is

Regression tests validate that already-working features remain intact after code changes. The goal is to catch regression bugs—failures in previously correct areas caused by new modifications. You can run them manually or automatically, though automation is far more efficient when changes happen frequently.

Regression testing isn’t a test type in itself, but rather a testing strategy. You decide which existing tests need to run again after a change. That selection can be comprehensive, risk-based, or driven by impact analysis.

When Do Regression Tests Run?

Regression tests activate whenever code or the software environment changes. Common triggers include:

• After code changes or bug fixes: Any modification can have side effects. Regression tests confirm that existing features haven’t been accidentally broken.
• After refactorings: When restructuring code, behavior should stay the same. Regression tests give confidence that nothing has shifted.
• After dependency updates: New library or framework versions can introduce incompatible changes. Regression tests verify the system still works correctly.
• Before merging to the main branch: Before integrating code into main, regression tests catch conflicts and side effects early.
• Before a release: A full regression suite runs just before shipping to confirm overall quality.
• During nightly builds: Automated overnight runs catch regressions that may have slipped through during the day.

Types of Regression Tests

Unit Regression Tests

Unit regression tests check individual functions or classes after changes. They’re fast and run frequently, ideally on every commit. They’re the first line of defense because they execute in isolation and complete in milliseconds. Example: A discount calculation function is modified. Unit tests verify all existing discount rules still produce the correct results.

Integration Regression Tests

Integration regression tests examine interfaces and interactions between modules that a change might affect. They’re slower than unit tests but crucial for catching side effects across module boundaries. Example: After modifying price calculations, you test whether the shopping cart, payment system, and invoice generation still work together correctly.

E2E Regression Tests

E2E regression tests validate critical user workflows, like the checkout process in an e-commerce platform. They’re the slowest and most resource-intensive tests, but they cover the entire chain. Deploy them strategically for business-critical processes, not for every minor feature.

Regression Testing Strategies

Full Regression

All existing tests run. This is the safest approach but becomes time-consuming with large test suites. It’s typically reserved for releases or nightly builds.

Risk-Based Regression

Only tests covering critical or frequently used features run. This strategy requires prioritizing tests by criticality and usage frequency.

Test Impact Analysis

Test impact analysis identifies which code areas a change affects, then runs only tests covering those areas. This requires tools that can analyze code dependencies.

Practical Example

The example below shows regression testing in action across multiple levels. It demonstrates how a focused change leads to a systematic testing approach.

A developer modifies the discount logic in checkout.

Regression tests:
- Unit tests for all discount rules
- Integration tests for shopping cart and price calculation
- E2E test for the complete checkout flow
- Exploratory testing of previous discount scenarios

Why these tests?

• Unit tests: Discount logic is an isolatable function. Unit tests validate each discount rule independently and ensure calculations stay correct across all prior cases.
• Integration tests: The shopping cart and price calculation depend directly on discount logic. Integration tests confirm the total price calculation works correctly with new discount rules.
• E2E test: The complete checkout flow is the most critical business process. An E2E test ensures users can complete purchases without errors.
• Exploratory testing: Automated tests cover known cases. Exploratory testing uncovers unexpected side effects that scripts might miss.

Pros and Cons

Pros	Cons
Protection against regression bugs	Test suite grows with each feature
Confidence in refactorings and code changes	Long execution times without a clear strategy
Safer, more frequent releases	Maintenance burden on tests
Early detection of side effects	Manual regression testing is expensive and slow
Living documentation of expected behavior	False positives erode confidence in the suite
Objective quality assessment before release	Initial effort to set up automation

Best Practices

• Automate: Manual regression testing doesn’t scale with frequent changes.
• Prioritize: Not all tests matter equally. Focus on critical paths first.
• Keep tests maintainable: Tests should adapt easily when code changes.
• Keep tests fast: Long test suites slow development. Parallelization and targeted selection help.
• Integrate into CI/CD: Regression tests should run automatically on every commit or merge.
• Avoid false positives: Flaky tests destroy confidence in the suite.
• Use test impact analysis: Run only affected tests to save time.

Comprehensive Practical Example: FastAPI + NiceGui Project with Docker and CI/CD

Project Context

The project consists of:

• FastAPI as the backend — provides API endpoints for text conversion tools (e.g., Markdown→HTML, JSON→YAML, Base64 en-/decoding, text statistics)
• NiceGui as the frontend — web-based UI that calls the API endpoints and displays results
• Docker — the tool runs locally during development and on the server via Docker containers
• Additional modules — for example, pydantic for validation, httpx for async HTTP requests, python-slugify for URL slug generation

The Vibecoding problem: AI-assisted code generation often unintentionally modifies other files, removes libraries, or changes imports. A function that worked yesterday suddenly breaks—without the developer noticing because the change happened somewhere completely different.

Project Structure

my-text-tools/
├── app/
│   ├── __init__.py
│   ├── main.py              # FastAPI app, registers routers
│   ├── routers/
│   │   ├── __init__.py
│   │   ├── markdown.py      # /api/markdown → HTML
│   │   ├── json_yaml.py     # /api/json-to-yaml, /api/yaml-to-json
│   │   ├── base64.py        # /api/base64/encode, /api/base64/decode
│   │   └── stats.py         # /api/stats (words, characters, lines)
│   ├── models/
│   │   ├── __init__.py
│   │   └── schemas.py       # Pydantic models for request/response
│   ├── services/
│   │   ├── __init__.py
│   │   ├── converter.py     # Core logic: conversion functions
│   │   └── slugify.py       # Slug generation
│   └── ui/
│       ├── __init__.py
│       └── nicegui_app.py   # NiceGui frontend, calls API
├── tests/
│   ├── __init__.py
│   ├── conftest.py           # pytest fixtures (test client, mocks)
│   ├── unit/
│   │   ├── test_converter.py
│   │   ├── test_slugify.py
│   │   └── test_models.py
│   ├── integration/
│   │   ├── test_markdown_api.py
│   │   ├── test_json_yaml_api.py
│   │   ├── test_base64_api.py
│   │   └── test_stats_api.py
│   └── e2e/
│       └── test_ui_flows.py  # NiceGui UI tests via Playwright
├── Dockerfile
├── docker-compose.yml
├── docker-compose.test.yml   # Test environment with isolated DB
├── requirements.txt
├── requirements-dev.txt      # pytest, pytest-asyncio, httpx, playwright
├── .github/
│   └── workflows/
│       ├── ci.yml            # Lint + unit + integration on every push
│       └── nightly.yml       # Full regression suite nightly
├── Makefile                  # make test, make test-unit, make test-e2e
└── pytest.ini

Step 1: Unit Regression Tests with pytest

Unit tests verify core logic in isolation—no database, no network, no UI.

# tests/unit/test_converter.py
import pytest
from app.services.converter import (
    markdown_to_html,
    json_to_yaml,
    yaml_to_json,
    base64_encode,
    base64_decode,
    text_statistics,
)

class TestMarkdownToHtml:
    """Regression tests for Markdown conversion."""

    def test_simple_heading(self):
        assert markdown_to_html("# Titel") == "<h1>Titel</h1>"

    def test_bold_text(self):
        assert markdown_to_html("**fett**") == "<strong>fett</strong>"

    def test_code_block(self):
        md = "```python\nprint('hello')\n```"
        result = markdown_to_html(md)
        assert "<code>" in result
        assert "print('hello')" in result

    def test_empty_input(self):
        assert markdown_to_html("") == ""

    def test_nested_lists(self):
        md = "- Item 1\n  - Subitem 1.1"
        result = markdown_to_html(md)
        assert "<ul>" in result
        assert "<li>Item 1" in result

    # Regression: Vibecoding removes library → ImportError
    def test_module_imports_successfully(self):
        """Ensures the converter remains importable.
        If Vibecoding removes, say, 'markdown' from requirements.txt,
        this test fails."""
        from app.services import converter
        assert hasattr(converter, 'markdown_to_html')


class TestJsonYamlConversion:
    """Regression tests for JSON↔YAML conversion."""

    def test_json_to_yaml_basic(self):
        json_input = '{"name": "test", "value": 42}'
        result = json_to_yaml(json_input)
        assert "name: test" in result
        assert "value: 42" in result

    def test_roundtrip_json_yaml_json(self):
        """JSON → YAML → JSON must be identical."""
        import json
        original = {"name": "test", "list": [1, 2, 3]}
        json_str = json.dumps(original)
        yaml_str = json_to_yaml(json_str)
        back = yaml_to_json(yaml_str)
        assert json.loads(back) == original

    def test_nested_structures(self):
        json_input = '{"outer": {"inner": {"deep": true}}}'
        result = json_to_yaml(json_input)
        assert "inner:" in result
        assert "deep: true" in result


class TestBase64:
    """Regression tests for Base64 en-/decoding."""

    def test_encode_decode_roundtrip(self):
        original = "Hello, World!"
        encoded = base64_encode(original)
        decoded = base64_decode(encoded)
        assert decoded == original

    def test_encode_empty_string(self):
        assert base64_encode("") == ""

    def test_decode_invalid_input_raises(self):
        with pytest.raises(Exception):
            base64_decode("!!!invalid_base64!!!")


class TestTextStatistics:
    """Regression tests for text statistics."""

    def test_word_count(self):
        stats = text_statistics("hello world foo")
        assert stats["words"] == 3

    def test_empty_text(self):
        stats = text_statistics("")
        assert stats["words"] == 0
        assert stats["characters"] == 0
        assert stats["lines"] == 0

    def test_multiline(self):
        stats = text_statistics("line1\nline2\nline3")
        assert stats["lines"] == 3

# tests/unit/test_slugify.py
from app.services.slugify import slugify

class TestSlugify:
    """Regression: If python-slugify is removed from requirements,
    this test fails—protecting against silent library loss."""

    def test_basic_slug(self):
        assert slugify("Hello World") == "hello-world"

    def test_german_umlauts(self):
        assert slugify("Übergröße") == "ubergrose"

    def test_special_characters(self):
        assert slugify("Python 3.12!@#") == "python-312"

    def test_empty_string(self):
        assert slugify("") == ""

Step 2: Integration Regression Tests with pytest + httpx

Integration tests verify API endpoints via FastAPI’s TestClient.

# tests/conftest.py
import pytest
from fastapi.testclient import TestClient
from app.main import app

@pytest.fixture
def client():
    """FastAPI test client for integration tests."""
    return TestClient(app)

@pytest.fixture
def sample_markdown():
    return "# Test\n\n**Bold** text with `code`."

# tests/integration/test_markdown_api.py
class TestMarkdownAPI:
    """Regression tests for the /api/markdown endpoint."""

    def test_markdown_to_html_success(self, client, sample_markdown):
        response = client.post("/api/markdown", json={"text": sample_markdown})
        assert response.status_code == 200
        data = response.json()
        assert "<h1>Test</h1>" in data["html"]
        assert "<strong>Bold</strong>" in data["html"]

    def test_markdown_empty_input(self, client):
        response = client.post("/api/markdown", json={"text": ""})
        assert response.status_code == 200
        assert response.json()["html"] == ""

    def test_markdown_missing_field(self, client):
        """Regression: Pydantic validation must stay active.
        If Vibecoding modifies the model, this test fails."""
        response = client.post("/api/markdown", json={})
        assert response.status_code == 422  # Validation Error

    def test_markdown_invalid_json(self, client):
        response = client.post("/api/markdown", data="not json")
        assert response.status_code == 422

# tests/integration/test_stats_api.py
class TestStatsAPI:
    """Regression tests for /api/stats."""

    def test_stats_basic(self, client):
        response = client.post("/api/stats", json={"text": "hello world"})
        assert response.status_code == 200
        data = response.json()
        assert data["words"] == 2
        assert data["characters"] == 11

    def test_stats_empty(self, client):
        response = client.post("/api/stats", json={"text": ""})
        assert response.status_code == 200
        assert response.json()["words"] == 0

    # Regression: Ensures the endpoint exists at all.
    # If Vibecoding removes a router, this fails.
    def test_stats_endpoint_exists(self, client):
        response = client.post("/api/stats", json={"text": "test"})
        assert response.status_code != 404

Step 3: End-to-End Regression Testing with Playwright

E2E tests verify the NiceGui interface through a real browser.

# tests/e2e/test_ui_flows.py
"""E2E regression tests for the NiceGui interface.
Requires the app to be running (docker-compose up -d)."""
import pytest
from playwright.sync_api import Page, expect

BASE_URL = "http://localhost:8080"

class TestMarkdownConverterUI:
    """Verifies that markdown conversion works through the UI."""

    def test_markdown_input_and_output(self, page: Page):
        page.goto(BASE_URL)
        page.fill("[data-testid='markdown-input']", "# Hello World")
        page.click("[data-testid='convert-button']")
        output = page.locator("[data-testid='markdown-output']")
        expect(output).to_contain_text("Hello World")
        expect(output).to_contain_text("<h1>")

    def test_clear_button(self, page: Page):
        page.goto(BASE_URL)
        page.fill("[data-testid='markdown-input']", "# Test")
        page.click("[data-testid='clear-button']")
        assert page.locator("[data-testid='markdown-input']").input_value() == ""

class TestNavigationRegression:
    """Ensures all UI tabs remain accessible.
    If Vibecoding removes a tab, this catches it."""

    @pytest.mark.parametrize("tab_name", [
        "Markdown", "JSON/YAML", "Base64", "Statistics"
    ])
    def test_tab_accessible(self, page: Page, tab_name):
        page.goto(BASE_URL)
        page.click(f"text={tab_name}")
        # Verifies that tab content is visible
        content = page.locator("[data-testid='tab-content']")
        expect(content).to_be_visible()

Step 4: Docker Test Setup

# Dockerfile
FROM python:3.12-slim

WORKDIR /app

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

COPY . .

# Start NiceGui + FastAPI
EXPOSE 8080
CMD ["python", "-m", "app.main"]

# docker-compose.test.yml
# Isolated test environment — no access to production data
version: "3.9"
services:
  app-test:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "8081:8080"  # Different port than production
    environment:
      - TESTING=true
      - LOG_LEVEL=DEBUG
    command: >
      bash -c "pip install -r requirements-dev.txt &&
               pytest tests/unit tests/integration -v --tb=short &&
               pytest tests/e2e -v --tb=short"

Step 5: CI/CD Pipeline with GitHub Actions

# .github/workflows/ci.yml
name: CI Regression Tests
on:
  push:
    branches: [main, dev]
  pull_request:
    branches: [main]

jobs:
  unit-and-integration:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt -r requirements-dev.txt
      - run: pytest tests/unit tests/integration -v --tb=short --junitxml=results.xml
      - uses: actions/upload-artifact@v4
        if: always()
        with:
          name: test-results
          path: results.xml

  e2e:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt -r requirements-dev.txt
      - run: playwright install --with-deps chromium
      - name: Start app in background
        run: |
          python -m app.main &
          sleep 5
      - run: pytest tests/e2e -v --tb=short

  docker-build-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: docker compose -f docker-compose.test.yml up --exit-code-from app-test

# .github/workflows/nightly.yml
name: Nightly Full Regression
on:
  schedule:
    - cron: "0 2 * * *"  # Every night at 02:00 UTC

jobs:
  full-regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt -r requirements-dev.txt
      - run: playwright install --with-deps chromium
      - name: Start app
        run: python -m app.main & sleep 5
      - name: Full regression suite
        run: pytest -v --tb=long --junitxml=nightly-results.xml
      - name: Notify on failure
        if: failure()
        run: |
          echo "::error::Nightly regression failed — check logs"

Step 6: Makefile for Local Test Execution

# Makefile
.PHONY: test test-unit test-integration test-e2e test-docker test-watch

test: test-unit test-integration
	@echo "✅ Unit + Integration tests passed"

test-unit:
	pytest tests/unit -v --tb=short

test-integration:
	pytest tests/integration -v --tb=short

test-e2e:
	pytest tests/e2e -v --tb=short

test-docker:
	docker compose -f docker-compose.test.yml up --exit-code-from app-test

test-watch:
	ptw -- -v --tb=short tests/

test-coverage:
	pytest --cov=app --cov-report=html tests/unit tests/integration
	@echo "📊 Coverage Report: htmlcov/index.html"

Step 7: Test Documentation

Each test file should be documented. Recommended structure:

docs/
├── testing.md              # Test strategy and guide
├── test-cases.md           # List of all test cases with descriptions
└── troubleshooting.md      # Known issues and solutions

docs/testing.md should include:

1. Test strategy — which tests run when (unit on commit, integration on PR, E2E nightly)
2. Execution — make test, make test-unit, make test-docker
3. CI/CD integration — which workflows exist and when they trigger
4. Adding new tests — conventions for file names, fixtures, assertions
5. Test coverage — make test-coverage and target coverage (e.g. ≥ 80%)
6. Troubleshooting — common failures and solutions

Step 8: Recommendations Specific to Vibecoding

When using Vibecoding (AI-assisted code generation), regressions often occur from:

• Removed imports — the AI deletes an import in one file that another file depends on
• Changed function signatures — parameters are renamed or removed
• Deleted utility functions — the AI thinks a function is “unused” and removes it
• Modified Pydantic models — fields are removed or renamed, API validation changes

Safeguards:

1. Import guard tests — each test file starts with a test that imports all relevant modules:

   def test_module_imports():
       """Regression: Ensures all modules remain importable."""
       import app.services.converter
       import app.services.slugify
       import app.routers.markdown
       import app.routers.stats

2. API contract tests — verify that endpoints respond with expected fields:

   def test_stats_response_schema(client):
       """Regression: Fails if Pydantic model is modified."""
       response = client.post("/api/stats", json={"text": "test"})
       data = response.json()
       assert "words" in data
       assert "characters" in data
       assert "lines" in data

3. Dependencies check — a test that parses requirements.txt and verifies all imported packages are listed:

   def test_all_imports_in_requirements():
       """Regression: Catches when Vibecoding removes an import
       but code still references it, or vice versa."""
       # Parse requirements.txt
       with open("requirements.txt") as f:
           requirements = {
               line.split("==")[0].split(">=")[0].strip().lower()
               for line in f if line.strip() and not line.startswith("#")
           }
       # Check critical packages
       assert "fastapi" in requirements
       assert "nicegui" in requirements
       assert "pydantic" in requirements
       assert "httpx" in requirements
       assert "python-slugify" in requirements

4. Pre-commit hook — runs unit tests before each commit:

   # .git/hooks/pre-commit
   #!/bin/bash
   pytest tests/unit -v --tb=short || exit 1

5. Git diff review — check git diff after each Vibecoding session:

   git diff --stat          # Which files changed?
   git diff requirements.txt # Were packages removed?
   git diff app/services/    # Was core logic modified?

Test Pyramid Summary for This Project

Level	Tool	When	Duration	Count
Unit	pytest	On every commit	< 5 s	30–50
Integration	pytest + TestClient	On every PR	< 30 s	15–25
E2E	Playwright	Nightly + before release	2–5 min	5–10
Docker	docker-compose.test	On every PR	1–2 min	1 Suite

Rule: When a code change modifies a function, unit and integration tests for that area must pass before committing.

Essential Tools

• JUnit / pytest / Jest: Frameworks for unit regression testing.
• Selenium / Cypress / Playwright: Tools for E2E regression testing.
• Postman / REST Assured: For API regression testing.
• TestNG / NUnit: Frameworks with advanced test management capabilities.
• Jenkins / GitHub Actions / GitLab CI: CI/CD pipelines for automated regression test execution.

Key Exam Points

• Regression test: A test ensuring that existing functionality continues to work after changes.
• Regression bug: A bug that arises from a new change in a previously working area.
• Triggers: Code changes, bugfixes, refactorings, dependency updates, merges, releases.
• Test levels: Unit, integration, and E2E all serve as regression tests.
• Strategies: Full regression, risk-based regression, test impact analysis.
• Automation: Essential for scalable and repeatable regression testing.
• Smoke test: A minimal test checking whether the system runs at all.
• Sanity test: A targeted test validating a small change.
• Nightly build: An automated overnight test run for early detection.
• CI/CD integration: Regression tests run automatically on every commit or merge.
• Maintenance: Tests must be kept current as code changes.
• False positives: Unstable tests can destroy confidence in the suite.

Key Sources

1. https://www.guru99.com/regression-testing.html
2. https://www.atlassian.com/continuous-delivery/software-testing/regression-testing
3. https://en.wikipedia.org/wiki/Regression_testing

Continue your Software Testing learning path

The next article in the Software Testing learning path covers acceptance testing — how acceptance tests ensure that requirements are met.