Initial commit: tdoc-crawler with comprehensive test suite and .env support

docs/source

# From https://raw.githubusercontent.com/github/gitignore/main/Python.gitignore

# Byte-compiled / optimized / DLL files

__pycache__/

*.py[codz]

*$py.class

# C extensions

*.so

# Distribution / packaging

.Python

build/

develop-eggs/

dist/

downloads/

eggs/

.eggs/

lib/

lib64/

parts/

sdist/

var/

wheels/

share/python-wheels/

*.egg-info/

.installed.cfg

*.egg

MANIFEST

# PyInstaller

#  Usually these files are written by a python script from a template

#  before PyInstaller builds the exe, so as to inject date/other infos into it.

*.manifest

*.spec

# Installer logs

pip-log.txt

pip-delete-this-directory.txt

# Unit test / coverage reports

htmlcov/

.tox/

.nox/

.coverage

.coverage.*

.cache

nosetests.xml

coverage.xml

*.cover

*.py.cover

.hypothesis/

.pytest_cache/

cover/

# Translations

*.mo

*.pot

# Django stuff:

*.log

local_settings.py

db.sqlite3

db.sqlite3-journal

# Flask stuff:

instance/

.webassets-cache

# Scrapy stuff:

.scrapy

# Sphinx documentation

docs/_build/

# PyBuilder

.pybuilder/

target/

# Jupyter Notebook

.ipynb_checkpoints

# IPython

profile_default/

ipython_config.py

# pyenv

#   For a library or package, you might want to ignore these files since the code is

#   intended to run in multiple environments; otherwise, check them in:

# .python-version

# pipenv

#   According to pypa/pipenv#598, it is recommended to include Pipfile.lock in version control.

#   However, in case of collaboration, if having platform-specific dependencies or dependencies

#   having no cross-platform support, pipenv may install dependencies that don't work, or not

#   install all needed dependencies.

#Pipfile.lock

# UV

#   Similar to Pipfile.lock, it is generally recommended to include uv.lock in version control.

#   This is especially recommended for binary packages to ensure reproducibility, and is more

#   commonly ignored for libraries.

#uv.lock

# poetry

#   Similar to Pipfile.lock, it is generally recommended to include poetry.lock in version control.

#   This is especially recommended for binary packages to ensure reproducibility, and is more

#   commonly ignored for libraries.

#   https://python-poetry.org/docs/basic-usage/#commit-your-poetrylock-file-to-version-control

#poetry.lock

#poetry.toml

# pdm

#   Similar to Pipfile.lock, it is generally recommended to include pdm.lock in version control.

#   pdm recommends including project-wide configuration in pdm.toml, but excluding .pdm-python.

#   https://pdm-project.org/en/latest/usage/project/#working-with-version-control

#pdm.lock

#pdm.toml

.pdm-python

.pdm-build/

# pixi

#   Similar to Pipfile.lock, it is generally recommended to include pixi.lock in version control.

#pixi.lock

#   Pixi creates a virtual environment in the .pixi directory, just like venv module creates one

#   in the .venv directory. It is recommended not to include this directory in version control.

.pixi

# PEP 582; used by e.g. github.com/David-OConnor/pyflow and github.com/pdm-project/pdm

__pypackages__/

# Celery stuff

celerybeat-schedule

celerybeat.pid

# SageMath parsed files

*.sage.py

# Environments

.env

.envrc

.venv

env/

venv/

ENV/

env.bak/

venv.bak/

# Spyder project settings

.spyderproject

.spyproject

# Rope project settings

.ropeproject

# mkdocs documentation

/site

# mypy

.mypy_cache/

.dmypy.json

dmypy.json

# Pyre type checker

.pyre/

# pytype static type analyzer

.pytype/

# Cython debug symbols

cython_debug/

# PyCharm

#  JetBrains specific template is maintained in a separate JetBrains.gitignore that can

#  be found at https://github.com/github/gitignore/blob/main/Global/JetBrains.gitignore

#  and can be added to the global gitignore or merged into this file.  For a more nuclear

#  option (not recommended) you can uncomment the following to ignore the entire idea folder.

#.idea/

# Abstra

# Abstra is an AI-powered process automation framework.

# Ignore directories containing user credentials, local state, and settings.

# Learn more at https://abstra.io/docs

.abstra/

# Visual Studio Code

#  Visual Studio Code specific template is maintained in a separate VisualStudioCode.gitignore

#  that can be found at https://github.com/github/gitignore/blob/main/Global/VisualStudioCode.gitignore

#  and can be added to the global gitignore or merged into this file. However, if you prefer,

#  you could uncomment the following to ignore the entire vscode folder

# .vscode/

# Ruff stuff:

.ruff_cache/

# PyPI configuration file

.pypirc

# Cursor

#  Cursor is an AI-powered code editor. `.cursorignore` specifies files/directories to

#  exclude from AI features like autocomplete and code analysis. Recommended for sensitive data

#  refer to https://docs.cursor.com/context/ignore-files

.cursorignore

.cursorindexingignore

# Marimo

marimo/_static/

marimo/_lsp/

__marimo__/

repos:

  - repo: https://github.com/pre-commit/pre-commit-hooks

    rev: "v5.0.0"

    hooks:

      - id: check-case-conflict

      - id: check-merge-conflict

      - id: check-toml

      - id: check-yaml

      - id: check-json

        exclude: ^.devcontainer/devcontainer.json

      - id: pretty-format-json

        exclude: ^.devcontainer/devcontainer.json

        args: [--autofix, --no-sort-keys]

      - id: end-of-file-fixer

      - id: trailing-whitespace

  - repo: https://github.com/astral-sh/ruff-pre-commit

    rev: "v0.12.7"

    hooks:

      - id: ruff-check

        args: [ --exit-non-zero-on-fix ]

      - id: ruff-format

3.13

 No newline at end of file

# Contributing to `tdoc-crawler`

Contributions are welcome, and they are greatly appreciated!

Every little bit helps, and credit will always be given.

You can contribute in many ways:

# Types of Contributions

## Report Bugs

Report bugs at https://github.com/Jan-Reimes_HEAD/tdoc-crawler/issues

If you are reporting a bug, please include:

- Your operating system name and version.

- Any details about your local setup that might be helpful in troubleshooting.

- Detailed steps to reproduce the bug.

## Fix Bugs

Look through the GitHub issues for bugs.

Anything tagged with "bug" and "help wanted" is open to whoever wants to implement a fix for it.

## Implement Features

Look through the GitHub issues for features.

Anything tagged with "enhancement" and "help wanted" is open to whoever wants to implement it.

## Write Documentation

tdoc-crawler could always use more documentation, whether as part of the official docs, in docstrings, or even on the web in blog posts, articles, and such.

## Submit Feedback

The best way to send feedback is to file an issue at https://github.com/Jan-Reimes_HEAD/tdoc-crawler/issues.

If you are proposing a new feature:

- Explain in detail how it would work.

- Keep the scope as narrow as possible, to make it easier to implement.

- Remember that this is a volunteer-driven project, and that contributions

  are welcome :)

# Get Started!

Ready to contribute? Here's how to set up `tdoc-crawler` for local development.

Please note this documentation assumes you already have `uv` and `Git` installed and ready to go.

1. Fork the `tdoc-crawler` repo on GitHub.

2. Clone your fork locally:

```bash

cd <directory_in_which_repo_should_be_created>

git clone git@github.com:YOUR_NAME/tdoc-crawler.git

```

3. Now we need to install the environment. Navigate into the directory

```bash

cd tdoc-crawler

```

Then, install and activate the environment with:

```bash

uv sync

```

4. Install pre-commit to run linters/formatters at commit time:

```bash

uv run pre-commit install

```

5. Create a branch for local development:

```bash

git checkout -b name-of-your-bugfix-or-feature

```

Now you can make your changes locally.

6. Don't forget to add test cases for your added functionality to the `tests` directory.

7. When you're done making changes, check that your changes pass the formatting tests.

```bash

make check

```

Now, validate that all unit tests are passing:

```bash

make test

```

9. Before raising a pull request you should also run tox.

   This will run the tests across different versions of Python:

```bash

tox

```

This requires you to have multiple versions of python installed.

This step is also triggered in the CI/CD pipeline, so you could also choose to skip this step locally.

10. Commit your changes and push your branch to GitHub:

```bash

git add .

git commit -m "Your detailed description of your changes."

git push origin name-of-your-bugfix-or-feature

```

11. Submit a pull request through the GitHub website.

# Pull Request Guidelines

Before you submit a pull request, check that it meets these guidelines:

1. The pull request should include tests.

2. If the pull request adds functionality, the docs should be updated.

   Put your new functionality into a function with a docstring, and add the feature to the list in `README.md`.