Commit b905624c authored by Jan Reimes's avatar Jan Reimes
Browse files

feat(executor): restore pool_executors package and remove upstream shim

parent e3854ea4
Loading
Loading
Loading
Loading
+28 −0
Original line number Diff line number Diff line
@@ -78,9 +78,37 @@ This package was created to:
1. **Unify executor creation**: Single factory function for all executor types with consistent interface
1. **Enable future extraction**: Designed as standalone package for easy repository separation

## Subinterpreter Support

### Why Subinterpreters?

Python's Global Interpreter Lock (GIL) and thread-safety issues in many C extensions limit the effectiveness of standard threading. Subinterpreters provide an alternative approach:

- **Thread Safety**: Each subinterpreter has its own GIL, providing true isolation between tasks
- **Performance**: Similar to real threading without the thread-safety concerns of shared-memory threading
- **Encapsulation**: Each subinterpreter runs in complete isolation with its own interpreter state

### Subinterpreters vs Other Execution Strategies

| Strategy | Pros | Cons |
|----------|------|------|
| **Serial** | No overhead, deterministic, easy debugging | No parallelism |
| **Threading** | Low overhead, shared memory | GIL contention, thread-safety issues in many libraries |
| **Multiprocessing** | True parallelism, no GIL | High inter-process communication overhead, especially on Windows |
| **Subinterpreter** | Thread-safety isolation, good performance | Python 3.14+ required, still maturing |

### Future Outlook

As Python 3.14+ matures and more libraries become subinterpreter-safe, this approach will provide a clean path toward concurrent execution without the overhead of multiprocessing. Until then, using well-encapsulated subinterpreters offers a good balance between thread-safety and performance characteristics similar to real multi-threading.

### Fallback Behavior

When `InterpreterPoolExecutor` is not available (Python < 3.14), the factory automatically falls back to `ProcessPoolExecutor` with a warning log message.

## Requirements

- Python 3.13 or higher
- Python 3.14+ recommended for subinterpreter support
- No external dependencies (stdlib only)

## License
+3 −3
Original line number Diff line number Diff line
"""Executor pool extensions with serial execution support."""

from pool_executors.factory import create_executor
from pool_executors.serial import SerialPoolExecutor
from pool_executors.types import ExecutorType
from .factory import create_executor
from .serial import SerialPoolExecutor
from .types import ExecutorType

__version__ = "0.1.0"

+4 −6
Original line number Diff line number Diff line
@@ -3,17 +3,15 @@
from __future__ import annotations

import logging
from concurrent.futures import (Executor, ProcessPoolExecutor,
                                ThreadPoolExecutor)
from concurrent.futures import Executor, ProcessPoolExecutor, ThreadPoolExecutor
from typing import Any

from pool_executors.serial import SerialPoolExecutor
from pool_executors.types import ExecutorType
from .serial import SerialPoolExecutor
from .types import ExecutorType

# Import InterpreterPoolExecutor for Python 3.14+, handle gracefully for older versions
try:
    from concurrent.futures import \
        InterpreterPoolExecutor  # type: ignore[attr-defined]  # noqa: PLC0415
    from concurrent.futures import InterpreterPoolExecutor  # noqa: PLC0415

    HAS_INTERPRETER_POOL_EXECUTOR = True
except (ImportError, AttributeError):
+0 −1
Original line number Diff line number Diff line
@@ -2,7 +2,6 @@

from __future__ import annotations

import contextlib
import logging
from collections.abc import Callable
from concurrent.futures import Executor, Future
+1 −1
Original line number Diff line number Diff line
@@ -7,7 +7,7 @@ from collections.abc import Callable
from concurrent.futures import Executor
from typing import Any, TypeVar

from tdoc_crawler.pool_executors import create_executor
from pool_executors.pool_executors import create_executor

T = TypeVar("T")

Loading