Python Environment Setup (venv, pip, poetry)

The Core Problem:

Imagine you have two AI projects. Project A needs TensorFlow 2.12 and Project B needs TensorFlow 2.16. If you install both globally, one project will break. This is called dependency hell.

Virtual environments solve this by giving each project its own isolated Python installation with its own packages. What happens in one environment stays in that environment.

Real-World Analogy - Swiggy Cloud Kitchens:

Think of Python environments like Swiggy cloud kitchens. Each kitchen (environment) has its own ingredients (packages), equipment (Python version), and recipes (code). Even though they are in the same building (your computer), they do not interfere with each other.

Without environments, it is like all chefs sharing one kitchen - someone needs sugar for dessert while another needs the counter for biryani. Chaos!

What We Will Cover:

• venv: Python's built-in virtual environment tool (simplest)
• pip: The default package installer for Python
• poetry: Modern dependency management with lock files (best for serious projects)
• conda: Popular in data science for managing Python versions + system libraries

What is venv?

venv is Python's built-in module (since Python 3.3) for creating lightweight virtual environments. No extra installation needed - it comes with Python.

When you create a venv, Python copies its interpreter and creates an isolated site-packages directory where pip installs packages.

How to Use venv - The Essential Commands:

# Create a virtual environment
python -m venv myproject-env

# Activate it (macOS/Linux)
source myproject-env/bin/activate

# Activate it (Windows)
myproject-envScriptsactivate

# Your terminal shows (myproject-env) prefix
# Now pip install goes into this environment only

# Install packages
pip install langchain openai

# Save dependencies
pip freeze > requirements.txt

# Deactivate when done
deactivate

# Recreate environment from requirements
pip install -r requirements.txt

venv Best Practices:

• Name convention: Use .venv or venv in your project root
• Git ignore: Always add your venv folder to .gitignore
• requirements.txt: Always maintain this file for reproducibility
• One venv per project: Never share environments between projects

venv Limitations:

• Cannot manage different Python versions (you need pyenv for that)
• requirements.txt does not lock transitive dependencies properly
• No built-in way to separate dev vs production dependencies
• pip freeze captures everything, including unused packages

What is pip?

pip (Pip Installs Packages) is Python's default package manager. It downloads packages from PyPI (Python Package Index) - the central repository with 500,000+ packages.

Think of PyPI like the Play Store for Python libraries. pip is the app that downloads and installs from it.

Essential pip Commands for AI/ML:

# Install a specific version
pip install transformers==4.38.0

# Install with version constraints
pip install "langchain>=0.1.0,<0.2.0"

# Install from requirements file
pip install -r requirements.txt

# Show installed packages
pip list

# Show details of a package
pip show torch

# Upgrade a package
pip install --upgrade openai

# Uninstall
pip uninstall tensorflow

# Install extra dependencies
pip install "langchain[all]"

requirements.txt - The Dependency File:

# requirements.txt - Pin exact versions for reproducibility
openai==1.12.0
langchain==0.1.9
chromadb==0.4.22
python-dotenv==1.0.0
fastapi==0.109.0
uvicorn==0.27.0

Pro tip: Always pin exact versions in production. Use >= only during development when you want the latest.

Common AI/ML Packages You Will Use:

Why Poetry Over pip?

Poetry solves the problems pip does not handle well:

• Lock file: poetry.lock captures exact versions of ALL dependencies (including transitive ones). pip freeze is not reliable for this.
• Dev dependencies: Separate dev tools (pytest, black) from production deps
• Dependency resolution: Checks for version conflicts BEFORE installing
• Project metadata: Single pyproject.toml for everything
• Virtual env management: Creates and manages venvs automatically

Poetry Workflow:

# Install poetry (one-time)
curl -sSL https://install.python-poetry.org | python3 -

# Create new project
poetry new my-ai-project

# Or initialize in existing project
poetry init

# Add dependencies
poetry add langchain openai chromadb

# Add dev dependencies
poetry add --group dev pytest black ruff

# Install all dependencies
poetry install

# Run commands inside the environment
poetry run python main.py

# Activate shell
poetry shell

# Update dependencies
poetry update

# Export to requirements.txt (for Docker)
poetry export -f requirements.txt --output requirements.txt

pyproject.toml - Poetry's Config File:

[tool.poetry]
name = "my-ai-project"
version = "0.1.0"
description = "AI-powered Flipkart review analyzer"

[tool.poetry.dependencies]
python = "^3.11"
langchain = "^0.1.9"
openai = "^1.12.0"
chromadb = "^0.4.22"

[tool.poetry.group.dev.dependencies]
pytest = "^8.0"
black = "^24.1"
ruff = "^0.2.0"

Poetry vs pip - When to Use Which:

Step 1: Install Python 3.11+

Download from python.org or use a version manager:

# macOS with Homebrew
brew install python@3.11

# Or use pyenv for multiple versions
brew install pyenv
pyenv install 3.11.8
pyenv global 3.11.8

# Verify
python --version  # Python 3.11.8

Step 2: Project Setup with Poetry

# Create project
mkdir flipkart-review-bot && cd flipkart-review-bot
poetry init --name flipkart-review-bot --python "^3.11"

# Add AI dependencies
poetry add langchain openai chromadb python-dotenv

# Add dev tools
poetry add --group dev pytest ruff

# Create project structure
mkdir -p src tests
touch src/__init__.py src/main.py
touch .env .gitignore

Step 3: Environment Variables (.env)

# .env - NEVER commit this file!
OPENAI_API_KEY=sk-proj-...
ANTHROPIC_API_KEY=sk-ant-...

# .gitignore
.env
.venv/
__pycache__/
*.pyc
.ruff_cache/

CRITICAL: Never commit API keys to git. Use .env files + python-dotenv to load them securely.

Step 4: VS Code Setup for AI Development

• Python extension: Provides IntelliSense, debugging, linting
• Jupyter extension: Run notebooks inside VS Code
• Select interpreter: Cmd+Shift+P > "Python: Select Interpreter" > choose your venv
• Ruff extension: Fast Python linter (replaces flake8, isort, black)

Problem 1: "Module not found" even after pip install

Cause: You installed the package in global Python but your terminal is using a different venv (or vice versa).

Fix: Check which Python is active: which python. Make sure it points to your venv.

Problem 2: torch/tensorflow fails to install

Cause: These are huge packages with platform-specific builds. Wrong OS/GPU version breaks things.

Fix: Always use the official install command from pytorch.org or tensorflow.org. For Mac M1/M2, use pip install torch torchvision torchaudio (MPS backend).

Problem 3: Version conflicts between packages

Cause: Package A needs numpy<1.24 but package B needs numpy>=1.25.

Fix: Use Poetry - it detects conflicts before installing. With pip, you get cryptic errors after installation.

Problem 4: Accidentally committed .env with API keys

Cause: Forgot to add .env to .gitignore before first commit.

Fix: (1) Immediately revoke and rotate the API key. (2) Remove from git history using git filter-branch or BFG Repo-Cleaner. (3) Add to .gitignore. The key is already exposed even if you delete the file - ALWAYS rotate.

Problem 5: Different behavior on teammate's machine

Cause: Different package versions, OS, or Python version.

Fix: Use poetry.lock (or requirements.txt with pinned versions). Use Docker for complete reproducibility. Specify Python version in pyproject.toml.

Q: Why should you use virtual environments in Python?

Q: What is the difference between pip freeze and a poetry.lock file?

Q: How do you manage API keys securely in a Python AI project?

Q: When would you choose conda over pip/poetry?

Q: How do you ensure reproducible environments across a team?