🚁 RL for Drone Delivery

A single PPO policy that flies a simulated quadrotor to deliver packages to multiple clients — managing battery, rejecting hidden wind, and dodging obstacles, with no hand-coded trajectory planning.

---

✨ Highlights

• 96% success / 4% collision for the final policy under simultaneous wind and obstacles.
• POMDP by design — the wind force is hidden from the agent, so the policy must infer it from its own motion.
• Five interlocking curricula + a four-campaign training pipeline that adds one new challenge at a time.
• Policy surgery that grows the observation from 13→17 dims while preserving every previously learned skill.
• Frozen-scenario evaluation — deterministic benchmarks so two checkpoints are compared fairly, with per-difficulty tier breakdowns.

New here? Start with docs/overview.md for the conceptual tour, then docs/getting_started.md for installation and first commands.

---

📊 Results at a glance

Campaign	Setting	Success	Collision	Mean reward
1 — Vanilla	calm air, energy budget	90%	—	450.7
2 — Wind	OU wind σ→0.50	98%	—	949.6
3 — Obstacles (K=2)	up to 5 spheres (500 scn)	91.2%	8.8%	736.0
4 — Combined	wind + obstacles	96%	4%	851.4

_{Figures are generated from the raw training logs and benchmark JSONs by python scripts/make_readme_figs.py.}

---

🧩 The Four Physical Challenges

1. Energy management — every step costs base_drain + speed_coeff × current_speed units of battery. The agent must trade speed against efficiency.
2. Wind disturbance — an Ornstein–Uhlenbeck process applies stochastic external forces. Wind is not in the observation vector, so the agent must learn implicit state estimation.
3. Obstacle avoidance — up to 5 static sphere colliders, with the K nearest encoded in the observation (K=1 → 13-dim obs, K=2 → 17-dim obs).
4. Multi-client routing — N clients (1 to 3–8 as curriculum advances) at random positions; deliveries register when the drone enters a delivery radius.

---

🏗️ How it works

The policy never touches the motors. A built-in solver picks the next target, the PPO network commits to a velocity, and a low-level PID controller realises it in PyBullet.

flowchart LR
    P[PyBullet physics] --> S[Simulator state]
    S --> TSP[Target selector<br/>TSP / nearest-neighbour]
    TSP -->|current target| O[Observation vector<br/>vel · energy · target · perception]
    O --> NN[PPO policy network<br/>shared MLP 256×256]
    NN -->|3-D velocity command| PID[PID flight controller]
    PID -->|motor RPMs| P
    W[Hidden OU wind] -.external force.-> P

Loading

Multi-Campaign Training Pipeline

Each campaign resumes from the previous campaign's best checkpoint, so new capabilities are layered on top of prior ones instead of being learned from scratch.

flowchart LR
    C1[1 · Vanilla<br/>baseline navigation]:::v --> C2[2 · Wind<br/>σ 0.01 → 0.50]:::w
    C2 --> C3a[3a · Obstacles K=1<br/>count 0 → 5]:::o
    C3a --> C3b[3b · Obstacles K=2<br/>policy surgery 13 → 17]:::o
    C2 --> C4[4 · Combined<br/>wind + obstacles]:::c
    C3b --> C4
    classDef v fill:#4C72B0,color:#fff,stroke:none;
    classDef w fill:#55A868,color:#fff,stroke:none;
    classDef o fill:#C44E52,color:#fff,stroke:none;
    classDef c fill:#8172B3,color:#fff,stroke:none;

Loading

Campaign	Description	Resumes from	Output dir
1 — Baseline navigation	Calm-air multi-client routing + energy mgmt	(scratch)	models/
2 — Wind robustness	--wind-anneal ramps σ from 0.01 → 0.50	Campaign 1	models_wind/
3a — Obstacles K=1	--obstacles ramps obstacle count 0 → 5	Campaign 2	models_finetune/
3b — Obstacles K=2	Policy surgery expands obs 13 → 17 dim	Campaign 3a	models_finetune_k2/
4 — Combined	Wind + obstacles fully active	Campaign 2 weights + 3b obs setup	models_wind_obs/

The K=1 → K=2 transition would normally fail because the first hidden layer's weight matrix changes shape from [256, 13] to [256, 17]. scripts/expand_obs_for_k2.py solves this by zero-padding the four new input columns — at initialisation the K=2 policy is mathematically identical to the K=1 policy, then learns to exploit the new perception channels during fine-tuning. See docs/training.md for the full walkthrough.

---

⚡ Installation

Prerequisites: Python 3.10+, OpenGL/display server (use xvfb on headless boxes), and a C++ toolchain to build PyBullet's native extensions.

pip install -r requirements.txt

stable-baselines3 pulls in torch and gymnasium automatically.

Verify the install — runs 3 random-policy episodes and opens the PyBullet GUI:

python scripts/run_env.py

---

🚀 Quick Start

Train (Campaign 1 — baseline navigation)

python training/train.py --config configs/config.yaml

Outputs models/best_model.zip + models/best_model_norm.pkl, regular checkpoints under models/, and TensorBoard event files under logs/.

tensorboard --logdir logs/

Useful flags: --output-dir DIR, --timesteps N, --seed N, --resume CHECKPOINT.zip, --resume-stage N, --wind-anneal, --obstacles.

Generate frozen evaluation datasets (run once)

Evaluations use deterministic, pre-generated scenario sets so that two checkpoints can be compared without stochastic variance. Each generator supports a --brief flag for a quick 50-scenario dataset and writes a 500-scenario "full" dataset by default. Outputs land under evaluation/.

# Vanilla — Campaign 1 (no wind, no obstacles)
python evaluation/create_vanilla_dataset.py             # 500 → evaluation/full_vanilla_dataset.json
python evaluation/create_vanilla_dataset.py --brief     #  50 → evaluation/brief_vanilla_dataset.json

# Wind — Campaign 2
python evaluation/create_wind_dataset.py                # 500 → evaluation/full_wind_dataset.json
python evaluation/create_wind_dataset.py --brief        #  50 → evaluation/brief_wind_dataset.json

# Obstacles — Campaign 3 (K=1 or K=2)
python evaluation/create_obstacles_dataset.py           # 500 → evaluation/full_obstacles_dataset.json
python evaluation/create_obstacles_dataset.py --brief   #  50 → evaluation/brief_obstacle_dataset.json

# Combined — Campaign 4 (writes BOTH brief and full in one run)
python evaluation/create_combined_dataset.py            # 50  → evaluation/brief_wind_obs_dataset.json
                                                        # 500 → evaluation/full_wind_obs_dataset.json

Common flags on all four generators: --out PATH to override the output file, --seed N to change the base random seed. The standalone generators also accept --num-scenarios N.

Evaluate a trained model

One harness per campaign — each loads <model>.zip plus its paired <model>_norm.pkl VecNormalize statistics (both required, see the warning in docs/training.md), iterates the dataset, and writes a tier-broken JSON report.

# Campaign 1 — baseline (no wind, no obstacles)
python evaluation/evaluate.py \
    --models-dir models/ --model best_model \
    --dataset evaluation/full_vanilla_dataset.json \
    --report reports/baseline.json

# Campaign 2 — wind robustness
python evaluation/evaluate_wind.py \
    --models-dir models_wind/models/ --model best_model \
    --dataset evaluation/full_wind_dataset.json \
    --report reports/wind.json

# Campaign 3 — obstacle avoidance (K=1 or K=2)
python evaluation/evaluate_obstacles.py \
    --models-dir models_finetune/models/ --model best_model \
    --dataset evaluation/full_obstacles_dataset.json \
    --report reports/obstacles.json

# Campaign 4 — combined wind + obstacles
python evaluation/evaluate_combined.py \
    --models-dir models_wind_obs/models/ --model best_model \
    --dataset evaluation/full_wind_obs_dataset.json \
    --report reports/combined.json

Swap full_* for brief_* to run against the 50-scenario quick datasets. Add --render to any harness to open a PyBullet GUI during the run.

Refresh the README figures & 2-D trajectory diagnostic

# Regenerate assets/*.png from logs/ and reports/ (numpy + matplotlib only)
python scripts/make_readme_figs.py

evaluation/render_2d.py plays one scenario from the obstacle dataset against the Campaign 3 model and writes a top-down Matplotlib animation — useful for why did this episode fail questions where aggregate metrics fall short.

---

📁 Repository Layout

env/            DroneDeliveryEnv, energy model, OU wind, obstacle field, PyBullet markers
agents/         PPO wrapper: MlpPolicy, LR schedule, checkpoint save/load
training/       Training orchestrator + SB3 callbacks (curriculum, eval, checkpointing)
evaluation/     Four benchmark harnesses, BenchmarkWrapper, frozen-dataset generators
configs/        Single YAML: env physics, rewards, curriculum, PPO hyperparams
scripts/        Policy surgery (K=1→K=2), diagnostics, run_env.py, README figure generator
docs/           Conceptual overview + per-subsystem technical references
assets/         Generated figures used in this README
logs/           TensorBoard event files + Monitor CSVs
reports/        JSON benchmark reports

---

📚 Documentation

The docs/ folder is the source of truth for the system's design. Each file is self-contained and cross-linked.

• Overview — Top-down tour: the four challenges, system architecture, training philosophy, the multi-campaign pipeline.
• Getting Started — Install, verify, train, monitor, and evaluate, with CLI flag reference.
• Environment — DroneDeliveryEnv, the Manager-Worker control scheme, observation space, and reward formulation.
• Physics Subsystems — Battery drain model, Ornstein–Uhlenbeck wind, sphere obstacle dynamics.
• Curriculum — The five interlocking curricula, dual-gated wind annealing, deterministic checkpoint recovery.
• Training — PPO configuration, the callback pipeline, multi-campaign sequencing, K=1 → K=2 policy surgery, full config.yaml reference.
• Evaluation — Frozen-scenario philosophy, the four benchmark harnesses, metric tier breakdowns, 2-D trajectory diagnostics.

---

👥 Authors

Aya Benali Khodja (Team Lead) · Ahmed Thebat Mazouz · Sarab Saidane · Hachem Safi Eddine Sekhsoukh · Youcef Guergour Supervised by Dr. Boulmerka Aissa — National School of Artificial Intelligence (ENSIA), 2025–2026.

📜 License

Released under the MIT License.