Skip to content

MykheiSilchuk/Web-Scrapper-Bookstore

BranchesTags

Repository files navigation

📚 Bookstore Scraper

A comprehensive web scraping solution for extracting book data from kennys.ie using Scrapy framework with MySQL persistence and clean architecture.

🎯 Project Overview

This project is built upon scrapy-boilerplate v2 and provides a robust solution for:

  • 🕷️ Web Crawling: Automated crawling of www.kennys.ie for book data from specified categories
  • 📊 Data Extraction: Comprehensive extraction of book details including publishing info, pricing, and cover images
  • 🗄️ Database Persistence: Secure data storage in MySQL using SQLAlchemy ORM
  • 🔧 Schema Management: Database migrations handled through Alembic
  • 📝 Advanced Logging: Clean, structured logging with custom formatters
  • 🌐 Proxy Support: Built-in proxy server integration for reliable scraping
  • 📦 Image Archiving: Utility for organizing and archiving collected book covers

✅ Requirements & Compliance

📋 General Requirements

Requirement Status Implementation Details
Code Quality ✅ PEP 8 compliant, English documentation
Architecture ✅ Based on scrapy-boilerplate v2
Task Management ✅ Decomposed into subtasks with daily commits
Database Integration ✅ SQLAlchemy Core with ORM models
Configuration ✅ Environment-based config (.env file)
Logging System ✅ Custom logger with dual output (console + file)
Proxy Integration ✅ Mandatory proxy with 16 concurrent threads
Git Workflow ✅ Git flow principles

🎯 Completion Criteria

  • ✅ Link Extraction: Extract all book detail page URLs
  • ✅ Field Parsing: Extract all fields defined in -bookstore_scraper/items.py-
  • ✅ Complete Results: Database and image archives contain all scraped data

🚀 Quick Start Guide

Prerequisites

Before starting, ensure you have:

  • 🐍 Python 3.8+ (recommended: Python 3.10+)
  • 📦 Poetry (Python package manager)
  • 🗃️ MySQL Server (Community Server or Docker)

1. Environment Setup

# Clone the repository
git clone https://gitlab.groupbwt.com/work.silchuk/task-1-bookstore-scraper.git
cd task-1-bookstore-scraper

# Install Poetry (if needed)
curl -sSL https://install.python-poetry.org | python3 -

# Install dependencies
poetry install

# Activate virtual environment
poetry shell

2. Database Configuration

Create MySQL Database

CREATE DATABASE IF NOT EXISTS bookstore_db 
CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;

-- Optional: Create dedicated user
CREATE USER 'scraper_user'@'localhost' IDENTIFIED BY 'your_password';
GRANT ALL PRIVILEGES ON bookstore_db.* TO 'scraper_user'@'localhost';
FLUSH PRIVILEGES;

Configure Environment Variables

Create .env file in project root:

``.env

Database Settings

DB_HOST=localhost DB_PORT=3306 DB_USERNAME=your_mysql_username DB_PASSWORD=your_mysql_password DB_DATABASE=bookstore_db

Proxy Configuration

PROXY=http://brd-customer-hl_f77c429c-zone-academy_s2025_3:h8xb6pk3oy8h@brd.superproxy.io:33335

Logging Level

LOG_LEVEL=CRITICAL


### 3. Database Migration

```bash
# Apply database migrations
poetry run alembic upgrade head

# Verify current revision
poetry run alembic current


## 📁 Project Architecture

📦 bookstore-scraper/ ├── 🗃️ alembic/ # Database migrations ├── 📁 scripts/ # Utility scripts ├── 🔧 src/ # Main application source │ ├── 📁 database/
| | models/ # Database layer │ │ └── 📄 base.py # SQLAlchemy base model │ ├── 📁 items/ # Scrapy item definitions │ │ └── 📄 book_item.py # Book data structure │ ├── 📁 middlewares/ # Scrapy middlewares │ ├── 📁 parsers/ # Data parsing utilities │ │ └── 📄 parser_functions.py # Extraction & cleaning functions │ ├── 📁 pipelines/ # Data processing pipelines │ │ ├── 📄 database_pipeline.py # Database persistence │ │ └── 📄 image_pipeline.py # Image download & processing │ ├── 🕷️ spiders/ # Scrapy spiders │ │ └── 📄 kennys_book_spider.py # Main book scraper │ ├── 📁 storage/ # Storage configurations │ ├── 📁 utils/ # General utilities │ │ └── 📄 custom_logger.py # Logging configuration | | __ 📄 mysql_connection_string.py # Special utility for connection to db │ ├── 📄 init.py # Package initializer │ ├── 📄 create_archive.py # Image archiving utility │ ├── 📄 scrapy.cfg # Scrapy configuration │ └── 📄 settings.py # Project settings ├── 📁 tests/ # Test suite ├── 📄 .dist.env # Environment template ├── 📄 .gitignore # Git ignore rules ├── 🐳 Dockerfile # Docker configuration ├── 📄 README.md # Documentation ├── 📄 alembic.ini # Alembic settings ├── 📄 compose.yaml # Docker Compose setup ├── 🔒 poetry.lock # Dependency lock file └── 📄 pyproject.toml # Poetry configuration

🎮 Usage Commands

Database Operations

Migration Management

# Generate new migration
poetry run alembic revision --autogenerate -m "Description"

# Apply migrations
poetry run alembic upgrade head

# Check current version
poetry run alembic current

# Rollback (use with caution)
poetry run alembic downgrade -1

Web Scraping

Run Complete Scraping

# Start full scraping process
poetry run scrapy crawl kennys_book_spider

# Limited scraping (for testing)
poetry run scrapy crawl kennys_book_spider -s CLOSESPIDER_ITEMCOUNT=10

Verification Steps

  1. ✅ Check console for -[DatabasePipeline]: Updating book:- messages
  2. ✅ Review -scrapy_log.txt- for detailed logs
  3. ✅ Verify database entries in -bookstore_db.books- table
  4. ✅ Confirm image downloads in -images/- directory

Image Archiving

# Create ZIP archive of book covers
poetry run python src/create_archive.py

# Output: deliverables/book_covers.zip
---

---

## 📊 Logging System

### 🎨 Log Format
---
YYYY-MM-DD HH:MM:SS [LoggerName]: Message 
*I use HH:MM just for the clearer terminal output*

---

### 📍 Output Destinations
**🖥️ Console**: Real-time feedback during execution
**📄 File**: Persistent logs in `scrapy.log`

### 🔧 Configuration Features

| Feature               | Implementation                                |
|-----------------------|-----------------------------------------------|
| **Custom Logger**     | `src/utils/custom_logger.py`                   |
| **Noise Suppression** | Scrapy internals set to CRITICAL level        |
| **Dual Output**       | Identical logs to console and file            |
| **No Duplicates**     | `propagate=False` prevents log multiplication |
| **Focused Output**    | Only spider and pipeline logs are prominent   |

---

## 🏗️ Architecture Highlights

### 🔍 Data Processing Pipeline

```mermaid
graph LR
    A[kennys.ie] --> B[Spider Crawler]
    B --> C[Data Extraction]
    C --> D[Item Pipeline]
    D --> E[Database Storage]
    D --> F[Image Download]
    E --> G[MySQL Database]
    F --> H[Local Storage]

🗄️ Database Design

Key Features

  • 📅 Date Handling: ISO 8601 format with datetime.date objects
  • 💰 Price Parsing: Currency-aware parsing with decimal precision
  • 📚 Multi-value Fields: JSON storage for authors and categories
  • 🖼️ Image Management: Automated filename sanitization and storage
  • ⏰ Timestamps: Automatic -created_at-/-updated_at- management
  • 🔒 Data Integrity: Proper NULL handling vs empty strings

Model Structure

---python

Example: Book model with key relationships

class Book: id: Primary Key external_id: Unique identifier from source title: Book title authors: JSON array of authors categories: JSON array of categories publication_date: DATE field (ISO 8601) price: DECIMAL with currency info cover_image_filename: Image reference created_at/updated_at: Automatic timestamps

🎯 Key Technical Decisions

🔧 Architecture Choices

Decision Rationale Implementation
Modular Parsing Clean separation of concerns Centralized in src/parsers/
SQLAlchemy ORM Type safety and migrations Core expressions with models
Environment Config Security and flexibility .env file with validation
Custom Logging Clean output control Dual-channel with formatting
Error Handling Graceful degradation Try-except with proper cleanup
Image Pipeline Automated asset management Filename sanitization and archiving
Docker Support Containerized deployment Dockerfile + compose configuration

🛡️ Security & Best Practices

  • 🔐 Credential Management: All sensitive data in environment variables
  • 🚫 SQL Injection Prevention: SQLAlchemy ORM expressions only
  • 📝 Comprehensive Logging: Full audit trail of operations
  • 🔄 Transaction Safety: Database operations with proper rollback
  • ⚡ Performance Optimization: Controlled concurrency with proxy limits

📋 Deliverables Checklist

📊 Final Project Requirements

Documentation

  • ✅ Complete README with setup instructions
  • ✅ Database schema documentation
  • ✅ Code architecture explanation
  • ✅ Requirements compliance verification

Results

  • ✅ Database Export: Complete SQL dump with all scraped data
  • ✅ Image Archive: ZIP file containing all book cover images
  • ✅ Source Code: Clean, documented codebase on master branch
  • ✅ Migration Scripts: All database schema changes tracked

Presentation Format

Choose one:

  • 📝 Text Report: Comprehensive project summary
  • 🎥 Video Demo: ~5 minutes showing setup and execution (recommended)

Presentation Content

  1. 🎯 Project Overview: Requirements and completion status
  2. 🚀 Live Demo: Setup from scratch following README
  3. 🏗️ Architecture Discussion: Key decisions and implementation details
  4. 📊 Results Showcase: Database content and image archives
  5. 🔍 Code Review: Highlight important technical aspects

🆘 Troubleshooting

Common Issues

Database Connection

---bash

Test database connectivity

poetry run python -c "from src.database.utils import mysql_connection_string; print('✅ DB config loaded')"

Proxy Issues

---bash

Verify proxy configuration

poetry run python -c "import os; print(f'Proxy: {os.getenv("PROXY")}')"

Migration Problems

---bash

Reset migrations (⚠️ Data loss warning)

poetry run alembic downgrade base poetry run alembic upgrade head

Log File Issues

  • Ensure write permissions in project directory
  • Check LOG_LEVEL setting in .env
  • Verify scrapy_log.txt is not locked by another process

📞 Support & Resources

About

No description, website, or topics provided.

Resources

Readme
Activity

Stars

1 star

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages