🎯 JobRadar

JobRadar is an automated job scraping and aggregation system built with .NET 10. It scrapes job listings from multiple sources (LinkedIn, HelloJob, Glorri), stores them in PostgreSQL, and exports results to CSV, Excel, or JSON formats.

---

📋 Features

• Multi-Source Scraping: Scrapes job listings from LinkedIn, HelloJob, and Glorri
• Smart Deduplication: Uses SHA-256 URL hashing to prevent duplicate job entries
• Scheduled Execution: Quartz.NET-based background jobs with configurable intervals
• Multiple Export Formats: Export data to CSV, Excel (XLSX), or JSON
• RESTful API: Query and filter job listings via minimal API endpoints
• Clean Architecture: Organized into SharedKernel, Features, Infrastructure, and API layers
• Fully Tested: Includes unit and integration tests using xUnit
• Structured Logging: Serilog integration for production-ready logging

---

🏗️ Architecture

JobRadar/
├── JobRadar.SharedKernel/       # Domain entities, enums, interfaces
├── JobRadar.Features/           # Scraping & export strategies (business logic)
├── JobRadar.Infrastructure/     # EF Core, repositories, PostgreSQL integration
├── JobRadar.API/               # Minimal API endpoints
├── JobRadar.Worker/            # Quartz background jobs
├── JobRadar.UnitTests/         # Unit tests
└── JobRadar.IntegrationTests/  # Integration tests

Design Patterns

• Strategy Pattern: For scrapers and exporters
• Page Object Model: For Playwright scraping logic
• Repository Pattern: For data access abstraction
• Options Pattern: For configuration management
• Result Pattern: Using ErrorOr for functional error handling

---

🚀 Getting Started

Prerequisites

• .NET 10 SDK
• PostgreSQL 14+
• Docker (optional, for containerized setup)

Installation

1. Clone the repository:

   git clone https://github.com/NigarOsmanova/JobRadar.git
   cd JobRadar

2. Install Playwright browsers:

   pwsh bin/Debug/net10.0/playwright.ps1 install chromium

3. Configure settings:

   • Copy appsettings.example.json to appsettings.json in both JobRadar.API and JobRadar.Worker
   • Update the PostgreSQL connection string
   • Configure scraper keywords, locations, and enabled sources

4. Apply database migrations:

   cd JobRadar.Infrastructure
   dotnet ef database update --startup-project ../JobRadar.Worker

5. Run with Docker Compose (recommended):

   docker-compose up -d

   Or run locally:

   # Terminal 1: Run the Worker
   cd JobRadar.Worker
   dotnet run
   
   # Terminal 2: Run the API
   cd JobRadar.API
   dotnet run

---

⚙️ Configuration

appsettings.json (Worker & API)

{
  "ConnectionStrings": {
	"Postgres": "Host=localhost;Database=JobRadar;Username=postgres;Password=yourpassword"
  },
  "ScraperSettings": {
	"SearchKeyword": ".NET Developer",
	"Location": "Baku",
	"IntervalMinutes": 120,
	"MaxJobsPerRun": 50,
	"EnabledSources": ["LinkedIn", "Hellojob", "Glorri"]
  },
  "ExportSettings": {
	"OutputPath": "C:\\JobRadarExports",
	"Formats": ["Csv", "Excel", "Json"]
  }
}

---

📡 API Endpoints

Base URL: https://localhost:5001 (or configured port)

Method	Endpoint	Description
GET	/jobs	Get paginated job listings
GET	/jobs/new	Get only new (unread) jobs
GET	/jobs/{id}	Get job by ID
GET	/jobs/source/{source}	Get jobs from specific source

Example Request

curl -X GET "https://localhost:5001/jobs?page=1&pageSize=20"

Response

{
  "items": [
	{
	  "id": "3fa85f64-5717-4562-b3fc-2c963f66afa6",
	  "title": ".NET Developer",
	  "company": "Acme Corp",
	  "location": "Baku, Azerbaijan",
	  "url": "https://linkedin.com/jobs/123",
	  "source": "LinkedIn",
	  "scrapedAt": "2025-01-15T10:30:00Z",
	  "postedAt": "2025-01-14T08:00:00Z",
	  "isNew": true
	}
  ],
  "page": 1,
  "pageSize": 20,
  "totalCount": 150,
  "totalPages": 8,
  "hasNextPage": true,
  "hasPreviousPage": false
}

---

🔧 Technologies Used

• .NET 10: Modern C# with minimal APIs
• PostgreSQL + EF Core 10: Relational data storage
• Playwright: Headless browser automation for scraping
• Quartz.NET: Job scheduling and background tasks
• Serilog: Structured logging
• ClosedXML: Excel file generation
• CsvHelper: CSV export
• xUnit: Unit and integration testing
• ErrorOr: Functional error handling

---

🧪 Running Tests

# Run all tests
dotnet test

# Run only unit tests
dotnet test --filter "FullyQualifiedName~UnitTests"

# Run only integration tests
dotnet test --filter "FullyQualifiedName~IntegrationTests"

---

📊 Database Schema

CREATE TABLE JobListings (
	Id UUID PRIMARY KEY,
	Title VARCHAR(300) NOT NULL,
	Company VARCHAR(200) NOT NULL,
	Location VARCHAR(500) NOT NULL,
	Url VARCHAR(1000) NOT NULL,
	UrlHash VARCHAR(64) NOT NULL UNIQUE,  -- SHA-256 hash for deduplication
	Source INT NOT NULL,
	ScrapedAt TIMESTAMP NOT NULL,
	PostedAt TIMESTAMP NULL,
	Description TEXT NULL,
	IsNew BOOLEAN NOT NULL
);

CREATE UNIQUE INDEX IX_JobListings_UrlHash ON JobListings(UrlHash);

---

🛠️ Development

Adding a New Scraper

1. Create a new folder in JobRadar.Features/Scraping/{SourceName}/
2. Implement IJobScraperStrategy by extending BaseJobScraper
3. Create Page Objects extending JobPage and JobCard
4. Register in DependencyInjection.cs
5. Add the source name to appsettings.json → EnabledSources

Adding a New Export Format

1. Create a new class in JobRadar.Features/Export/Strategies/
2. Implement IJobExportStrategy
3. Register in ExportExtensions.cs
4. Add the format to appsettings.json → Formats

---

🐳 Docker Support

The project includes a docker-compose.yml for easy setup:

docker-compose up -d

This starts:

• PostgreSQL container
• pgAdmin (optional, for database management)

---

📝 License

This project is open-source and available under the MIT License.

---

👤 Author

Nigar Osmanova

• GitHub: @NigarOsmanova
• LinkedIn: Nigar Osmanova

---

🙏 Acknowledgments

• Playwright for robust browser automation
• Quartz.NET for reliable job scheduling
• ErrorOr for functional error handling patterns

---

🔮 Future Enhancements

• Web dashboard UI — VakansiyaRadar (Next.js + TypeScript)
• Add more job sources (Indeed, Glassdoor, etc.)
• Email notifications for new jobs
• Dockerize the entire application
• Add CI/CD pipeline (GitHub Actions)
• Implement rate limiting for API
• Add filtering by keywords/salary in API

Disclaimer

This project is intended for educational and portfolio purposes only.

It demonstrates how to build an automated job aggregation system using .NET, Playwright, PostgreSQL, and scheduled background processing. Users are responsible for ensuring that their use of this project complies with the Terms of Service, robots.txt policies, and applicable laws of any websites they choose to scrape.

The author does not encourage or endorse unauthorized scraping, excessive request volumes, bypassing access controls, or collecting data in violation of third-party website policies.