DEMOCICD/README.md

# Comparison Project - Backend Documentation

A Flask-based web application for comparing and managing construction project data (excavation and manhole work) between subcontractors and clients. The system imports data from Excel files, stores it in a database, and generates comparison reports.

## Table of Contents

- [Project Overview](#project-overview)
- [Tech Stack](#tech-stack)
- [Project Structure](#project-structure)
- [Database Schema](#database-schema)
- [Application Flow](#application-flow)
- [API Routes & Endpoints](#api-routes--endpoints)
- [Setup & Installation](#setup--installation)

---

## Project Overview

The Comparison Project is designed to:
- **Manage subcontractors** and their excavation data (Manhole Excavation, Trench Excavation, Domestic Chambers)
- **Import client-side project data** for comparison
- **Compare subcontractor work** against client specifications
- **Generate detailed reports** highlighting differences between client and subcontractor data
- **User authentication** with login/registration system

### Key Features

✅ User Authentication (Register/Login/Logout)  
✅ Subcontractor Management (Add/Edit/List/Delete)  
✅ File Import (Excel/CSV) for both Client and Subcontractor data  
✅ Data validation and duplicate detection  
✅ Comparison Reports with Excel export  
✅ Dashboard with file management  

---

## Tech Stack

**Backend Framework**: Flask  
**Database**: SQL Database (MySQL/PostgreSQL/SQLite configured via environment variables)  
**ORM**: SQLAlchemy  
**File Processing**: Pandas, OpenPyXL, XlsxWriter  
**Authentication**: Werkzeug (Password hashing)  
**Configuration**: Python Dotenv  

### Dependencies

```
Flask
pandas
openpyxl
xlrd
Werkzeug
python-dotenv
cryptography
xlsxwriter
matplotlib 

```

---

## Project Structure

```
Comparison_Project/
├── run.py                          # Application entry point
├── requirements.txt                # Python dependencies
├── .env                            # Environment variables (DB, secrets)
├── README.md                       # This file
│
├── app/
│   ├── __init__.py                # Flask app initialization
│   ├── config.py                  # Configuration settings
│   │
│   ├── models/                    # Database models (SQLAlchemy ORM)
│   │   ├── user_model.py          # User table schema
│   │   ├── subcontractor_model.py # Subcontractor table schema
│   │   ├── manhole_excavation_model.py        # Subcontractor MH excavation
│   │   ├── trench_excavation_model.py         # Subcontractor Trench excavation
│   │   ├── manhole_domestic_chamber_model.py  # Subcontractor Domestic chamber
│   │   ├── mh_ex_client_model.py   # Client MH excavation
│   │   ├── tr_ex_client_model.py   # Client Trench excavation
│   │   └── mh_dc_client_model.py   # Client Domestic chamber
│   │
│   ├── routes/                    # Flask Blueprints (API endpoints)
│   │   ├── auth.py                # Login, Register, Logout
│   │   ├── dashboard.py           # Dashboard page
│   │   ├── file_import.py         # File upload endpoints
│   │   ├── file_report.py         # Fetch and format data for reports
│   │   ├── generate_comparison_report.py  # Comparison logic & Excel export
│   │   ├── subcontractor_routes.py        # CRUD for subcontractors
│   │   ├── user.py                # User management routes
│   │   └── file_format.py         # File format validation
│   │
│   ├── services/                  # Business logic layer
│   │   ├── db_service.py          # Database connection & SQLAlchemy init
│   │   ├── file_service.py        # File parsing & data insertion
│   │   └── user_service.py        # User authentication logic
│   │
│   ├── utils/                     # Helper functions
│   │   ├── file_utils.py          # File path utilities
│   │   ├── helpers.py             # Decorators, common helpers
│   │   └── __pycache__/
│   │
│   ├── static/                    # Static assets
│   │   ├── css/                   # Stylesheets
│   │   ├── images/                # Images
│   │   ├── uploads/               # Uploaded files directory
│   │   │   ├── Client_Bill_1/
│   │   │   └── sub_1/
│   │   └── downloads/             # Generated reports
│   │
│   ├── templates/                 # Jinja2 HTML templates
│   │   ├── base.html              # Base template
│   │   ├── login.html
│   │   ├── register.html
│   │   ├── dashboard.html
│   │   ├── file_import.html
│   │   ├── file_import_client.html
│   │   ├── file_format.html
│   │   ├── file_report.html
│   │   ├── generate_comparison_report.html
│   │   ├── generate_comparison_client_vs_subcont.html
│   │   ├── list_user.html
│   │   ├── report.html
│   │   ├── users.htm
│   │   └── subcontractor/
│   │       ├── add.html
│   │       ├── edit.html
│   │       └── list.html
│   │
│   └── logs/                      # Application logs
│
├── instance/                      # Instance folder (DB, temp files)
└── logs/                          # Root level logs
```

---

## Database Schema

### 1. **Users Table** (`users`)
Stores user authentication data.

| Column | Type | Constraints | Purpose |
|--------|------|-------------|---------|
| `id` | Integer | Primary Key | User identifier |
| `name` | String(120) | NOT NULL | User's full name |
| `email` | String(120) | UNIQUE, NOT NULL | User's email |
| `password_hash` | String(255) | NOT NULL | Hashed password (Werkzeug) |

**Key Methods:**
- `set_password(password)` - Hash and store password
- `check_password(password)` - Verify password

---

### 2. **Subcontractors Table** (`subcontractors`)
Stores subcontractor company information.

| Column | Type | Constraints | Purpose |
|--------|------|-------------|---------|
| `id` | Integer | Primary Key | Subcontractor ID |
| `subcontractor_name` | String(255) | NOT NULL | Company name |
| `address` | String(500) | - | Company address |
| `gst_no` | String(50) | - | GST number |
| `pan_no` | String(50) | - | PAN number |
| `mobile_no` | String(20) | - | Contact phone |
| `email_id` | String(150) | - | Contact email |
| `contact_person` | String(150) | - | Contact person name |
| `status` | String(20) | Default: "Active" | Active/Inactive status |
| `created_at` | DateTime | Default: today | Record creation date |

**Relationships:**
- One-to-Many with `manhole_excavation`
- One-to-Many with `trench_excavation`
- One-to-Many with `manhole_domestic_chamber`

---

### 3. **Manhole Excavation (Subcontractor)** (`manhole_excavation`)
Stores manhole excavation work data from subcontractors.

| Column Type | Purpose |
|-----|---------|
| `id` | Primary Key |
| `subcontractor_id` | Foreign Key → Subcontractor |
| `Location` | Worksite location |
| `MH_NO` | Manhole number |
| `RA_Bill_No` | RA Bill reference (for grouping) |
| `Upto_IL_Depth`, `Cutting_Depth`, `ID_of_MH_m`, `Ex_Dia_of_Manhole`, `Area_of_Manhole` | Dimension fields |
| **Excavation Categories** (by depth ranges): | |
| Soft_Murum_0_to_1_5, Soft_Murum_1_5_to_3_0, etc. | Material type + depth range |
| Hard_Murum_0_to_1_5, Hard_Murum_1_5_to_3_0, etc. | Hard mudstone layers |
| Soft_Rock_0_to_1_5, Soft_Rock_1_5_to_3_0, etc. | Soft rock layers |
| Hard_Rock_0_to_1_5 through Hard_Rock_6_0_to_7_5 | Hard rock layers |
| **Totals** | Sum per category |
| `Total` | Grand total excavation |
| `Remarks`, `created_at` | Notes and timestamp |

---

### 4. **Trench Excavation (Subcontractor)** (`trench_excavation`)
Stores trench/sewer line excavation work data from subcontractors.

**Similar structure to Manhole Excavation with additional fields:**
- Width categories: `Width_0_to_2_5`, `Width_2_5_to_3_0`, etc.
- `Avg_Depth`, `Actual_Trench_Length`, `Pipe_Dia_mm`

---

### 5. **Manhole Domestic Chamber (Subcontractor)** (`manhole_domestic_chamber`)
Stores domestic chamber construction data.

| Key Fields | Purpose |
|-----------|---------|
| `id`, `subcontractor_id` | Primary & Foreign Key |
| `Location`, `MH_NO`, `RA_Bill_No` | Identification |
| `Depth_of_MH`, `MH_TOP_LEVEL`, `MH_IL_LEVEL` | Dimensions |
| `d_0_to_1_5` through `d_6_0_to_6_5` | Depth-based excavation categories |
| `Domestic_Chambers` | Count of chambers |
| `DWC_Pipe_Length`, `UPVC_Pipe_Length` | Pipe lengths |

---

### 6-8. **Client Data Tables** (Similar structure to subcontractor models)

- **`mh_ex_client`** - Manhole Excavation (Client specifications)
- **`tr_ex_client`** - Trench Excavation (Client specifications)
- **`mh_dc_client`** - Manhole Domestic Chamber (Client specifications)

**Note:** Client tables do NOT have `subcontractor_id` as they represent client baseline data.

---

## Application Flow

### 1. **User Authentication Flow**
```
User Access (/) 
  ↓
Redirect to /login
  ↓
[Login Page]
  ├─ POST /auth/login
  │   ↓
  │   UserService.validate_login(email, password)
  │   ↓
  │   Query User table, verify password
  │   ↓
  │   Set session["user_id"], session["user_name"]
  ↓
Redirect to /dashboard
```

### 2. **Subcontractor File Import Flow**
```
User → /file/import [GET]
  ↓
Display form with subcontractor dropdown
  ↓
User selects subcontractor & uploads Excel file
  ↓
POST /file/import
  ↓
FileService.handle_file_upload()
  ├─ Validate file type (xlsx, xls, csv)
  ├─ Create upload folder: static/uploads/sub_{id}/
  ├─ Save file
  │
  ├─ Read Excel sheets:
  │  ├─ "Tr.Ex." (Trench Excavation)
  │  ├─ "MH Ex." (Manhole Excavation)
  │  └─ "MH & DC" (Manhole & Domestic Chamber)
  │
  ├─ Process each sheet:
  │  ├─ Normalize column names
  │  ├─ Clean data (handle NaN, empty values)
  │  ├─ Check for duplicates by (Location, MH_NO, RA_Bill_No, subcontractor_id)
  │  ├─ Insert records into respective tables
  │  └─ Commit to database
  │
  └─ Return success/error message
```

### 3. **Client File Import Flow**
```
User → /file/import_client [GET]
  ↓
Display form for client file upload
  ↓
User uploads Excel file with client specifications
  ↓
POST /file/import_client
  ↓
FileService.handle_client_file_upload()
  ├─ Validate & save file
  ├─ Read sheets: "Tr.Ex.", "MH Ex.", "MH & DC"
  ├─ Process and insert into:
  │  ├─ mh_ex_client
  │  ├─ tr_ex_client
  │  └─ mh_dc_client
  └─ Return status
```

### 4. **Comparison Report Generation Flow**
```
User → /report/generate_comparison [GET]
  ↓
Display form to select RA Bill No.
  ↓
User selects bill number
  ↓
POST /report/generate_comparison
  ↓
generate_report_bp routes request
  ├─ Fetch client data (mh_ex_client, tr_ex_client, mh_dc_client)
  ├─ Fetch subcontractor data (by RA_Bill_No)
  ├─ For each record type:
  │  ├─ Create lookup table by (Location, MH_NO)
  │  ├─ Match client records with subcontractor records
  │  ├─ Calculate totals for each category
  │  ├─ Compute differences/variances
  │  └─ Format for Excel output
  │
  ├─ Generate Excel file with comparison results:
  │  ├─ Summary sheet
  │  ├─ Detailed Trench Excavation comparison
  │  ├─ Detailed Manhole Excavation comparison
  │  └─ Detailed Domestic Chamber comparison
  │
  └─ Send file to client as download
```

### 5. **Dashboard & Data Retrieval Flow**
```
User → /dashboard [GET]
  ↓
Check session["user_id"] (authentication)
  ↓
Render dashboard.html
  ├─ Display available reports
  ├─ List recent RA Bills
  ├─ Show subcontractor list
  └─ Provide navigation to:
     ├─ File import
     ├─ Reports
     ├─ Subcontractor management
     └─ Logout
```

---

## API Routes & Endpoints

### Authentication Routes (`/auth`)
| Method | Endpoint | Purpose | Auth |
|--------|----------|---------|------|
| GET/POST | `/login` | Login form & validation | ❌ |
| GET/POST | `/register` | User registration | ❌ |
| GET | `/logout` | Clear session & logout | ✅ |

### Dashboard Route (`/dashboard`)
| Method | Endpoint | Purpose | Auth |
|--------|----------|---------|------|
| GET | `/dashboard/` | Main dashboard | ✅ |

### Subcontractor Routes (`/subcontractor`)
| Method | Endpoint | Purpose | Auth |
|--------|----------|---------|------|
| GET | `/subcontractor/add` | Add form | ✅ |
| POST | `/subcontractor/save` | Save new subcontractor | ✅ |
| GET | `/subcontractor/list` | List all subcontractors | ✅ |
| GET | `/subcontractor/edit/<id>` | Edit form | ✅ |
| POST | `/subcontractor/update/<id>` | Update subcontractor | ✅ |
| GET | `/subcontractor/delete/<id>` | Delete subcontractor | ✅ |

### File Import Routes (`/file`)
| Method | Endpoint | Purpose | Auth |
|--------|----------|---------|------|
| GET/POST | `/file/import` | Subcontractor file upload | ✅ |
| GET/POST | `/file/import_client` | Client file upload | ✅ |
| GET/POST | `/file/report` | View imported data report | ✅ |

### Report Routes (`/report`)
| Method | Endpoint | Purpose | Auth |
|--------|----------|---------|------|
| GET/POST | `/report/generate_comparison` | Generate comparison report | ✅ |
| GET/POST | `/report/generate_comparison_client_vs_subcont` | Alternative comparison | ✅ |

### User Routes (`/user`)
| Method | Endpoint | Purpose | Auth |
|--------|----------|---------|------|
| GET | `/user/list` | List all users | ✅ |
| Other | (Check routes) | User management | ✅ |

### File Format Route (`/format`)
| Method | Endpoint | Purpose | Auth |
|--------|----------|---------|------|
| GET/POST | `/format/...` | File format reference | ✅ |

---

## Setup & Installation

### Prerequisites
- Python 3.7+
- MySQL/PostgreSQL (or SQLite for development)
- pip (Python package manager)

### 1. Clone & Install Dependencies
```bash
cd d:\New folder\Comparison\Comparison_Project
pip install -r requirements.txt
```

### 2. Configure Environment Variables
Create a `.env` file in the project root:
```env
# -----------------------------
# Flask App Configuration
# -----------------------------
FLASK_ENV=development
FLASK_DEBUG=True
FLASK_HOST='0.0.0.0'
FLASK_PORT=5001

# -----------------------------
# Security
# -----------------------------
SECRET_KEY=change-this-to-strong-secret-key

# -----------------------------
# Database Configuration
# -----------------------------
DB_DIALECT=mysql
DB_DRIVER=pymysql
DB_HOST=127.0.0.1
DB_PORT=3306
DB_NAME=comparisondb
DB_USER=root
DB_PASSWORD=your_password


```

### 3. Initialize Database
```bash
python run.py
```
This will:
- Load environment variables
- Create Flask app with configuration
- Initialize SQLAlchemy
- Create all tables (if not exist)
- Start Flask development server

### 4. Access Application
Open browser: `http://127.0.0.1:5000/`
- First time: Redirect to `/login`
- Register a new account
- Login and start using the application

---

## Key Services & Utilities

### FileService (`app/services/file_service.py`)
**Purpose:** Handles file uploads, parsing, and data validation

**Key Methods:**
- `handle_file_upload(file, subcontractor_id, RA_Bill_No)` - Upload & process subcontractor file
- `handle_client_file_upload(file, RA_Bill_No)` - Upload & process client file
- `process_trench_excavation(df, subcontractor_id, RA_Bill_No)` - Parse & insert trench data
- `process_manhole_excavation(df, subcontractor_id, RA_Bill_No)` - Parse & insert manhole data
- `process_manhole_domestic_chamber(df, subcontractor_id, RA_Bill_No)` - Parse & insert chamber data

### UserService (`app/services/user_service.py`)
**Purpose:** User authentication & management

**Key Methods:**
- `register_user(name, email, password)` - Register new user
- `validate_login(email, password)` - Authenticate user
- `get_all_users()` - Fetch all users

### DBService (`app/services/db_service.py`)
**Purpose:** Database connection & SQLAlchemy initialization

**Exports:**
- `db` - SQLAlchemy instance for ORM operations
- `migrate` - Flask-Migrate for schema migrations

---

## Authentication & Security

### Session Management
- Uses Flask `session` object
- Stores `user_id` and `user_name` after successful login
- `@login_required` decorator validates authenticated requests

### Password Security
- Passwords hashed using Werkzeug's `generate_password_hash()`
- Verification via `check_password_hash()`
- No plaintext passwords stored in database

### File Security
- File uploads sanitized with `secure_filename()`
- Only allowed extensions: `.xlsx`, `.xls`, `.csv`
- Files stored in user-specific subfolders

---

## Error Handling & Validation

### File Import Validation
1. **File Type Check** - Only CSV/XLS/XLSX allowed
2. **Sheet Validation** - Required sheets must exist
3. **Column Normalization** - Auto-fixes column name inconsistencies
4. **Data Type Conversion** - Converts to appropriate types
5. **Duplicate Detection** - Prevents duplicate records by (Location, MH_NO, RA_Bill_No)
6. **Null Handling** - Converts empty/NaN values to None

### Database Constraints
- Foreign key relationships enforced
- Unique email constraint on users
- NOT NULL constraints on critical fields

---

## Example: Typical User Workflow

```
1. User registers/logs in
   → POST /auth/register (new user) or /auth/login
   
2. User adds subcontractors
   → GET /subcontractor/add
   → POST /subcontractor/save
   
3. User uploads subcontractor data (Excel file with 3 sheets)
   → GET /file/import
   → POST /file/import (file, subcontractor_id, RA_Bill_No)
   → Database populated with excavation data
   
4. User uploads client specifications (Excel file with same 3 sheets)
   → GET /file/import_client
   → POST /file/import_client (file, RA_Bill_No)
   → Database populated with client data
   
5. User generates comparison report
   → GET /report/generate_comparison
   → POST /report/generate_comparison (RA_Bill_No)
   → System compares client vs subcontractor data
   → Generates Excel file showing differences
   → User downloads report
   
6. User logs out
   → GET /auth/logout
   → Session cleared, redirected to login
```

---

## Future Enhancements

- [ ] Email notifications for report generation
- [ ] Data visualization dashboards
- [ ] Role-based access control (Admin, User, Viewer)
- [ ] Bulk report generation
- [ ] Data export to PDF format
- [ ] Audit logs for data modifications
- [ ] API documentation (Swagger/OpenAPI)
- [ ] Unit & integration tests

---

## Support & Contribution

For issues, feature requests, or contributions, please contact the development team.

**Last Updated:** January 2026