Comparison Project - Backend Documentation

A Flask-based web application for comparing and managing construction project data (excavation and manhole work) between subcontractors and clients. The system imports data from Excel files, stores it in a database, and generates comparison reports.

Table of Contents

• Project Overview
• Tech Stack
• Project Structure
• Database Schema
• Application Flow
• API Routes & Endpoints
• Setup & Installation

---

Project Overview

The Comparison Project is designed to:

• Manage subcontractors and their excavation data (Manhole Excavation, Trench Excavation, Domestic Chambers)
• Import client-side project data for comparison
• Compare subcontractor work against client specifications
• Generate detailed reports highlighting differences between client and subcontractor data
• User authentication with login/registration system

Key Features

✅ User Authentication (Register/Login/Logout)
✅ Subcontractor Management (Add/Edit/List/Delete)
✅ File Import (Excel/CSV) for both Client and Subcontractor data
✅ Data validation and duplicate detection
✅ Comparison Reports with Excel export
✅ Dashboard with file management

---

Tech Stack

Backend Framework: Flask
Database: SQL Database (MySQL/PostgreSQL/SQLite configured via environment variables)
ORM: SQLAlchemy
File Processing: Pandas, OpenPyXL, XlsxWriter
Authentication: Werkzeug (Password hashing)
Configuration: Python Dotenv

Dependencies

Flask
pandas
openpyxl
xlrd
Werkzeug
python-dotenv
cryptography
xlsxwriter
matplotlib 

---

Project Structure

Comparison_Project/
├── run.py                          # Application entry point
├── requirements.txt                # Python dependencies
├── .env                            # Environment variables (DB, secrets)
├── README.md                       # This file
│
├── app/
│   ├── __init__.py                # Flask app initialization
│   ├── config.py                  # Configuration settings
│   │
│   ├── models/                    # Database models (SQLAlchemy ORM)
│   │   ├── user_model.py          # User table schema
│   │   ├── subcontractor_model.py # Subcontractor table schema
│   │   ├── manhole_excavation_model.py        # Subcontractor MH excavation
│   │   ├── trench_excavation_model.py         # Subcontractor Trench excavation
│   │   ├── manhole_domestic_chamber_model.py  # Subcontractor Domestic chamber
│   │   ├── mh_ex_client_model.py   # Client MH excavation
│   │   ├── tr_ex_client_model.py   # Client Trench excavation
│   │   └── mh_dc_client_model.py   # Client Domestic chamber
│   │
│   ├── routes/                    # Flask Blueprints (API endpoints)
│   │   ├── auth.py                # Login, Register, Logout
│   │   ├── dashboard.py           # Dashboard page
│   │   ├── file_import.py         # File upload endpoints
│   │   ├── file_report.py         # Fetch and format data for reports
│   │   ├── generate_comparison_report.py  # Comparison logic & Excel export
│   │   ├── subcontractor_routes.py        # CRUD for subcontractors
│   │   ├── user.py                # User management routes
│   │   └── file_format.py         # File format validation
│   │
│   ├── services/                  # Business logic layer
│   │   ├── db_service.py          # Database connection & SQLAlchemy init
│   │   ├── file_service.py        # File parsing & data insertion
│   │   └── user_service.py        # User authentication logic
│   │
│   ├── utils/                     # Helper functions
│   │   ├── file_utils.py          # File path utilities
│   │   ├── helpers.py             # Decorators, common helpers
│   │   └── __pycache__/
│   │
│   ├── static/                    # Static assets
│   │   ├── css/                   # Stylesheets
│   │   ├── images/                # Images
│   │   ├── uploads/               # Uploaded files directory
│   │   │   ├── Client_Bill_1/
│   │   │   └── sub_1/
│   │   └── downloads/             # Generated reports
│   │
│   ├── templates/                 # Jinja2 HTML templates
│   │   ├── base.html              # Base template
│   │   ├── login.html
│   │   ├── register.html
│   │   ├── dashboard.html
│   │   ├── file_import.html
│   │   ├── file_import_client.html
│   │   ├── file_format.html
│   │   ├── file_report.html
│   │   ├── generate_comparison_report.html
│   │   ├── generate_comparison_client_vs_subcont.html
│   │   ├── list_user.html
│   │   ├── report.html
│   │   ├── users.htm
│   │   └── subcontractor/
│   │       ├── add.html
│   │       ├── edit.html
│   │       └── list.html
│   │
│   └── logs/                      # Application logs
│
├── instance/                      # Instance folder (DB, temp files)
└── logs/                          # Root level logs

---

Database Schema

1. Users Table (users)

Stores user authentication data.

Column	Type	Constraints	Purpose
id	Integer	Primary Key	User identifier
name	String(120)	NOT NULL	User's full name
email	String(120)	UNIQUE, NOT NULL	User's email
password_hash	String(255)	NOT NULL	Hashed password (Werkzeug)

Key Methods:

• set_password(password) - Hash and store password
• check_password(password) - Verify password

---

2. Subcontractors Table (subcontractors)

Stores subcontractor company information.

Column	Type	Constraints	Purpose
id	Integer	Primary Key	Subcontractor ID
subcontractor_name	String(255)	NOT NULL	Company name
address	String(500)	-	Company address
gst_no	String(50)	-	GST number
pan_no	String(50)	-	PAN number
mobile_no	String(20)	-	Contact phone
email_id	String(150)	-	Contact email
contact_person	String(150)	-	Contact person name
status	String(20)	Default: "Active"	Active/Inactive status
created_at	DateTime	Default: today	Record creation date

Relationships:

• One-to-Many with manhole_excavation
• One-to-Many with trench_excavation
• One-to-Many with manhole_domestic_chamber

---

3. Manhole Excavation (Subcontractor) (manhole_excavation)

Stores manhole excavation work data from subcontractors.

Column Type	Purpose
id	Primary Key
subcontractor_id	Foreign Key → Subcontractor
Location	Worksite location
MH_NO	Manhole number
RA_Bill_No	RA Bill reference (for grouping)
Upto_IL_Depth, Cutting_Depth, ID_of_MH_m, Ex_Dia_of_Manhole, Area_of_Manhole	Dimension fields
Excavation Categories (by depth ranges):
Soft_Murum_0_to_1_5, Soft_Murum_1_5_to_3_0, etc.	Material type + depth range
Hard_Murum_0_to_1_5, Hard_Murum_1_5_to_3_0, etc.	Hard mudstone layers
Soft_Rock_0_to_1_5, Soft_Rock_1_5_to_3_0, etc.	Soft rock layers
Hard_Rock_0_to_1_5 through Hard_Rock_6_0_to_7_5	Hard rock layers
Totals	Sum per category
Total	Grand total excavation
Remarks, created_at	Notes and timestamp

---

4. Trench Excavation (Subcontractor) (trench_excavation)

Stores trench/sewer line excavation work data from subcontractors.

Similar structure to Manhole Excavation with additional fields:

• Width categories: Width_0_to_2_5, Width_2_5_to_3_0, etc.
• Avg_Depth, Actual_Trench_Length, Pipe_Dia_mm

---

5. Manhole Domestic Chamber (Subcontractor) (manhole_domestic_chamber)

Stores domestic chamber construction data.

Key Fields	Purpose
id, subcontractor_id	Primary & Foreign Key
Location, MH_NO, RA_Bill_No	Identification
Depth_of_MH, MH_TOP_LEVEL, MH_IL_LEVEL	Dimensions
d_0_to_1_5 through d_6_0_to_6_5	Depth-based excavation categories
Domestic_Chambers	Count of chambers
DWC_Pipe_Length, UPVC_Pipe_Length	Pipe lengths

---

6-8. Client Data Tables (Similar structure to subcontractor models)

• mh_ex_client - Manhole Excavation (Client specifications)
• tr_ex_client - Trench Excavation (Client specifications)
• mh_dc_client - Manhole Domestic Chamber (Client specifications)

Note: Client tables do NOT have subcontractor_id as they represent client baseline data.

---

Application Flow

1. User Authentication Flow

User Access (/) 
  ↓
Redirect to /login
  ↓
[Login Page]
  ├─ POST /auth/login
  │   ↓
  │   UserService.validate_login(email, password)
  │   ↓
  │   Query User table, verify password
  │   ↓
  │   Set session["user_id"], session["user_name"]
  ↓
Redirect to /dashboard

2. Subcontractor File Import Flow

User → /file/import [GET]
  ↓
Display form with subcontractor dropdown
  ↓
User selects subcontractor & uploads Excel file
  ↓
POST /file/import
  ↓
FileService.handle_file_upload()
  ├─ Validate file type (xlsx, xls, csv)
  ├─ Create upload folder: static/uploads/sub_{id}/
  ├─ Save file
  │
  ├─ Read Excel sheets:
  │  ├─ "Tr.Ex." (Trench Excavation)
  │  ├─ "MH Ex." (Manhole Excavation)
  │  └─ "MH & DC" (Manhole & Domestic Chamber)
  │
  ├─ Process each sheet:
  │  ├─ Normalize column names
  │  ├─ Clean data (handle NaN, empty values)
  │  ├─ Check for duplicates by (Location, MH_NO, RA_Bill_No, subcontractor_id)
  │  ├─ Insert records into respective tables
  │  └─ Commit to database
  │
  └─ Return success/error message

3. Client File Import Flow

User → /file/import_client [GET]
  ↓
Display form for client file upload
  ↓
User uploads Excel file with client specifications
  ↓
POST /file/import_client
  ↓
FileService.handle_client_file_upload()
  ├─ Validate & save file
  ├─ Read sheets: "Tr.Ex.", "MH Ex.", "MH & DC"
  ├─ Process and insert into:
  │  ├─ mh_ex_client
  │  ├─ tr_ex_client
  │  └─ mh_dc_client
  └─ Return status

4. Comparison Report Generation Flow

User → /report/generate_comparison [GET]
  ↓
Display form to select RA Bill No.
  ↓
User selects bill number
  ↓
POST /report/generate_comparison
  ↓
generate_report_bp routes request
  ├─ Fetch client data (mh_ex_client, tr_ex_client, mh_dc_client)
  ├─ Fetch subcontractor data (by RA_Bill_No)
  ├─ For each record type:
  │  ├─ Create lookup table by (Location, MH_NO)
  │  ├─ Match client records with subcontractor records
  │  ├─ Calculate totals for each category
  │  ├─ Compute differences/variances
  │  └─ Format for Excel output
  │
  ├─ Generate Excel file with comparison results:
  │  ├─ Summary sheet
  │  ├─ Detailed Trench Excavation comparison
  │  ├─ Detailed Manhole Excavation comparison
  │  └─ Detailed Domestic Chamber comparison
  │
  └─ Send file to client as download

5. Dashboard & Data Retrieval Flow

User → /dashboard [GET]
  ↓
Check session["user_id"] (authentication)
  ↓
Render dashboard.html
  ├─ Display available reports
  ├─ List recent RA Bills
  ├─ Show subcontractor list
  └─ Provide navigation to:
     ├─ File import
     ├─ Reports
     ├─ Subcontractor management
     └─ Logout

---

API Routes & Endpoints

Authentication Routes (/auth)

Method	Endpoint	Purpose	Auth
GET/POST	/login	Login form & validation	❌
GET/POST	/register	User registration	❌
GET	/logout	Clear session & logout	✅

Dashboard Route (/dashboard)

Method	Endpoint	Purpose	Auth
GET	/dashboard/	Main dashboard	✅

Subcontractor Routes (/subcontractor)

Method	Endpoint	Purpose	Auth
GET	/subcontractor/add	Add form	✅
POST	/subcontractor/save	Save new subcontractor	✅
GET	/subcontractor/list	List all subcontractors	✅
GET	/subcontractor/edit/<id>	Edit form	✅
POST	/subcontractor/update/<id>	Update subcontractor	✅
GET	/subcontractor/delete/<id>	Delete subcontractor	✅

File Import Routes (/file)

Method	Endpoint	Purpose	Auth
GET/POST	/file/import	Subcontractor file upload	✅
GET/POST	/file/import_client	Client file upload	✅
GET/POST	/file/report	View imported data report	✅

Report Routes (/report)

Method	Endpoint	Purpose	Auth
GET/POST	/report/generate_comparison	Generate comparison report	✅
GET/POST	/report/generate_comparison_client_vs_subcont	Alternative comparison	✅

User Routes (/user)

Method	Endpoint	Purpose	Auth
GET	/user/list	List all users	✅
Other	(Check routes)	User management	✅

File Format Route (/format)

Method	Endpoint	Purpose	Auth
GET/POST	/format/...	File format reference	✅

---

Setup & Installation

Prerequisites

• Python 3.7+
• MySQL/PostgreSQL (or SQLite for development)
• pip (Python package manager)

1. Clone & Install Dependencies

cd d:\New folder\Comparison\Comparison_Project
pip install -r requirements.txt

2. Configure Environment Variables

Create a .env file in the project root:

# -----------------------------
# Flask App Configuration
# -----------------------------
FLASK_ENV=development
FLASK_DEBUG=True
FLASK_HOST='0.0.0.0'
FLASK_PORT=5001

# -----------------------------
# Security
# -----------------------------
SECRET_KEY=change-this-to-strong-secret-key

# -----------------------------
# Database Configuration
# -----------------------------
DB_DIALECT=mysql
DB_DRIVER=pymysql
DB_HOST=127.0.0.1
DB_PORT=3306
DB_NAME=comparisondb
DB_USER=root
DB_PASSWORD=your_password

3. Initialize Database

python run.py

This will:

• Load environment variables
• Create Flask app with configuration
• Initialize SQLAlchemy
• Create all tables (if not exist)
• Start Flask development server

4. Access Application

Open browser: http://127.0.0.1:5000/

• First time: Redirect to /login
• Register a new account
• Login and start using the application

---

Key Services & Utilities

FileService (app/services/file_service.py)

Purpose: Handles file uploads, parsing, and data validation

Key Methods:

• handle_file_upload(file, subcontractor_id, RA_Bill_No) - Upload & process subcontractor file
• handle_client_file_upload(file, RA_Bill_No) - Upload & process client file
• process_trench_excavation(df, subcontractor_id, RA_Bill_No) - Parse & insert trench data
• process_manhole_excavation(df, subcontractor_id, RA_Bill_No) - Parse & insert manhole data
• process_manhole_domestic_chamber(df, subcontractor_id, RA_Bill_No) - Parse & insert chamber data

UserService (app/services/user_service.py)

Purpose: User authentication & management

Key Methods:

• register_user(name, email, password) - Register new user
• validate_login(email, password) - Authenticate user
• get_all_users() - Fetch all users

DBService (app/services/db_service.py)

Purpose: Database connection & SQLAlchemy initialization

Exports:

• db - SQLAlchemy instance for ORM operations
• migrate - Flask-Migrate for schema migrations

---

Authentication & Security

Session Management

• Uses Flask session object
• Stores user_id and user_name after successful login
• @login_required decorator validates authenticated requests

Password Security

• Passwords hashed using Werkzeug's generate_password_hash()
• Verification via check_password_hash()
• No plaintext passwords stored in database

File Security

• File uploads sanitized with secure_filename()
• Only allowed extensions: .xlsx, .xls, .csv
• Files stored in user-specific subfolders

---

Error Handling & Validation

File Import Validation

1. File Type Check - Only CSV/XLS/XLSX allowed
2. Sheet Validation - Required sheets must exist
3. Column Normalization - Auto-fixes column name inconsistencies
4. Data Type Conversion - Converts to appropriate types
5. Duplicate Detection - Prevents duplicate records by (Location, MH_NO, RA_Bill_No)
6. Null Handling - Converts empty/NaN values to None

Database Constraints

• Foreign key relationships enforced
• Unique email constraint on users
• NOT NULL constraints on critical fields

---

Example: Typical User Workflow

1. User registers/logs in
   → POST /auth/register (new user) or /auth/login
   
2. User adds subcontractors
   → GET /subcontractor/add
   → POST /subcontractor/save
   
3. User uploads subcontractor data (Excel file with 3 sheets)
   → GET /file/import
   → POST /file/import (file, subcontractor_id, RA_Bill_No)
   → Database populated with excavation data
   
4. User uploads client specifications (Excel file with same 3 sheets)
   → GET /file/import_client
   → POST /file/import_client (file, RA_Bill_No)
   → Database populated with client data
   
5. User generates comparison report
   → GET /report/generate_comparison
   → POST /report/generate_comparison (RA_Bill_No)
   → System compares client vs subcontractor data
   → Generates Excel file showing differences
   → User downloads report
   
6. User logs out
   → GET /auth/logout
   → Session cleared, redirected to login

---

Future Enhancements

• Email notifications for report generation
• Data visualization dashboards
• Role-based access control (Admin, User, Viewer)
• Bulk report generation
• Data export to PDF format
• Audit logs for data modifications
• API documentation (Swagger/OpenAPI)
• Unit & integration tests

---

Support & Contribution

For issues, feature requests, or contributions, please contact the development team.

Last Updated: January 2026