Design and Implementation of an Email Spam and Phishing Detection System

Design and Implementation of an Email Spam and Phishing Detection System
Abstract
Email remains the primary vector for spam and phishing attacks, with over 90% of data breaches originating from phishing campaigns and spam accounting for more than 45% of global email traffic . Traditional rule-based filters struggle against sophisticated obfuscation, misleading language, and spoofed sender identities . This project presents the design and implementation of an integrated email spam and phishing detection system that combines deep learning, machine learning, and rule-based heuristics in a multi-layered detection architecture. The system evaluates transformer models (BERT, DistilBERT, TinyBERT) alongside traditional classifiers (Logistic Regression, SVM, Naive Bayes) on email text and subject analysis. Experimental results demonstrate that transformer-based models achieve superior detection performance, with hybrid architectures integrating BERT embeddings and BiLSTM sequences reaching F1-scores of 0.97 . A lightweight TinyBERT model with undersampling achieves an optimal balance between accuracy and computational efficiency for resource-constrained deployment . The system implements a Flask-based API with Chrome extension integration for Gmail real-time detection and a management dashboard for monitoring and threshold adjustment. Rule-based heuristics including safe-sender whitelisting and keyword detection complement machine learning models to reduce false positives. Validation on a dataset of 108,685 samples from merged open-source repositories demonstrates strong generalization capability .
Keywords: Phishing Detection; Spam Filtering; Transformer Models; BERT; Hybrid Deep Learning; Email Security
Chapter 1: Introduction
1.1 Research Background
Email serves as the predominant digital communication medium, with more than 4.6 billion users worldwide sending and receiving approximately 360 billion emails daily in 2025 . However, the ubiquity that makes email an essential communication tool also renders it an ideal attack vector. Three of the most significant email-based threats—spam, phishing, and business email compromise (BEC)—drain productivity and pose substantial financial and security risks. Recent security studies indicate that over 90% of all data breaches are caused by phishing attacks, while spam still accounts for more than 45% of total email traffic worldwide .
Traditional email security measures including keyword blocklists, sender blacklists, and heuristic rules have proven inadequate against modern threats. Attackers employ sophisticated obfuscation mechanisms, misleading language, and spoofed sending identities to circumvent rule-based and statistical spam filters . As noted in recent research, “phishing attacks represent an evolving threat to modern organizations, often bypassing traditional rule-based email security systems” . Corporate-targeted phishing leverages personalized details and social engineering tactics to achieve significantly higher success rates than generic campaigns .
The evolution of phishing attacks has accelerated with the adoption of generative AI tools, enabling attackers to create grammatically correct, highly contextualized content that evades keyword-based detection. This necessitates a shift toward semantic understanding approaches capable of capturing the intent and context of email communications rather than surface-level patterns.
1.2 Problem Statement
Current email spam and phishing detection systems face several critical challenges:
Dataset Heterogeneity and Label Quality: Existing research suffers from inconsistent dataset quality, where spam emails are often mislabeled as phishing despite significant semantic differences. Many papers do not publish source code, preventing reproducibility. Dataset naming is inconsistent across papers, with the same dataset appearing under different names .
Concept Drift and Class Imbalance: Phishing strategies evolve rapidly, causing static models to degrade quickly. Additionally, phishing emails represent a small fraction of total email volume in real-world environments, creating class imbalance that biases classifiers toward the majority (benign) class .
Semantic Understanding Gaps: Keyword-based approaches cannot identify semantically equivalent but differently worded phishing content. Traditional spam filters rely on surface features and fail to capture contextual and behavioral cues .
Computational Efficiency: Advanced transformer models achieve high accuracy but require significant computational resources, limiting deployment in resource-constrained environments. The challenge lies in balancing detection performance with inference speed .
1.3 Research Objectives
This project aims to design and implement an email spam and phishing detection system with the following objectives:
1. Compare transformer models (BERT, DistilBERT, TinyBERT) with traditional classifiers (Logistic Regression, SVM, Naive Bayes, Random Forest) on email text and subject classification.
2. Implement a hybrid detection architecture integrating deep learning, traditional machine learning, and rule-based heuristics.
3. Provide a Chrome browser extension for real-time Gmail detection with local processing capabilities.
4. Build a management dashboard supporting threshold adjustment, log export, and performance monitoring.
5. Address class imbalance and dataset heterogeneity through undersampling and multi-source data integration.
1.4 Thesis Structure
This thesis comprises five chapters. Chapter 1 introduces research background and objectives; Chapter 2 reviews related work in email phishing detection, machine learning techniques, and existing systems; Chapter 3 presents the system architecture and design; Chapter 4 describes implementation details and experimental results; Chapter 5 concludes and discusses future work.
Chapter 2: Literature Review
2.1 Evolution of Email Spam and Phishing Detection
Email spam and phishing detection techniques have evolved from rule-based matching to machine learning and, most recently, deep learning approaches. Early systems relied on keyword blacklists, sender blacklists, and heuristic rules—methods that are fast but adapt poorly to evolving attack strategies. Attackers bypass these filters through character obfuscation, URL shortening, and HTML spoofing .
Machine learning approaches introduced feature engineering methods including TF-IDF vectorization, n-gram features, and email header metadata. A comprehensive review of machine learning for spam filtering examined approaches including Naive Bayes, Support Vector Machines, and ensemble methods . However, traditional machine learning methods generally struggle with generalization and exhibit high false positive rates .
The advent of transformer-based deep learning models has significantly improved detection capabilities. BERT (Bidirectional Encoder Representations from Transformers) provides contextual understanding of text, capturing subtle linguistic patterns that indicate phishing intent. Studies demonstrate that BERT-based models achieve superior performance compared to traditional machine learning approaches .
2.2 Transformer Models for Phishing Detection
Recent research has extensively evaluated transformer models for email phishing detection. A study on deep learning with adaptive optimization demonstrated BERT’s effectiveness, with the model achieving high accuracy in identifying phishing emails . The contextual embeddings generated by BERT capture semantic relationships that simpler models miss, enabling detection of paraphrased phishing content .
Hybrid architectures combining BERT with sequential models have shown particular promise. Research combining BERT with BiLSTM achieved an F1-score of 0.97, significantly outperforming standalone BiLSTM (0.93) . The BERT component provides contextual understanding while the LSTM captures sequential dependencies in email structure . Another study integrating BERT with TF-IDF found that hybrid models consistently outperformed single-method approaches, with TF-IDF + BERT achieving the highest accuracy on smaller datasets .