开源标书查重系统构建：文档解析与相似度算法实战

标书查重文档相似度开源技术

于 2026-08-01 04:11:58 修改

·本内容遵循CC 4.0 BY-SA版权协议

在招投标过程中，标书查重是确保公平竞争的重要环节。无论是投标方自查还是招标方审核，都需要高效、准确的文档相似度检查工具。本文将详细介绍如何利用开源技术构建免费的标书查重系统，涵盖Word、PDF文档解析、文本相似度算法和完整实现方案。

1. 标书查重的背景与核心价值

标书查重是指通过技术手段检测不同标书之间的相似程度，主要应用于以下场景：

招标方审核：防止投标人相互串通、围标串标，确保招标过程的公正性。招标方需要对所有投标文件进行相似度检查，发现异常高度相似的标书。

投标方自查：投标企业在提交标书前进行自查，避免因模板使用不当或内容重复导致被认定为不合格标书。

学术与研究机构：虽然主要针对商业标书，但同样的技术可应用于学术论文、项目申报书等文档的原创性检查。

传统的商业查重软件价格昂贵，且可能存在数据隐私风险。开源解决方案不仅能节省成本，还能根据具体需求进行定制化开发。

2. 技术选型与环境准备

2.1 核心技术与工具

文档解析库：

Apache POI：处理Word文档（.doc, .docx）
PDFBox：解析PDF文档内容
Tika：通用文档内容提取工具

文本处理与相似度计算：

Lucene/Solr：全文检索与相似度分析
SimHash：局部敏感哈希算法
Cosine Similarity：余弦相似度计算
Jaccard Similarity：杰卡德相似系数

开发环境：

Java 8+ 或 Python 3.6+
Maven 3.6+ 或 pip
IDE：IntelliJ IDEA 或 VS Code

2.2 项目依赖配置

Maven项目配置示例：

XML

<groupId>org.apache.poi</groupId>

</dependency>

<groupId>org.apache.poi</groupId>

<artifactId>poi-ooxml</artifactId>

</dependency>

<groupId>org.apache.pdfbox</groupId>

<artifactId>pdfbox</artifactId>

</dependency>

<groupId>org.apache.tika</groupId>

</dependency>

<groupId>org.apache.commons</groupId>

<artifactId>commons-text</artifactId>

</dependency>

</dependencies>

Python环境配置：

PYTHON

# requirements.txt

python-docx==0.8.11

PyPDF2==3.0.1

pdfplumber==0.9.0

scikit-learn==1.3.0

nltk==3.8.1

jellyfish==0.11.2

3. 文档解析与文本提取技术

3.1 Word文档解析实现

Word文档包含复杂的格式信息，需要专门的处理库来提取纯文本内容。

Java实现示例：

JAVA

// WordParser.java

import org.apache.poi.xwpf.extractor.XWPFWordExtractor;

import org.apache.poi.xwpf.usermodel.XWPFDocument;

import java.io.FileInputStream;

import java.io.IOException;

public class WordParser {

public static String parseWord(String filePath) throws IOException {

try (FileInputStream fis = new FileInputStream(filePath);

XWPFDocument document = new XWPFDocument(fis);

XWPFWordExtractor extractor = new XWPFWordExtractor(document)) {

return extractor.getText();

}

// 测试方法

public static void main(String[] args) {

try {

String content = parseWord("投标文件.docx");

System.out.println("提取的文本内容：");

System.out.println(content.substring(0, Math.min(500, content.length())) + "...");

} catch (IOException e) {

System.err.println("文档解析失败：" + e.getMessage());

}

Python实现示例：

PYTHON

# word_parser.py

from docx import Document

def parse_word(file_path):

"""

解析Word文档，提取纯文本内容

"""

try:

doc = Document(file_path)

full_text = []

for paragraph in doc.paragraphs:

full_text.append(paragraph.text)

return '\n'.join(full_text)

except Exception as e:

print(f"Word文档解析错误: {e}")

return ""

# 测试

if __name__ == "__main__":

content = parse_word("投标文件.docx")

print("提取的文本内容前500字符：")

print(content[:500] + "...")

3.2 PDF文档解析技术

PDF文档解析相对复杂，需要处理页面布局、字体编码等问题。

Java PDF解析：

JAVA

// PDFParser.java

import org.apache.pdfbox.pdmodel.PDDocument;

import org.apache.pdfbox.text.PDFTextStripper;

import java.io.File;

import java.io.IOException;

public class PDFParser {

public static String parsePDF(String filePath) throws IOException {

try (PDDocument document = PDDocument.load(new File(filePath))) {

PDFTextStripper stripper = new PDFTextStripper();

stripper.setSortByPosition(true); // 按位置排序

stripper.setStartPage(1); // 从第一页开始

stripper.setEndPage(document.getNumberOfPages()); // 到最后一页

return stripper.getText(document);

}

public static void main(String[] args) {

try {

String content = parsePDF("技术方案.pdf");

System.out.println("PDF内容提取成功，字符数：" + content.length());

} catch (IOException e) {

System.err.println("PDF解析失败：" + e.getMessage());

}

Python PDF解析：

PYTHON

# pdf_parser.py

import pdfplumber

def parse_pdf(file_path):

"""

使用pdfplumber解析PDF文档

"""

full_text = []

try:

with pdfplumber.open(file_path) as pdf:

for page in pdf.pages:

text = page.extract_text()

if text:

full_text.append(text)

return '\n'.join(full_text)

except Exception as e:

print(f"PDF解析错误: {e}")

return ""

def parse_pdf_pypdf2(file_path):

"""

使用PyPDF2作为备选方案

"""

from PyPDF2 import PdfReader

try:

reader = PdfReader(file_path)

text = ""

for page in reader.pages:

text += page.extract_text() + "\n"

return text

except Exception as e:

print(f"PyPDF2解析错误: {e}")

return ""

# 测试

if __name__ == "__main__":

content = parse_pdf("招标文件.pdf")

print(f"提取到{len(content)}个字符")

3.3 文本预处理与清洗

提取的原始文本需要清洗和标准化处理，提高查重准确性。

PYTHON

# text_processor.py

import re

import jieba

from nltk.corpus import stopwords

import nltk

# 下载停用词资源（首次运行需要）

# nltk.download('stopwords')

class TextProcessor:

def __init__(self):

self.stop_words = set(stopwords.words('chinese'))

# 添加标书特定停用词

self.stop_words.update(['有限公司', '有限责任公司', '股份有限公司', '项目', '招标', '投标'])

def clean_text(self, text):

"""文本清洗"""

# 移除特殊字符和数字

text = re.sub(r'[^\u4e00-\u9fa5a-zA-Z]', ' ', text)

# 转换为小写

text = text.lower()

# 移除多余空格

text = re.sub(r'\s+', ' ', text).strip()

return text

def tokenize_chinese(self, text):

"""中文分词"""

return list(jieba.cut(text))

def remove_stopwords(self, tokens):

"""移除停用词"""

return [token for token in tokens if token not in self.stop_words and len(token) > 1]

def process(self, text):

"""完整的文本处理流程"""

cleaned = self.clean_text(text)

tokens = self.tokenize_chinese(cleaned)

filtered = self.remove_stopwords(tokens)

return ' '.join(filtered)

# 测试

processor = TextProcessor()

sample_text = "本公司致力于提供优质的工程项目服务，具有丰富的施工经验。"

processed = processor.process(sample_text)

print(f"原始文本: {sample_text}")

print(f"处理后: {processed}")

4. 相似度算法原理与实现

4.1 余弦相似度算法

余弦相似度通过计算两个向量的夹角余弦值来衡量相似度，适合处理文本数据。

PYTHON

# similarity_calculator.py

from sklearn.feature_extraction.text import TfidfVectorizer

from sklearn.metrics.pairwise import cosine_similarity

import numpy as np

class CosineSimilarityCalculator:

def __init__(self):

self.vectorizer = TfidfVectorizer()

def calculate_similarity(self, text1, text2):

"""计算两段文本的余弦相似度"""

# 创建文档列表

documents = [text1, text2]

# 生成TF-IDF向量

tfidf_matrix = self.vectorizer.fit_transform(documents)

# 计算相似度

similarity = cosine_similarity(tfidf_matrix[0:1], tfidf_matrix[1:2])

return similarity[0][0]

def batch_calculate(self, text_list):

"""批量计算相似度矩阵"""

tfidf_matrix = self.vectorizer.fit_transform(text_list)

similarity_matrix = cosine_similarity(tfidf_matrix)

return similarity_matrix

# 测试示例

calculator = CosineSimilarityCalculator()

text_a = "本项目采用先进的技术方案确保工程质量"

text_b = "我们使用创新的技术方法来保证项目质量"

similarity = calculator.calculate_similarity(text_a, text_b)

print(f"余弦相似度: {similarity:.4f}")

4.2 SimHash算法实现

SimHash是一种局部敏感哈希算法，适合海量文本的近似查重。

PYTHON

# simhash_algorithm.py

import hashlib

import numpy as np

class SimHash:

def __init__(self, dimensions=64):

self.dimensions = dimensions

def get_hash(self, text):

"""计算文本的SimHash值"""

# 初始化特征向量

v = np.zeros(self.dimensions)

# 分词处理（简单按字符分割，实际应用应使用分词器）

words = text.split()

for word in words:

# 计算单词的hash值

h = hashlib.md5(word.encode('utf-8')).hexdigest()

# 将hash转换为二进制

binary_hash = bin(int(h, 16))[2:].zfill(128)[:self.dimensions]

# 更新特征向量

for i, bit in enumerate(binary_hash):

if bit == '1':

v[i] += 1

else:

v[i] -= 1

# 生成SimHash

simhash = ''.join(['1' if x > 0 else '0' for x in v])

return simhash

def calculate_distance(self, hash1, hash2):

"""计算两个SimHash的汉明距离"""

if len(hash1) != len(hash2):

raise ValueError("Hash长度不一致")

distance = 0

for i in range(len(hash1)):

if hash1[i] != hash2[i]:

distance += 1

return distance

def calculate_similarity(self, text1, text2):

"""基于SimHash计算相似度"""

hash1 = self.get_hash(text1)

hash2 = self.get_hash(text2)

distance = self.calculate_distance(hash1, hash2)

# 相似度 = 1 - 归一化距离

similarity = 1 - (distance / self.dimensions)

return similarity

# 测试

simhash = SimHash()

text1 = "建筑工程质量安全管理规范"

text2 = "建筑施工质量与安全管理制度"

similarity = simhash.calculate_similarity(text1, text2)

print(f"SimHash相似度: {similarity:.4f}")

4.3 Jaccard相似系数

Jaccard系数通过计算交集与并集的比例来衡量相似度。

PYTHON

# jaccard_similarity.py

def jaccard_similarity(text1, text2):

"""计算Jaccard相似系数"""

# 将文本转换为词集合

set1 = set(text1.split())

set2 = set(text2.split())

# 计算交集和并集

intersection = set1.intersection(set2)

union = set1.union(set2)

if len(union) == 0:

return 0.0

return len(intersection) / len(union)

def jaccard_similarity_with_ngrams(text1, text2, n=2):

"""使用n-gram计算Jaccard相似度"""

def get_ngrams(text, n):

words = text.split()

ngrams = []

for i in range(len(words) - n + 1):

ngrams.append(' '.join(words[i:i+n]))

return set(ngrams)

ngrams1 = get_ngrams(text1, n)

ngrams2 = get_ngrams(text2, n)

intersection = ngrams1.intersection(ngrams2)

union = ngrams1.union(ngrams2)

if len(union) == 0:

return 0.0

return len(intersection) / len(union)

# 测试

text_a = "项目质量管理体系认证"

text_b = "工程项目质量管理制度"

similarity1 = jaccard_similarity(text_a, text_b)

similarity2 = jaccard_similarity_with_ngrams(text_a, text_b, 2)

print(f"Jaccard相似度: {similarity1:.4f}")

print(f"2-gram Jaccard相似度: {similarity2:.4f}")

5. 完整标书查重系统实现

5.1 系统架构设计

构建一个完整的标书查重系统需要包含以下模块：

TEXT

标书查重系统架构：

1. 文档输入层：支持Word、PDF上传

2. 解析处理层：文档解析、文本提取、预处理

3. 算法计算层：多种相似度算法并行计算

4. 结果展示层：相似度报告、详细对比

5.2 核心查重引擎实现

PYTHON

# bid_checker.py

import os

from datetime import datetime

from typing import List, Dict, Tuple

class BidDocumentChecker:

def __init__(self):

self.processor = TextProcessor()

self.cosine_calc = CosineSimilarityCalculator()

self.simhash_calc = SimHash()

def process_document(self, file_path: str) -> str:

"""处理单个文档，返回清洗后的文本"""

if file_path.lower().endswith('.docx'):

from word_parser import parse_word

content = parse_word(file_path)

elif file_path.lower().endswith('.pdf'):

from pdf_parser import parse_pdf

content = parse_pdf(file_path)

else:

raise ValueError(f"不支持的文档格式: {file_path}")

return self.processor.process(content)

def check_similarity(self, file1: str, file2: str) -> Dict[str, float]:

"""检查两个文档的相似度"""

try:

text1 = self.process_document(file1)

text2 = self.process_document(file2)

# 多种算法计算相似度

cosine_sim = self.cosine_calc.calculate_similarity(text1, text2)

simhash_sim = self.simhash_calc.calculate_similarity(text1, text2)

jaccard_sim = jaccard_similarity(text1, text2)

# 加权平均相似度

weighted_avg = (cosine_sim * 0.5 + simhash_sim * 0.3 + jaccard_sim * 0.2)

return {

'cosine_similarity': cosine_sim,

'simhash_similarity': simhash_sim,

'jaccard_similarity': jaccard_sim,

'weighted_average': weighted_avg,

'text1_length': len(text1),

'text2_length': len(text2)

}

except Exception as e:

print(f"查重过程出错: {e}")

return {}

def batch_check(self, directory: str) -> List[Tuple[str, str, Dict]]:

"""批量检查目录中的所有文档"""

results = []

supported_extensions = ['.docx', '.pdf']

# 获取所有支持的文件

files = [f for f in os.listdir(directory)

if any(f.lower().endswith(ext) for ext in supported_extensions)]

print(f"找到 {len(files)} 个文档进行查重")

# 两两比较

for i in range(len(files)):

for j in range(i + 1, len(files)):

file1 = os.path.join(directory, files[i])

file2 = os.path.join(directory, files[j])

similarity_result = self.check_similarity(file1, file2)

results.append((files[i], files[j], similarity_result))

return results

# 使用示例

if __name__ == "__main__":

checker = BidDocumentChecker()

# 单个文件对比

result = checker.check_similarity("标书1.docx", "标书2.pdf")

print("单个对比结果:")

for key, value in result.items():

print(f"{key}: {value}")

# 批量查重

directory = "bid_documents"

if os.path.exists(directory):

batch_results = checker.batch_check(directory)

print(f"\n批量查重完成，共 {len(batch_results)} 组对比")

5.3 结果报告生成

PYTHON

# report_generator.py

from datetime import datetime

import json

class ReportGenerator:

@staticmethod

def generate_text_report(results: List[Tuple[str, str, Dict]]) -> str:

"""生成文本格式的查重报告"""

report = []

report.append("=" * 60)

report.append(" 标书查重报告")

report.append("=" * 60)

report.append(f"生成时间: {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}")

report.append(f"对比文档数量: {len(results)} 组")

report.append("")

for i, (file1, file2, similarity) in enumerate(results, 1):

report.append(f"对比组 {i}: {file1} vs {file2}")

report.append(f" 余弦相似度: {similarity.get('cosine_similarity', 0):.4f}")

report.append(f" SimHash相似度: {similarity.get('simhash_similarity', 0):.4f}")

report.append(f" Jaccard相似度: {similarity.get('jaccard_similarity', 0):.4f}")

report.append(f" 加权平均相似度: {similarity.get('weighted_average', 0):.4f}")

# 相似度评级

avg_similarity = similarity.get('weighted_average', 0)

if avg_similarity > 0.8:

rating = "⚠️ 高度相似"

elif avg_similarity > 0.6:

rating = "🔍 中度相似"

elif avg_similarity > 0.3:

rating = "📊 低度相似"

else:

rating = "✅ 基本不相似"

report.append(f" 相似度评级: {rating}")

report.append("")

return '\n'.join(report)

@staticmethod

def generate_json_report(results: List[Tuple[str, str, Dict]]) -> str:

"""生成JSON格式的详细报告"""

report_data = {

"generated_time": datetime.now().isoformat(),

"total_comparisons": len(results),

"comparisons": []

}

for file1, file2, similarity in results:

comparison = {

"file1": file1,

"file2": file2,

"similarity_scores": similarity,

"risk_level": ReportGenerator._assess_risk_level(

similarity.get('weighted_average', 0)

)

}

report_data["comparisons"].append(comparison)

return json.dumps(report_data, ensure_ascii=False, indent=2)

@staticmethod

def _assess_risk_level(similarity: float) -> str:

"""评估风险等级"""

if similarity > 0.8:

return "high_risk"

elif similarity > 0.6:

return "medium_risk"

elif similarity > 0.3:

return "low_risk"

else:

return "no_risk"

# 使用示例

if __name__ == "__main__":

# 模拟查重结果

sample_results = [

("标书A.docx", "标书B.pdf", {

'cosine_similarity': 0.75,

'simhash_similarity': 0.68,

'jaccard_similarity': 0.72,

'weighted_average': 0.72

}),

("标书A.docx", "标书C.docx", {

'cosine_similarity': 0.25,

'simhash_similarity': 0.18,

'jaccard_similarity': 0.22,

'weighted_average': 0.22

})

]

# 生成文本报告

text_report = ReportGenerator.generate_text_report(sample_results)

print(text_report)

# 生成JSON报告

json_report = ReportGenerator.generate_json_report(sample_results)

print("\nJSON报告:")

print(json_report)

6. 系统部署与Web界面

6.1 基于Flask的Web应用

PYTHON

# app.py

from flask import Flask, request, render_template, jsonify, send_file

import os

from werkzeug.utils import secure_filename

from bid_checker import BidDocumentChecker

from report_generator import ReportGenerator

app = Flask(__name__)

app.config['UPLOAD_FOLDER'] = 'uploads'

app.config['MAX_CONTENT_LENGTH'] = 16 * 1024 * 1024 # 16MB限制

# 确保上传目录存在

os.makedirs(app.config['UPLOAD_FOLDER'], exist_ok=True)

checker = BidDocumentChecker()

@app.route('/')

def index():

return render_template('index.html')

@app.route('/upload', methods=['POST'])

def upload_files():

"""处理文件上传和查重"""

if 'files' not in request.files:

return jsonify({'error': '没有选择文件'}), 400

files = request.files.getlist('files')

if len(files) < 2:

return jsonify({'error': '请至少上传两个文件'}), 400

saved_files = []

for file in files:

if file.filename == '':

continue

filename = secure_filename(file.filename)

filepath = os.path.join(app.config['UPLOAD_FOLDER'], filename)

file.save(filepath)

saved_files.append(filepath)

# 执行查重

results = []

for i in range(len(saved_files)):

for j in range(i + 1, len(saved_files)):

similarity = checker.check_similarity(saved_files[i], saved_files[j])

results.append({

'file1': os.path.basename(saved_files[i]),

'file2': os.path.basename(saved_files[j]),

'similarity': similarity

})

# 清理上传的文件

for filepath in saved_files:

os.remove(filepath)

return jsonify({'results': results})

@app.route('/api/check', methods=['POST'])

def api_check():

"""API接口，接收文本直接计算相似度"""

data = request.json

text1 = data.get('text1', '')

text2 = data.get('text2', '')

if not text1 or not text2:

return jsonify({'error': '缺少文本参数'}), 400

# 直接计算文本相似度

from similarity_calculator import CosineSimilarityCalculator

calculator = CosineSimilarityCalculator()

similarity = calculator.calculate_similarity(text1, text2)

return jsonify({'similarity': similarity})

if __name__ == '__main__':

app.run(debug=True, host='0.0.0.0', port=5000)

6.2 前端界面模板

HTML

<!DOCTYPE html>

<head>

<title>开源标书查重系统</title>

<style>

body { font-family: Arial, sans-serif; margin: 40px; }

.container { max-width: 800px; margin: 0 auto; }

.upload-area { border: 2px dashed #ccc; padding: 40px; text-align: center; margin: 20px 0; }

.results { margin-top: 30px; }

.similarity-item { border: 1px solid #ddd; padding: 15px; margin: 10px 0; }

.high-similarity { background-color: #ffebee; border-left: 4px solid #f44336; }

.medium-similarity { background-color: #fff3e0; border-left: 4px solid #ff9800; }

.low-similarity { background-color: #e8f5e8; border-left: 4px solid #4caf50; }

</style>

</head>

<body>

<h1>开源标书查重系统</h1>

<p>支持Word(.docx)和PDF文档的相似度检查</p>

</div>

</div>

async function uploadFiles() {

const fileInput = document.getElementById('fileInput');

const files = fileInput.files;

if (files.length < 2) {

alert('请至少选择两个文件');

return;

}

const formData = new FormData();

for (let file of files) {

formData.append('files', file);

}

try {

const response = await fetch('/upload', {

method: 'POST',

body: formData

});

const data = await response.json();

displayResults(data.results);

} catch (error) {

console.error('上传失败:', error);

alert('查重失败，请重试');

}

function displayResults(results) {

const resultsDiv = document.getElementById('results');

resultsDiv.innerHTML = '<h2>查重结果</h2>';

results.forEach(result => {

const similarity = result.similarity.weighted_average;

let className = 'similarity-item';

if (similarity > 0.8) className += ' high-similarity';

else if (similarity > 0.6) className += ' medium-similarity';

else className += ' low-similarity';

resultsDiv.innerHTML += `

<h3>${result.file1} vs ${result.file2}</h3>

<p>加权平均相似度: ${similarity.toFixed(4)}</p>

<p>余弦相似度: ${result.similarity.cosine_similarity.toFixed(4)}</p>

<p>SimHash相似度: ${result.similarity.simhash_similarity.toFixed(4)}</p>

</div>

});

}

</script>

</body>

</html>

7. 性能优化与大规模处理

7.1 基于数据库的批量处理

对于大量标书文件，需要数据库支持进行高效管理。

PYTHON

# database_manager.py

import sqlite3

from datetime import datetime

import hashlib

class DocumentDatabase:

def __init__(self, db_path="bid_documents.db"):

self.db_path = db_path

self._init_database()

def _init_database(self):

"""初始化数据库表结构"""

conn = sqlite3.connect(self.db_path)

cursor = conn.cursor()

cursor.execute('''

CREATE TABLE IF NOT EXISTS documents (

id INTEGER PRIMARY KEY AUTOINCREMENT,

filename TEXT UNIQUE,

file_hash TEXT,

content_text TEXT,

word_count INTEGER,

created_time DATETIME DEFAULT CURRENT_TIMESTAMP

)

''')

cursor.execute('''

CREATE TABLE IF NOT EXISTS similarity_results (

id INTEGER PRIMARY KEY AUTOINCREMENT,

doc1_id INTEGER,

doc2_id INTEGER,

cosine_similarity REAL,

simhash_similarity REAL,

jaccard_similarity REAL,

weighted_average REAL,

calculated_time DATETIME DEFAULT CURRENT_TIMESTAMP,

FOREIGN KEY (doc1_id) REFERENCES documents (id),

FOREIGN KEY (doc2_id) REFERENCES documents (id)

)

''')

conn.commit()

conn.close()

def add_document(self, filename, content):

"""添加文档到数据库"""

file_hash = hashlib.md5(content.encode()).hexdigest()

word_count = len(content.split())

conn = sqlite3.connect(self.db_path)

cursor = conn.cursor()

cursor.execute('''

INSERT OR REPLACE INTO documents

(filename, file_hash, content_text, word_count)

VALUES (?, ?, ?, ?)

''', (filename, file_hash, content, word_count))

conn.commit()

doc_id = cursor.lastrowid

conn.close()

return doc_id

def get_similar_documents(self, threshold=0.7):

"""获取相似度超过阈值的文档对"""

conn = sqlite3.connect(self.db_path)

cursor = conn.cursor()

cursor.execute('''

SELECT d1.filename, d2.filename, sr.weighted_average

FROM similarity_results sr

JOIN documents d1 ON sr.doc1_id = d1.id

JOIN documents d2 ON sr.doc2_id = d2.id

WHERE sr.weighted_average > ?

ORDER BY sr.weighted_average DESC

''', (threshold,))

results = cursor.fetchall()

conn.close()

return results

7.2 多线程并行处理

PYTHON

# parallel_processor.py

import concurrent.futures

from threading import Lock

import time

class ParallelDocumentProcessor:

def __init__(self, max_workers=4):

self.max_workers = max_workers

self.results = []

self.lock = Lock()

def process_document_pair(self, file_pair):

"""处理文档对的任务函数"""

file1, file2 = file_pair

checker = BidDocumentChecker()

try:

result = checker.check_similarity(file1, file2)

return (file1, file2, result)

except Exception as e:

print(f"处理 {file1} 和 {file2} 时出错: {e}")

return None

def parallel_batch_check(self, file_pairs):

"""并行处理多个文档对"""

start_time = time.time()

with concurrent.futures.ThreadPoolExecutor(max_workers=self.max_workers) as executor:

future_to_pair = {

executor.submit(self.process_document_pair, pair): pair

for pair in file_pairs

}

for future in concurrent.futures.as_completed(future_to_pair):

pair = future_to_pair[future]

try:

result = future.result()

if result:

with self.lock:

self.results.append(result)

except Exception as e:

print(f"任务执行出错: {e}")

end_time = time.time()

print(f"并行处理完成，耗时: {end_time - start_time:.2f}秒")

return self.results

# 使用示例

if __name__ == "__main__":

import os

import itertools

# 模拟文件列表

document_dir = "documents"

files = [os.path.join(document_dir, f) for f in os.listdir(document_dir)

if f.endswith(('.docx', '.pdf'))]

# 生成所有文档对

file_pairs = list(itertools.combinations(files, 2))

print(f"开始处理 {len(file_pairs)} 个文档对")

processor = ParallelDocumentProcessor(max_workers=8)

results = processor.parallel_batch_check(file_pairs)

print(f"处理完成，成功 {len(results)} 个结果")

8. 常见问题与解决方案

8.1 文档解析问题排查

问题1：PDF解析乱码

原因：PDF字体编码问题或加密保护
解决方案：尝试不同的PDF解析库，处理前检查文档属性

PYTHON

def check_pdf_accessible(file_path):

"""检查PDF是否可解析"""

try:

with open(file_path, 'rb') as f:

# 检查文件头

header = f.read(10)

if b'%PDF' not in header:

return False, "不是有效的PDF文件"

# 检查是否加密

f.seek(0)

content = f.read(1000)

if b'/Encrypt' in content:

return False, "PDF文件已加密"

return True, "文件可解析"

except Exception as e:

return False, f"文件访问错误: {e}"

问题2：Word文档格式丢失

原因：复杂格式和表格处理不完整
解决方案：使用专业的文档处理库，分块提取内容

8.2 相似度算法选择指南

场景	推荐算法	优点	缺点
精确对比	余弦相似度	准确性高，考虑词频	计算复杂度高
海量数据	SimHash	速度快，适合去重	精度相对较低
短文本	Jaccard系数	计算简单，直观