Python数据分析实战：从零构建猫眼电影数据采集与可视化系统

Python数据分析猫眼电影数据采集pandas数据清洗

于 2026-07-07 15:21:29 修改

·本内容遵循CC 4.0 BY-SA版权协议

如果你正在学习Python数据分析，或者想要一个完整的实战项目来提升技能，那么猫眼电影数据分析系统绝对值得你投入时间。这个项目看似简单，却涵盖了从数据采集、清洗、存储到可视化分析的完整数据流水线，是检验你Python综合能力的绝佳试金石。

很多初学者在学完Python基础后，往往陷入"不知道下一步该做什么"的困境。单独学习Flask、requests或Pandas时感觉不难，但如何将这些技术组合起来解决实际问题却是个挑战。本文将带你从零开始构建一个完整的猫眼电影数据分析系统，每个步骤都有详细代码和解释，即使是编程新手也能跟着做出来。

1. 为什么选择猫眼电影数据分析项目？

在众多数据源中，为什么猫眼电影特别适合作为数据分析的入门项目？这背后有几个关键考量：

数据质量高且稳定：猫眼电影作为国内主流电影平台，数据更新及时、字段规范，包含电影名称、评分、票房、演员、导演等丰富信息，非常适合进行分析挖掘。

技术栈覆盖面广：这个项目涉及爬虫技术(requests+BeautifulSoup)、数据处理(pandas)、Web框架(Flask)、数据库(MySQL)、数据可视化(ECharts)等核心技术，正好构成一个完整的数据分析技术生态。

商业价值明显：电影数据分析有实际应用场景，比如票房预测、观众偏好分析、市场趋势判断等，学完后你掌握的不仅是技术，更是解决实际问题的能力。

适中的技术难度：相比一些复杂的商业系统，猫眼电影数据分析项目难度适中，既有挑战性又不会让初学者望而却步，是积累项目经验的理想选择。

2. 环境准备与技术栈说明

在开始编码前，我们需要准备好开发环境。以下是本项目所需的主要技术组件：

2.1 核心依赖库及版本建议

PYTHON

# requirements.txt

Flask==2.3.3

requests==2.31.0

pandas==2.0.3

beautifulsoup4==4.12.2

pymysql==1.1.0

sqlalchemy==2.0.20

echarts==1.0.0

2.2 环境配置步骤

BASH

# 创建虚拟环境（推荐）

python -m venv maoyan_env

source maoyan_env/bin/activate # Linux/Mac

maoyan_env\Scripts\activate # Windows

# 安装依赖

pip install -r requirements.txt

# 数据库准备（MySQL）

# 确保已安装MySQL并创建数据库

mysql -u root -p

CREATE DATABASE maoyan_movies CHARACTER SET utf8mb4;

2.3 技术选型理由说明

Flask：轻量级Web框架，学习曲线平缓，适合快速开发原型
requests：简单易用的HTTP库，处理网络请求更加方便
pandas：数据分析核心工具，提供强大的数据清洗和转换能力
BeautifulSoup：HTML解析库，适合处理猫眼电影页面的结构化数据

3. 数据采集模块实现

数据采集是整个项目的基础，我们需要从猫眼电影网站获取原始数据。这里要特别注意反爬虫机制的应对策略。

3.1 爬虫核心代码实现

PYTHON

import requests

from bs4 import BeautifulSoup

import time

import random

import csv

class MaoyanSpider:

def __init__(self):

self.headers = {

'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36',

'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8'

}

self.base_url = 'https://maoyan.com/films'

def get_movie_list(self, offset=0):

"""获取电影列表页数据"""

url = f'{self.base_url}?offset={offset}'

try:

response = requests.get(url, headers=self.headers, timeout=10)

response.encoding = 'utf-8'

if response.status_code == 200:

soup = BeautifulSoup(response.text, 'html.parser')

movies = []

# 解析电影列表

movie_items = soup.find_all('div', class_='movie-item')

for item in movie_items:

movie = {}

# 提取电影基本信息

movie['name'] = item.find('span', class_='name').text.strip()

movie['score'] = item.find('span', class_='score').text.strip() if item.find('span', class_='score') else '暂无评分'

movie['actors'] = item.find('div', class_='actors').text.strip().replace('主演：', '')

movie['release_time'] = item.find('div', class_='releasetime').text.strip().replace('上映时间：', '')

# 获取详情页链接

detail_link = item.find('a')['href']

movie['detail_url'] = f'https://maoyan.com{detail_link}'

movies.append(movie)

return movies

else:

print(f'请求失败，状态码：{response.status_code}')

return []

except Exception as e:

print(f'爬取过程中出现错误：{e}')

return []

def get_movie_detail(self, detail_url):

"""获取电影详情数据"""

try:

response = requests.get(detail_url, headers=self.headers, timeout=10)

response.encoding = 'utf-8'

if response.status_code == 200:

soup = BeautifulSoup(response.text, 'html.parser')

detail = {}

# 解析详情页信息

detail['box_office'] = soup.find('span', class_='box-office').text.strip() if soup.find('span', class_='box-office') else '未知'

detail['movie_type'] = soup.find('div', class_='movie-brief-container').find_all('li')[0].text.strip()

detail['country'] = soup.find('div', class_='movie-brief-container').find_all('li')[1].text.strip()

detail['duration'] = soup.find('div', class_='movie-brief-container').find_all('li')[2].text.strip()

return detail

else:

print(f'详情页请求失败：{response.status_code}')

return {}

except Exception as e:

print(f'详情页爬取错误：{e}')

return {}

def save_to_csv(self, movies, filename='maoyan_movies.csv'):

"""保存数据到CSV文件"""

with open(filename, 'w', newline='', encoding='utf-8-sig') as f:

writer = csv.writer(f)

# 写入表头

writer.writerow(['电影名称', '评分', '主演', '上映时间', '票房', '类型', '国家', '时长'])

for movie in movies:

writer.writerow([

movie.get('name', ''),

movie.get('score', ''),

movie.get('actors', ''),

movie.get('release_time', ''),

movie.get('box_office', ''),

movie.get('movie_type', ''),

movie.get('country', ''),

movie.get('duration', '')

])

def run(self, pages=5):

"""主运行函数"""

all_movies = []

for page in range(pages):

print(f'正在爬取第{page+1}页...')

offset = page * 30 # 每页30部电影

movies = self.get_movie_list(offset)

for movie in movies:

# 获取详情信息

detail = self.get_movie_detail(movie['detail_url'])

movie.update(detail)

all_movies.append(movie)

# 添加随机延迟，避免被封IP

time.sleep(random.uniform(1, 3))

print(f'第{page+1}页爬取完成，共{len(movies)}部电影')

# 保存数据

self.save_to_csv(all_movies)

print(f'所有数据爬取完成，共{len(all_movies)}部电影，已保存到maoyan_movies.csv')

return all_movies

# 使用示例

if __name__ == '__main__':

spider = MaoyanSpider()

movies = spider.run(pages=3) # 爬取3页数据

3.2 反爬虫策略应对

在实际爬取过程中，你可能会遇到各种反爬虫机制。以下是几个实用的应对策略：

PYTHON

# 高级爬虫配置示例

class AdvancedMaoyanSpider(MaoyanSpider):

def __init__(self):

super().__init__()

self.session = requests.Session()

# 设置更真实的请求头

self.headers.update({

'Accept-Language': 'zh-CN,zh;q=0.9,en;q=0.8',

'Connection': 'keep-alive',

'Referer': 'https://maoyan.com/'

})

def get_with_retry(self, url, max_retries=3):

"""带重试机制的请求函数"""

for attempt in range(max_retries):

try:

response = self.session.get(url, headers=self.headers, timeout=15)

if response.status_code == 200:

return response

elif response.status_code == 429: # 请求过于频繁

wait_time = (2 ** attempt) + random.random()

print(f'请求过于频繁，等待{wait_time}秒后重试...')

time.sleep(wait_time)

else:

print(f'请求失败，状态码：{response.status_code}')

except Exception as e:

print(f'请求异常：{e}')

if attempt < max_retries - 1:

sleep_time = random.uniform(2, 5)

time.sleep(sleep_time)

return None

4. 数据清洗与存储模块

获取到的原始数据往往存在各种问题，需要进行清洗和标准化处理后才能用于分析。

4.1 数据清洗核心代码

PYTHON

import pandas as pd

import numpy as np

import re

class DataCleaner:

def __init__(self, csv_file):

self.df = pd.read_csv(csv_file)

def clean_score(self):

"""清洗评分数据"""

# 将"暂无评分"替换为NaN

self.df['评分'] = self.df['评分'].replace('暂无评分', np.nan)

# 提取数字评分

self.df['评分'] = self.df['评分'].apply(lambda x: float(x) if pd.notna(x) and x.replace('.', '').isdigit() else np.nan)

def clean_box_office(self):

"""清洗票房数据"""

def parse_box_office(value):

if pd.isna(value) or value == '未知':

return np.nan

# 处理票房格式：如"1.2亿"、"3500万"

if '亿' in value:

return float(re.findall(r'[\d.]+', value)[0]) * 10000

elif '万' in value:

return float(re.findall(r'[\d.]+', value)[0])

else:

return np.nan

self.df['票房_万'] = self.df['票房'].apply(parse_box_office)

def clean_release_time(self):

"""清洗上映时间"""

# 提取年份和月份

self.df['上映年份'] = self.df['上映时间'].str.extract(r'(\d{4})')

self.df['上映月份'] = self.df['上映时间'].str.extract(r'(\d{1,2})月')

# 转换为数值类型

self.df['上映年份'] = pd.to_numeric(self.df['上映年份'], errors='coerce')

self.df['上映月份'] = pd.to_numeric(self.df['上映月份'], errors='coerce')

def clean_movie_type(self):

"""清洗电影类型"""

# 将类型字符串拆分为列表

self.df['类型列表'] = self.df['类型'].str.split('/')

def remove_duplicates(self):

"""去除重复数据"""

self.df = self.df.drop_duplicates(subset=['电影名称'], keep='first')

def handle_missing_values(self):

"""处理缺失值"""

# 数值列用中位数填充

numeric_cols = ['评分', '票房_万', '上映年份', '上映月份']

for col in numeric_cols:

if col in self.df.columns:

self.df[col] = self.df[col].fillna(self.df[col].median())

# 文本列用"未知"填充

text_cols = ['主演', '国家', '时长']

for col in text_cols:

if col in self.df.columns:

self.df[col] = self.df[col].fillna('未知')

def run_cleaning(self):

"""执行完整的清洗流程"""

print('开始数据清洗...')

print(f'原始数据形状：{self.df.shape}')

self.clean_score()

self.clean_box_office()

self.clean_release_time()

self.clean_movie_type()

self.remove_duplicates()

self.handle_missing_values()

print(f'清洗后数据形状：{self.df.shape}')

print('数据清洗完成！')

return self.df

# 使用示例

cleaner = DataCleaner('maoyan_movies.csv')

cleaned_df = cleaner.run_cleaning()

cleaned_df.to_csv('cleaned_maoyan_movies.csv', index=False, encoding='utf-8-sig')

4.2 数据存储到MySQL

PYTHON

import pymysql

from sqlalchemy import create_engine

class DataStorage:

def __init__(self, host='localhost', user='root', password='', database='maoyan_movies'):

self.connection_string = f'mysql+pymysql://{user}:{password}@{host}/{database}'

self.engine = create_engine(self.connection_string)

def create_tables(self):

"""创建数据表"""

create_table_sql = """

CREATE TABLE IF NOT EXISTS movies (

id INT AUTO_INCREMENT PRIMARY KEY,

name VARCHAR(255) NOT NULL,

score FLOAT,

actors TEXT,

release_time VARCHAR(100),

box_office VARCHAR(100),

movie_type VARCHAR(100),

country VARCHAR(100),

duration VARCHAR(50),

box_office_wan FLOAT,

release_year INT,

release_month INT,

created_time TIMESTAMP DEFAULT CURRENT_TIMESTAMP

)

"""

try:

with self.engine.connect() as conn:

conn.execute(create_table_sql)

print('数据表创建成功！')

except Exception as e:

print(f'创建数据表失败：{e}')

def save_to_database(self, df):

"""保存数据到数据库"""

try:

# 选择需要存储的列

columns_to_save = [

'电影名称', '评分', '主演', '上映时间', '票房',

'类型', '国家', '时长', '票房_万', '上映年份', '上映月份'

]

# 重命名列以匹配数据库表结构

df_to_save = df[columns_to_save].copy()

df_to_save.columns = [

'name', 'score', 'actors', 'release_time', 'box_office',

'movie_type', 'country', 'duration', 'box_office_wan', 'release_year', 'release_month'

]

# 保存到数据库

df_to_save.to_sql('movies', self.engine, if_exists='replace', index=False)

print('数据保存到数据库成功！')

except Exception as e:

print(f'保存到数据库失败：{e}')

def read_from_database(self):

"""从数据库读取数据"""

try:

df = pd.read_sql('SELECT * FROM movies', self.engine)

return df

except Exception as e:

print(f'从数据库读取数据失败：{e}')

return None

# 使用示例

storage = DataStorage(host='localhost', user='root', password='your_password', database='maoyan_movies')

storage.create_tables()

storage.save_to_database(cleaned_df)

5. Flask Web应用搭建

现在我们来构建Web应用，提供数据查询和可视化界面。

5.1 Flask应用基础结构

PYTHON

from flask import Flask, render_template, request, jsonify

import pymysql

import pandas as pd

import json

app = Flask(__name__)

# 数据库配置

DB_CONFIG = {

'host': 'localhost',

'user': 'root',

'password': 'your_password',

'database': 'maoyan_movies',

'charset': 'utf8mb4'

}

def get_db_connection():

"""获取数据库连接"""

return pymysql.connect(**DB_CONFIG)

@app.route('/')

def index():

"""首页"""

return render_template('index.html')

@app.route('/api/movies')

def get_movies():

"""获取电影数据API"""

page = request.args.get('page', 1, type=int)

limit = request.args.get('limit', 10, type=int)

offset = (page - 1) * limit

conn = get_db_connection()

try:

with conn.cursor(pymysql.cursors.DictCursor) as cursor:

# 获取总数

cursor.execute('SELECT COUNT(*) as total FROM movies')

total = cursor.fetchone()['total']

# 获取分页数据

cursor.execute('SELECT * FROM movies ORDER BY box_office_wan DESC LIMIT %s OFFSET %s', (limit, offset))

movies = cursor.fetchall()

return jsonify({

'code': 0,

'msg': '成功',

'count': total,

'data': movies

})

finally:

conn.close()

@app.route('/api/box_office_analysis')

def box_office_analysis():

"""票房分析API"""

year = request.args.get('year', 'all')

conn = get_db_connection()

try:

with conn.cursor(pymysql.cursors.DictCursor) as cursor:

if year == 'all':

# 按类型统计总票房

sql = '''

SELECT movie_type, SUM(box_office_wan) as total_box_office

FROM movies

WHERE box_office_wan IS NOT NULL

GROUP BY movie_type

ORDER BY total_box_office DESC

'''

else:

# 按年份和类型统计

sql = f'''

SELECT movie_type, SUM(box_office_wan) as total_box_office

FROM movies

WHERE release_year = {year} AND box_office_wan IS NOT NULL

GROUP BY movie_type

ORDER BY total_box_office DESC

'''

cursor.execute(sql)

result = cursor.fetchall()

# 格式化数据供ECharts使用

types = [item['movie_type'] for item in result]

values = [float(item['total_box_office']) for item in result]

return jsonify({

'types': types,

'values': values

})

finally:

conn.close()

@app.route('/api/actor_analysis')

def actor_analysis():

"""演员分析API"""

top_n = request.args.get('top_n', 10, type=int)

# 使用pandas进行复杂的数据分析

conn = get_db_connection()

try:

df = pd.read_sql('SELECT actors FROM movies', conn)

# 拆分演员列并统计

all_actors = []

for actors_str in df['actors'].dropna():

actors = actors_str.split('/')

all_actors.extend([actor.strip() for actor in actors])

# 统计演员出现次数

from collections import Counter

actor_counts = Counter(all_actors)

# 获取出现次数最多的演员

top_actors = actor_counts.most_common(top_n)

actors = [actor[0] for actor in top_actors]

counts = [actor[1] for actor in top_actors]

return jsonify({

'actors': actors,

'counts': counts

})

finally:

conn.close()

if __name__ == '__main__':

app.run(debug=True, host='0.0.0.0', port=5000)

5.2 前端页面模板

HTML

<!DOCTYPE html>

<head>

<title>猫眼电影数据分析系统</title>

</head>

<body>

<a class="navbar-brand" href="#">猫眼电影数据分析系统</a>

</div>

</nav>

<h5>电影数据总览</h5>

</div>

<thead>

<tr>

</tr>

</thead>

</table>

</div>

<h5>票房类型分布</h5>

</div>

</div>

<h5>演员作品数量排行</h5>

</div>

</div>

// 初始化ECharts实例

const boxOfficeChart = echarts.init(document.getElementById('boxOfficeChart'));

const actorChart = echarts.init(document.getElementById('actorChart'));

// 票房分析图表配置

const boxOfficeOption = {

tooltip: {

trigger: 'item',

formatter: '{a} <br/>{b}: {c}万 ({d}%)'

legend: {

orient: 'vertical',

left: 10,

data: []

series: [

{

name: '票房分布',

type: 'pie',

radius: ['50%', '70%'],

avoidLabelOverlap: false,

label: {

show: false,

position: 'center'

emphasis: {

label: {

show: true,

fontSize: '18',

fontWeight: 'bold'

}

labelLine: {

show: false

data: []

}

]

};

// 演员分析图表配置

const actorOption = {

tooltip: {

trigger: 'axis',

axisPointer: {

type: 'shadow'

}

xAxis: {

type: 'category',

data: []

yAxis: {

type: 'value'

series: [{

data: [],

type: 'bar',

itemStyle: {

color: '#5470c6'

}

}]

};

// 加载数据

function loadData() {

// 加载票房数据

fetch('/api/box_office_analysis')

.then(response => response.json())

.then(data => {

boxOfficeOption.legend.data = data.types;

boxOfficeOption.series[0].data = data.types.map((type, index) => {

return { value: data.values[index], name: type };

});

boxOfficeChart.setOption(boxOfficeOption);

});

// 加载演员数据

fetch('/api/actor_analysis?top_n=10')

.then(response => response.json())

.then(data => {

actorOption.xAxis.data = data.actors;

actorOption.series[0].data = data.counts;

actorChart.setOption(actorOption);

});

}

// 页面加载完成后执行

window.addEventListener('load', loadData);

window.addEventListener('resize', function() {

boxOfficeChart.resize();

actorChart.resize();

});

</script>

</body>

</html>

6. 数据可视化深度分析

除了基础的可视化，我们还可以进行更深入的数据分析，挖掘更有价值的洞察。

6.1 高级分析功能实现

PYTHON

import matplotlib.pyplot as plt

import seaborn as sns

from wordcloud import WordCloud

class AdvancedAnalysis:

def __init__(self, df):

self.df = df

def score_distribution(self):

"""评分分布分析"""

plt.figure(figsize=(10, 6))

sns.histplot(self.df['评分'].dropna(), kde=True, bins=20)

plt.title('电影评分分布')

plt.xlabel('评分')

plt.ylabel('数量')

plt.savefig('score_distribution.png', dpi=300, bbox_inches='tight')

plt.close()

def box_office_vs_score(self):

"""票房与评分关系分析"""

plt.figure(figsize=(10, 6))

sns.scatterplot(data=self.df, x='评分', y='票房_万', alpha=0.6)

plt.title('票房与评分关系')

plt.xlabel('评分')

plt.ylabel('票房（万）')

plt.savefig('box_office_vs_score.png', dpi=300, bbox_inches='tight')

plt.close()

def genre_analysis(self):

"""类型分析"""

# 展开类型列表

genre_list = []

for genres in self.df['类型列表'].dropna():

genre_list.extend(genres)

from collections import Counter

genre_counts = Counter(genre_list)

# 绘制类型分布

plt.figure(figsize=(12, 6))

genres, counts = zip(*genre_counts.most_common(10))

sns.barplot(x=list(genres), y=list(counts))

plt.title('电影类型分布')

plt.xticks(rotation=45)

plt.savefig('genre_distribution.png', dpi=300, bbox_inches='tight')

plt.close()

def generate_wordcloud(self):

"""生成演员词云"""

# 合并所有演员

all_actors = ' '.join(self.df['主演'].dropna().astype(str))

wordcloud = WordCloud(

font_path='simhei.ttf',

width=800,

height=400,

background_color='white'

).generate(all_actors)

plt.figure(figsize=(12, 6))

plt.imshow(wordcloud, interpolation='bilinear')

plt.axis('off')

plt.title('演员词云图')

plt.savefig('actor_wordcloud.png', dpi=300, bbox_inches='tight')

plt.close()

def run_analysis(self):

"""执行完整分析"""

print('开始高级数据分析...')

self.score_distribution()

self.box_office_vs_score()

self.genre_analysis()

self.generate_wordcloud()

print('分析完成，图表已保存！')

# 使用示例

analysis = AdvancedAnalysis(cleaned_df)

analysis.run_analysis()

6.2 时间趋势分析

PYTHON

def time_trend_analysis(df):

"""时间趋势分析"""

# 按年份统计

yearly_stats = df.groupby('上映年份').agg({

'票房_万': 'sum',

'评分': 'mean',

'电影名称': 'count'

}).reset_index()

yearly_stats.columns = ['年份', '总票房', '平均评分', '电影数量']

# 绘制趋势图

fig, (ax1, ax2) = plt.subplots(2, 1, figsize=(12, 10))

# 票房趋势

ax1.plot(yearly_stats['年份'], yearly_stats['总票房'], marker='o', linewidth=2)

ax1.set_title('年度票房趋势')

ax1.set_xlabel('年份')

ax1.set_ylabel('总票房（万）')

ax1.grid(True, alpha=0.3)

# 评分趋势

ax2.plot(yearly_stats['年份'], yearly_stats['平均评分'], marker='s', color='red', linewidth=2)

ax2.set_title('年度平均评分趋势')

ax2.set_xlabel('年份')

ax2.set_ylabel('平均评分')

ax2.grid(True, alpha=0.3)

plt.tight_layout()

plt.savefig('time_trend_analysis.png', dpi=300, bbox_inches='tight')

plt.close()

return yearly_stats

# 执行分析

trend_data = time_trend_analysis(cleaned_df)

7. 项目部署与优化

完成开发后，我们需要考虑如何部署和优化这个系统。

7.1 生产环境部署配置

PYTHON

# config.py

import os

class Config:

SECRET_KEY = os.environ.get('SECRET_KEY') or 'your-secret-key-here'

MYSQL_HOST = os.environ.get('MYSQL_HOST') or 'localhost'

MYSQL_USER = os.environ.get('MYSQL_USER') or 'root'

MYSQL_PASSWORD = os.environ.get('MYSQL_PASSWORD') or ''

MYSQL_DATABASE = os.environ.get('MYSQL_DATABASE') or 'maoyan_movies'

class ProductionConfig(Config):

DEBUG = False

TESTING = False

class DevelopmentConfig(Config):

DEBUG = True

TESTING = True

# 根据环境选择配置

if os.environ.get('FLASK_ENV') == 'production':

config = ProductionConfig()

else:

config = DevelopmentConfig()

7.2 性能优化建议

PYTHON

# 数据库查询优化

def optimized_queries():

"""优化数据库查询"""

# 1. 添加索引

index_sqls = [

"CREATE INDEX idx_box_office ON movies(box_office_wan)",

"CREATE INDEX idx_release_year ON movies(release_year)",

"CREATE INDEX idx_score ON movies(score)"

]

# 2. 使用连接池

from DBUtils.PooledDB import PooledDB

pool = PooledDB(

creator=pymysql,

maxconnections=10,

host=config.MYSQL_HOST,

user=config.MYSQL_USER,

password=config.MYSQL_PASSWORD,

database=config.MYSQL_DATABASE,

charset='utf8mb4'

)

# 3. 缓存常用数据

import redis

redis_client = redis.Redis(host='localhost', port=6379, db=0)

def get_cached_data(key, expire_time=3600):

"""获取缓存数据"""

cached = redis_client.get(key)

if cached:

return json.loads(cached)

return None

def set_cached_data(key, data, expire_time=3600):

"""设置缓存数据"""

redis_client.setex(key, expire_time, json.dumps(data))

8. 常见问题与解决方案

在实际开发过程中，你可能会遇到以下常见问题：

8.1 爬虫相关问题

问题1：请求被拒绝，返回429状态码

PYTHON

# 解决方案：添加更完善的请求控制

def smart_request(url, headers, max_retries=5):

for i in range(max_retries):

try:

response = requests.get(url, headers=headers, timeout=15)

if response.status_code == 429:

wait_time = 2 ** i + random.random()

time.sleep(wait_time)

continue

return response

except Exception as e:

if i == max_retries - 1:

raise e

time.sleep(1)

问题2：网页结构变化导致解析失败

PYTHON

# 解决方案：使用更灵活的解析策略

def flexible_parsing(soup):

# 尝试多种选择器

selectors = [

'span.name',

'.movie-name',

'h1.movie-title'

]

for selector in selectors:

element = soup.select_one(selector)

if element:

return element.text.strip()

return '未知'

8.2 数据处理问题

问题3：数据清洗时内存不足

PYTHON

# 解决方案：分块处理大数据集

def process_large_file(filename, chunk_size=1000):

for chunk in pd.read_csv(filename, chunksize=chunk_size):

# 处理每个数据块

processed_chunk = clean_chunk(chunk)

yield processed_chunk

# 或者使用Dask进行分布式处理

import dask.dataframe as dd

ddf = dd.read_csv('large_file.csv')

result = ddf.map_partitions(clean_chunk).compute()

问题4：特殊字符编码问题

PYTHON

# 解决方案：统一编码处理

def safe_encode(text):

if isinstance(text, str):

return text.encode('utf-8', errors='ignore').decode('utf-8')

return str(text)

# 在读取文件时指定编码

df = pd.read_csv('file.csv', encoding='utf-8-sig')

9. 项目扩展与进阶方向

完成基础版本后，你可以考虑以下扩展方向来提升项目的价值：

9.1 机器学习集成

PYTHON

from sklearn.ensemble import RandomForestRegressor

from sklearn.model_selection import train_test_split

from sklearn.metrics import mean_absolute_error

def predict_box_office(df):

"""票房预测模型"""

# 特征工程

features = df[['评分', '上映年份', '上映月份']].copy()

features = pd.get_dummies(features, columns=['上映月份'])

# 目标变量

target = df['票房_万']

# 划分训练测试集

X_train, X_test, y_train, y_test = train_test_split(

features, target, test_size=0.2, random_state=42

)

# 训练模型

model = RandomForestRegressor(n_estimators=100, random_state=42)

model.fit(X_train, y_train)

# 评估模型

predictions = model.predict(X_test)

mae = mean_absolute_error(y_test, predictions)

print(f'模型MAE