分页接口数据重复问题

記的打卡· 2023-10-24 22:37:20

requests库如何循环获取分页接口的全部数据
由于框架限制page最大上限为1000,默认10条
部分代码
author = 'Leo'
import requests
import json
import time

url = "http://order-full-process/list?page=0&size=10"
headers = heade_D()
ir = ApiRequest()
data = {
'''''''
}
result = ir.run_method(url=url, method='post', data=data, headers=headers)

json_data = json.loads(result) # 转换
orderno_list = [i['orderNo'] for i in json_data.get('content', [])]
Status = [i['orderStatus'] for i in json_data.get('content', [])]

sums = len(orderno_list)
print(f'订单数量：{sums}')

//试了循环调用
def url_info():
url_list = ['http://order-detail?page={}&size=10&orderNo='.format(i) for i in range(200)]
return url_list

这样返回的数据会有重复值,请问啥原因,?

...全文

112 3 打赏收藏转发到动态举报

写回复

用AI写文章

3 条回复

切换为时间正序

请发表友善的回复…

发表回复

点灯大神 2023-10-25

打赏
举报


import requests
import json

base_url = "http://order-full-process/list"
headers = heade_D()  # 你的headers函数，假设它会返回正确的头部信息

def get_total_pages():
    # 请求第一页数据来获取总的数据量或分页数
    response = requests.post(base_url, params={'page': 0, 'size': 10}, headers=headers)
    data = response.json()
    total_count = data.get('totalElements', 0)  # 假设接口返回的json中有一个叫做'totalElements'的字段，这个字段表示总的数据条数
    total_pages = total_count // 10 + (1 if total_count % 10 != 0 else 0)  # 计算总的分页数
    return total_pages

def get_data_for_page(page, size=10):
    response = requests.post(base_url, params={'page': page, 'size': size}, headers=headers)
    return response.json()

total_pages = get_total_pages()

all_data = []
for page in range(total_pages):
    data = get_data_for_page(page)
    all_data.extend(data.get('content', []))

# 打印结果
print(f'总订单数量：{len(all_data)}')

記的打卡· 2023-10-25

@点灯大神返回数据: content: [{id: "501712619678683136", dispatchCode: "MD00000271", dispatchName: "106",…},…] empty: false number: 0 numberOfElements: 10 size: 10 totalElements: 31 totalPages: 4

記的打卡· 2023-10-25