在使用进程池时,遇到了如下错误: multiprocessing.pool.MaybeEncodingError: Error sending result:
在使用进程池时,遇到了如下错误:
multiprocessing.pool.MaybeEncodingError: Error sending result: '<multiprocessing.pool.ExceptionWithTraceback object at 0x0000022149D84F98>'. Reason: 'TypeError("cannot serialize '_io.BufferedReader' object",)'
在stackoverflow搜索了一些答案,但是感觉和我这个情况不是很像
https://stackoverflow.com/questions/48761983/multiprocessing-pool-maybeencodingerror-error-sending-result-reason-typeerro
https://stackoverflow.com/questions/54736710/multiprocessing-pool-maybeencodingerror-typeerrorcannot-serialize-io-buffe?noredirect=1
这里数据集是论文的一些信息,process_paper主要是针对作者论文信息构建一些特征,以下是数据来源:
https://www.biendata.xyz/competition/chaindream_nd_task1/
def pre_process_pubs(pubs, save_path, pl):
papers = pl.starmap(process_paper, zip(pubs.values()))
paper_ids = list(pubs.keys())
pubs = dict(zip(paper_ids, papers))
json.dump(pubs, open(save_path, 'w'), indent=4)
if __name__ == '__main__':
pool = mlp.Pool(4)
# 训练集
_, pub = load_original_train(0, 1)
pre_process_pubs(pub, './data/preprocess_pubs/cleaned_train_pubs.json', pool)
del pub
# 验证集
pub = json.load(open('./data/OAG-v2-track1/valid/sna_valid_pub.json', 'r', encoding='utf-8'))
pre_process_pubs(pub, './data/preprocess_pubs/cleaned_test_pubs.json', pool)
del pub
pool.close()
第一次用进程池,现在完全不知道从哪里下手去找问题,希望有大佬能解答。