爬虫高手来看看这个带数组的POST怎么配置requests

洗洗睡去 2020-06-26 10:22:46
最近准备爬取一个网站的数据，chrome80 DevTool里可以看到我需要数据的POST请求各种参数，

但是自己在python里用requests库始终获取不到数据，一直报JSON解析错误，个人感觉应该是那个modules数组配置不对，不应该有引号分号逗号之类的符号。请各位指点


url = 'http://www.zjsq.net.cn:8010/MobileService/Process/getMultiProcessDataTest.action?time=1593172244742'

r = requests.post(url,{"modules":[{"id":"70311540","type":["P","Z","Q","RW"]}],"dt":"60","hours":24,"et":"","st":""})

print(r)

print(r.content)








Struts Problem Report

Struts has detected an unhandled exception:



Messages:	

解析json错误

File:	com/sucsoft/forecast/utils/Jacksons.java

Line number:	157

Stacktraces

java.lang.RuntimeException: 解析json错误

    com.sucsoft.forecast.utils.Jacksons.fromJsonToMap(Jacksons.java:157)

    com.sucsoft.ms.action.CommonAction.getMultiProcessDataTest(CommonAction.java:503)

    sun.reflect.GeneratedMethodAccessor592.invoke(Unknown Source)

    sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)

    java.lang.reflect.Method.invoke(Unknown Source)

    ognl.OgnlRuntime.invokeMethod(OgnlRuntime.java:892)

    ognl.OgnlRuntime.callAppropriateMethod(OgnlRuntime.java:1294)

    ognl.ObjectMethodAccessor.callMethod(ObjectMethodAccessor.java:68)

    com.opensymphony.xwork2.ognl.accessor.XWorkMethodAccessor.callMethodWithDebugInfo(XWorkMethodAccessor.java:117)

    com.opensymphony.xwork2.ognl.accessor.XWorkMethodAccessor.callMethod(XWorkMethodAccessor.java:108)

    ognl.OgnlRuntime.callMethod(OgnlRuntime.java:1370)

    ognl.ASTMethod.getValueBody(ASTMethod.java:91)

    ognl.SimpleNode.evaluateGetValueBody(SimpleNode.java:212)

    ognl.SimpleNode.getValue(SimpleNode.java:258)

    ognl.Ognl.getValue(Ognl.java:467)

    ognl.Ognl.getValue(Ognl.java:431)

    com.opensymphony.xwork2.ognl.OgnlUtil$3.execute(OgnlUtil.java:352)

    com.opensymphony.xwork2.ognl.OgnlUtil.compileAndExecuteMethod(OgnlUtil.java:404)

    com.opensymphony.xwork2.ognl.OgnlUtil.callMethod(OgnlUtil.java:350)

    com.opensymphony.xwork2.DefaultActionInvocation.invokeAction(DefaultActionInvocation.java:430)

    com.opensymphony.xwork2.DefaultActionInvocation.invokeActionOnly(DefaultActionInvocation.java:290)

    com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:251)

    org.apache.struts2.interceptor.DeprecationInterceptor.intercept(DeprecationInterceptor.java:41)

    com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)

    org.apache.struts2.interceptor.debugging.DebuggingInterceptor.intercept(DebuggingInterceptor.java:256)

    com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)

    com.opensymphony.xwork2.interceptor.DefaultWorkflowInterceptor.doIntercept(DefaultWorkflowInterceptor.java:168)

    com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

    com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)

    com.opensymphony.xwork2.validator.ValidationInterceptor.doIntercept(ValidationInterceptor.java:265)

    org.apache.struts2.interceptor.validation.AnnotationValidationInterceptor.doIntercept(AnnotationValidationInterceptor.java:76)

    com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

    com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)

    com.opensymphony.xwork2.interceptor.ConversionErrorInterceptor.intercept(ConversionErrorInterceptor.java:138)

    com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)

    com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:229)

    com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

    com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)

    com.opensymphony.xwork2.interceptor.ParametersInterceptor.doIntercept(ParametersInterceptor.java:229)

    com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

    com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)

    com.opensymphony.xwork2.interceptor.StaticParametersInterceptor.intercept(StaticParametersInterceptor.java:191)

    com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)

    org.apache.struts2.interceptor.MultiselectInterceptor.intercept(MultiselectInterceptor.java:73)

    com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)

    org.apache.struts2.interceptor.DateTextFieldInterceptor.intercept(DateTextFieldInterceptor.java:125)

    com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)

    org.apache.struts2.interceptor.CheckboxInterceptor.intercept(CheckboxInterceptor.java:91)

    com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)

    org.apache.struts2.interceptor.FileUploadInterceptor.intercept(FileUploadInterceptor.java:253)

    com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)

    com.opensymphony.xwork2.interceptor.ModelDrivenInterceptor.intercept(ModelDrivenInterceptor.java:100)

    com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)

    com.opensymphony.xwork2.interceptor.ScopedModelDrivenInterceptor.intercept(ScopedModelDrivenInterceptor.java:141)

    com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)

    com.opensymphony.xwork2.interceptor.ChainingInterceptor.intercept(ChainingInterceptor.java:145)

    com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)

    com.opensymphony.xwork2.interceptor.PrepareInterceptor.doIntercept(PrepareInterceptor.java:171)

    com.opensymphony.xwork2.interceptor.MethodFilterInterceptor.intercept(MethodFilterInterceptor.java:98)

    com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)

    com.opensymphony.xwork2.interceptor.I18nInterceptor.intercept(I18nInterceptor.java:140)

    com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)

    org.apache.struts2.interceptor.ServletConfigInterceptor.intercept(ServletConfigInterceptor.java:164)

    com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)

    com.opensymphony.xwork2.interceptor.AliasInterceptor.intercept(AliasInterceptor.java:193)

    com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)

    com.opensymphony.xwork2.interceptor.ExceptionMappingInterceptor.intercept(ExceptionMappingInterceptor.java:189)

    com.opensymphony.xwork2.DefaultActionInvocation.invoke(DefaultActionInvocation.java:245)

    org.apache.struts2.impl.StrutsActionProxy.execute(StrutsActionProxy.java:54)

    org.apache.struts2.dispatcher.Dispatcher.serviceAction(Dispatcher.java:575)

    org.apache.struts2.dispatcher.ng.ExecuteOperations.executeAction(ExecuteOperations.java:81)

    org.apache.struts2.dispatcher.ng.filter.StrutsPrepareAndExecuteFilter.doFilter(StrutsPrepareAndExecuteFilter.java:99)

    org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)

    org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

    com.sucsoft.common.filter.NoCacheFilter.doFilter(NoCacheFilter.java:35)

    org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:192)

    org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:165)

    org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)

    org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)

    org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:474)

    org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:140)

    org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:79)

    org.apache.catalina.valves.AbstractAccessLogValve.invoke(AbstractAccessLogValve.java:624)

    org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)

    org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:349)

    org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:495)

    org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)

    org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:767)

    org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1347)

    org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)

    java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)

    java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)

    org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)

    java.lang.Thread.run(Unknown Source)

    

You are seeing this page because development mode is enabled. Development mode, or devMode, enables extra debugging behaviors and reports to assist developers. To disable this mode, set:



  struts.devMode=false

in your WEB-INF/classes/struts.properties file.
...全文
200 6 打赏 收藏 转发到动态 举报
写回复
用AI写文章
6 条回复
切换为时间正序
请发表友善的回复…
发表回复
xujibicool 2020-06-29
  • 打赏
  • 举报
 回复 
url = 'http://www.zjsq.net.cn:8010/MobileService/Process/getMultiProcessDataTest.action?'
data = {'data':'{"modules":[{"id":"70311540","type":["P","Z","Q","RW"]}],"dt":"60","hours":24,"et":"","st":""}'}

params=(('time','1593172244742'),)
r = requests.post(url,data=data,params=params)
print(r)
print(r.content)
这个直接复制浏览请求头是这样配置的，你搞得参数配置就有问题
xujibicool 2020-06-28
  • 打赏
  • 举报
 回复
http://www.zjsq.net.cn:8010/MobileService/Process/getMultiProcessDataTest.action?time=1593172244742 这网址直接浏览器也得不到数据，参数传递没问题，浏览器打开也是这内容
洗洗睡去 2020-06-28
  • 打赏
  • 举报
 回复
引用 1 楼 xujibicool 的回复:
url = 'http://www.zjsq.net.cn:8010/MobileService/Process/getMultiProcessDataTest.action?time=1593172244742'
r = requests.post(url,json={"modules":[{"id":"70311540","type":["P","Z","Q","RW"]}],"dt":"60","hours":24,"et":"","st":""})
print(r)
print(r.content)
参数嵌套 json=data
试过了 一样不行
xujibicool 2020-06-28
  • 打赏
  • 举报
 回复
你这个网页接口哪里来的？
洗洗睡去 2020-06-28
  • 打赏
  • 举报
 回复
引用 3 楼 xujibicool 的回复:
http://www.zjsq.net.cn:8010/MobileService/Process/getMultiProcessDataTest.action?time=1593172244742 这网址直接浏览器也得不到数据，参数传递没问题，浏览器打开也是这内容
网站里要具体选择查看的站点之后 才能跟踪到这个地址 ，直接访问是没有的
xujibicool 2020-06-27
  • 打赏
  • 举报
 回复 
url = 'http://www.zjsq.net.cn:8010/MobileService/Process/getMultiProcessDataTest.action?time=1593172244742'
r = requests.post(url,json={"modules":[{"id":"70311540","type":["P","Z","Q","RW"]}],"dt":"60","hours":24,"et":"","st":""})
print(r)
print(r.content)
参数嵌套 json=data
post 爬虫request_爬虫初识和request使用
一.什么是爬虫爬虫的概念:通过编写程序,模拟浏览器上网,让其去互联网上爬取数据的过程.爬虫的工作流程:模拟浏览器发送请求->下载网页代码->只提取有用的数据->存放于数据库或文件中爬虫的分类:通用爬虫:爬取全部的页面数据.聚焦爬虫: 抓取页面中局部的页面数据增量式爬虫:爬取网站中更新出的数据反爬机制门户网站会通过制定相关的技术手段阻止爬虫程序进行数据的爬取反反爬策略:robots...
如何使用request.post(Python)直接发送数组类型的方式
1。
2.17.3-Python爬虫(requests登录)
爬虫整理，使用python，目前写到了requests部分，包含requests的get和post、session登录。 后续会把selenium部分补充起来
Python爬虫之Requests
![在这里插入图片描述](https://img-blog.csdnimg.cn/20210503212726757.png?x-oss-process=image/watermark,type_ZmFuZ3poZW5naGVpdGk,shadow_10,text_aHR0cHM6Ly9ibG9nLmNzZG4ubmV0L3FxXzQ1ODA3MDMy,size_16,color_FFFFFF,t_70) 1. requests简介 获取网页最主要的就是如何模拟浏览器向服务器发送请求，而r第三方库requ
[Python从零到壹] 七.网络爬虫之Requests爬取作者个人博客网站及CSV存储
前一篇文章讲述了 BeautifulSoup 爬取豆瓣TOP250电影，通过案例的方式让大家熟悉Python网络爬虫。这篇文章将详细讲解Requests库爬取个人博客信息，并存储至CSV文件。该案例非常适合入门的案例，也能普及简单的预处理知识。 希望对您有所帮助，本文参考了作者CSDN的文章和学生杨友的博客，从学生的角度实现网络爬虫，可能对读者更友好。
脚本语言

37,722

社区成员

34,239

社区内容

发帖
与我相关
我的任务
社区描述
JavaScript，VBScript，AngleScript，ActionScript，Shell，Perl，Ruby，Lua，Tcl，Scala，MaxScript 等脚本语言交流。
社区管理员
  • 脚本语言(Perl/Python)社区
  • IT.BOB
加入社区
  • 近7日
  • 近30日
  • 至今

加载中

查看更多榜单

试试用AI创作助手写篇文章吧

+ 用AI写文章