版主大大求助，如何突破服务器对我这段代码的限制呢？

fojiao8745 2018-03-18 08:25:58

版主大大求助啊，代码如下，我自己写的爬虫，去年运行还一切正常，现在想更新一下，重新抓取已经全部失效了，VFP有没有办法突破服务器对这种爬虫的封锁呢？真心求教！！谢谢！！

PUBLIC lncount,roothtml,imghtml,lccover,lccoverurl

lncount  = 0

imghtml = ''

roothtml = '<html><head><link rel="stylesheet" type="text/css" href="..\css\main.css"><meta http-equiv="expires" content="60">'+'<meta http-equiv="content-type" content="text/html;charset=utf-8"><meta name="keywords" content="美女,性感,写真,诱惑,大尺度,丝袜"><title>Beautiful Girl!!!</title></head><body><ul>'

IF !DIRECTORY('.\img')

	MKDIR('.\img')

ENDIF

IF !DIRECTORY('.\cover')

	MKDIR('.\cover')

ENDIF

IF !DIRECTORY('.\html')

	MKDIR('.\html')

ENDIF

FOR pg = 1 TO 2    &&这个地方可以将100分成10等分，然后编译出10个EXE，利用多进程，加快爬去速度。

	lnnum = 30

	IF pg < 2

		lcpgurl = 'https://www.nvshens.com/gallery/yazhou'

	ELSE

		lcpgurl = 'https://www.nvshens.com/gallery/yazhou/'+ALLTRIM(STR(pg))+'.html'

	ENDIF

	LOCAL oxhttp AS Microsoft.xmlhttp

	oxhttp=CREATEOBJECT("Microsoft.xmlhttp")

	oxhttp.OPEN("GET",lcpgurl,.F.)

	oxhttp.SEND()

	SourceCode=STRCONV(oxhttp.responseBody,11)

	RELEASE oxhttp

	FOR a = 1 TO lnnum

		lchtml = 'https://www.nvshens.com'+STREXTRACT(SourceCode,"'><a href='","' class='caption'>",a)

		lccoverurl = "https://t1.onvshen.com:85/gallery/" + STREXTRACT(SourceCode,"https://t1.onvshen.com:85/gallery/","'",a)

		IF (lchtml == "") = .F.

			LOCAL oxhttp1 AS Microsoft.xmlhttp

			oxhttp1=CREATEOBJECT("Microsoft.xmlhttp")

			oxhttp1.OPEN("GET",lchtml,.F.)

			oxhttp1.SEND()

			SourceCode1=STRCONV(oxhttp1.responseBody,11)

			lnpicnum = VAL(STREXTRACT(SourceCode1,"#DB0909'>","张照片",1))

			lcfolder = ALLTRIM(STREXTRACT(SourceCode1,'<title>','</title>',1))

			lcfolder = STRTRAN(lcfolder,'_宅男女神','')

			lcfolder = STRTRAN(lcfolder,'图片','')

			lcfolder = STRTRAN(lcfolder,'|','')

			lcfolder = STRTRAN(lcfolder,'<','')

			lcfolder = STRTRAN(lcfolder,'>','')

			lcfolder = STRTRAN(lcfolder,'/','')

			lcfolder = STRTRAN(lcfolder,'\','')

			lcfolder = STRTRAN(lcfolder,':','')

			lcfolder = STRTRAN(lcfolder,'"','')

			lcfolder = STRTRAN(lcfolder,'*','')

			lcfolder = STRTRAN(lcfolder,'?','')

			IF ("404" $ lcfolder) = .F.

				imghtml = imghtml + '<html><head><link rel="stylesheet" type="text/css" href="..\..\css\main.css"><meta http-equiv="expires" content="60"><meta http-equiv="content-type" content="text/html;charset=utf-8"><meta name="keywords" content="美女,性感,写真,诱惑,大尺度,丝袜"><title>'+lcfolder+'</title></head><body>'

				lncount = lncount + 1

				IF !DIRECTORY('.\img\'+lcfolder)

					MKDIR('.\img\'+lcfolder)

				ELSE

					lcfolder = lcfolder  + '_NEW'

					MKDIR('.\img\'+lcfolder)

				ENDIF

				lcpicurlbase = STREXTRACT(SourceCode1,"<img src='","0.jpg",1)

				RELEASE oxhttp1

				****************获取cover******************

				LOCAL oxhttp1 AS Microsoft.xmlhttp

				oxhttp1=CREATEOBJECT("Microsoft.xmlhttp")

				oxhttp1.OPEN("GET",lccoverurl,.F.)

				oxhttp1.SEND()

				STRTOFILE(oxhttp1.responseBody,".\cover\" + lcfolder + ".jpg")

				RELEASE oxhttp1

				*******************************************

				FOR b = 1 TO lnpicnum

					IF b < 2

						lcpicurl = lcpicurlbase + "0.jpg"

					ELSE

						DO CASE

							CASE b < 11

								lcpicurl = lcpicurlbase + '00' + ALLTRIM(STR(b-1)) + '.jpg'

							CASE b < 101

								lcpicurl = lcpicurlbase + '0' + ALLTRIM(STR(b-1)) + '.jpg'

							OTHERWISE

								lcpicurl = lcpicurlbase + ALLTRIM(STR(b-1)) + '.jpg'

						ENDCASE

					ENDIF

					LOCAL oxhttp2 AS Microsoft.xmlhttp

					oxhttp2=CREATEOBJECT("Microsoft.xmlhttp")

					oxhttp2.OPEN("GET",lcpicurl,.F.)

					oxhttp2.SEND()

					IF !ISNULL(oxhttp2.responseBody)

						STRTOFILE(oxhttp2.responseBody,'.\img\'+lcfolder+'\'+ALLTRIM(STR(b))+'.jpg')

						imghtml = imghtml + '<p align="center"><img src="..\img\'+lcfolder+'\'+ALLTRIM(STR(b))+'.jpg"'+' alt="'+'jpg'+ALLTRIM(STR(b))+'"></p>'

					ENDIF

					RELEASE oxhttp2

				ENDFOR

				imghtml  = imghtml + '<p align="center"><a href="..\beauty.html">返回</a></p></body></html>'

				STRTOFILE(STRCONV(imghtml,9),'.\html\'+ALLTRIM(STR(lncount))+'.html')

				imghtml  = ''

				roothtml = roothtml + '<li class="pic"><div class="pic_div"><a class="pic_link" href="' + '.\html\'+ALLTRIM(STR(lncount))+'.html';

				+'" target="view_window"><img alt="'+lcfolder+'" src="'+'.\cover\'+lcfolder+'.jpg'+'"  title="'+lcfolder+'" /></a></div><div class="pic_title"><a href="';

				+'.\html\'+ALLTRIM(STR(lncount))+'.html'+'" target="view_window" class="caption">'+lcfolder+'</a></div></li>'

			ENDIF

		ENDIF

	ENDFOR

ENDFOR

roothtml = roothtml + '</ul></body></html>'

STRTOFILE(STRCONV(roothtml,9),'.\beauty.html')

MESSAGEBOX("All done!!",48,"Tips")

...全文

537 6 打赏收藏转发到动态举报

写回复

用AI写文章

6 条回复

切换为时间正序

请发表友善的回复…

发表回复

xuzuning 2018-03-22

打赏
举报

你这是在和人讨论问题吗？

fojiao8745 2018-03-22

打赏
举报

引用 2 楼 xuzuning 的回复:

你先看看 lccoverurl 的值是什么

url一切都正常，就是服务器会检测这次访问是正常浏览器访问，还是爬虫的访问，应该是检测header和cookie还有没有别的我就不清楚了。

xuzuning 2018-03-22

打赏
举报

你认为不是吗？ https://www.nvshens.com/gallery/yazhou/ 的第一张图片是 https://t1.onvshen.com:85/gallery/19864/25551/cover/0.jpg 不过就是让你核实一下正确与否，为什么要含糊其辞呢？图片来源网站有简单的防盗链措施需要在 http 头中加入来源 REFERER 字段，值为, https://www.nvshens.com/ 你可以查看 https://www.baidu.com/s?wd=%E7%BB%99+xmlhttp+%E5%8A%A0+REFERER&ie=UTF-8 查找可能的解决办法比较可行的方法是通过 webbrowse 组件获取

fojiao8745 2018-03-22