关于 pyspider 的 post 请求中 data 被转义导致无法抓取

用 requests 请求时可以正常返回结果:
import requests
import json
headers = {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "zh-CN,zh;q=0.9",
"Connection": "keep-alive",
"Content-Length": "56",
"Content-Type": "application/json",
"Host": "www.bw30.com",
"Origin": "http://www.bw30.com",
"Referer": "http://www.bw30.com/",
"User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36",
}
data = json.dumps({"channeltype": "shortvideo", "tid": 8, "index": 0, "size": 10})
a = requests.post("http://www.bw30.com/videoserver/RPCACBCABF433DB30A893DC6C5863E3102C?tid=7", data=data, headers=headers)
print(a.text)

同样的代码逻辑放到 pyspider 中则无法正常返回数据:
def on_start(self):
for tag, tid in {"美食": 7, "资讯": 8, "体育": 10, "亲子": 9, "微记录": 11}.items():
url = "http://www.bw30.com/videoserver/RPCACBCABF433DB30A893DC6C5863E3102C?tid={}".format(tid)
headers = {
"Accept": "*/*",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "zh-CN,zh;q=0.9",
"Connection": "keep-alive",
"Content-Length": "56",
"Content-Type": "application/json",
"Host": "www.bw30.com",
"Origin": "http://www.bw30.com",
"Referer": "http://www.bw30.com/",
"User-Agent": "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.131 Safari/537.36",
}
data = {"channeltype": "shortvideo", "tid": tid, "index": 0, "size": 10}
data = str(data)
self.crawl(url, callback=self.index_page,save={'column_name':tag},method="POST",data=data, headers=headers)

Data

Headers

JSON

tid

3 条回复 2019-05-13 09:42:40 +08:00

foxyier

2019-05-09 15:19:55 +08:00

emmmm, 有没有大佬协助一下, 我在网上搜过之后, 只有提问的，没有解答的。。

Sum0l

2019-05-12 17:47:12 +08:00 via Android

data 不用转 str 的吧？多看看文档

foxyier

2019-05-13 09:42:40 +08:00

@Sum0l 原网站的 data 是需要转的,一般转不转 str 取决于原网站对于 data 的封装方式。同样的代码在 python 下是可以走通的,可是在 pyspider 里边就会请求失败。