微信公众号爬虫笔记
1. 使用Selenium启动web browser driver从而通过程序模拟web application在web browser上的操作。一般来说,selenium是用于测试web application的操作, 在这里使用selenium模拟人为操作登陆微信公众平台从而获取cookies。
使用步骤:
a. Selenium下载安装包
b. Google developer site下载 chromdriver.exe.
需要注意的是在mac环境下,安装的时候需要使用sudo(admin权限) ,并且为了使python能够识别chromedriver.exe, 需要把这个执行文件放置于Library/Python/2.6/site-package下面。
第一部分代码
from selenium import webdriver
import time
import json
from pprint import pprint
post = {}
driver = webdriver.Chrome('/Library/Python/2.7/site-packages/selenium/webdriver/chromedriver')
driver.get('https://mp.weixin.qq.com/')
time.sleep(2)
driver.find_element_by_name("account").clear()
driver.find_element_by_name("account").send_keys('账号')
driver.find_element_by_name("password").clear()
driver.find_element_by_name("password").send_keys('密码')
使用步骤:
a. Selenium下载安装包
b. Google developer site下载 chromdriver.exe.
需要注意的是在mac环境下,安装的时候需要使用sudo(admin权限) ,并且为了使python能够识别chromedriver.exe, 需要把这个执行文件放置于Library/Python/2.6/site-package下面。
第一部分代码
from selenium import webdriver
import time
import json
from pprint import pprint
post = {}
driver = webdriver.Chrome('/Library/Python/2.7/site-packages/selenium/webdriver/chromedriver')
driver.get('https://mp.weixin.qq.com/')
time.sleep(2)
driver.find_element_by_name("account").clear()
driver.find_element_by_name("account").send_keys('账号')
driver.find_element_by_name("password").clear()
driver.find_element_by_name("password").send_keys('密码')
time.sleep(5)
driver.find_element_by_xpath("//a[@class='btn_login']").click()
time.sleep(15)
driver.get('https://mp.weixin.qq.com/')
cookie_items = driver.get_cookies()
driver.get('https://mp.weixin.qq.com/')
cookie_items = driver.get_cookies()
for cookie_item in cookie_items:
post[cookie_item['name']] = cookie_item['value']
cookie_str = json.dumps(post)
with open('cookie.txt', 'w+') as f:
f.write(cookie_str)
2. 获取微信文章永久链接的方式是通过post文章的入口进入到文章搜索处。这样获取到的链接是永久链接,但却无法为已有的微信post带来流量。
import requests
import redis
import json
import re
import random
import time
gzlist = ['summithunter']
url = 'https://mp.weixin.qq.com'
post[cookie_item['name']] = cookie_item['value']
cookie_str = json.dumps(post)
with open('cookie.txt', 'w+') as f:
f.write(cookie_str)
2. 获取微信文章永久链接的方式是通过post文章的入口进入到文章搜索处。这样获取到的链接是永久链接,但却无法为已有的微信post带来流量。
http://mp.weixin.qq.com/s?__biz=MzIyMTk2MzkwMQ==&mid=2247483673&idx=1&sn=7dbe18c8dab482d07387b693f8714b09&chksm=e835f20bdf427b1df9c5b3d104702f1003880a272fd78fae63f1c044696e897dc7a9dc314c5c#rd
第二部分代码如下:
import redis
import json
import re
import random
import time
gzlist = ['summithunter']
url = 'https://mp.weixin.qq.com'
header = {
"HOST": "mp.weixin.qq.com",
"User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0"
}
with open('cookie.txt', 'r') as f:
cookie = f.read()
cookies = json.loads(cookie)
response = requests.get(url=url, cookies=cookies)
token = re.findall(r'token=(\d+)', str(response.url))[0]
for query in gzlist:
query_id = {
'action': 'search_biz',
'token' : token,
"HOST": "mp.weixin.qq.com",
"User-Agent": "Mozilla/5.0 (Windows NT 6.1; WOW64; rv:53.0) Gecko/20100101 Firefox/53.0"
}
with open('cookie.txt', 'r') as f:
cookie = f.read()
cookies = json.loads(cookie)
response = requests.get(url=url, cookies=cookies)
token = re.findall(r'token=(\d+)', str(response.url))[0]
for query in gzlist:
query_id = {
'action': 'search_biz',
'token' : token,
'lang': 'zh_CN',
'f': 'json',
'ajax': '1',
'random': random.random(),
'query': query,
'begin': '0',
'count': '5',
}
search_url = 'https://mp.weixin.qq.com/cgi-bin/searchbiz?'
search_response = requests.get(search_url, cookies=cookies, headers=header, params=query_id)
lists = search_response.json().get('list')[0]
fakeid = lists.get('fakeid')
query_id_data = {
'token': token,
'lang': 'zh_CN',
'f': 'json',
'ajax': '1',
'random': random.random(),
'action': 'list_ex',
'begin': '0',
'count': '5',
'query': '',
'fakeid': fakeid,
'type': '9'
}
appmsg_url = 'https://mp.weixin.qq.com/cgi-bin/appmsg?'
appmsg_response = requests.get(appmsg_url, cookies=cookies, headers=header, params=query_id_data)
max_num = appmsg_response.json().get('app_msg_cnt')
# 分页
num = int(int(max_num) / 5)
begin = 0
while num + 1 > 0 :
query_id_data = {
'token': token,
'lang': 'zh_CN',
'f': 'json',
'ajax': '1',
'random': random.random(),
'action': 'list_ex',
'begin': '{}'.format(str(begin)),
'count': '5',
'query': '',
'fakeid': fakeid,
'type': '9'
}
query_fakeid_response = requests.get(appmsg_url, cookies=cookies, headers=header, params=query_id_data)
fakeid_list = query_fakeid_response.json().get('app_msg_list')
for item in fakeid_list:
print(item.get('link'))
num -= 1
begin = int(begin)
begin+=5
time.sleep(2)
'f': 'json',
'ajax': '1',
'random': random.random(),
'query': query,
'begin': '0',
'count': '5',
}
search_url = 'https://mp.weixin.qq.com/cgi-bin/searchbiz?'
search_response = requests.get(search_url, cookies=cookies, headers=header, params=query_id)
lists = search_response.json().get('list')[0]
fakeid = lists.get('fakeid')
query_id_data = {
'token': token,
'lang': 'zh_CN',
'f': 'json',
'ajax': '1',
'random': random.random(),
'action': 'list_ex',
'begin': '0',
'count': '5',
'query': '',
'fakeid': fakeid,
'type': '9'
}
appmsg_url = 'https://mp.weixin.qq.com/cgi-bin/appmsg?'
appmsg_response = requests.get(appmsg_url, cookies=cookies, headers=header, params=query_id_data)
max_num = appmsg_response.json().get('app_msg_cnt')
# 分页
num = int(int(max_num) / 5)
begin = 0
while num + 1 > 0 :
query_id_data = {
'token': token,
'lang': 'zh_CN',
'f': 'json',
'ajax': '1',
'random': random.random(),
'action': 'list_ex',
'begin': '{}'.format(str(begin)),
'count': '5',
'query': '',
'fakeid': fakeid,
'type': '9'
}
query_fakeid_response = requests.get(appmsg_url, cookies=cookies, headers=header, params=query_id_data)
fakeid_list = query_fakeid_response.json().get('app_msg_list')
for item in fakeid_list:
print(item.get('link'))
num -= 1
begin = int(begin)
begin+=5
time.sleep(2)
Comments
Post a Comment