[教程]揭秘高效小说网站爬虫：Python攻略，轻松整站小说一键抓取

3. 代码编写

以下是一个简单的Python爬虫代码示例：

import requests
from bs4 import BeautifulSoup
def fetch_novel_list(url): headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3' } response = requests.get(url, headers=headers) soup = BeautifulSoup(response.text, 'lxml') novel_list = soup.find_all('a', class_='l') for novel in novel_list: print(novel.get_text())
def fetch_novel_content(url): headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.3' } response = requests.get(url, headers=headers) soup = BeautifulSoup(response.text, 'lxml') content = soup.find('div', id='content') return content.get_text()
if __name__ == '__main__': novel_list_url = 'https://www.xbiquge.la/xiaoshuo/' novel_url = 'https://www.xbiquge.la/xiaoshuo/1_1.html' fetch_novel_list(novel_list_url) novel_content = fetch_novel_content(novel_url) print(novel_content)

4. 调试与优化

根据实际情况，对爬虫代码进行调试和优化。例如，处理反爬虫策略、提高抓取速度等。

总结

通过以上步骤，我们可以轻松地编写一个小说网站爬虫，实现整站小说的一键抓取。在实际应用中，需要根据目标网站的具体情况，不断调整和优化爬虫代码。