1.MongoDB的安装
MongoDB是一种非关系型数据库
选择你的系统对应的版本下载安装即可
2.MongoDB配置
a.在C盘或者D盘建一个文件夹如图mongodb
b.安装成功后里面会有bin文件然后再文件夹里面新建一个data文件夹,data文件里面新建db文件夹
c.在bin文件路径下打开命令行工具执行下面的命令
mongod --dbpath C:\mongdb\data\db
d.此时在打开一个命令行在bin路径下执行下面的代码
mongo
3.安装第三方库pymongo(连接MongoDB)
pip3 install pymongo
4.安装Mongodb可视化管理工具Robomongo
安装成功之后启动Robomongo,在空白处点击,然后选择Add命令,单击Save,最后点击Connect按钮连接到MongoDB数据库
5.案例代码
引入相应的模块
import requestsfrom lxml import etreeimport reimport pymongoimport time
连接mongodb数据库
client = pymongo.MongoClient('localhost', 27017)mydb = client['mydb']musictop = mydb['musictop']
案例完整代码
import requestsfrom lxml import etreeimport reimport pymongoimport timeclient = pymongo.MongoClient('localhost', 27017)mydb = client['mydb']musictop = mydb['musictop']headers = { 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/66.0.3359.181 Safari/537.36'}def get_url_music(url): html = requests.get(url, headers=headers) selector = etree.HTML(html.text) music_hrefs = selector.xpath('//a[@class="nbg"]/@href') for music_href in music_hrefs: get_music_info(music_href)def get_music_info(url): html = requests.get(url, headers=headers) selector = etree.HTML(html.text) name = selector.xpath('//*[@id="wrapper"]/h1/span/text()')[0] author = re.findall('表演者:.*?>(.*?)', html.text,re.S)[0] styles = re.findall('流派: (.*?)',html.text,re.S) if len(styles) == 0: style = '未知' else: style = styles[0].strip() time = re.findall('发行时间: (.*?)', html.text, re.S)[0].strip() publishers = re.findall('出版者:.*?>(.*?)', html.text, re.S) if len(publishers) == 0: publishers = '未知' else: publishers = publishers[0].strip() score = selector.xpath('//*[@id="interest_sectl"]/div/div[2]/strong/text()')[0] print(name, author, style, time, publishers, score) info = { 'name': name, 'author': author, 'style': style, 'time': time, 'publisher': publishers, 'score': score } musictop.insert_one(info)if __name__ == '__main__': urls = ['https://music.douban.com/top250?start={}'.format(str(i)) for i in range(0, 250, 25)] for url in urls: get_url_music(url) time.sleep(2)