3.2 Yahoo奇摩電影本週新片- Python web crawler note - GitBook

Yahoo電影也算滿好爬的, 這邊要做的事情是把本週新片及其相關資訊都列出來, 並且寫入json檔案 ... Y_MOVIE_URL = 'https://tw.movies.yahoo.com/movie_thisweek.html'. 📘📘📘📘PythonwebcrawlernoteSearch…📘📘📘📘PythonwebcrawlernoteIntroduction1.環境安裝與爬蟲的基本1.1環境安裝1.2IDE設定1.3一隻很原始的爬蟲1.4幫爬蟲裝煞車2.用BeautifuSoup來分析網頁2.1BeautifulSoup範例-12.2BeautifulSoup說明2.3BeautifulSoup範例-22.4加入RegularExpression2.5Dcard今日十大熱門文章3.更多實際的應用3.1PTT八卦版今日熱門文章3.2Yahoo奇摩電影本週新片3.3蘋果日報/自由時報今日焦點3.4GoogleFinance個股資訊3.5Yahoo奇摩字典4.基於API的爬蟲4.1八卦版鄉民從哪來?4.2FacebookGraphAPI4.3imdb電影資訊查詢4.4GoogleFinanceAPI4.5台灣證券交易所API5.資料儲存5.1痴漢爬蟲(PTT表特版下載器)5.2儲存成CSV檔案5.3儲存至SQLite6.不同編碼/文件類型的爬蟲6.1非UTF-8編碼的文件6.2XML文件7.比價爬蟲7.1momo購物網爬蟲7.2PChome24hAPI爬蟲7.3比價圖表程式8.處理POST請求/登入頁面8.1空氣品質監測網9.動態網頁爬蟲9.1台銀法拍屋資訊查詢10.自然語言處理10.1歌詞頻率與歌詞雲PoweredByGitBook3.2Yahoo奇摩電影本週新片Yahoo電影也算滿好爬的,這邊要做的事情是把本週新片及其相關資訊都列出來,並且寫入json檔案裡:1importrequests2importre3importjson4frombs4importBeautifulSoup5​6​7Y_MOVIE_URL='https://tw.movies.yahoo.com/movie_thisweek.html'8​9Y_MOVIE_INFO_URL='https://tw.movies.yahoo.com/movieinfo_main.htm'10Y_MOVIE_PHOTO_URL='https://tw.movies.yahoo.com/movieinfo_photos.html'11Y_MOVIE_TIME_URL='https://tw.movies.yahoo.com/movietime_result.html'12​13​14defget_web_page(url):15resp=requests.get(url)16ifresp.status_code!=200:17print('Invalidurl:',resp.url)18returnNone19else:20returnresp.text21​22​23defget_movies(dom):24soup=BeautifulSoup(dom,'html5lib')25movies=[]26rows=soup.find_all('div','clearfixrow')27forrowinrows:28movie=dict()29movie['expectation']=row.find(id='ymvle').find('div','bdclearfix').em.text30movie['ch_name']=row.find('div','text').h4.text31movie['en_name']=row.find('div','text').h5.text32movie['movie_id']=get_movie_id(row.find('div','text').h4.a['href'])33movie['poster_url']=row.find('div','img').img['src'].replace('mpost4','mpost')34movie['release_date']=get_date(row.find('div','text').span.text)35movie['intro']=row.find('div','text').p.text.replace(u'...詳全文','').replace('\n','')36trailer_li=row.find('div','text').find('li','trailer')37movie['trailer_url']=get_trailer_url(trailer_li.a['href'])iftrailer_lielse''38movies.append(movie)39returnmovies40​41​42defget_date(date_str):43#e.g."上映日期：2017-03-23"->match.group(0):"2017-03-23"44pattern='\d+-\d+-\d+'45#re.compileAPIDOC:https://docs.python.org/3/library/re.html#re.compile46#re.searchAPIDOC:https://docs.python.org/3/library/re.html#search-vs-match47match=re.search(pattern,date_str)48ifmatchisNone:49returndate_str50else:51returnmatch.group(0)52​53​54defget_movie_id(url):55#e.g."https://tw.rd.yahoo.com/referurl/movie/thisweek/info/*https://tw.movies.yahoo.com/movieinfo_main.html/id=6707"56#->match.group(0):"/id=6707"57pattern='/id=\d+'58match=re.search(pattern,url)59ifmatchisNone:60returnurl61else:62returnmatch.group(0).replace('/id=','')63​64​65defget_trailer_url(url):66#e.g.,'https://tw.rd.yahoo.com/referurl/movie/thisweek/trailer/*https://tw.movies.yahoo.com/video/美女與野獸-最終版預告-024340912.html'67returnurl.split('*')[1]68​69​70defget_complete_intro(movie_id):71page=get_web_page(Y_MOVIE_INFO_URL+'/id='+movie_id)72ifpage:73soup=BeautifulSoup.get(page,'html5lib')74div_text_show=soup.find('div','textshow')75ifdiv_text_show:76print(div_text_show.text)77div_text_full=soup.find('div','textfull')78ifdiv_text_full:79print(div_text_full.text)80returnNone81​82​83defmain():84page=get_web_page(Y_MOVIE_URL)85ifpage:86movies=get_movies(page)87formovieinmovies:88print(movie)89withopen('movie.json','w',encoding='UTF-8')asfile:90json.dump(movies,file,indent=2,sort_keys=True,ensure_ascii=False)91​92​93if__name__=='__main__':94main()Copied!輸出如下:1{'expectation':'98','ch_name':'加勒比海盜神鬼奇航：死無對證','en_name':'PiratesoftheCaribbean:DeadMenTellNoTales','movie_id':'6534','poster_url':'https://s.yimg.com/vu/movies/fp/mpost/65/34/6534.jpg','release_date':'2017-05-24','intro':'位處墨西哥灣與加勒比海的神秘百慕達三角洲，發生過大量令人匪疑所思的失蹤事件，不僅讓這個地區蒙上一層神祕的色彩，更被烙上「魔鬼三角」','trailer_url':'https://tw.movies.yahoo.com/video/加勒比海盜-神鬼奇航-死無對證-強尼戴普現身迪士尼樂園-013706487.html'}2{'expectation':'94','ch_name':'海灘救護隊','en_name':'Baywatch','movie_id':'6682','poster_url':'https://s.yimg.com/vu/movies/fp/mpost/66/82/6682.jpg','release_date':'2017-05-25','intro':'改編自1989年同名電視劇。

兩位不稱頭的肌肉猛男救生員準備到美國加州海灘應徵救護隊成員。

','trailer_url':'https://tw.movies.yahoo.com/video/海灘救護隊-海灘篇-031153665.html'}3{'expectation':'71','ch_name':'大釣哥','en_name':'HankyPanky','movie_id':'6657','poster_url':'https://s.yimg.com/vu/movies/fp/mpost/66/57/6657.jpg','release_date':'2017-05-26','intro':'★萬眾矚目！坐擁10億票房以上紀錄不敗天王豬哥亮2017賀歲強檔！★卡司陣容耳目一新，打造台灣賀歲全新類型！★影帝影后聯袂登場','trailer_url':'https://tw.movies.yahoo.com/video/大釣哥-刪減感人片段篇-092443147.html'}4{'expectation':'95','ch_name':'墨利斯的情人經典數位修復','en_name':'Maurice','movie_id':'6816','poster_url':'https://s.yimg.com/vu/movies/fp/mpost/68/16/6816.jpg','release_date':'2017-05-26','intro':'★2017柏林影展經典單元★威尼斯影展銀獅獎、最佳男演員、GoldenOsella最佳電影音樂★奧斯卡最佳服裝設計入圍兩','trailer_url':'https://tw.movies.yahoo.com/video/墨利斯的情人經典數位修復-中文預告-101351858.html'}5{'expectation':'96','ch_name':'愛在回家時','en_name':'AFamilyMan','movie_id':'6827','poster_url':'https://s.yimg.com/vu/movies/fp/mpost/68/27/6827.jpg','release_date':'2017-05-26','intro':'★傑瑞德巴特勒從影至今最動人演出，情感層次豐富更勝《P.S.我愛妳》★《大法官》《會計師》編劇、《藥命俱樂部》製片聯手出擊，從','trailer_url':'https://tw.movies.yahoo.com/video/愛在回家時-中文預告-094916516.html'}6{'expectation':'97','ch_name':'玩命鎗火','en_name':'FreeFire','movie_id':'6865','poster_url':'https://s.yimg.com/vu/movies/fp/mpost/68/65/6865.jpg','release_date':'2017-05-26','intro':'★黑色幽默×鎗林彈雨×極致暴力×爽度破表×笑到岔氣！★《華爾街之狼》《隔離島》金獎名導馬丁史柯西斯超殺監製！★暴力','trailer_url':'https://tw.movies.yahoo.com/video/玩命鎗火-6秒看完-玩命鎗火-021640376.html'}7{'expectation':'80','ch_name':'2017臺北文學．閱影展','en_name':'\u3000','movie_id':'6871','poster_url':'https://s.yimg.com/vu/movies/fp/mpost/68/71/6871.jpg','release_date':'2017-05-26','intro':'【策展緣起】每一次的「文學閱影展」都是一種召喚。

召喚觀眾翻開書頁，召喚讀者走入戲院，召喚所有人在電影中看見文學之美。

正如文學','trailer_url':''}8{'expectation':'92','ch_name':'為妳唱的歌','en_name':'SongtoSong','movie_id':'6876','poster_url':'https://s.yimg.com/vu/movies/fp/mpost/68/76/6876.jpg','release_date':'2017-05-26','intro':'兩對戀人，一場音樂季，浪漫四角戀激情上演…。

菲(魯妮瑪拉飾)愛玩音樂，夢想成為創作歌手，談戀愛不受羈絆，她遇見了音樂人”BV','trailer_url':'https://tw.movies.yahoo.com/video/為妳唱的歌-雷恩深情獻唱20秒預告-095213971.html'}9{'expectation':'40','ch_name':'我就是要結婚！','en_name':'ThroughtheWall','movie_id':'6889','poster_url':'https://s.yimg.com/vu/movies/fp/mpost/68/89/6889.jpg','release_date':'2017-05-26','intro':'只想被愛的蜜涵，能否找到完美另一半？！誰會在這麼短的時間裡，成為她生命的另一半呢？年過三十的蜜涵沒有心儀對象，為了擁有一生一','trailer_url':'https://tw.movies.yahoo.com/video/我就是要結婚-中文預告-103015177.html'}10{'expectation':'100','ch_name':'雙面法蘭茲','en_name':'Frantz','movie_id':'6894','poster_url':'https://s.yimg.com/vu/movies/fp/mpost/68/94/6894.jpg','release_date':'2017-05-26','intro':'兩國的無情戰場\u3000炸出三人的熾烈情場次世界大戰德法交火，在殊死交鋒之際，一股超越敵對的溫熱情感，在壕溝中默默滋長。

這一切','trailer_url':'https://tw.movies.yahoo.com/video/雙面法蘭茲-中文預告-093941023.html'}11{'expectation':'33','ch_name':'回聲戀習曲','en_name':'OneStep','movie_id':'6903','poster_url':'https://s.yimg.com/vu/movies/fp/mpost/69/03/6903.jpg','release_date':'2017-05-26','intro':'★韓國超人氣女子團體2NE1前成員Dara，首次大銀幕一展歌喉，詮釋試圖找回過去記憶的魅力女主角，讓影迷心疼又尖叫！★南韓名導金基德盛','trailer_url':'https://tw.movies.yahoo.com/video/回聲戀習曲-中文預告-150821910.html'}12{'expectation':'100','ch_name':'藍心狂想曲','en_name':'TheBlueHearts','movie_id':'6904','poster_url':'https://s.yimg.com/vu/movies/fp/mpost/69/04/6904.jpg','release_date':'2017-05-26','intro':'★2016高雄電影節口碑爆棚★2016日本夕張國際奇幻影展★2016夏威夷國際電影節★日本傳奇龐克天團「藍心樂團」三十週年出道','trailer_url':'https://tw.movies.yahoo.com/video/藍心狂想曲-中文預告-150821276.html'}13{'expectation':'75','ch_name':'吃吃的愛','en_name':"Didi'sDream",'movie_id':'6870','poster_url':'https://s.yimg.com/vu/movies/fp/mpost/68/70/6870.jpg','release_date':'2017-05-27','intro':'想要在巨星姐姐面前證明自己的臨時演員上官娣娣，和多年期待真愛卻在最後被狠狠出賣的太空黑鳥麵館老板娘許春梅，當兩個人的世界以想象不到','trailer_url':'https://tw.movies.yahoo.com/video/吃吃的愛-洗腦神曲鯉魚歌-012223571.html'}14​15Processfinishedwithexitcode0Copied!產生的檔案:1[2{3"ch_name":"加勒比海盜神鬼奇航：死無對證",4"en_name":"PiratesoftheCaribbean:DeadMenTellNoTales",5"expectation":"98",6"intro":"位處墨西哥灣與加勒比海的神秘百慕達三角洲，發生過大量令人匪疑所思的失蹤事件，不僅讓這個地區蒙上一層神祕的色彩，更被烙上「魔鬼三角」",7"movie_id":"6534",8"poster_url":"https://s.yimg.com/vu/movies/fp/mpost/65/34/6534.jpg",9"release_date":"2017-05-24",10"trailer_url":"https://tw.movies.yahoo.com/video/加勒比海盜-神鬼奇航-死無對證-強尼戴普現身迪士尼樂園-013706487.html"11},12{13"ch_name":"海灘救護隊",14"en_name":"Baywatch",15"expectation":"94",16"intro":"改編自1989年同名電視劇。

兩位不稱頭的肌肉猛男救生員準備到美國加州海灘應徵救護隊成員。

",17"movie_id":"6682",18"poster_url":"https://s.yimg.com/vu/movies/fp/mpost/66/82/6682.jpg",19"release_date":"2017-05-25",20"trailer_url":"https://tw.movies.yahoo.com/video/海灘救護隊-海灘篇-031153665.html"21},22{23"ch_name":"大釣哥",24"en_name":"HankyPanky",25"expectation":"71",26"intro":"★萬眾矚目！坐擁10億票房以上紀錄不敗天王豬哥亮2017賀歲強檔！★卡司陣容耳目一新，打造台灣賀歲全新類型！★影帝影后聯袂登場",27"movie_id":"6657",28"poster_url":"https://s.yimg.com/vu/movies/fp/mpost/66/57/6657.jpg",29"release_date":"2017-05-26",30"trailer_url":"https://tw.movies.yahoo.com/video/大釣哥-刪減感人片段篇-092443147.html"31},32{33"ch_name":"墨利斯的情人經典數位修復",34"en_name":"Maurice",35"expectation":"95",36"intro":"★2017柏林影展經典單元★威尼斯影展銀獅獎、最佳男演員、GoldenOsella最佳電影音樂★奧斯卡最佳服裝設計入圍兩",37"movie_id":"6816",38"poster_url":"https://s.yimg.com/vu/movies/fp/mpost/68/16/6816.jpg",39"release_date":"2017-05-26",40"trailer_url":"https://tw.movies.yahoo.com/video/墨利斯的情人經典數位修復-中文預告-101351858.html"41},42{43"ch_name":"愛在回家時",44"en_name":"AFamilyMan",45"expectation":"96",46"intro":"★傑瑞德巴特勒從影至今最動人演出，情感層次豐富更勝《P.S.我愛妳》★《大法官》《會計師》編劇、《藥命俱樂部》製片聯手出擊，從",47"movie_id":"6827",48"poster_url":"https://s.yimg.com/vu/movies/fp/mpost/68/27/6827.jpg",49"release_date":"2017-05-26",50"trailer_url":"https://tw.movies.yahoo.com/video/愛在回家時-中文預告-094916516.html"51},52{53"ch_name":"玩命鎗火",54"en_name":"FreeFire",55"expectation":"97",56"intro":"★黑色幽默×鎗林彈雨×極致暴力×爽度破表×笑到岔氣！★《華爾街之狼》《隔離島》金獎名導馬丁史柯西斯超殺監製！★暴力",57"movie_id":"6865",58"poster_url":"https://s.yimg.com/vu/movies/fp/mpost/68/65/6865.jpg",59"release_date":"2017-05-26",60"trailer_url":"https://tw.movies.yahoo.com/video/玩命鎗火-6秒看完-玩命鎗火-021640376.html"61},62{63"ch_name":"2017臺北文學．閱影展",64"en_name":"　",65"expectation":"80",66"intro":"【策展緣起】每一次的「文學閱影展」都是一種召喚。

召喚觀眾翻開書頁，召喚讀者走入戲院，召喚所有人在電影中看見文學之美。

正如文學",67"movie_id":"6871",68"poster_url":"https://s.yimg.com/vu/movies/fp/mpost/68/71/6871.jpg",69"release_date":"2017-05-26",70"trailer_url":""71},72{73"ch_name":"為妳唱的歌",74"en_name":"SongtoSong",75"expectation":"92",76"intro":"兩對戀人，一場音樂季，浪漫四角戀激情上演…。

菲(魯妮瑪拉飾)愛玩音樂，夢想成為創作歌手，談戀愛不受羈絆，她遇見了音樂人”BV",77"movie_id":"6876",78"poster_url":"https://s.yimg.com/vu/movies/fp/mpost/68/76/6876.jpg",79"release_date":"2017-05-26",80"trailer_url":"https://tw.movies.yahoo.com/video/為妳唱的歌-雷恩深情獻唱20秒預告-095213971.html"81},82{83"ch_name":"我就是要結婚！",84"en_name":"ThroughtheWall",85"expectation":"40",86"intro":"只想被愛的蜜涵，能否找到完美另一半？！誰會在這麼短的時間裡，成為她生命的另一半呢？年過三十的蜜涵沒有心儀對象，為了擁有一生一",87"movie_id":"6889",88"poster_url":"https://s.yimg.com/vu/movies/fp/mpost/68/89/6889.jpg",89"release_date":"2017-05-26",90"trailer_url":"https://tw.movies.yahoo.com/video/我就是要結婚-中文預告-103015177.html"91},92{93"ch_name":"雙面法蘭茲",94"en_name":"Frantz",95"expectation":"100",96"intro":"兩國的無情戰場　炸出三人的熾烈情場次世界大戰德法交火，在殊死交鋒之際，一股超越敵對的溫熱情感，在壕溝中默默滋長。

這一切",97"movie_id":"6894",98"poster_url":"https://s.yimg.com/vu/movies/fp/mpost/68/94/6894.jpg",99"release_date":"2017-05-26",100"trailer_url":"https://tw.movies.yahoo.com/video/雙面法蘭茲-中文預告-093941023.html"101},102{103"ch_name":"回聲戀習曲",104"en_name":"OneStep",105"expectation":"33",106"intro":"★韓國超人氣女子團體2NE1前成員Dara，首次大銀幕一展歌喉，詮釋試圖找回過去記憶的魅力女主角，讓影迷心疼又尖叫！★南韓名導金基德盛",107"movie_id":"6903",108"poster_url":"https://s.yimg.com/vu/movies/fp/mpost/69/03/6903.jpg",109"release_date":"2017-05-26",110"trailer_url":"https://tw.movies.yahoo.com/video/回聲戀習曲-中文預告-150821910.html"111},112{113"ch_name":"藍心狂想曲",114"en_name":"TheBlueHearts",115"expectation":"100",116"intro":"★2016高雄電影節口碑爆棚★2016日本夕張國際奇幻影展★2016夏威夷國際電影節★日本傳奇龐克天團「藍心樂團」三十週年出道",117"movie_id":"6904",118"poster_url":"https://s.yimg.com/vu/movies/fp/mpost/69/04/6904.jpg",119"release_date":"2017-05-26",120"trailer_url":"https://tw.movies.yahoo.com/video/藍心狂想曲-中文預告-150821276.html"121},122{123"ch_name":"吃吃的愛",124"en_name":"Didi'sDream",125"expectation":"75",126"intro":"想要在巨星姐姐面前證明自己的臨時演員上官娣娣，和多年期待真愛卻在最後被狠狠出賣的太空黑鳥麵館老板娘許春梅，當兩個人的世界以想象不到",127"movie_id":"6870",128"poster_url":"https://s.yimg.com/vu/movies/fp/mpost/68/70/6870.jpg",129"release_date":"2017-05-27",130"trailer_url":"https://tw.movies.yahoo.com/video/吃吃的愛-洗腦神曲鯉魚歌-012223571.html"131}132]Copied!原始碼點我​Previous3.1PTT八卦版今日熱門文章Next3.3蘋果日報/自由時報今日焦點Lastmodified2yragoCopylink