Skip to content
Projects
Groups
Snippets
Help
This project
Loading...
Sign in / Register
Toggle navigation
M
media_data_crawler
Overview
Overview
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
zhiwei
media_data_crawler
Commits
630e8f87
Commit
630e8f87
authored
Feb 27, 2018
by
zhiwei
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
修复百度新闻采集问题
parent
f21ed458
Show whitespace changes
Inline
Side-by-side
Showing
2 changed files
with
5 additions
and
4 deletions
+5
-4
src/main/java/com/zhiwei/media_data_crawler/crawler/BaiduNewsCrawlerParse.java
+3
-2
src/main/java/com/zhiwei/media_data_crawler/crawler/SougouNewsCrawlerParse.java
+2
-2
No files found.
src/main/java/com/zhiwei/media_data_crawler/crawler/BaiduNewsCrawlerParse.java
View file @
630e8f87
...
@@ -224,9 +224,10 @@ public class BaiduNewsCrawlerParse extends HttpClientTemplateOK {
...
@@ -224,9 +224,10 @@ public class BaiduNewsCrawlerParse extends HttpClientTemplateOK {
NewsData
newsData
=
new
NewsData
(
link
,
title
,
source
,
time
,
content
,
pt
,
word
);
NewsData
newsData
=
new
NewsData
(
link
,
title
,
source
,
time
,
content
,
pt
,
word
);
list
.
add
(
newsData
);
list
.
add
(
newsData
);
/**采集相同新闻链接**/
/**采集相同新闻链接**/
if
(
element
.
select
(
"div.c-row"
).
select
(
"a.c-more_link"
)!=
null
)
String
otherUrl
=
element
.
select
(
"div.c-row"
).
select
(
"a.c-more_link"
).
attr
(
"href"
);
if
(
otherUrl
!=
null
&&
!
otherUrl
.
equals
(
""
))
{
{
String
otherLink
=
"http://news.baidu.com"
+
element
.
select
(
"div.c-row"
).
select
(
"a.c-more_link"
).
attr
(
"href"
);
String
otherLink
=
"http://news.baidu.com"
+
element
.
select
(
"div.c-row"
).
select
(
"a.c-more_link"
).
attr
(
"href"
);
List
<
NewsData
>
otherDataList
=
getOherBaiduNewsData
(
otherLink
,
word
,
proxy
);
List
<
NewsData
>
otherDataList
=
getOherBaiduNewsData
(
otherLink
,
word
,
proxy
);
list
.
addAll
(
otherDataList
);
list
.
addAll
(
otherDataList
);
ZhiWeiTools
.
sleep
(
100
);
ZhiWeiTools
.
sleep
(
100
);
...
...
src/main/java/com/zhiwei/media_data_crawler/crawler/SougouNewsCrawlerParse.java
View file @
630e8f87
...
@@ -103,7 +103,7 @@ public class SougouNewsCrawlerParse extends HttpClientTemplateOK {
...
@@ -103,7 +103,7 @@ public class SougouNewsCrawlerParse extends HttpClientTemplateOK {
//获取通用请求头
//获取通用请求头
Map
<
String
,
String
>
headerMap
=
HeaderTool
.
getCommonHead
();
Map
<
String
,
String
>
headerMap
=
HeaderTool
.
getCommonHead
();
//获取链接地址
//获取链接地址
String
url
=
getUrl
(
word
,
mode
,
page
);
String
url
=
get
MediaNews
Url
(
word
,
mode
,
page
);
headerMap
.
put
(
"Host"
,
"news.sogou.com"
);
headerMap
.
put
(
"Host"
,
"news.sogou.com"
);
headerMap
.
put
(
"Referer"
,
url
.
split
(
"&page="
)[
0
]+
"&page="
+(
page
-
1
));
headerMap
.
put
(
"Referer"
,
url
.
split
(
"&page="
)[
0
]+
"&page="
+(
page
-
1
));
//下载数据页面
//下载数据页面
...
@@ -263,7 +263,7 @@ public class SougouNewsCrawlerParse extends HttpClientTemplateOK {
...
@@ -263,7 +263,7 @@ public class SougouNewsCrawlerParse extends HttpClientTemplateOK {
* @param @return 设定文件
* @param @return 设定文件
* @return String 返回类型
* @return String 返回类型
*/
*/
private
static
String
getUrl
(
String
word
,
int
mode
,
int
page
){
private
static
String
get
MediaNews
Url
(
String
word
,
int
mode
,
int
page
){
String
url
=
null
;
String
url
=
null
;
if
(
word
!=
null
){
if
(
word
!=
null
){
url
=
"http://news.sogou.com/news?mode="
+
mode
+
"&media=&query="
url
=
"http://news.sogou.com/news?mode="
+
mode
+
"&media=&query="
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment