Skip to content
Projects
Groups
Snippets
Help
This project
Loading...
Sign in / Register
Toggle navigation
M
media_data_crawler
Overview
Overview
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
zhiwei
media_data_crawler
Commits
7adbc5b2
Commit
7adbc5b2
authored
Sep 11, 2018
by
yangchen
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
zhiwei-tool 修改为0.0.5版本
parent
eb2d52f1
Show whitespace changes
Inline
Side-by-side
Showing
3 changed files
with
5 additions
and
4 deletions
+5
-4
pom.xml
+2
-1
src/main/java/com/zhiwei/media_data_crawler/crawler/BaiduNewsCrawlerParse.java
+1
-1
src/test/java/com/zhiwei/media_data_crawler/test/DataCrawlerTest.java
+2
-2
No files found.
pom.xml
View file @
7adbc5b2
...
...
@@ -65,7 +65,7 @@
<dependency>
<groupId>
com.zhiwei.tools
</groupId>
<artifactId>
zhiwei-tools
</artifactId>
<version>
0.0.
4
-SNAPSHOT
</version>
<version>
0.0.
5
-SNAPSHOT
</version>
</dependency>
</dependencies>
</project>
\ No newline at end of file
src/main/java/com/zhiwei/media_data_crawler/crawler/BaiduNewsCrawlerParse.java
View file @
7adbc5b2
...
...
@@ -247,7 +247,7 @@ public class BaiduNewsCrawlerParse {
for
(
int
i
=
1
;
i
<=
3
;
i
++)
{
try
{
Response
response
=
HttpBoot
.
syncCall
(
RequestUtils
.
wrapGet
(
url
,
headerMap
),
proxy
,
false
);
return
response
.
body
().
toS
tring
();
return
response
.
body
().
s
tring
();
}
catch
(
Exception
e
)
{
logger
.
error
(
"获取数据时出现问题,问题为:{}"
,
e
.
fillInStackTrace
());
if
(
i
==
3
){
...
...
src/test/java/com/zhiwei/media_data_crawler/test/DataCrawlerTest.java
View file @
7adbc5b2
...
...
@@ -28,7 +28,7 @@ public class DataCrawlerTest {
Proxy
proxy
=
null
;
//代理IP,不用可不填写
try
{
// //百度新闻采集demo
// List<NewsData> baiduNewsL
ist = DataCrawler.getBaiduNewsData(word, startTime, endTime, proxy);
List
<
NewsData
>
l
ist
=
DataCrawler
.
getBaiduNewsData
(
word
,
startTime
,
endTime
,
proxy
);
// //搜狗新闻关键词采集demo
// List<NewsData> sogouNewsList = DataCrawler.getSougouNewsData(word, proxy);
// //360新闻采集demo
...
...
@@ -46,7 +46,7 @@ public class DataCrawlerTest {
// List<DouBanData> list = DataCrawler.getDouBanData(word, type, proxy);
List
<
NewsData
>
list
=
DataCrawler
.
getSoData
(
"京东"
,
"www.toutiao.com"
,
"d"
,
proxy
);
//
List<NewsData> list = DataCrawler.getSoData("京东", "www.toutiao.com", "d", proxy);
for
(
NewsData
newsData
:
list
)
{
System
.
out
.
println
(
newsData
);
}
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment