Skip to content
Projects
Groups
Snippets
Help
This project
Loading...
Sign in / Register
Toggle navigation
M
media_data_crawler
Overview
Overview
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
zhiwei
media_data_crawler
Commits
a7d988a7
Commit
a7d988a7
authored
Sep 11, 2018
by
yangchen
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
页面获取失败修改
parent
7adbc5b2
Hide whitespace changes
Inline
Side-by-side
Showing
7 changed files
with
9 additions
and
9 deletions
+9
-9
src/main/java/com/zhiwei/media_data_crawler/crawler/BaiduTiebaCrawlerParse.java
+1
-1
src/main/java/com/zhiwei/media_data_crawler/crawler/DoubanCrawlerParse.java
+1
-1
src/main/java/com/zhiwei/media_data_crawler/crawler/SoCrawlerParse.java
+1
-1
src/main/java/com/zhiwei/media_data_crawler/crawler/SoNewsCrawlerParse.java
+1
-1
src/main/java/com/zhiwei/media_data_crawler/crawler/SougouNewsCrawlerParse.java
+2
-2
src/main/java/com/zhiwei/media_data_crawler/crawler/SougouZhihuCrawlerParse.java
+2
-2
src/main/java/com/zhiwei/media_data_crawler/crawler/TianYaCrawlerParse.java
+1
-1
No files found.
src/main/java/com/zhiwei/media_data_crawler/crawler/BaiduTiebaCrawlerParse.java
View file @
a7d988a7
...
@@ -256,7 +256,7 @@ public class BaiduTiebaCrawlerParse {
...
@@ -256,7 +256,7 @@ public class BaiduTiebaCrawlerParse {
for
(
int
i
=
1
;
i
<=
3
;
i
++)
{
for
(
int
i
=
1
;
i
<=
3
;
i
++)
{
try
{
try
{
Response
response
=
HttpBoot
.
syncCall
(
RequestUtils
.
wrapGet
(
url
,
headerMap
),
proxy
,
false
);
Response
response
=
HttpBoot
.
syncCall
(
RequestUtils
.
wrapGet
(
url
,
headerMap
),
proxy
,
false
);
return
response
.
body
().
toS
tring
();
return
response
.
body
().
s
tring
();
}
catch
(
Exception
e
)
{
}
catch
(
Exception
e
)
{
logger
.
error
(
"获取数据时出现问题,问题为:{}"
,
e
.
fillInStackTrace
());
logger
.
error
(
"获取数据时出现问题,问题为:{}"
,
e
.
fillInStackTrace
());
if
(
i
==
3
){
if
(
i
==
3
){
...
...
src/main/java/com/zhiwei/media_data_crawler/crawler/DoubanCrawlerParse.java
View file @
a7d988a7
...
@@ -94,7 +94,7 @@ public class DoubanCrawlerParse {
...
@@ -94,7 +94,7 @@ public class DoubanCrawlerParse {
for
(
int
i
=
1
;
i
<=
3
;
i
++)
{
for
(
int
i
=
1
;
i
<=
3
;
i
++)
{
try
{
try
{
Response
response
=
HttpBoot
.
syncCall
(
RequestUtils
.
wrapGet
(
url
,
headerMap
),
proxy
,
false
);
Response
response
=
HttpBoot
.
syncCall
(
RequestUtils
.
wrapGet
(
url
,
headerMap
),
proxy
,
false
);
return
response
.
body
().
toS
tring
();
return
response
.
body
().
s
tring
();
}
catch
(
Exception
e
)
{
}
catch
(
Exception
e
)
{
logger
.
error
(
"获取数据时出现问题,问题为:{}"
,
e
.
fillInStackTrace
());
logger
.
error
(
"获取数据时出现问题,问题为:{}"
,
e
.
fillInStackTrace
());
if
(
i
==
3
){
if
(
i
==
3
){
...
...
src/main/java/com/zhiwei/media_data_crawler/crawler/SoCrawlerParse.java
View file @
a7d988a7
...
@@ -104,7 +104,7 @@ public class SoCrawlerParse {
...
@@ -104,7 +104,7 @@ public class SoCrawlerParse {
for
(
int
i
=
1
;
i
<=
3
;
i
++)
{
for
(
int
i
=
1
;
i
<=
3
;
i
++)
{
try
{
try
{
Response
response
=
HttpBoot
.
syncCall
(
RequestUtils
.
wrapGet
(
url
,
headerMap
),
proxy
,
false
);
Response
response
=
HttpBoot
.
syncCall
(
RequestUtils
.
wrapGet
(
url
,
headerMap
),
proxy
,
false
);
return
response
.
body
().
toS
tring
();
return
response
.
body
().
s
tring
();
}
catch
(
Exception
e
)
{
}
catch
(
Exception
e
)
{
logger
.
error
(
"获取360新闻数据时出现问题,问题为:{}"
,
e
.
fillInStackTrace
());
logger
.
error
(
"获取360新闻数据时出现问题,问题为:{}"
,
e
.
fillInStackTrace
());
if
(
i
==
3
){
if
(
i
==
3
){
...
...
src/main/java/com/zhiwei/media_data_crawler/crawler/SoNewsCrawlerParse.java
View file @
a7d988a7
...
@@ -136,7 +136,7 @@ public class SoNewsCrawlerParse {
...
@@ -136,7 +136,7 @@ public class SoNewsCrawlerParse {
for
(
int
i
=
1
;
i
<=
3
;
i
++)
{
for
(
int
i
=
1
;
i
<=
3
;
i
++)
{
try
{
try
{
Response
response
=
HttpBoot
.
syncCall
(
RequestUtils
.
wrapGet
(
url
,
headerMap
),
proxy
,
false
);
Response
response
=
HttpBoot
.
syncCall
(
RequestUtils
.
wrapGet
(
url
,
headerMap
),
proxy
,
false
);
return
response
.
body
().
toS
tring
();
return
response
.
body
().
s
tring
();
}
catch
(
Exception
e
)
{
}
catch
(
Exception
e
)
{
logger
.
error
(
"获取360新闻数据时出现问题,问题为:{}"
,
e
.
fillInStackTrace
());
logger
.
error
(
"获取360新闻数据时出现问题,问题为:{}"
,
e
.
fillInStackTrace
());
if
(
i
==
3
){
if
(
i
==
3
){
...
...
src/main/java/com/zhiwei/media_data_crawler/crawler/SougouNewsCrawlerParse.java
View file @
a7d988a7
...
@@ -129,7 +129,7 @@ public class SougouNewsCrawlerParse {
...
@@ -129,7 +129,7 @@ public class SougouNewsCrawlerParse {
for
(
int
i
=
1
;
i
<=
3
;
i
++){
for
(
int
i
=
1
;
i
<=
3
;
i
++){
try
{
try
{
Response
response
=
HttpBoot
.
syncCall
(
RequestUtils
.
wrapGet
(
url
,
headerMap
),
proxy
,
false
);
Response
response
=
HttpBoot
.
syncCall
(
RequestUtils
.
wrapGet
(
url
,
headerMap
),
proxy
,
false
);
return
response
.
body
().
toS
tring
();
return
response
.
body
().
s
tring
();
}
catch
(
Exception
e
)
{
}
catch
(
Exception
e
)
{
logger
.
error
(
"获取搜狗新闻数据时出现问题,问题为:{}"
,
e
.
fillInStackTrace
());
logger
.
error
(
"获取搜狗新闻数据时出现问题,问题为:{}"
,
e
.
fillInStackTrace
());
if
(
i
==
3
){
if
(
i
==
3
){
...
@@ -153,7 +153,7 @@ public class SougouNewsCrawlerParse {
...
@@ -153,7 +153,7 @@ public class SougouNewsCrawlerParse {
for
(
int
i
=
1
;
i
<=
3
;
i
++){
for
(
int
i
=
1
;
i
<=
3
;
i
++){
try
{
try
{
Response
response
=
HttpBoot
.
syncCall
(
RequestUtils
.
wrapGet
(
url
,
headerMap
),
proxy
,
false
);
Response
response
=
HttpBoot
.
syncCall
(
RequestUtils
.
wrapGet
(
url
,
headerMap
),
proxy
,
false
);
return
response
.
body
().
toS
tring
();
return
response
.
body
().
s
tring
();
}
catch
(
Exception
e
)
{
}
catch
(
Exception
e
)
{
logger
.
error
(
"获取搜狗新闻数据时出现问题,问题为:{}"
,
e
.
fillInStackTrace
());
logger
.
error
(
"获取搜狗新闻数据时出现问题,问题为:{}"
,
e
.
fillInStackTrace
());
if
(
i
==
3
){
if
(
i
==
3
){
...
...
src/main/java/com/zhiwei/media_data_crawler/crawler/SougouZhihuCrawlerParse.java
View file @
a7d988a7
...
@@ -97,7 +97,7 @@ public class SougouZhihuCrawlerParse{
...
@@ -97,7 +97,7 @@ public class SougouZhihuCrawlerParse{
for
(
int
i
=
1
;
i
<=
3
;
i
++){
for
(
int
i
=
1
;
i
<=
3
;
i
++){
try
{
try
{
Response
response
=
HttpBoot
.
syncCall
(
RequestUtils
.
wrapGet
(
url
,
headerMap
),
proxy
,
false
);
Response
response
=
HttpBoot
.
syncCall
(
RequestUtils
.
wrapGet
(
url
,
headerMap
),
proxy
,
false
);
return
response
.
body
().
toS
tring
();
return
response
.
body
().
s
tring
();
}
catch
(
Exception
e
)
{
}
catch
(
Exception
e
)
{
logger
.
error
(
"获取搜狗新闻数据时出现问题,问题为:{}"
,
e
.
fillInStackTrace
());
logger
.
error
(
"获取搜狗新闻数据时出现问题,问题为:{}"
,
e
.
fillInStackTrace
());
if
(
i
==
3
){
if
(
i
==
3
){
...
@@ -124,7 +124,7 @@ public class SougouZhihuCrawlerParse{
...
@@ -124,7 +124,7 @@ public class SougouZhihuCrawlerParse{
for
(
int
i
=
1
;
i
<=
3
;
i
++){
for
(
int
i
=
1
;
i
<=
3
;
i
++){
try
{
try
{
Response
response
=
HttpBoot
.
syncCall
(
RequestUtils
.
wrapGet
(
url
,
headerMap
),
proxy
,
false
);
Response
response
=
HttpBoot
.
syncCall
(
RequestUtils
.
wrapGet
(
url
,
headerMap
),
proxy
,
false
);
return
response
.
body
().
toS
tring
();
return
response
.
body
().
s
tring
();
}
catch
(
Exception
e
)
{
}
catch
(
Exception
e
)
{
logger
.
error
(
"获取搜狗新闻数据时出现问题,问题为:{}"
,
e
.
fillInStackTrace
());
logger
.
error
(
"获取搜狗新闻数据时出现问题,问题为:{}"
,
e
.
fillInStackTrace
());
if
(
i
==
3
){
if
(
i
==
3
){
...
...
src/main/java/com/zhiwei/media_data_crawler/crawler/TianYaCrawlerParse.java
View file @
a7d988a7
...
@@ -87,7 +87,7 @@ public class TianYaCrawlerParse {
...
@@ -87,7 +87,7 @@ public class TianYaCrawlerParse {
for
(
int
i
=
1
;
i
<=
3
;
i
++)
{
for
(
int
i
=
1
;
i
<=
3
;
i
++)
{
try
{
try
{
Response
response
=
HttpBoot
.
syncCall
(
RequestUtils
.
wrapGet
(
url
,
headerMap
),
proxy
,
false
);
Response
response
=
HttpBoot
.
syncCall
(
RequestUtils
.
wrapGet
(
url
,
headerMap
),
proxy
,
false
);
return
response
.
body
().
toS
tring
();
return
response
.
body
().
s
tring
();
}
catch
(
Exception
e
)
{
}
catch
(
Exception
e
)
{
logger
.
error
(
"获取数据时出现问题,问题为:{}"
,
e
.
fillInStackTrace
());
logger
.
error
(
"获取数据时出现问题,问题为:{}"
,
e
.
fillInStackTrace
());
if
(
i
==
3
){
if
(
i
==
3
){
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment