Skip to content
Projects
Groups
Snippets
Help
This project
Loading...
Sign in / Register
Toggle navigation
W
weibohotcrawler
Overview
Overview
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
chenweiyang
weibohotcrawler
Commits
a40e931c
Commit
a40e931c
authored
Oct 09, 2018
by
[zhangzhiwei]
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
删除无用的log
parent
d9b8652d
Show whitespace changes
Inline
Side-by-side
Showing
3 changed files
with
2 additions
and
5 deletions
+2
-5
src/main/java/com/zhiwei/weibocrawler/crawler/DataUpdate.java
+2
-2
src/main/java/com/zhiwei/weibocrawler/httpclient/HttpClientDemo.java
+0
-2
src/main/java/com/zhiwei/weibocrawler/rsidClient/RsidClientDAO.java
+0
-1
No files found.
src/main/java/com/zhiwei/weibocrawler/crawler/DataUpdate.java
View file @
a40e931c
...
...
@@ -45,7 +45,7 @@ public class DataUpdate implements Runnable{
List
<
String
>
midList
=
DataQueue
.
get
(
48
);
if
(
midList
!=
null
&&
midList
.
size
()
>
0
)
{
List
<
Status
>
list
=
WeiboCrawlerAnalysis
.
getWeiboData
(
midList
,
token
);
logger
.
info
(
"更新数据量"
+
list
.
size
());
//
logger.info("更新数据量" + list.size());
for
(
Status
status
:
list
)
{
Date
date
=
new
Date
(
new
Date
().
getTime
()-
24
*
60
*
60
*
1000
);
if
(
status
.
getCreatedAt
().
after
(
date
)){
...
...
@@ -55,7 +55,7 @@ public class DataUpdate implements Runnable{
}
i
=
1
;
}
else
{
logger
.
error
(
"目前数据量不足50,目前队列中得数据量为:::{}"
,
DataQueue
.
linkQueue
.
size
());
// logger.info
("目前数据量不足50,目前队列中得数据量为:::{}", DataQueue.linkQueue.size());
i
++;
}
}
catch
(
Exception
e
)
{
...
...
src/main/java/com/zhiwei/weibocrawler/httpclient/HttpClientDemo.java
View file @
a40e931c
...
...
@@ -23,11 +23,9 @@ public class HttpClientDemo {
headerMap
.
put
(
"Content-Type"
,
"application/x-www-form-urlencoded"
);
headerMap
.
put
(
"Host"
,
"weibo.com"
);
try
{
System
.
out
.
println
(
"开始下载"
);
// Response response = HttpBoot.syncCall(RequestUtils.wrapGet(url, headerMap));
// result = response.body().string();
result
=
HttpClientTemplateOK
.
get
(
url
,
null
,
headerMap
);
System
.
out
.
println
(
"下载结束"
);
return
result
;
}
catch
(
Exception
e
)
{
...
...
src/main/java/com/zhiwei/weibocrawler/rsidClient/RsidClientDAO.java
View file @
a40e931c
...
...
@@ -44,7 +44,6 @@ public class RsidClientDAO {
for
(
int
i
=
0
;
i
<
3
;
i
++){
try
{
boolean
f
=
client
.
contains
(
mid
,
PTENUM
.
COMMON
);
System
.
out
.
println
(
mid
+
"==========="
+
f
);
return
f
;
}
catch
(
Exception
e
)
{
logger
.
error
(
"判断此条微博消息是否存在出现问题"
,
e
.
fillInStackTrace
());
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment