Skip to content
Projects
Groups
Snippets
Help
This project
Loading...
Sign in / Register
Toggle navigation
S
source_forward
Overview
Overview
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
zhiwei
source_forward
Commits
9fcfba2d
Commit
9fcfba2d
authored
Aug 13, 2020
by
zhiwei
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
各个采集验证添加休眠,避免数据过多导致程序阻塞
parent
aa059934
Show whitespace changes
Inline
Side-by-side
Showing
5 changed files
with
7 additions
and
1 deletions
+7
-1
src/main/java/com/zhiwei/source_forward/crawler/ContentCrawler.java
+2
-0
src/main/java/com/zhiwei/source_forward/crawler/MediaSelfSourceCrawler.java
+2
-0
src/main/java/com/zhiwei/source_forward/crawler/SourceForwardCrawler.java
+2
-0
src/main/java/com/zhiwei/source_forward/crawler/UrlLiveCrawler.java
+1
-0
src/main/java/com/zhiwei/source_forward/util/MatchSource.java
+0
-1
No files found.
src/main/java/com/zhiwei/source_forward/crawler/ContentCrawler.java
View file @
9fcfba2d
...
...
@@ -2,6 +2,7 @@ package com.zhiwei.source_forward.crawler;
import
java.util.Objects
;
import
com.zhiwei.tools.tools.ZhiWeiTools
;
import
org.apache.logging.log4j.LogManager
;
import
org.apache.logging.log4j.Logger
;
...
...
@@ -47,6 +48,7 @@ public class ContentCrawler {
ContentDataCallback
callback
,
String
...
urls
)
{
if
(
urls
!=
null
&&
urls
.
length
>
0
)
{
for
(
String
url
:
urls
)
{
ZhiWeiTools
.
sleep
(
100
);
if
(
url
!=
null
)
{
try
{
search
(
counter
,
url
,
Attribution
.
of
(
url
),
callback
);
...
...
src/main/java/com/zhiwei/source_forward/crawler/MediaSelfSourceCrawler.java
View file @
9fcfba2d
...
...
@@ -5,6 +5,7 @@ import java.util.List;
import
java.util.Map
;
import
java.util.Objects
;
import
com.zhiwei.tools.tools.ZhiWeiTools
;
import
org.apache.logging.log4j.LogManager
;
import
org.apache.logging.log4j.Logger
;
import
org.jsoup.Jsoup
;
...
...
@@ -64,6 +65,7 @@ public class MediaSelfSourceCrawler {
private
void
start
(
GroupSync
counter
,
MediaSelfSourceDataCallBack
callback
,
String
...
urls
)
{
if
(
urls
!=
null
&&
urls
.
length
>
0
)
{
for
(
String
url
:
urls
)
{
ZhiWeiTools
.
sleep
(
100
);
counter
.
add
();
if
(
url
!=
null
)
{
try
{
...
...
src/main/java/com/zhiwei/source_forward/crawler/SourceForwardCrawler.java
View file @
9fcfba2d
...
...
@@ -5,6 +5,7 @@ import java.util.List;
import
java.util.Map
;
import
java.util.Objects
;
import
com.zhiwei.tools.tools.ZhiWeiTools
;
import
org.apache.logging.log4j.LogManager
;
import
org.apache.logging.log4j.Logger
;
import
org.jsoup.Jsoup
;
...
...
@@ -46,6 +47,7 @@ public class SourceForwardCrawler {
if
(
urls
!=
null
&&
urls
.
length
>
0
)
{
for
(
String
url
:
urls
)
{
counter
.
add
();
ZhiWeiTools
.
sleep
(
100
);
if
(
url
!=
null
)
{
try
{
search
(
counter
,
url
,
Attribution
.
of
(
url
),
callback
);
...
...
src/main/java/com/zhiwei/source_forward/crawler/UrlLiveCrawler.java
View file @
9fcfba2d
...
...
@@ -50,6 +50,7 @@ public class UrlLiveCrawler {
if
(
nonNull
(
urls
)
&&
urls
.
length
>
0
)
{
for
(
String
url
:
urls
)
{
counter
.
add
();
ZhiWeiTools
.
sleep
(
100
);
if
(
nonNull
(
url
))
{
try
{
// ZhiWeiTools.sleep(3000);
...
...
src/main/java/com/zhiwei/source_forward/util/MatchSource.java
View file @
9fcfba2d
...
...
@@ -325,7 +325,6 @@ public class MatchSource {
source
=
source
.
replaceAll
(
".*来源:|)"
,
""
);
}
}
if
(
Objects
.
nonNull
(
source
)
&&
source
.
length
()
!=
0
)
{
return
source
;
}
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment