Skip to content
Projects
Groups
Snippets
Help
This project
Loading...
Sign in / Register
Toggle navigation
T
toutiao
Overview
Overview
Details
Activity
Cycle Analytics
Repository
Repository
Files
Commits
Branches
Tags
Contributors
Graph
Compare
Charts
Issues
0
Issues
0
List
Board
Labels
Milestones
Merge Requests
0
Merge Requests
0
CI / CD
CI / CD
Pipelines
Jobs
Schedules
Charts
Wiki
Wiki
Snippets
Snippets
Members
Members
Collapse sidebar
Close sidebar
Activity
Graph
Charts
Create a new issue
Jobs
Commits
Issue Boards
Open sidebar
zhiwei
toutiao
Commits
a5f5a270
Commit
a5f5a270
authored
Jul 22, 2019
by
[zhangzhiwei]
Browse files
Options
Browse Files
Download
Email Patches
Plain Diff
修改部分命名不规范及升级版本
parent
22ad3587
Hide whitespace changes
Inline
Side-by-side
Showing
4 changed files
with
9 additions
and
10 deletions
+9
-10
pom.xml
+1
-1
src/main/java/com/zhiwei/toutiao/parse/TouTiaoAccountParse.java
+6
-6
src/main/java/com/zhiwei/toutiao/parse/TouTiaoCommentParse.java
+2
-2
src/main/java/com/zhiwei/toutiao/parse/TouTiaoParse.java
+0
-1
No files found.
pom.xml
View file @
a5f5a270
...
...
@@ -3,7 +3,7 @@
<modelVersion>
4.0.0
</modelVersion>
<groupId>
com.zhiwei
</groupId>
<artifactId>
toutiao
</artifactId>
<version>
0.3.
6
-SNAPSHOT
</version>
<version>
0.3.
7
-SNAPSHOT
</version>
<dependencies>
<dependency>
...
...
src/main/java/com/zhiwei/toutiao/parse/TouTiaoAccountParse.java
View file @
a5f5a270
...
...
@@ -139,12 +139,12 @@ public class TouTiaoAccountParse {
* @return List<TouTiaoAccount> 返回类型
*/
@Deprecated
public
static
List
<
TouTiaoAccount
>
getFriendsList
(
String
user
i
d
,
Proxy
proxy
,
long
sleep
){
public
static
List
<
TouTiaoAccount
>
getFriendsList
(
String
user
I
d
,
Proxy
proxy
,
long
sleep
){
List
<
TouTiaoAccount
>
ttaList
=
new
ArrayList
<>();
boolean
more
=
true
;
int
page
=
0
;
while
(
more
){
String
url
=
"http://it-hl.snssdk.com/user/relation/following/v2/?user_id="
+
userid
+
"&device_id=54560738994&cursor=&iid=53238029655&offset="
+
page
*
50
+
"&count=50&ts="
+
System
.
currentTimeMillis
()/
1000
;
String
url
=
"http://it-hl.snssdk.com/user/relation/following/v2/?user_id="
+
userId
+
"&device_id=54560738994&cursor=&iid=53238029655&offset="
+
page
*
50
+
"&count=50&ts="
+
System
.
currentTimeMillis
()/
1000
;
page
++;
headerMap
=
Tools
.
getTouTiaoHeader
();
headerMap
.
put
(
"User-Agent"
,
"Dalvik/2.1.0 (Linux; U; Android 8.1.0; MI 8 MIUI/V10.0.11.0.OEACNFH) NewsArticle/7.0.1 cronet/TTNetVersion:pre_blink_merge-277498-gd2bb364e 2018-08-24"
);
...
...
@@ -173,15 +173,15 @@ public class TouTiaoAccountParse {
}
public
static
List
<
TouTiaoAccount
>
getFriendsList
(
String
user
i
d
,
ProxyHolder
proxy
){
public
static
List
<
TouTiaoAccount
>
getFriendsList
(
String
user
I
d
,
ProxyHolder
proxy
){
List
<
TouTiaoAccount
>
ttaList
=
new
ArrayList
<>();
Signature
signature
=
new
Signature
(
user
i
d
,
"0"
);
Signature
signature
=
new
Signature
(
user
I
d
,
"0"
);
String
signatureStr
=
signature
.
getSignature
();
boolean
more
=
true
;
while
(
more
){
String
url
=
"https://www.toutiao.com/c/user/following/?user_id="
+
userid
+
"&cursor=0&count=100&_signature="
+
signatureStr
;
String
url
=
"https://www.toutiao.com/c/user/following/?user_id="
+
userId
+
"&cursor=0&count=100&_signature="
+
signatureStr
;
headerMap
=
Tools
.
getTouTiaoHeader
();
headerMap
.
put
(
"referer"
,
"ihttps://www.toutiao.com/c/user/relation/"
+
user
i
d
+
"/?tab=following"
);
headerMap
.
put
(
"referer"
,
"ihttps://www.toutiao.com/c/user/relation/"
+
user
I
d
+
"/?tab=following"
);
headerMap
.
put
(
"user-agent"
,
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/71.0.3578.80 Safari/537.36"
);
for
(
int
i
=
0
;
i
<
3
;
i
++){
try
{
...
...
src/main/java/com/zhiwei/toutiao/parse/TouTiaoCommentParse.java
View file @
a5f5a270
...
...
@@ -138,9 +138,9 @@ public class TouTiaoCommentParse {
* @return int 返回类型
* @throws Exception
*/
private
static
int
getPage
(
String
group
_i
d
,
Proxy
proxy
)
throws
Exception
private
static
int
getPage
(
String
group
I
d
,
Proxy
proxy
)
throws
Exception
{
String
urlNew
=
"http://www.toutiao.com/api/comment/list/?group_id="
+
group_id
+
"&item_id=0&count=20&offset=0"
;
String
urlNew
=
"http://www.toutiao.com/api/comment/list/?group_id="
+
groupId
+
"&item_id=0&count=20&offset=0"
;
//设置头信息
Map
<
String
,
String
>
headerMap
=
Tools
.
getTouTiaoHeader
();
String
htmlBody
=
downloadHtml
(
urlNew
,
proxy
,
headerMap
);
...
...
src/main/java/com/zhiwei/toutiao/parse/TouTiaoParse.java
View file @
a5f5a270
...
...
@@ -24,7 +24,6 @@ import org.jsoup.nodes.Document;
import
com.alibaba.fastjson.JSONArray
;
import
com.alibaba.fastjson.JSONObject
;
import
com.zhiwei.crawler.core.HttpBoot
;
import
com.zhiwei.tools.timeparse.TimeParse
;
import
com.zhiwei.toutiao.bean.TouTiaoArticle
;
...
...
Write
Preview
Markdown
is supported
0%
Try again
or
attach a new file
Attach a file
Cancel
You are about to add
0
people
to the discussion. Proceed with caution.
Finish editing this message first!
Cancel
Please
register
or
sign in
to comment