coder-xiaomo/tools
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity GitHub Gitee

合并仓库前的准备

Browse Source
This commit is contained in:
小墨
2022-10-17 13:09:59 +08:00
parent c11080a6f6
commit 04eb563794
29 changed files with 78 additions and 78 deletions
Download Patch File Download Diff File Expand all files Collapse all files

20
hotband/.env.example Normal file
View File

@@ -0,0 +1,20 @@
# 调试模式
# 1为开启调试
DEBUG_MODE=1
# 爬取数据保存的文件夹
# 目录开头与结尾的 [./] [/] [\] [\\] 均可带可不带
# 默认为 data 文件夹
DATA_FOLDER=data
# 是否在程序刚一启动时就抓取一次数据
# 1为是
EXECUTE_AT_STARTUP=1
# 数据是否推送到Git仓库
# 1为是
PUSH_TO_GIT=0
# 是否仅保存 latest.json 而不保存其他文件作为存档
# 1为是
LATEST_DATA_ONLY=0

8
hotband/.gitignore vendored Normal file
View File

@@ -0,0 +1,8 @@
.DS_Store
data/*
.env
node_modules
.VSCodeCounter
test.js

247
hotband/README.md Normal file
View File

@@ -0,0 +1,247 @@
# 热搜数据爬取工具
> 本仓库中代码仅供学习研究使用，不得用于违法用途，学习使用完毕后请于24小时内删除。
>
> 数据来自微博、B站（详见下方「数据来源」），本项目不对数据真实性做验证，使用数据时请遵守相关平台的相关限制要求。
## 简介
您可以将本项目代码部署在服务器上（在本地运行也可），程序会每隔一分钟拉取一次热搜数据，并保存为 `json` 格式文件。
## 数据预览
在部署并启动项目后，您可以在浏览器中打开 `html/index.html` 文件实时预览当前热搜。
## 数据来源
**微博热搜**
页面：https://weibo.com/hot/search
接口：https://weibo.com/ajax/statuses/hot_band
**B站热搜**
页面：https://www.bilibili.com/blackboard/activity-trending-topic.html
接口：https://app.bilibili.com/x/v2/search/trending/ranking
**B站排行榜**
页面：https://www.bilibili.com/v/popular/rank/all
接口：https://api.bilibili.com/x/web-interface/ranking/v2?type=all
(切换到其他榜单再切换回来会调用此接口)
## 运行环境
原理上来说 Windows 下和 Linux 都可运行，目前仅在 Windows 下测试过，暂未在 Linux 系统下测试。
项目使用 node 开发，以下部署流程默认您已安装了 `Git`、`Nodejs`。
## 部署
1. 克隆仓库（或直接下载压缩包）
```bash
git clone https://git.only4.work/coder-xiaomo/weibo-hotband
```
2. 安装依赖
```bash
npm i
```
3. 修改配置文件
将项目目录下的 `.env.example` 文件复制一份，并改名为 `.env`，使用文本编辑器打开（例如：记事本、VS Code、vim等均可），根据其中的注释说明来进行配置即可。
> 如果不创建 .env 文件，项目启动时会报如下错误并退出。
>
> ```bash
> [ERROR] .env file not found!
> ```
4. 启动项目
```bash
# 直接运行
# node index.js
# 使用 pm2
# pm2 start index.js --name weibo-hotband-bot
```
5. 停止项目
```bash
# 使用 node index.js 命令直接运行的项目可以通过 `Ctrl + C` 停止
# 使用 pm2 运行的可以使用以下两行命令来停止和从列表中删除项目
# pm2 stop weibo-hotband-bot
# pm2 delete weibo-hotband-bot
```
## 说明
项目爬取的数据默认保存在项目目录下的 data 文件夹中，您也可以通过修改 `.env` 文件中的 `DATA_FOLDER` 参数值来自定义数据保存路径。
### 微博热搜榜
> 微博热搜在 `weibo_hotband` 子文件夹下
在程序运行后，该文件夹下会出现 `latest.json` 文件及其余几个文件夹，这些子文件夹中的文件按照以下格式保存：`年/月/日/年月日_时分.json`。
每次爬取后，`latest.json`中的数据都会被覆盖为最新的热搜数据。
`origin` 文件夹中的数据是通过Api接口获取到的原始数据，没有经过任何处理。
<!--
`simplify` 文件夹中的数据是在原始数据的基础上，去除了部分冗余数据。
-->
`final` 文件夹中的数据是从原始数据中抽离出的有用数据，并重新整理得到的。
<!--
`regulation` 文件夹中的数据主要用于观测原始值与显示值不同的热搜，这部分热搜猜测可能是经过微博平台调控的。（这部分数据没有太大意义，可以忽略）
-->
### B站热搜榜
> 微博热搜在 `bilibili_hotband` 子文件夹下
在程序运行后，该文件夹下会出现 `latest.json` 文件及其余几个文件夹，这些子文件夹中的文件按照以下格式保存：`年/月/日/年月日_时分.json`。
每次爬取后，`latest.json`中的数据都会被覆盖为最新的热搜数据。
`origin` 文件夹中的数据是通过Api接口获取到的原始数据，此处仅仅去除了 `trackid`。
`final` 文件夹中的数据是从原始数据中抽离出的有用数据，并重新整理得到的。
### B站排行榜
> 微博热搜在 `bilibili_rank` 子文件夹下
在程序运行后，该文件夹下会出现 `latest.json` 文件及其余几个文件夹，这些子文件夹中的文件按照以下格式保存：`年/月/日/年月日_时分.json`。
每次爬取后，`latest.json`中的数据都会被覆盖为最新的热搜数据。
`origin` 文件夹中的数据是通过Api接口获取到的原始数据，没有经过任何处理。
## 目录结构
### 项目目录结构
```bash
hotband // 本项目
├─ data // 爬取的数据（启动项目后自动创建）
├─ html // html 页面
│ ├─ assets
│ │ ├─ css // CSS 样式
│ │ │ └─
│ │ ├─ image // 前端图片资源
│ │ │ ├─ ...
│ │ └─ js
│ │ └─ isMobile.js
│ ├─ bilibili_hotband.html
│ ├─ bilibili_rank.html
│ └─ weibo_hotband.html
├─ src // 数据爬取核心代码
│ ├─ utils // 工具类代码
│ │ ├─ fileUtils.js
│ │ └─ requestUtils.js
│ ├─ execute_command.js // 执行命令行脚本（暂时没用到）
│ ├─ get_bilibili_hotband.js // 获取 B站热搜榜 代码
│ ├─ get_bilibili_rank.js // 爬取 B站排行榜 代码
│ └─ get_weibo_hotband.js // 爬取 微博热搜榜 代码
├─ .env.example // 项目配置文件模板
├─ .env // 项目配置文件（需要自行创建）
├─ index.html // html 页面打开文件
├─ index.js // node 项目启动入口文件
├─ nodemon.json
├─ package-lock.json
├─ package.json
├─ pm2 restart.bat
├─ pm2 restart.sh
├─ pm2 start.bat
├─ pm2 start.sh
├─ pm2 stop.bat
├─ pm2 stop.sh
└─ README.md // 项目自述文件
```
### data 目录结构
data 文件夹下的目录结构如下
```bash
data
├─ bilibili-hotband
│ ├─ final / origin
│ │ └─ xxxx // 年
│ │ └─ xx // 月
│ │ └─ xx // 日
│ │ ├─ xxxxxxxx_xxxx.min.json // 年月日_时分秒.min.json
│ └─ latest.json // 最新的json文件
├─ bilibili-rank
│ ├─ origin
│ │ └─ xxxx // 年
│ │ └─ xx // 月
│ │ └─ xx // 日
│ │ ├─ xxxxxxxx_xxxx.min.json // 年月日_时分秒.min.json
│ └─ latest.json // 最新的json文件
└─ weibo-hotband
├─ origin / final / simplify
│ └─ xxxx // 年
│ └─ xx // 月
│ └─ xx // 日
│ ├─ xxxxxxxx_xxxx.min.json // 年月日_时分秒.min.json
├─ regulation
│ └─ xxxx // 年
│ └─ xx // 月
│ └─ xx // 日
│ ├─ xxxxxxxx_xxxx.json // 年月日_时分秒.json
└─ latest.json // 最新的json文件
```
### 题外话：怎么生成目录结构？
> 有很多小伙伴在问像上方的目录结构是如何生成的，这里跟大家说下：
>
> 1. Windows 下可以通过 `tree` 命令来生成，例如：
>
> ```bash
> tree /f > xxx.txt
> ```
>
> 2. 使用 VS Code 插件
>
> 我使用的是 [tree-generator](https://marketplace.visualstudio.com/items?itemName=xboxyan.tree-generator) 这个插件，安装之后直接在文件夹上右键即可生成
>
> 3. 另外还有一些其他方法也可以生成，大家可以自己探索。

34
hotband/html/assets/css/main.css Normal file
View File

@@ -0,0 +1,34 @@
#list {
width: 100%;
text-align: center;
border-spacing: 0;
border: 0.4px solid black;
}
#list tr {
height: min(1.85rem, 50px);
}
#list td {
margin: 0;
border: 0.4px solid black;
}
/* 热搜的 label 样式 */
.hotband-label {
color: white;
padding: 3px;
border-radius: 6px;
font-size: 10px;
display: inline-block;
}
.bottom-placeholder {
height: 90px;
font-size:12px;
color: #999;
display: grid;
place-items: center;
text-align: center;
line-height: 1.7em;
}

1
hotband/html/assets/image/bilibili.svg Normal file
View File

@@ -0,0 +1 @@
<?xml version="1.0" standalone="no"?><!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg viewBox="0 0 1129 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="220" height="200"><path d="M234.909 9.656a80.468 80.468 0 0 1 68.398 0 167.374 167.374 0 0 1 41.843 30.578l160.937 140.82h115.07l160.936-140.82a168.983 168.983 0 0 1 41.843-30.578A80.468 80.468 0 0 1 930.96 76.445a80.468 80.468 0 0 1-17.703 53.914 449.818 449.818 0 0 1-35.406 32.187 232.553 232.553 0 0 1-22.531 18.508h100.585a170.593 170.593 0 0 1 118.289 53.109 171.397 171.397 0 0 1 53.914 118.288v462.693a325.897 325.897 0 0 1-4.024 70.007 178.64 178.64 0 0 1-80.468 112.656 173.007 173.007 0 0 1-92.539 25.75h-738.7a341.186 341.186 0 0 1-72.421-4.024A177.835 177.835 0 0 1 28.91 939.065a172.202 172.202 0 0 1-27.36-92.539V388.662a360.498 360.498 0 0 1 0-66.789A177.03 177.03 0 0 1 162.487 178.64h105.414c-16.899-12.07-31.383-26.555-46.672-39.43a80.468 80.468 0 0 1-25.75-65.984 80.468 80.468 0 0 1 39.43-63.57M216.4 321.873a80.468 80.468 0 0 0-63.57 57.937 108.632 108.632 0 0 0 0 30.578v380.615a80.468 80.468 0 0 0 55.523 80.469 106.218 106.218 0 0 0 34.601 5.632h654.208a80.468 80.468 0 0 0 76.444-47.476 112.656 112.656 0 0 0 8.047-53.109v-354.06a135.187 135.187 0 0 0 0-38.625 80.468 80.468 0 0 0-52.304-54.719 129.554 129.554 0 0 0-49.89-7.242H254.22a268.764 268.764 0 0 0-37.82 0z m0 0" fill="#20B0E3"></path><path d="M348.369 447.404a80.468 80.468 0 0 1 55.523 18.507 80.468 80.468 0 0 1 28.164 59.547v80.468a80.468 80.468 0 0 1-16.094 51.5 80.468 80.468 0 0 1-131.968-9.656 104.609 104.609 0 0 1-10.46-54.719v-80.468a80.468 80.468 0 0 1 70.007-67.593z m416.02 0a80.468 80.468 0 0 1 86.102 75.64v80.468a94.148 94.148 0 0 1-12.07 53.11 80.468 80.468 0 0 1-132.773 0 95.757 95.757 0 0 1-12.875-57.133V519.02a80.468 80.468 0 0 1 70.007-70.812z m0 0" fill="#20B0E3"></path></svg>
Side by Side

After

Width:  |  Height:  |  Size: 1.9 KiB

BIN
hotband/html/assets/image/icon_rank.png Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.9 KiB

1
hotband/html/assets/image/weibo.svg Normal file
View File

@@ -0,0 +1 @@
<?xml version="1.0" standalone="no"?><!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN" "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd"><svg viewBox="0 0 1026 1024" version="1.1" xmlns="http://www.w3.org/2000/svg" p-id="2278" xmlns:xlink="http://www.w3.org/1999/xlink" width="200" height="200"><path d="M1012.49 451.553v0.159c-6.697 20.66-28.861 31.99-49.449 25.288a39.352 39.352 0 0 1-25.287-49.582l-0.067-0.031c20.536-63.6 7.516-136.156-40.315-189.363-47.892-53.212-118.502-73.554-183.731-59.659-21.222 4.537-42.133-9.047-46.638-30.3-4.506-21.253 9.021-42.194 30.239-46.73 91.709-19.563 191.114 8.98 258.467 83.881 67.36 74.839 85.515 176.85 56.781 266.337z" fill="#D32024"></path><path d="M740.429 304.348v-0.03c-18.217 3.973-36.178-7.732-40.06-26.01-3.947-18.31 7.763-36.373 25.98-40.254 44.692-9.548 93.143 4.322 125.885 40.781 32.866 36.496 41.631 86.17 27.607 129.772a33.833 33.833 0 0 1-42.562 21.847c-17.782-5.76-27.484-24.914-21.724-42.69h-0.062c6.887-21.346 2.565-45.635-13.46-63.473-16.026-17.818-39.752-24.546-61.604-19.943z m30.05 192.184c-14.46-4.352-24.352-7.326-16.774-26.352 16.333-41.313 18.027-76.964 0.317-102.385-33.31-47.734-124.451-45.133-228.838-1.28 0-0.061-32.799 14.367-24.412-11.704 16.056-51.774 13.645-95.186-11.361-120.192-56.658-56.878-207.304 2.12-336.477 131.64C56.187 463.32 0 566.14 0 655.1 0 825.18 217.503 928.594 430.28 928.594c278.917 0 464.527-162.504 464.527-291.59 0-77.936-65.546-122.193-124.329-140.472zM430.842 867.62c-169.774 16.84-316.35-60.155-327.368-171.96-11.049-111.74 117.72-216.034 287.488-232.873 169.805-16.84 316.355 60.16 327.368 171.904 11.018 111.866-117.683 216.09-287.488 232.929z" fill="#D32024"></path><path d="M447.805 548.859c-80.783-21.09-172.119 19.287-207.206 90.65-35.743 72.862-1.188 153.681 80.44 180.1 84.578 27.357 184.233-14.525 218.88-93.148 34.181-76.81-8.478-155.94-92.114-177.602zM386.12 734.792c-16.43 26.29-51.584 37.806-78.065 25.661-26.107-11.889-33.833-42.44-17.403-68.045 16.215-25.538 50.207-36.869 76.498-25.856 26.604 11.392 35.087 41.687 18.97 68.24z" fill="#D32024"></path></svg>
Side by Side

After

Width:  |  Height:  |  Size: 2.0 KiB

24
hotband/html/assets/js/isMobile.js Normal file
View File

@@ -0,0 +1,24 @@
function isMobile() {
var userAgentInfo = navigator.userAgent;
var mobileAgents = ["Android", "iPhone", "SymbianOS", "Windows Phone", "iPad", "iPod"];
var mobile_flag = false;
//根据userAgent判断是否是手机
for (var v = 0; v < mobileAgents.length; v++) {
if (userAgentInfo.indexOf(mobileAgents[v]) > 0) {
mobile_flag = true;
break;
}
}
// var screen_width = window.screen.width;
// var screen_height = window.screen.height;
// //根据屏幕分辨率判断是否是手机
// if (screen_width > 325 && screen_height < 750) {
// mobile_flag = true;
// }
return mobile_flag;
}

310
hotband/html/bilibili_hotband.html Normal file
View File

@@ -0,0 +1,310 @@
<!DOCTYPE html>
<html lang="cn">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>B站热搜</title>
<link rel="stylesheet" href="./assets/css/main.css">
<style>
#list td {
font-size: 14px;
}
#mobile-info {
color: grey;
font-size: 13px;
text-align: center;
}
</style>
</head>
<body>
<div style="text-align: center;">
<h1>B站热搜榜</h1>
<hr>
<div>
显示字段：
<label for="show_keyword">
<input type="checkbox" class="filter_checkbox" name="show_keyword" id="show_keyword" checked="true"
detailed-checked="true">关键词
</label>
<label for="show_word_type">
<input type="checkbox" class="filter_checkbox" name="show_word_type" id="show_word_type"
detailed-checked="true">word_type
</label>
<label for="show_hot_id">
<input type="checkbox" class="filter_checkbox" name="show_hot_id" id="show_hot_id"
detailed-checked="true">hot_id
</label>
<br>
<button id="btn_show_all">全选</button>
<button id="btn_show_none">全不选</button>
|
<button id="btn_show_default">普通</button>
<button id="btn_show_detailed">详细</button>
</div>
</div>
<p id="latestUpdateTime" style="font-size: 12px; display: inline-block; vertical-align: middle;"></p>
<nobr>
<button id="btn_refresh">重新拉取</button>
</nobr>
<nobr>
<label for="auto_refresh">
<input type="checkbox" name="auto_refresh" id="auto_refresh">自动拉取<span id="auto_refresh_countdown"></span>
</label>
</nobr>
<nobr>
<span id="update-finish-info" style="color: green; font-weight: bold; display: none;">拉取成功，数据已更新</span>
</nobr>
<table id="list"></table>
<p id="mobile-info"></p>
<div class="bottom-placeholder">
<p>
— 到底啦 —<br>
数据来源: https://www.bilibili.com/blackboard/activity-trending-topic.html
</p>
</div>
<script src="./assets/js/isMobile.js"></script>
<script>
let mobileFlag = isMobile();
document.getElementById('mobile-info').innerHTML = `手机、电脑端热搜链接不同，当前为您呈现 <span style="font-weight: bold;">${mobileFlag ? '手机端' : '电脑端'}</span> 链接`;
</script>
<script>
let iconMapper = {
"http://i0.hdslb.com/bfs/feed-admin/e9e7a2d8497d4063421b685e72680bf1cfb99a0d.png": ["热", "#FF895C"],
"http://i0.hdslb.com/bfs/feed-admin/4d579fb61f9655316582db193118bba3a721eec0.png": ["新", "#F87399"],
}
function getIconText(iconUrl) {
if (!iconUrl) {
return '';
}
else if (Object.keys(iconMapper).indexOf(iconUrl) != -1) {
return iconMapper[iconUrl];
} else {
console.log("未知 iconUrl:", iconUrl);
return ["&nbsp;?&nbsp;", 'grey'];
}
}
</script>
<script>
/**
* 全局变量
*/
// 拉取下来的数据
let hotBandData;
// 按钮
const btnShowAll = document.getElementById('btn_show_all');
const btnShowNone = document.getElementById('btn_show_none');
const btnShowDefault = document.getElementById('btn_show_default');
const btnShowDetailed = document.getElementById('btn_show_detailed');
const btnRefresh = document.getElementById('btn_refresh');
// 复选框
const filterCheckbox = document.getElementsByClassName("filter_checkbox");
const showKeyword = document.getElementById("show_keyword");
const showWordType = document.getElementById("show_word_type");
const showHotId = document.getElementById("show_hot_id");
const autoRefresh = document.getElementById("auto_refresh");
// 绑定按钮点击事件
btnShowAll.addEventListener('click', function () {
for (let i = 0; i < filterCheckbox.length; i++) {
const element = filterCheckbox[i];
element.checked = true;
}
render();
});
btnShowNone.addEventListener('click', function () {
for (let i = 0; i < filterCheckbox.length; i++) {
const element = filterCheckbox[i];
element.checked = false;
}
render();
});
btnShowDefault.addEventListener('click', function () {
for (let i = 0; i < filterCheckbox.length; i++) {
const element = filterCheckbox[i];
element.checked = element.getAttribute('checked') === 'true';
}
render();
});
btnShowDetailed.addEventListener('click', function () {
for (let i = 0; i < filterCheckbox.length; i++) {
const element = filterCheckbox[i];
element.checked = element.getAttribute('detailed-checked') === 'true';
}
render();
});
btnRefresh.onclick = function () {
getData();
document.getElementById("update-finish-info").style.display = "";
// btnRefresh.style.display = "none";
btnRefresh.style.visibility = "hidden";
setTimeout(function () {
document.getElementById("update-finish-info").style.display = "none";
// btnRefresh.style.display = "";
btnRefresh.style.visibility = "";
}, 1000);
};
// 绑定复选框改变事件
for (let i = 0; i < filterCheckbox.length; i++) {
// console.log(filterCheckbox[i]);
filterCheckbox[i].onchange = function () {
render();
};
}
let autoRefreshIntreval = null;
const autoRefreshCountDownElement = document.getElementById('auto_refresh_countdown');
const countDown = 20; // 自动拉取间隔时间，单位：秒
let autoRefreshCountDown = countDown;
autoRefresh.onchange = function () {
if (autoRefresh.checked) {
btnRefresh.style.display = "none";
btnRefresh.click();
autoRefreshIntreval = setInterval(function () {
if ((--autoRefreshCountDown) > 0) {
autoRefreshCountDownElement.innerHTML = `(${autoRefreshCountDown}s)`;
} else {
autoRefreshCountDown = countDown;
btnRefresh.click();
autoRefreshCountDownElement.innerHTML = ``;
}
}, 1000);
} else {
clearInterval(autoRefreshIntreval);
btnRefresh.style.display = "";
autoRefreshCountDownElement.innerHTML = ``;
}
};
// 根据屏幕判断要显示哪些字段
// 此时还未拉取数据，所以进入 render 函数会直接返回，不会多次渲染
let initWidth = document.body.offsetWidth;
// console.log(initWidth);
if (initWidth < 400) {
btnShowNone.click();
btnShowNone.innerHTML += "(默认)";
} else if (initWidth < 780) {
btnShowDefault.click();
btnShowDefault.innerHTML += "(默认)";
} else {
btnShowDetailed.click();
btnShowDetailed.innerHTML += "(默认)";
}
// 网页加载后加载榜单
getData();
// 定时刷新
// setInterval(getData, 10 * 1000);
function getData() {
var xhr = new XMLHttpRequest();
xhr.open("GET", "../data/bilibili-hotband/latest.json?t=" + Date.now(), true);
xhr.send();
xhr.onreadystatechange = function () {
if (xhr.readyState !== 4) return;
if (xhr.status == 200) {
try {
hotBandData = JSON.parse(xhr.responseText);
if (!hotBandData.data || typeof hotBandData.data !== 'object')
throw new Error("data is undefined or not an object");
} catch (e) {
console.error("[error]", "\n", e, "\n", "\n", "[xhr.responseText]", "\n", xhr.responseText);
alert("latest.json 文件解析失败，请检查文件");
return;
}
console.log(hotBandData);
// 更新时间
document.getElementById("latestUpdateTime").innerHTML =
"数据拉取时间：" + new Date().toLocaleString() + "<br/>" +
"热榜更新时间：" + new Date(hotBandData.update_time).toLocaleString();
// 渲染榜单
render();
} else if (xhr.status == 404) {
alert("data 目录下未找到 latest.json 文件，可能的原因：\n您还没有运行脚本拉取数据，请先运行脚本，然后刷新页面");
}
}
}
function render() {
if (!hotBandData) return;
/**
* 渲染热搜列表
*/
let hotBandList = hotBandData.data;
var str = [];
// 渲染表格
str.push(`<thead>
<tr class="thead" style="top: 0; background-color: white; position: sticky;">
<td>编号</td>
<td>热搜</td>
${showKeyword.checked ? "<td>关键词</td>" : ""}
${showWordType.checked ? "<td>word_type</td>" : ""}
${showHotId.checked ? "<td>hot_id</td>" : ""}
</tr>
</thead>`);
str.push(`<tbody>`);
for (var i = 0; i < hotBandList.length; i++) {
const hotBand = hotBandList[i];
let linkUrl = mobileFlag
? `https://m.bilibili.com/search?keyword=${encodeURIComponent(hotBand.keyword)}`
: `https://search.bilibili.com/all?keyword=${encodeURIComponent(hotBand.keyword)}`;
str.push(`<tr>
<!-- 编号 -->
<td>${hotBand.position}</td>
<!-- 热搜 -->
<td style="text-align: left;">
<nobr>
<div style="min-width: 20px; display: inline-block;">
<span class="hotband-label" style="background-color: ${getIconText(hotBand.icon)[1]}; ${hotBand.icon ? "" : "display: none;"}">${getIconText(hotBand.icon)[0]}</span>
</div>
<span>
<a href="${linkUrl}" target="_blank">${hotBand.show_name}</a>
</span>
${hotBand.show_live_icon ? `<span class="hotband-label" style="background-color: #f69; padding: 1px 5px;">直播中</span>` : ""}
</nobr>
</td>
${showKeyword.checked ? `
<!-- 关键词 -->
<td>${hotBand.keyword}</td>
` : ""}
${showWordType.checked ? `
<!-- word_type -->
<td>${hotBand.word_type}</td>
` : ""}
${showHotId.checked ? `
<!-- hot_id -->
<td>${hotBand.hot_id}</td>
` : ""}
</tr >`);
}
str.push(`</tbody>`);
document.getElementById('list').innerHTML = str.join('');
}
</script>
</body>
</html>

466
hotband/html/bilibili_rank.html Normal file
View File

@@ -0,0 +1,466 @@
<!DOCTYPE html>
<html lang="cn">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<!-- 不携带 referer -->
<meta name="referrer" content="never">
<title>B站排行</title>
<link rel="stylesheet" href="./assets/css/main.css">
<style>
#title {
margin-bottom: 0;
}
#dynamic-note {
color: grey;
margin: 5px auto;
font-size: 13px;
}
td {
font-size: 12px;
max-width: 200px;
word-wrap: break-word;
}
</style>
</head>
<body>
<div style="text-align: center;">
<h1 id="title">B站排行榜</h1>
<p id="dynamic-note"></p>
<hr>
<div style="display: none;">
显示字段：
<label for="show_emoticon">
<input type="checkbox" class="filter_checkbox" name="show_emoticon" id="show_emoticon"
detailed-checked="true">热搜表情
</label>
<label for="show_num">
<input type="checkbox" class="filter_checkbox" name="show_num" id="show_num" checked="true"
concise-checked="true" detailed-checked="true">热度
</label>
<label for="show_category">
<input type="checkbox" class="filter_checkbox" name="show_category" id="show_category" checked="true"
detailed-checked="true">分类
</label>
<label for="show_onboard_time">
<input type="checkbox" class="filter_checkbox" name="show_onboard_time" id="show_onboard_time"
checked="true" detailed-checked="true">上线时间
</label>
<label for="show_is_new">
<input type="checkbox" class="filter_checkbox" name="show_is_new" id="show_is_new"
detailed-checked="true">是否新热搜
</label>
<label for="show_detail">
<input type="checkbox" class="filter_checkbox" name="show_detail" id="show_detail"
detailed-checked="true">热搜详情
</label>
<label for="show_mid">
<input type="checkbox" class="filter_checkbox" name="show_mid" id="show_mid">mid
</label>
<br>
<button id="btn_show_all">全选</button>
<button id="btn_show_none">全不选</button>
|
<button id="btn_show_concise">简洁</button>
<button id="btn_show_default">普通</button>
<button id="btn_show_detailed">详细</button>
</div>
</div>
<p id="latestUpdateTime" style="font-size: 12px; display: inline-block; vertical-align: middle;"></p>
<nobr>
<button id="btn_refresh">重新拉取</button>
</nobr>
<nobr>
<label for="auto_refresh">
<input type="checkbox" name="auto_refresh" id="auto_refresh">自动拉取<span id="auto_refresh_countdown"></span>
</label>
</nobr>
<nobr>
<span id="update-finish-info" style="color: green; font-weight: bold; display: none;">拉取成功，数据已更新</span>
</nobr>
<table id="list"></table>
<div class="bottom-placeholder">
<p>
— 到底啦 —<br>
数据来源: https://www.bilibili.com/v/popular/rank/all
</p>
</div>
<script>
/**
* 全局变量
*/
// 拉取下来的数据
let hotBandData;
// 按钮
const btnShowAll = document.getElementById('btn_show_all');
const btnShowNone = document.getElementById('btn_show_none');
const btnShowConcise = document.getElementById('btn_show_concise');
const btnShowDefault = document.getElementById('btn_show_default');
const btnShowDetailed = document.getElementById('btn_show_detailed');
const btnRefresh = document.getElementById('btn_refresh');
// 复选框
const filterCheckbox = document.getElementsByClassName("filter_checkbox");
const showEmoticon = document.getElementById("show_emoticon");
const showNum = document.getElementById("show_num");
const showCategory = document.getElementById("show_category");
const showOnboardTime = document.getElementById("show_onboard_time");
const showIsNew = document.getElementById("show_is_new");
const showDetail = document.getElementById("show_detail");
const showMid = document.getElementById("show_mid");
const autoRefresh = document.getElementById("auto_refresh");
// 绑定按钮点击事件
btnShowAll.addEventListener('click', function () {
for (let i = 0; i < filterCheckbox.length; i++) {
const element = filterCheckbox[i];
element.checked = true;
}
render();
});
btnShowNone.addEventListener('click', function () {
for (let i = 0; i < filterCheckbox.length; i++) {
const element = filterCheckbox[i];
element.checked = false;
}
render();
});
btnShowConcise.addEventListener('click', function () {
for (let i = 0; i < filterCheckbox.length; i++) {
const element = filterCheckbox[i];
element.checked = element.getAttribute('concise-checked') === 'true';
}
render();
});
btnShowDefault.addEventListener('click', function () {
for (let i = 0; i < filterCheckbox.length; i++) {
const element = filterCheckbox[i];
element.checked = element.getAttribute('checked') === 'true';
}
render();
});
btnShowDetailed.addEventListener('click', function () {
for (let i = 0; i < filterCheckbox.length; i++) {
const element = filterCheckbox[i];
element.checked = element.getAttribute('detailed-checked') === 'true';
}
render();
});
btnRefresh.onclick = function () {
getData();
document.getElementById("update-finish-info").style.display = "";
// btnRefresh.style.display = "none";
btnRefresh.style.visibility = "hidden";
setTimeout(function () {
document.getElementById("update-finish-info").style.display = "none";
// btnRefresh.style.display = "";
btnRefresh.style.visibility = "";
}, 1000);
};
// 绑定复选框改变事件
for (let i = 0; i < filterCheckbox.length; i++) {
// console.log(filterCheckbox[i]);
filterCheckbox[i].onchange = function () {
render();
};
}
let autoRefreshIntreval = null;
const autoRefreshCountDownElement = document.getElementById('auto_refresh_countdown');
const countDown = 20; // 自动拉取间隔时间，单位：秒
let autoRefreshCountDown = countDown;
autoRefresh.onchange = function () {
if (autoRefresh.checked) {
btnRefresh.style.display = "none";
btnRefresh.click();
autoRefreshIntreval = setInterval(function () {
if ((--autoRefreshCountDown) > 0) {
autoRefreshCountDownElement.innerHTML = `(${autoRefreshCountDown}s)`;
} else {
autoRefreshCountDown = countDown;
btnRefresh.click();
autoRefreshCountDownElement.innerHTML = ``;
}
}, 1000);
} else {
clearInterval(autoRefreshIntreval);
btnRefresh.style.display = "";
autoRefreshCountDownElement.innerHTML = ``;
}
};
// 根据屏幕判断要显示哪些字段
// 此时还未拉取数据，所以进入 render 函数会直接返回，不会多次渲染
let initWidth = document.body.offsetWidth;
// console.log(initWidth);
/* if (initWidth < 400) {
btnShowNone.click();
btnShowNone.innerHTML += "(默认)";
} else */ if (initWidth < 600) {
btnShowConcise.click();
btnShowConcise.innerHTML += "(默认)";
} else if (initWidth < 1900) {
btnShowDefault.click();
btnShowDefault.innerHTML += "(默认)";
} else {
btnShowDetailed.click();
btnShowDetailed.innerHTML += "(默认)";
}
// 网页加载后加载榜单
getData();
// 定时刷新
// setInterval(getData, 10 * 1000);
function getData() {
var xhr = new XMLHttpRequest();
xhr.open("GET", "../data/bilibili-rank/latest.json?t=" + Date.now(), true);
xhr.send();
xhr.onreadystatechange = function () {
if (xhr.readyState !== 4) return;
if (xhr.status == 200) {
try {
hotBandData = JSON.parse(xhr.responseText);
if (!hotBandData.data || typeof hotBandData.data !== 'object')
throw new Error("data is undefined or not an object");
} catch (e) {
console.error("[error]", "\n", e, "\n", "\n", "[xhr.responseText]", "\n", xhr.responseText);
alert("latest.json 文件解析失败，请检查文件");
return;
}
console.log(hotBandData);
// 更新动态 note
document.getElementById("dynamic-note").innerHTML = hotBandData.note;
// 更新时间
document.getElementById("latestUpdateTime").innerHTML =
"数据拉取时间：" + new Date().toLocaleString() + "<br/>" +
"热榜更新时间：" + new Date(hotBandData.update_time).toLocaleString();
// 渲染榜单
render();
} else if (xhr.status == 404) {
alert("data 目录下未找到 latest.json 文件，可能的原因：\n您还没有运行脚本拉取数据，请先运行脚本，然后刷新页面");
}
}
}
function render() {
if (!hotBandData) return;
/**
* 渲染热搜列表
*/
let hotBandList = hotBandData.data;
var str = [];
// 渲染表格
str.push(`<thead>
<tr class="thead" style="top: 0; background-color: white; position: sticky;">
<td>编号</td>
<td>标题</td>
<td>时长</td>
<td>封面</td>
<td>第一帧</td>
<td>视频ID</td>
<td>分类</td>
<td>发布时间</td>
<td>简介</td>
<td>作者(mid)</td>
<td>统计</td>
<td>视频宽高</td>
<td>定位</td>
<td>videos</td>
<td>tid</td>
<td>copyright</td>
<td>state</td>
<td>rights</td>
<td>dynamic</td>
<td>score</td>
<td>mission_id</td>
<td>season_id</td>
<td>up_from_v2</td>
<td>others</td>
</tr>
</thead>`);
str.push(`<tbody>`);
for (var i = 0; i < hotBandList.length; i++) {
const hotBand = hotBandList[i];
let link = hotBand.short_link == hotBand.short_link_v2
? hotBand.short_link
: `${hotBand.short_link}<br/>${hotBand.short_link_v2}`;
/**
* 视频总秒数转化为友好显示时间
*/
// refer: https://blog.csdn.net/weixin_43838488/article/details/122337474
function formatZero(num, len) {
if (String(num).length > len) {
return num;
}
return (Array(len).join(0) + num).slice(-len)
}
let duration = hotBand.duration - 1; // 根据观测，基本上所有视频都是少1s(有些少了2s或者其他)，所以这里减1
let durationArr = [0, 0, 0];
durationArr[0] = Math.floor(duration / (60 * 60));
durationArr[1] = Math.floor((duration - durationArr[0] * 60 * 60) / 60);
durationArr[2] = Math.floor(duration - durationArr[0] * 60 * 60 - durationArr[1] * 60);
let durationStr = "";
if (durationArr[0] === 0) {
// 小时为0
durationStr = `${formatZero(durationArr[1], 2)}:${formatZero(durationArr[2], 2)}`;
} else {
// 小时不为0
durationStr = `${formatZero(durationArr[0], 2)}:${formatZero(durationArr[1], 2)}:${formatZero(durationArr[2], 2)}`;
}
//功能：求最大公约数
//参数： x 、y number
//返回值： number
function gcd(x, y) {
if (isNaN(x) || isNaN(y)) return null;
if (x % y === 0) {
return y;
}
return gcd(y, x % y)
//三目运算符写法：
//return x % y === 0 ? y : gcd(y , x % y) ;
}
let dimension_gcd = gcd(hotBand.dimension.width, hotBand.dimension.height);
str.push(`<tr>
<!-- 编号 -->
<td>${i + 1}</td>
<!-- 标题 -->
<td>
<a href="${link}" target="_blank">${hotBand.title}</a>
</td>
<!-- 时长 -->
<td>${durationStr}<br>(${hotBand.duration}s)</td>
<!-- 封面 -->
<td>
<img src="${hotBand.pic}" style="width: 120px;"/>
</td>
<!-- 第一帧 -->
<td>
<img src="${hotBand.first_frame}" style="width: 120px;"/>
</td>
<!-- aid -->
<td style="text-align: left;">
<nobr>aid: ${hotBand.aid}</nobr>
<nobr>bvid: ${hotBand.bvid}</nobr>
<nobr>cid: ${hotBand.cid}</nobr>
</td>
<!-- 分类 -->
<td>${hotBand.tname}</td>
<!-- 发布时间 -->
<td style="font-size: 10px;">
<nobr>pubdate: ${new Date(hotBand.pubdate * 1000).toLocaleString()}</nobr>
<nobr>ctime: ${new Date(hotBand.ctime * 1000).toLocaleString()}</nobr>
</td>
<!-- 简介 -->
<td>${hotBand.desc}</td>
<!-- 作者 -->
<td>
<img src="${hotBand.owner.face}" style="width: 30px;"/><br>
${hotBand.owner.name}<br>
(${hotBand.owner.mid})
<!-- ${JSON.stringify(hotBand.owner)} -->
</td>
<!-- 统计 -->
<td style="text-align: left;">
<nobr>播放: ${hotBand.stat.view}</nobr>
<nobr>弹幕: ${hotBand.stat.danmaku}</nobr>
<nobr>评论: ${hotBand.stat.reply}</nobr>
<nobr>喜欢: ${hotBand.stat.favorite}</nobr>
<nobr>投币: ${hotBand.stat.coin}</nobr>
<nobr>分享: ${hotBand.stat.share}</nobr>
<!--<nobr>当前排名: ${hotBand.stat.now_rank}</nobr>-->
<nobr>历史<!--最高-->排名: ${hotBand.stat.his_rank}</nobr>
<nobr>喜欢数: ${hotBand.stat.like}</nobr>
<nobr>不喜欢数: ${hotBand.stat.dislike}</nobr>
<!-- ${JSON.stringify(hotBand.stat)} -->
</td>
<!-- 视频宽高 -->
<td>
${hotBand.dimension.width}:${hotBand.dimension.height}<br>
(${hotBand.dimension.width / dimension_gcd}:${hotBand.dimension.height / dimension_gcd})
<!-- ${hotBand.dimension.rotate} -->
<!-- ${JSON.stringify(hotBand.dimension)} -->
</td>
<!-- 定位 -->
<td>
<nobr>${hotBand.pub_location}</nobr>
</td>
<!-- videos -->
<td>${hotBand.videos}</td>
<!-- tid -->
<td>${hotBand.tid}</td>
<!-- copyright -->
<td>${hotBand.copyright}</td>
<!-- state -->
<td>${hotBand.state}</td>
<!-- rights -->
<td>${JSON.stringify(hotBand.rights)}</td>
<!-- dynamic -->
<td>${hotBand.dynamic}</td>
<!-- score -->
<td>${hotBand.score}</td>
<!-- mission_id -->
<td>${hotBand.mission_id ? hotBand.mission_id : ''}</td>
<!-- season_id -->
<td>${hotBand.season_id ? hotBand.season_id : ''}</td>
<!-- up_from_v2 -->
<td>${hotBand.up_from_v2 ? hotBand.up_from_v2 : ''}</td>
<!-- others -->
<td>
${hotBand.others
? `<div style="max-height: 200px; overflow: scroll;"><span>${JSON.stringify(hotBand.others)}</span></div>`
: ''}
</td>
</tr >`);
}
str.push(`</tbody>`);
document.getElementById('list').innerHTML = str.join('');
}
</script>
</body>
</html>

333
hotband/html/weibo_hotband.html Normal file
View File

@@ -0,0 +1,333 @@
<!DOCTYPE html>
<html lang="cn">
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>微博热搜</title>
<link rel="stylesheet" href="./assets/css/main.css">
</head>
<body>
<div style="text-align: center;">
<h1>微博热搜榜</h1>
<hr>
<div>
显示字段：
<label for="show_emoticon">
<input type="checkbox" class="filter_checkbox" name="show_emoticon" id="show_emoticon"
detailed-checked="true">热搜表情
</label>
<label for="show_num">
<input type="checkbox" class="filter_checkbox" name="show_num" id="show_num" checked="true"
concise-checked="true" detailed-checked="true">热度
</label>
<label for="show_category">
<input type="checkbox" class="filter_checkbox" name="show_category" id="show_category" checked="true"
detailed-checked="true">分类
</label>
<label for="show_onboard_time">
<input type="checkbox" class="filter_checkbox" name="show_onboard_time" id="show_onboard_time"
checked="true" detailed-checked="true">上线时间
</label>
<label for="show_is_new">
<input type="checkbox" class="filter_checkbox" name="show_is_new" id="show_is_new"
detailed-checked="true">是否新热搜
</label>
<label for="show_detail">
<input type="checkbox" class="filter_checkbox" name="show_detail" id="show_detail"
detailed-checked="true">热搜详情
</label>
<label for="show_mid">
<input type="checkbox" class="filter_checkbox" name="show_mid" id="show_mid">mid
</label>
<br>
<button id="btn_show_all">全选</button>
<button id="btn_show_none">全不选</button>
|
<button id="btn_show_concise">简洁</button>
<button id="btn_show_default">普通</button>
<button id="btn_show_detailed">详细</button>
</div>
</div>
<p id="latestUpdateTime" style="font-size: 12px; display: inline-block; vertical-align: middle;"></p>
<nobr>
<button id="btn_refresh">重新拉取</button>
</nobr>
<nobr>
<label for="auto_refresh">
<input type="checkbox" name="auto_refresh" id="auto_refresh">自动拉取<span id="auto_refresh_countdown"></span>
</label>
</nobr>
<nobr>
<span id="update-finish-info" style="color: green; font-weight: bold; display: none;">拉取成功，数据已更新</span>
</nobr>
<table id="list"></table>
<div class="bottom-placeholder">
<p>
— 到底啦 —<br>
数据来源: https://weibo.com/hot/search
</p>
</div>
<script>
/**
* 全局变量
*/
// 拉取下来的数据
let hotBandData;
// 按钮
const btnShowAll = document.getElementById('btn_show_all');
const btnShowNone = document.getElementById('btn_show_none');
const btnShowConcise = document.getElementById('btn_show_concise');
const btnShowDefault = document.getElementById('btn_show_default');
const btnShowDetailed = document.getElementById('btn_show_detailed');
const btnRefresh = document.getElementById('btn_refresh');
// 复选框
const filterCheckbox = document.getElementsByClassName("filter_checkbox");
const showEmoticon = document.getElementById("show_emoticon");
const showNum = document.getElementById("show_num");
const showCategory = document.getElementById("show_category");
const showOnboardTime = document.getElementById("show_onboard_time");
const showIsNew = document.getElementById("show_is_new");
const showDetail = document.getElementById("show_detail");
const showMid = document.getElementById("show_mid");
const autoRefresh = document.getElementById("auto_refresh");
// 绑定按钮点击事件
btnShowAll.addEventListener('click', function () {
for (let i = 0; i < filterCheckbox.length; i++) {
const element = filterCheckbox[i];
element.checked = true;
}
render();
});
btnShowNone.addEventListener('click', function () {
for (let i = 0; i < filterCheckbox.length; i++) {
const element = filterCheckbox[i];
element.checked = false;
}
render();
});
btnShowConcise.addEventListener('click', function () {
for (let i = 0; i < filterCheckbox.length; i++) {
const element = filterCheckbox[i];
element.checked = element.getAttribute('concise-checked') === 'true';
}
render();
});
btnShowDefault.addEventListener('click', function () {
for (let i = 0; i < filterCheckbox.length; i++) {
const element = filterCheckbox[i];
element.checked = element.getAttribute('checked') === 'true';
}
render();
});
btnShowDetailed.addEventListener('click', function () {
for (let i = 0; i < filterCheckbox.length; i++) {
const element = filterCheckbox[i];
element.checked = element.getAttribute('detailed-checked') === 'true';
}
render();
});
btnRefresh.onclick = function () {
getData();
document.getElementById("update-finish-info").style.display = "";
// btnRefresh.style.display = "none";
btnRefresh.style.visibility = "hidden";
setTimeout(function () {
document.getElementById("update-finish-info").style.display = "none";
// btnRefresh.style.display = "";
btnRefresh.style.visibility = "";
}, 1000);
};
// 绑定复选框改变事件
for (let i = 0; i < filterCheckbox.length; i++) {
// console.log(filterCheckbox[i]);
filterCheckbox[i].onchange = function () {
render();
};
}
let autoRefreshIntreval = null;
const autoRefreshCountDownElement = document.getElementById('auto_refresh_countdown');
const countDown = 20; // 自动拉取间隔时间，单位：秒
let autoRefreshCountDown = countDown;
autoRefresh.onchange = function () {
if (autoRefresh.checked) {
btnRefresh.style.display = "none";
btnRefresh.click();
autoRefreshIntreval = setInterval(function () {
if ((--autoRefreshCountDown) > 0) {
autoRefreshCountDownElement.innerHTML = `(${autoRefreshCountDown}s)`;
} else {
autoRefreshCountDown = countDown;
btnRefresh.click();
autoRefreshCountDownElement.innerHTML = ``;
}
}, 1000);
} else {
clearInterval(autoRefreshIntreval);
btnRefresh.style.display = "";
autoRefreshCountDownElement.innerHTML = ``;
}
};
// 根据屏幕判断要显示哪些字段
// 此时还未拉取数据，所以进入 render 函数会直接返回，不会多次渲染
let initWidth = document.body.offsetWidth;
// console.log(initWidth);
/* if (initWidth < 400) {
btnShowNone.click();
btnShowNone.innerHTML += "(默认)";
} else */ if (initWidth < 600) {
btnShowConcise.click();
btnShowConcise.innerHTML += "(默认)";
} else if (initWidth < 1900) {
btnShowDefault.click();
btnShowDefault.innerHTML += "(默认)";
} else {
btnShowDetailed.click();
btnShowDetailed.innerHTML += "(默认)";
}
// 网页加载后加载榜单
getData();
// 定时刷新
// setInterval(getData, 10 * 1000);
function getData() {
var xhr = new XMLHttpRequest();
xhr.open("GET", "../data/weibo-hotband/latest.json?t=" + Date.now(), true);
xhr.send();
xhr.onreadystatechange = function () {
if (xhr.readyState !== 4) return;
if (xhr.status == 200) {
try {
hotBandData = JSON.parse(xhr.responseText);
if (!hotBandData.data || typeof hotBandData.data !== 'object')
throw new Error("data is undefined or not an object");
} catch (e) {
console.error("[error]", "\n", e, "\n", "\n", "[xhr.responseText]", "\n", xhr.responseText);
alert("latest.json 文件解析失败，请检查文件");
return;
}
console.log(hotBandData);
// 更新时间
document.getElementById("latestUpdateTime").innerHTML =
"数据拉取时间：" + new Date().toLocaleString() + "<br/>" +
"热榜更新时间：" + new Date(hotBandData.update_time).toLocaleString();
// 渲染榜单
render();
} else if (xhr.status == 404) {
alert("data 目录下未找到 latest.json 文件，可能的原因：\n您还没有运行脚本拉取数据，请先运行脚本，然后刷新页面");
}
}
}
function render() {
if (!hotBandData) return;
/**
* 渲染热搜列表
*/
let hotBandList = hotBandData.data;
var str = [];
// 渲染表格
str.push(`<thead>
<tr class="thead" style="top: 0; background-color: white; position: sticky;">
<td>编号</td>
<td>热搜</td>
${showEmoticon.checked ? "<td>表情</td>" : ""}
${showNum.checked ? '<td>热度<br/><span style="font-size: 10px;">(展示/真实)</span></td>' : ""}
${showCategory.checked ? "<td>分类</td>" : ""}
${showOnboardTime.checked ? "<td>上线时间</td>" : ""}
${showIsNew.checked ? "<td>是否新热搜</td>" : ""}
${showDetail.checked ? "<td>热搜详情</td>" : ""}
${showMid.checked ? "<td>mid</td>" : ""}
</tr>
</thead>`);
str.push(`<tbody>`);
for (var i = 0; i < hotBandList.length; i++) {
const hotBand = hotBandList[i];
let hotDelta = hotBand.num - hotBand.raw_hot;
str.push(`<tr>
<!-- 编号 -->
<td>${i + 1}</td>
<!-- 热搜 -->
<td style="text-align: left; font-size: 14px;">
<nobr>
<div style="min-width: 20px; display: inline-block;">
<span class="hotband-label" style="background-color: ${hotBand.more.icon_desc_color};">${hotBand.label_name}</span>
</div>
<a href="${hotBand.url}" target="_blank">${hotBand.word}</a>
</nobr>
</td>
${showEmoticon.checked ? `
<!-- 表情 -->
<td>${hotBand.emoticon}</td>
` : ""}
${showNum.checked ? `
<!-- 热度 -->
<td style="line-height: 12px;">
<nobr><span style="font-size: 14px;">${hotDelta == 0 ? hotBand.num : `${hotBand.num} / ${hotBand.raw_hot}`}</span></nobr><br/>
<nobr>
<span style="font-size: 10px; color: ${hotDelta > 0 ? "red" : "green"}; font-weight: bold;">
${hotDelta != 0 ? `(官方调控 ${hotDelta > 0 ? "+" : ""}${hotDelta})` : ""}
</span>
</nobr>
</td>
` : ""}
${showCategory.checked ? `
<!-- 分类 -->
<td style="font-size: 10px;">${hotBand.category.map((c) => `<nobr>${c}</nobr>`).join('；')}</td>
` : ""}
${showOnboardTime.checked ? `
<!-- 热搜上线时间 -->
<td style="font-size: 10px;">${new Date(hotBand.onboard_time * 1000).toLocaleString()}</td>
` : ""}
${showIsNew.checked ? `
<!-- 是否新热搜 -->
<td>${hotBand.more.is_new == 1 ? "是" : ""}</td>
` : ""}
${showDetail.checked ? `
<!-- 热搜详情 -->
<td>
<div style="font-size:10px; max-width: 300px; display: inline-block;">${hotBand.more.detail}</div>
</td>
` : ""}
${showMid.checked ? `
<!-- mid -->
<td style="font-size: 14px;">${hotBand.more.mid}</td>
` : ""}
</tr >`);
}
str.push(`</tbody>`);
document.getElementById('list').innerHTML = str.join('');
}
</script>
</body>
</html>

191
hotband/index.html Normal file
View File

@@ -0,0 +1,191 @@
<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>热搜榜单</title>
<style>
* {
margin: 0;
padding: 0;
}
html,
body {
height: 100%;
overflow: hidden;
}
.container {
display: grid;
grid-template-rows: 1fr 50px;
width: 100%;
height: 100%;
}
.iframe-container,
iframe {
width: 100%;
height: 100%;
}
#float-container.bt {
bottom: 0;
right: 0;
left: 0;
height: 100%;
background-color: #fdfdfd;
box-shadow: 0 -2px 10px 0 rgb(0 0 0 / 10%);
display: grid;
/* grid-template-columns: repeat(3, 1fr); */
place-items: center;
}
#float-container.bt .navbar-item {
text-align: center;
cursor: pointer;
width: 80px;
user-select: none;
}
#float-container.bt .navbar-item .navbar-image {
width: 22px;
height: 22px;
filter: grayscale(1);
}
#float-container.bt .navbar-item .navbar-title {
font-size: 11px;
color: #666;
margin-top: -3px;
}
#float-container.bt .navbar-item.active .navbar-image {
filter: initial;
}
</style>
</head>
<body>
<div class="container">
<div id="iframe-container">
<!-- <iframe id="main-iframe" src="html/weibo.html" frameborder="0"></iframe> -->
</div>
<div id="float-container" class="bt">
<!-- <div class="navbar-item" targetPage="weibo_hotband">
<img class="navbar-image" src="./html/assets/image/weibo.svg" />
<p class="navbar-title">微博热搜</p>
</div> -->
</div>
</div>
<script>
let pages = {
'weibo_hotband': {
title: '微博热搜',
// url: 'html/weibo_hotband.html',
icon: './html/assets/image/weibo.svg',
icon_scale: 1,
},
'bilibili_hotband': {
title: 'B站热搜',
// url: 'html/bilibili_hotband.html',
icon: './html/assets/image/bilibili.svg',
icon_scale: 0.87,
},
'bilibili_rank': {
title: 'B站排行',
// url: 'html/bilibili_rank.html',
icon: './html/assets/image/icon_rank.png',
icon_scale: 0.87,
},
};
// 渲染导航栏
for (let key in pages) {
let page = pages[key];
let navbarItem = document.createElement('div');
navbarItem.className = 'navbar-item';
navbarItem.setAttribute('targetPage', key);
navbarItem.innerHTML = `
<img class="navbar-image" src="${page.icon}" ${page.icon_scale != 1 ? `style="transform: scale(${page.icon_scale});"` : ''} />
<p class="navbar-title"> ${page.title}</p>
`;
document.getElementById('float-container').style.gridTemplateColumns = `repeat(${Object.keys(pages).length}, 1fr)`;
document.getElementById('float-container').appendChild(navbarItem);
}
</script>
<script>
let navbarItems = document.querySelectorAll('.navbar-item');
let iframeContainer = document.querySelector('#iframe-container');
let iframe = document.querySelector('#main-iframe');
navbarItems.forEach(item => {
item.addEventListener('click', () => {
// 已选中项再次点击，不执行操作
if (item.classList.contains('active')) return;
navbarItems.forEach(item => {
item.classList.remove('active');
});
item.classList.add('active');
let target = item.getAttribute('targetPage');
switchPage(target, true);
});
});
// 切换页面
// 传入参数：目标页面，是否更新网址
function switchPage(target, pushState) {
let url = `html/${target}.html`;
if (pushState) {
// 更新url但不刷新页面
history.pushState(null, null, `./index.html?target=${target}`);
}
// iframe 进入新页面
if (iframe)
iframe.remove();
iframe = document.createElement('iframe');
iframe.id = "main-iframe";
iframe.src = url;
iframe.setAttribute('frameborder', '0');
iframeContainer.appendChild(iframe);
iframe.addEventListener('load', function (event) {
console.log(event);
document.title = document.getElementById("main-iframe").contentWindow.document.title + " | 热搜榜单"
}, true);
}
// 根据 URL 参数切换页面
function locationToPageByUrlParam() {
// 获取 url 中的 target 参数
let url = new URL(window.location.href);
let target = url.searchParams.get('target');
if (!Object.keys(pages).includes(target)) {
target = 'weibo_hotband';
history.replaceState(null, null, `./index.html?target=${target}`);
}
// 切换页面
switchPage(target, false);
// 更新下方导航栏选中状态
let oldActive = document.querySelector('.active')
if (oldActive) oldActive.classList.remove('active');
navbarItems.forEach(item => {
if (item.getAttribute('targetPage') === target) {
item.classList.add('active');
}
});
}
window.addEventListener("popstate", function (e) {
console.log("浏览器返回事件", e);
locationToPageByUrlParam();
}, false);
locationToPageByUrlParam();
</script>
</body>
</html>

110
hotband/index.js Normal file
View File

@@ -0,0 +1,110 @@
'use strict';
const dotenv = require('dotenv');
const schedule = require('node-schedule');
const path = require('path');
const os = require('os');
const fs = require('fs');
/**
* 环境变量
*/
if (!fs.existsSync('.env')) {
// 如果没有 .env 文件，则报错并退出
console.error('[ERROR] .env file not found!');
return;
}
process.env = {}; // 清除系统自带的环境变量
dotenv.config('./.env'); // 导入 .env 文件中的环境变量
// console.log(process.env);
const DEBUG_MODE = process.env.DEBUG_MODE == true;
const EXECUTE_AT_STARTUP = process.env.EXECUTE_AT_STARTUP == true;
const PUSH_TO_GIT = process.env.PUSH_TO_GIT == true;
const ROOT_PATH = path.join(__dirname, process.env.DATA_FOLDER ?? 'data');
/**
* 调试模式
*/
if (DEBUG_MODE) {
console.log('DEBUG_MODE is on');
console.log('Environment variables: ', process.env);
}
/**
* 引入模块
*/
const get_weibo_hotband = require('./src/get_weibo_hotband');
const get_bilibili_hotband = require('./src/get_bilibili_hotband');
const get_bilibili_rank = require('./src/get_bilibili_rank');
const execute_command = require('./src/execute_command');
/**
* 开始运行
*/
console.log("Start running ...");
/**
* 程序主函数
*/
async function start() {
// 爬取热搜数据
await get_weibo_hotband.main();
await get_bilibili_hotband.main();
await get_bilibili_rank.main();
// 调试模式下
if (DEBUG_MODE) {
// 推送到 Git 仓库
await pushToGitRepo();
}
}
// 调试模式下，程序一启动就首先运行一次
if (EXECUTE_AT_STARTUP) {
process.stdout.write("程序启动时，立即运行一次\t");
start();
}
// 每分钟的第 5 秒执行一次
// 这里指定第 5 秒是为了稍微与微博服务器热榜更新时间错开，避免因为微秒级误差造成拉取两次相同的热榜数据
// refer: https://www.npmjs.com/package/node-schedule
const scheduleJob = schedule.scheduleJob('05 * * * * *', start);
/**
* 定时将热搜数据推送到 Git 仓库
*/
async function pushToGitRepo() {
if (!PUSH_TO_GIT) return;
let commands = [
'git status',
'git pull',
'git add .',
`git commit -m "${new Date(Date.now() + 8 * 3600 * 1000).toISOString().substring(0, 19).replace('T', ' ')} update"`,
`git push origin master`,
'git status',
];
switch (os.type()) {
case 'Windows_NT': // Windows
commands.unshift('dir');
break;
case 'Darwin': // Mac OS X
case 'Linux': // Linux
default:
commands.unshift('pwd');
break;
}
let outputs = await execute_command.execute(ROOT_PATH, commands);
console.log(commands, outputs);
}
// 每个小时同步一次
schedule.scheduleJob('0 0 * * * *', pushToGitRepo);

8
hotband/nodemon.json Normal file
View File

@@ -0,0 +1,8 @@
{
"ignore": [
".git",
".svn",
"node_modules/**/node_modules"
],
"ext": "js"
}

40
hotband/pack.bat Normal file
View File

@@ -0,0 +1,40 @@
set f_year=2022
set f_month=10
ren data data_for_backup
cd ./data_for_backup
cd ./bilibili-hotband
del /f /s/q latest.json
cd ./final/%f_year%/%f_month%
for /d %%i in (*) do ( "C:\Users\Administrator\Desktop\7-Zip（绿色版）\7z.exe" a "%%~ni.zip" "%%i" -sdel )
cd ../../../
cd ./origin/%f_year%/%f_month%
for /d %%i in (*) do ( "C:\Users\Administrator\Desktop\7-Zip（绿色版）\7z.exe" a "%%~ni.zip" "%%i" -sdel )
cd ../../../
cd ../
cd ./bilibili-rank
del /f /s/q latest.json
cd ./origin/%f_year%/%f_month%
for /d %%i in (*) do ( "C:\Users\Administrator\Desktop\7-Zip（绿色版）\7z.exe" a "%%~ni.zip" "%%i" -sdel )
cd ../../../
cd ../
cd ./weibo-hotband
del /f /s/q latest.json
cd ./final/%f_year%/%f_month%
for /d %%i in (*) do ( "C:\Users\Administrator\Desktop\7-Zip（绿色版）\7z.exe" a "%%~ni.zip" "%%i" -sdel )
cd ../../../
cd ./final/%f_year%/%f_month%
for /d %%i in (*) do ( "C:\Users\Administrator\Desktop\7-Zip（绿色版）\7z.exe" a "%%~ni.zip" "%%i" -sdel )
cd ../../../
cd ../../
pause

1
hotband/pm2 restart.bat Normal file
View File

@@ -0,0 +1 @@
pm2 restart weibo-hotband-bot

1
hotband/pm2 restart.sh Normal file
View File

@@ -0,0 +1 @@
pm2 restart weibo-hotband-bot

1
hotband/pm2 start.bat Normal file
View File

@@ -0,0 +1 @@
pm2 start index.js --name weibo-hotband-bot

1
hotband/pm2 start.sh Normal file
View File

@@ -0,0 +1 @@
pm2 start index.js --name weibo-hotband-bot

2
hotband/pm2 stop.bat Normal file
View File

@@ -0,0 +1,2 @@
pm2 stop weibo-hotband-bot
pm2 delete weibo-hotband-bot

2
hotband/pm2 stop.sh Normal file
View File

@@ -0,0 +1,2 @@
pm2 stop weibo-hotband-bot
pm2 delete weibo-hotband-bot

40
hotband/src/execute_command.js Normal file
View File

@@ -0,0 +1,40 @@
'use strict';
const child_process = require('child_process');
const iconv = require("iconv-lite");
const encoding = "cp936";
const bufferEncoding = "binary";
async function execute(rootPath, cmds) {
let outputs = [];
for (let cmd of cmds) {
let result = await new Promise(function (resolve) {
// refer: https://www.webhek.com/post/execute-a-command-line-binary-with-node-js/
child_process.exec(cmd, {
cwd: rootPath, // 脚本执行目录
encoding: bufferEncoding
}, function (err, stdout, stderr) {
if (err) {
resolve({
cmd: cmd,
err: err,
// err_stack: iconv.decode(Buffer.from(err.stack, bufferEncoding), encoding),
// err_message: iconv.decode(Buffer.from(err.message, bufferEncoding), encoding),
});
} else {
// 获取命令执行的输出
resolve({
cmd: cmd,
stdout: iconv.decode(Buffer.from(stdout, bufferEncoding), encoding),
stderr: iconv.decode(Buffer.from(stderr, bufferEncoding), encoding),
});
}
});
});
outputs.push(result);
}
return outputs;
}
exports.execute = execute;

99
hotband/src/get_bilibili_hotband.js Normal file
View File

@@ -0,0 +1,99 @@
'use strict';
const fs = require('fs');
const path = require('path');
const fileUtils = require('./utils/fileUtils');
const requestUtils = require('./utils/requestUtils');
const API_URL = "https://app.bilibili.com/x/v2/search/trending/ranking";
const SUB_FOLDER = "bilibili-hotband";
const DATA_FOLDER = path.join(path.dirname(__dirname), process.env.DATA_FOLDER ?? 'data', SUB_FOLDER);
console.log("DATA_FOLDER", DATA_FOLDER);
fileUtils.createFolder(DATA_FOLDER); // 程序运行就保证 data 目录存在
async function main() {
let requestTimestamp = Date.now();
let now = new Date(requestTimestamp + 8 * 3600 * 1000).toISOString();
let result = await requestUtils.getApiResult(API_URL);
if (result.code != 0) {
console.log(new Date(Date.now() + 8 * 60 * 60 * 1000).toISOString(), SUB_FOLDER, "请求成功，但服务器处理失败，等待3s后重试。");
await new Promise((resolve) => {
setTimeout(resolve, 3000); // 等待3秒
});
result = await requestUtils.getApiResult(API_URL);
if (result.ok != 1) {
console.log(new Date(Date.now() + 8 * 60 * 60 * 1000).toISOString(), SUB_FOLDER, "请求成功，但服务器处理失败，保存失败信息。");
// ok 不为 1，那么久直接保存便于后续分析，不进行后续处理
fileUtils.saveJSON({
saveFolder: DATA_FOLDER,
now: now,
fileNameSuffix: `origin-error`,
object: result,
compress: true,
uncompress: false
});
return;
}
}
console.log(new Date(Date.now() + 8 * 60 * 60 * 1000).toISOString(), SUB_FOLDER, "请求成功");
// console.log("result", result);
let data = result.data;
// 去除 trackid
delete data["trackid"];
// console.log(data);
/**
* 保存原始数据
*/
fileUtils.saveJSON({
saveFolder: DATA_FOLDER,
now: now,
fileNameSuffix: `origin`,
object: result,
compress: true,
uncompress: false
});
/**
* 获取需要的数据，进行转换
*/
let convert = [];
data.list.forEach(item => {
// {
// "position": 1,
// "keyword": "关键词",
// "show_name": "热搜名称",
// "word_type": 8,
// "icon": "热搜的图标，也可能没有",
// "hot_id": 7399 // 热搜id
// }
convert.push(item);
});
fileUtils.saveJSON({
saveFolder: DATA_FOLDER,
now: now,
fileNameSuffix: `final`,
object: convert,
compress: true,
uncompress: false,
});
/**
* 更新最新的
*/
fs.writeFileSync(`${DATA_FOLDER}/latest.json`, JSON.stringify({
update_time: requestTimestamp,
update_time_friendly: now.substring(0, 19).replace(/T/g, " "),
data: data.list,
exp_str: data.exp_str,
}));
}
exports.main = main;

90
hotband/src/get_bilibili_rank.js Normal file
View File

@@ -0,0 +1,90 @@
'use strict';
const fs = require('fs');
const path = require('path');
const fileUtils = require('./utils/fileUtils');
const requestUtils = require('./utils/requestUtils');
const API_URL = "https://api.bilibili.com/x/web-interface/ranking/v2?type=all";
const SUB_FOLDER = "bilibili-rank";
const DATA_FOLDER = path.join(path.dirname(__dirname), process.env.DATA_FOLDER ?? 'data', SUB_FOLDER);
console.log("DATA_FOLDER", DATA_FOLDER);
fileUtils.createFolder(DATA_FOLDER); // 程序运行就保证 data 目录存在
async function main() {
let requestTimestamp = Date.now();
let now = new Date(requestTimestamp + 8 * 3600 * 1000).toISOString();
let result = await requestUtils.getApiResult(API_URL);
if (result.code != 0) {
console.log(new Date(Date.now() + 8 * 60 * 60 * 1000).toISOString(), SUB_FOLDER, "请求成功，但服务器处理失败，等待3s后重试。");
await new Promise((resolve) => {
setTimeout(resolve, 3000); // 等待3秒
});
result = await requestUtils.getApiResult(API_URL);
if (result.ok != 1) {
console.log(new Date(Date.now() + 8 * 60 * 60 * 1000).toISOString(), SUB_FOLDER, "请求成功，但服务器处理失败，保存失败信息。");
// ok 不为 1，那么久直接保存便于后续分析，不进行后续处理
fileUtils.saveJSON({
saveFolder: DATA_FOLDER,
now: now,
fileNameSuffix: `origin-error`,
object: result,
compress: true,
uncompress: false
});
return;
}
}
console.log(new Date(Date.now() + 8 * 60 * 60 * 1000).toISOString(), SUB_FOLDER, "请求成功");
// console.log("result", result);
let data = result.data;
// // 去除 trackid
// delete data["trackid"];
// console.log(data);
/**
* 保存原始数据
*/
fileUtils.saveJSON({
saveFolder: DATA_FOLDER,
now: now,
fileNameSuffix: `origin`,
object: result,
compress: true,
uncompress: false
});
// /**
// * 获取需要的数据，进行转换
// */
// let convert = [];
// data.list.forEach(item => {
// convert.push(item);
// });
// fileUtils.saveJSON({
// saveFolder: DATA_FOLDER,
// now: now,
// fileNameSuffix: `final`,
// object: convert,
// compress: true,
// uncompress: false,
// });
/**
* 更新最新的
*/
fs.writeFileSync(`${DATA_FOLDER}/latest.json`, JSON.stringify({
update_time: requestTimestamp,
update_time_friendly: now.substring(0, 19).replace(/T/g, " "),
note: data.note,
data: data.list,
}));
}
exports.main = main;

228
hotband/src/get_weibo_hotband.js Normal file
View File

@@ -0,0 +1,228 @@
'use strict';
const fs = require('fs');
const path = require('path');
const fileUtils = require('./utils/fileUtils');
const requestUtils = require('./utils/requestUtils');
const API_URL = "https://weibo.com/ajax/statuses/hot_band";
const SUB_FOLDER = "weibo-hotband";
const DATA_FOLDER = path.join(path.dirname(__dirname), process.env.DATA_FOLDER ?? 'data', SUB_FOLDER);
console.log("DATA_FOLDER", DATA_FOLDER);
fileUtils.createFolder(DATA_FOLDER); // 程序运行就保证 data 目录存在
async function main() {
let requestTimestamp = Date.now();
let now = new Date(requestTimestamp + 8 * 3600 * 1000).toISOString();
let result = await requestUtils.getApiResult(API_URL);
if (result.ok != 1) {
console.log(new Date(Date.now() + 8 * 60 * 60 * 1000).toISOString(), SUB_FOLDER, "请求成功，但服务器处理失败，等待3s后重试。");
await new Promise((resolve) => {
setTimeout(resolve, 3000); // 等待3秒
});
result = await requestUtils.getApiResult(API_URL);
if (result.ok != 1) {
console.log(new Date(Date.now() + 8 * 60 * 60 * 1000).toISOString(), SUB_FOLDER, "请求成功，但服务器处理失败，保存失败信息。");
// ok 不为 1，那么就直接保存便于后续分析，不进行后续处理
fileUtils.saveJSON({
saveFolder: DATA_FOLDER,
now: now,
fileNameSuffix: `origin-error`,
object: result,
compress: true,
uncompress: false
});
return;
}
}
console.log(new Date(Date.now() + 8 * 60 * 60 * 1000).toISOString(), SUB_FOLDER, "请求成功");
// console.log("result", result);
/**
* 保存原始数据
*/
fileUtils.saveJSON({
saveFolder: DATA_FOLDER,
now: now,
fileNameSuffix: `origin`,
object: result,
compress: true,
uncompress: false
});
let data = JSON.parse(JSON.stringify(result.data));
if (!data) {
fileUtils.saveJSON({
saveFolder: DATA_FOLDER,
now: now,
fileNameSuffix: `origin-parse-error`,
object: result,
compress: true,
uncompress: false
});
return;
}
/**
* 过滤掉不需要的数据
*/
// hotgov
if (data.hotgov) {
delete data.hotgov["mblog"];
// 重复字段只保留一个
delete data.hotgov["note"]; // note word
delete data.hotgov["small_icon_desc"]; // icon_desc small_icon_desc
delete data.hotgov["small_icon_desc_color"]; // icon_desc_color small_icon_desc_color
}
// band_list
for (let i = 0; i < data.band_list.length; i++) {
const item = data.band_list[i];
// 过滤广告
if (item.is_ad) {
data.band_list.splice(i, 1);
i--;
}
// 过滤空字段
delete item["ad_info"];
// 重复字段只保留一个
delete item["note"]; // note word
delete item["icon_desc"]; delete item["small_icon_desc"]; // label_name icon_desc small_icon_desc
delete item["small_icon_desc_color"]; // icon_desc_color small_icon_desc_color
delete item["flag_desc"]; // flag_desc subject_label 这两个有值的时候相同，没有值的时候，前一个为 undefined，后一个为 ""
}
/**
* 获取需要的数据，进行转换
*/
let convert = [];
data.band_list.forEach(item => {
let detail = "";
let pic_ids = [];
if (item.mblog) { // 有些热搜没有 mblog
var regex = /(<([^>]+)>)/ig
detail = item.mblog.text.replace(regex, "");
if (item.mblog.pics) {
pic_ids = item.mblog.pics.map(pic => `${pic}`);
}
}
convert.push({
// 热搜排行顺序
rank: item.rank,
realpos: item.realpos,
// 热搜信息
word: item.word, // 热搜标题
word_scheme: item.word_scheme, // 热搜话题 "#热搜标题#"
emoticon: item.emoticon, // 热搜小表情，如 "[泪]"
label_name: item.label_name, // 热搜标签，如 "爆" "热" "新" ""
onboard_time: item.onboard_time, // 热搜上线时间，秒级时间戳，如 1658565575
/**
* 热搜数据
*
* 大部分的 num 和 raw_hot 是相同的，页面上显示的是 num，可能是人工调控的热搜
*
* 两者差值通过观测似乎最大是 1250000
* 例如 【爆】唐山打架事件8名违法嫌疑人已到案 这条热搜一开始 delta 首先不断增大，最大达到 1250000
* 然后热搜数量增加到 12600000 左右的时候，delta 逐渐减小到 1040000 左右
*/
num: item.num,
raw_hot: item.raw_hot,
detla: item.num - item.raw_hot, // 计算值
url: `https://s.weibo.com/weibo?q=${encodeURIComponent(item.word_scheme)}`, // 热搜话题链接
// 分类
category: item.category ? item.category.split(',') : "",
subject_label: item.subject_label,
// 其他
more: {
is_new: item.is_new,
subject_querys: item.subject_querys,
mid: item.mid,
icon_desc_color: item.icon_desc_color,
detail: detail,
},
});
});
fileUtils.saveJSON({
saveFolder: DATA_FOLDER,
now: now,
fileNameSuffix: `final`,
object: convert,
compress: true,
// uncompress: true,
uncompress: false,
});
// /**
// * 只统计微博调控信息
// */
// let convert2 = [];
// let total = 0;
// data.band_list.forEach(item => {
// total += item.num;
// total -= item.raw_hot;
// if (item.num - item.raw_hot == 0) return;
// convert2.push([
// `[${item.realpos}] ${item.word}【${item.label_name}】`,
// `原始:${item.raw_hot} 显示:${item.num} 调控: ${item.num - item.raw_hot}`
// ]);
// });
// fileUtils.saveJSON({
// saveFolder: DATA_FOLDER,
// now: now,
// fileNameSuffix: `regulation`,
// object: {
// total_delta: total, // 所有调控值之和
// data: convert2
// },
// compress: false,
// uncompress: true
// });
// /**
// * 保存预处理后数据
// */
// // 过滤掉不需要的数据
// // band_list
// data.band_list.forEach(function (item) {
// delete item["mblog"];
// });
// fileUtils.saveJSON({
// saveFolder: DATA_FOLDER,
// now: now,
// fileNameSuffix: `simplify`,
// object: data,
// compress: true,
// // uncompress: true,
// // compress: false,
// uncompress: false,
// });
/**
* 更新最新的
*/
fs.writeFileSync(`${DATA_FOLDER}/latest.json`, JSON.stringify({
update_time: requestTimestamp,
update_time_friendly: now.substring(0, 19).replace(/T/g, " "),
// regulation: convert2,
data: convert
}));
}
exports.main = main;

51
hotband/src/utils/fileUtils.js Normal file
View File

@@ -0,0 +1,51 @@
const fs = require('fs');
const path = require('path');
const LATEST_DATA_ONLY = process.env.LATEST_DATA_ONLY == true;
// 创建目录
async function createFolder(folderToCreate) {
let currentFolder = path.join(folderToCreate);
let parentFolder = path.join(currentFolder, '../');
// console.log({ currentFolder: currentFolder, parentFolder: parentFolder });
if (!fs.existsSync(currentFolder)) {
// 文件夹不存在，创建文件夹
createFolder(parentFolder); // 保证父级文件夹存在
fs.mkdirSync(currentFolder); // 创建当前级文件夹
} else {
// 否则就什么也不做
}
}
// 保存 JSON
function saveJSON({ saveFolder, now, fileNameSuffix, object, compress = true, uncompress = true }) {
if (LATEST_DATA_ONLY) return;
let year = now.substring(0, 4);
let month = now.substring(5, 7);
let day = now.substring(8, 10);
let hour = now.substring(11, 13);
let minute = now.substring(14, 16);
// console.log(now);
// console.log( "year, month, day, hour, minute: " + year + ", " + month + ", " + day + ", " + hour + ", " + minute);
// 创建当前文件夹
let folder = `${saveFolder}/${fileNameSuffix}/${year}/${month}/${day}`;
createFolder(folder);
let fileName = `${folder}/${year}${month}${day}_${hour}${minute}`;
// 生成文件名
// '2022-07-23T10:11:38.650Z' => '20220723_1011'
// let fileName = now.replace(/T/, '_').replace(/:\d{2}.\d{3}Z/, '').replace(/[-:]/g, '');
// console.log(`fileName is ${fileName}`);
if (compress)
fs.writeFileSync(`${fileName}.min.json`, JSON.stringify(object));
if (uncompress)
fs.writeFileSync(`${fileName}.json`, JSON.stringify(object, "", "\t"));
}
module.exports = {
createFolder,
saveJSON,
}

27
hotband/src/utils/requestUtils.js Normal file
View File

@@ -0,0 +1,27 @@
const request = require('request');
// 请求 APi 接口
async function getApiResult(url) {
var return_data = await new Promise((resolve) => {
request({
method: 'GET',
url: url,
json: true,
}, (error, response, result) => {
if (!error && (response.statusCode == 200)) {
// 请求成功
resolve(result);
} else {
// 请求失败
console.log(`error is ${error}`);
resolve({});
}
});
});
// console.log(`return_data is ${JSON.stringify(return_data)}`);
return return_data;
}
module.exports = {
getApiResult,
}