修正一些逻辑细节

READEME.md

@@ -1,51 +0,0 @@
-# wxbot
-
-微信聊天记录自动存sqlite数据库，基于微信网页版。注意，Web版获取不到微信用户的微信ID，所得到的用户名/ID是仅当前会话有效的，因此只有根据备注名在数据库中识别用户比较靠谱。
-
-项目fork自[@Urinx/WeixinBot](https://github.com/Urinx/WeixinBot)，我只是加了个存到sqlite的壳，感谢老哥的网页版微信API。
-
-## 数据表结构
-| 字段名 | 类型 | 说明 |
-|  --   | --  | --   |
-| msg_id| TEXT | 消息ID |
-|create_time | INTEGER | 创建时间，Unix时间戳 |
-| msg_type | INTEGER | 消息类型，1-文本，3-图片，34-语音，43-视频，0-其它
-| content | TEXT | 消息的文本内容
-| from_id | TEXT | 消息发送者ID（不是微信ID，没啥用），如果是来自群的消息此字段为空
-| from_name | TEXT | 消息发送者的备注，没备注就昵称，如果是来自群的消息此字段为空
-| from_nickname | TEXT | 消息发送者的昵称，如果是来自群的消息此字段为空
-| to_id | TEXT | 接收者ID，如果是发送到群的消息此字段为空
-| to_name | TEXT | 接收者备注，没备注就昵称，如果是发送到群的消息此字段为空
-| to_nickname | TEXT | 接收者昵称，如果是发送到群的消息此字段为空
-| group_id | TEXT | 如果是群消息则为群的ID否则为空，照样，没啥用
-| group_name | TEXT | 如果是群消息则为群聊名称（应该吧🤣我不关注群消息），否则为空
-
-消息里的图片等文件以`img_`，`voice_`等前缀 + 消息ID存储于命令行参数给定的数据目录中。
-
-其它信息请参考[项目@Urinx/WeixinBot](https://github.com/Urinx/WeixinBot)。
-
-## 使用
-### 克隆仓库并安装python依赖
-```shell
-# 不适用win系统
-git clone https://github.com/mattuylee/wxbot.git
-cd wxbot
-python3 -m venv ./venv
-source ./venv/bin/activate
-pip3 install -r ./requirements.txt
-```
-
-### 运行
-`python ./main.py -f <DB_FILE> -d <DATA_DIR>`
-其中`DB_FILE`是sqlite数据库文件路径，`DATA_DIR`是图片视频等数据的存储目录。
-
-支持sqlcipher加密，程序启动会要求输入密码，如不需要加密可留空直接回车。注意，以后每次启动必须输入相同的密码。暂不支持修改密码，可自行安装sqlcipher打开数据库修改密码。
-
-未加密数据库可直接通过sqlite3打开，加密数据库需要sqlcipher才能打开。程序只负责记录数据，没有读取能力。
-
-### 其它
-关于表情包，如果是从表情商店里添加的表情包网页版微信是获取不到的，为了保护创作者的权益。这倒是可以理解。
-
-如果发现无法登录先试试在浏览器登录网页版微信，有可能是腾讯为了“安全”限制了你的微信帐号在网页登录。
-
-最后，祝腾讯早点倒闭。

README.md

@@ -7,20 +7,20 @@
 ## 数据表结构
 | 字段名 | 类型 | 说明 |
 |  --   | --  | --   |
-| msg_id| TEXT | 消息ID |
+| msg_id| TEXT | 消息ID，结构为创建时间+原微信消息ID |
 |create_time | INTEGER | 创建时间，Unix时间戳 |
-| msg_type | INTEGER | 消息类型，1-文本，3-图片，34-语音，43-视频，0-其它
+| msg_type | INTEGER | 消息类型，1-文本，3-图片，34-语音，43-视频，0-其它 |
-| content | TEXT | 消息的文本内容
+| content | TEXT | 消息的文本内容 |
-| from_id | TEXT | 消息发送者ID（不是微信ID，没啥用），如果是来自群的消息此字段为空
+| from_id | TEXT | 消息发送者ID（不是微信ID，没啥用），如果是来自群的消息此字段为空 |
-| from_name | TEXT | 消息发送者的备注，没备注就昵称，如果是来自群的消息此字段为空
+| from_name | TEXT | 消息发送者的备注，没备注就昵称，如果是来自群的消息此字段为空 |
-| from_nickname | TEXT | 消息发送者的昵称，如果是来自群的消息此字段为空
+| from_nickname | TEXT | 消息发送者的昵称，如果是来自群的消息此字段为空 |
-| to_id | TEXT | 接收者ID，如果是发送到群的消息此字段为空
+| to_id | TEXT | 接收者ID，如果是发送到群的消息此字段为空 |
-| to_name | TEXT | 接收者备注，没备注就昵称，如果是发送到群的消息此字段为空
+| to_name | TEXT | 接收者备注，没备注就昵称，如果是发送到群的消息此字段为空 |
-| to_nickname | TEXT | 接收者昵称，如果是发送到群的消息此字段为空
+| to_nickname | TEXT | 接收者昵称，如果是发送到群的消息此字段为空 |
-| group_id | TEXT | 如果是群消息则为群的ID否则为空，照样，没啥用
+| group_id | TEXT | 如果是群消息则为群的ID否则为空，照样，没啥用 |
-| group_name | TEXT | 如果是群消息则为群聊名称（应该吧🤣我不关注群消息），否则为空
+| group_name | TEXT | 如果是群消息则为群聊名称（应该吧🤣我不关注群消息），否则为空 |
 
-消息里的图片等文件以`img_`，`voice_`等前缀 + 消息ID存储于命令行参数给定的数据目录中。
+消息里的图片等文件以消息ID存储于命令行参数给定的数据目录中，并根据年份划分子目录。
 
 其它信息请参考[项目@Urinx/WeixinBot](https://github.com/Urinx/WeixinBot)。
 

main.py

@@ -1,7 +1,10 @@
+#!/usr/bin/env python3
+# coding: utf-8
 import argparse
 import getpass
 import logging
 import sys
+import time
 import os
 from weixin import WebWeixin
 from sqlcipher3 import dbapi2 as sqlite
@@ -9,8 +12,9 @@ from sqlcipher3 import dbapi2 as sqlite
 
 def parse_args():
     parser = argparse.ArgumentParser()
-    parser.add_argument("-f", "--db-file", required=True, help="数据库文件")
+    parser.add_argument('-f', '--db-file', required=True, help='数据库文件')
-    parser.add_argument("-d", "--data-dir",required=True, help="文件存储目录")
+    parser.add_argument('-d', '--data-dir',required=True, help='文件存储目录')
+    parser.add_argument('--debug', action='store_true', help='调试模式')
     args = parser.parse_args()
     return args
 
@@ -24,69 +28,116 @@ class WXBot(WebWeixin):
                 return member['NickName']
         return None
 
+    def composeMessageID(self, msgID, timestamp):
+        return time.strftime('%Y%m%d%H%M%S', time.localtime(timestamp)) + msgID
+
+    def saveMessageImage(self, msgid, composedID):
+        url = self.base_uri + \
+            '/webwxgetmsgimg?MsgID=%s&skey=%s' % (msgid, self.skey)
+        data = self._get(url, 'webwxgetmsgimg')
+        return self.saveFile(composedID[:4], composedID + '.jpg', data)
+
+    def saveVoice(self, msgid, composedID):
+        url = self.base_uri + \
+            '/webwxgetvoice?msgid=%s&skey=%s' % (msgid, self.skey)
+        data = self._get(url, api='webwxgetvoice')
+        return self.saveFile(composedID[:4], composedID + '.mp3', data)
+
+    def saveVideo(self, msgid, composedID):
+        url = self.base_uri + \
+            '/webwxgetvideo?msgid=%s&skey=%s' % (msgid, self.skey)
+        data = self._get(url, api='webwxgetvideo')
+        return self.saveFile(composedID[:4], composedID + '.mp4', data)
+
+    def saveFile(self, subdir, filename, data):
+        if data == '':
+            return False
+        dirName = os.path.join(dataDir, subdir)
+        if not os.path.exists(dirName):
+            os.makedirs(dirName)
+        fn = os.path.join(dirName, filename)
+        with open(fn, 'wb') as f:
+            f.write(data)
+            f.close()
+        return True
+
     def handleMsg(self, r):
+        if self.DEBUG:
+            print("处理消息, NewMessageCount =", r['AddMsgCount'])
         for msg in r['AddMsgList']:
             msgID = msg['MsgId']
+            composedID = self.composeMessageID(msgID, msg['CreateTime'])
             msgType = msg['MsgType']
+            if msg['FromUserName'] in self.SpecialUsers or msg['ToUserName'] in self.SpecialUsers:
+                # 过滤特殊帐号消息
+                continue
+            elif msgType in (50, 51, 52, 53, 9999):
+                #无用消息
+                continue
             sql = "INSERT INTO msg (\
                 msg_id,\
                 create_time,\
                 msg_type,\
                 content,\
-                from_id,\
                 from_name,\
                 from_nickname,\
-                to_id,\
                 to_name,\
                 to_nickname,\
-                group_id,\
                 group_name\
             ) VALUES (\
                 :msg_id,\
                 :create_time,\
                 :msg_type,\
                 :content,\
-                :from_id,\
                 :from_name,\
                 :from_nickname,\
-                :to_id,\
                 :to_name,\
                 :to_nickname,\
-                :group_id,\
                 :group_name\
             )"
             params = {
-                "msg_id": msgID,
+                "msg_id": composedID,
                 "create_time": msg['CreateTime'],
                 "msg_type": msgType,
                 "content": None,
-                "from_id": msg['FromUserName'],
                 "from_name": self.getUserRemarkName(msg['FromUserName']),
                 "from_nickname": self.getUserNickName(msg['FromUserName']),
-                "to_id": msg['ToUserName'],
                 "to_name": self.getUserRemarkName(msg['ToUserName']),
                 "to_nickname": self.getUserNickName(msg['ToUserName']),
-                "group_id": None,
                 "group_name": None
             }
-            # 过滤特殊帐号消息
+            if msg['FromUserName'] == self.User['UserName']:
-            if msg['FromUserName'] in self.SpecialUsers or msg['ToUserName'] in self.SpecialUsers:
+                params['from_name'] = '_self'
-                continue
+            elif msg['ToUserName'] == self.User['UserName']:
+                params['to_name'] = '_self'
+            if msg['FromUserName'][:2] == '@@':
+                # 群消息
+                if ":<br/>" in content:
+                    [people, content] = msg['Content'].split(':<br/>', 1)
+                    params['group_name'] = self.getUserRemarkName(msg['FromUserName'])
+                    params['from_name'] = self.getUserRemarkName(people)
+                    params['to_name'] = params['to_nickname'] = None
+            elif msg['ToUserName'][:2] == '@@':
+                # 发送的群消息
+                params['group_name'] = self.getUserRemarkName(msg['ToUserName'])
+                params['from_name'] = self.User['NickName']
+                params['to_name'] = params['to_nickname'] = None
             if msgType == 1:
+                # 文字消息
                 params['content'] = msg['Content'].replace('&lt;', '<').replace('&gt;', '>')
             elif msgType == 3:
-                params['content'] = "[图片](%s)" % msgID
+                params['content'] = "[图片](%s)" % composedID
-                self.webwxgetmsgimg(msgID)
+                self.saveMessageImage(msgID, composedID)
             elif msgType == 34:
-                params['content'] = "[语音](%s)" % msgID
+                params['content'] = "[语音](%s)" % composedID
-                self.webwxgetvoice(msgID)
+                self.saveVoice(msgID, composedID)
             elif msgType == 43:
-                params['content'] = "[视频](%s)" % msgID
+                params['content'] = "[视频](%s)" % composedID
-                self.webwxgetvideo(msgID)
+                self.saveVideo(msgID, composedID)
             elif msgType == 62:
                 params['msg_type'] = 43
-                params['content'] = "[小视频](%s)" % msgID
+                params['content'] = "[小视频](%s)" % composedID
-                self.webwxgetvideo(msgID)
+                self.saveVideo(msgID, composedID)
             elif msgType == 42:
                 params['msg_type'] = 0
                 params['content'] = "[名片](%s)" % msg['RecommendInfo']['NickName']
@@ -97,84 +148,70 @@ class WXBot(WebWeixin):
                     params['content'] = "[表情包图片]"
                 else:
                     params['msg_type'] = 3
-                    params['content'] = "[动画表情](%s)" % msgID
+                    params['content'] = "[动画表情](%s)" % composedID
-                    self.webwxgetmsgimg(msgID)
+                    self.saveMessageImage(msgID, composedID)
             elif msgType == 49:
                 params['msg_type'] = 0
                 params['content'] = "[链接：%s](%s)" % (msg['FileName'], msg['Url'])
-            elif (50, 51, 52, 53, 9999, 10000, 10002).index(msgType) >= 0:
+            elif msgType == 10000:
-                continue
+                params['msg_type'] = 0
+                params['content'] = "[系统消息](%s)" % (msg['Content'])
+            elif msgType == 10002:
+                params['msg_type'] = 0
+                params['content'] = "[系统消息](%s撤回了一条消息)" % params['from_name']
             else:
                 params['msg_type'] = 0
-                params['content'] = ""
+                params['content'] = "[未识别消息]" + msg['Content']
-                continue
-            # 群消息
-            if msg['FromUserName'][:2] == '@@':
-                if ":<br/>" in content:
-                    [people, content] = msg['Content'].split(':<br/>', 1)
-                    params['group_name'] = self.getUserRemarkName(msg['FromUserName'])
-                    params['from_id'] = people
-                    params['from_name'] = self.getUserRemarkName(people)
-                    params['to_id'] = params['to_name'] = params['to_nickname'] = None
-            elif msg['ToUserName'][:2] == '@@':
-                params['group_name'] = self.getUserRemarkName(msg['ToUserName'])
-                params['from_id'] = self.User['UserName']
-                params['from_name'] = self.User['NickName']
-                params['to_id'] = params['to_name'] = params['to_nickname'] = None
             conn.execute(sql, params)
             conn.commit()
 
 
-args = parse_args()
-dataDir = args.data_dir
-if not os.access(os.path.dirname(args.db_file), os.X_OK) and not os.access(args.db_file, os.W_OK):
-    print("无法创建或打开数据库文件")
-    exit()
-elif not os.access(dataDir, os.X_OK):
-    print("数据目录不存在或权限不足")
-    exit()
-conn = sqlite.connect(args.db_file)
-
-while True:
-    try:
-        passwd = getpass.getpass("输入数据库密码：")
-        if passwd != '':
-            conn.execute("PRAGMA KEY = %s" % passwd)
-        conn.execute("CREATE TABLE IF NOT EXISTS msg (\
-            msg_id TEXT NOT NULL,\
-            create_time INTEGER NOT NULL,\
-            msg_type INTEGER NOT NULL,\
-            content TEXT,\
-            from_id TEXT,\
-            from_name TEXT,\
-            from_nickname TEXT,\
-            to_id TEXT,\
-            to_name TEXT,\
-            to_nickname TEXT,\
-            group_id TEXT,\
-            group_name TEXT\
-        );")
-        break
-    except sqlite.DatabaseError as e:
-        if 'file is not a database' in e.args:
-            print("密码错误，请重试")
-        else:
-            print("打开数据库失败", e)
-    except KeyboardInterrupt:
-        exit(1)
-    except Exception as e:
-        print("打开数据库失败", e)
-        exit(1)
-
-
-if sys.stdout.encoding == 'cp936':
-    sys.stdout = UnicodeStreamFilter(sys.stdout)
-
 if __name__ == '__main__':
+    if sys.stdout.encoding == 'cp936':
+        sys.stdout = UnicodeStreamFilter(sys.stdout)
+
     logger = logging.getLogger(__name__)
     if not sys.platform.startswith('win'):
         import coloredlogs
         coloredlogs.install(level='INFO')
+
+    args = parse_args()
+    dataDir = args.data_dir
+    if not os.access(os.path.dirname(args.db_file), os.X_OK) and not os.access(args.db_file, os.W_OK):
+        print("无法创建或打开数据库文件")
+        exit()
+    elif not os.access(dataDir, os.X_OK):
+        print("数据目录不存在或权限不足")
+        exit()
+    conn = sqlite.connect(args.db_file)
+    while True:
+        try:
+            passwd = getpass.getpass("输入数据库密码：")
+            if passwd != '':
+                conn.execute("PRAGMA KEY = %s" % passwd)
+            conn.execute("CREATE TABLE IF NOT EXISTS msg (\
+                msg_id TEXT NOT NULL,\
+                create_time INTEGER NOT NULL,\
+                msg_type INTEGER NOT NULL,\
+                content TEXT,\
+                from_name TEXT,\
+                from_nickname TEXT,\
+                to_name TEXT,\
+                to_nickname TEXT,\
+                group_name TEXT\
+            );")
+            break
+        except sqlite.DatabaseError as e:
+            if 'file is not a database' in e.args:
+                print("密码错误，请重试")
+            else:
+                print("打开数据库失败", e)
+        except KeyboardInterrupt:
+            exit(1)
+        except Exception as e:
+            print("打开数据库失败", e)
+            exit(1)
     wxbot = WXBot()
+    wxbot.DEBUG = args.debug
     wxbot.saveFolder = dataDir
     wxbot.start()

weixin.py

@@ -118,7 +118,6 @@ class WebWeixin(object):
         self.memberCount = 0
         self.SpecialUsers = ['newsapp', 'fmessage', 'filehelper', 'weibo', 'qqmail', 'fmessage', 'tmessage', 'qmessage', 'qqsync', 'floatbottle', 'lbsapp', 'shakeapp', 'medianote', 'qqfriend', 'readerapp', 'blogapp', 'facebookapp', 'masssendapp', 'meishiapp', 'feedsapp',
                              'voip', 'blogappweixin', 'weixin', 'brandsessionholder', 'weixinreminder', 'wxid_novlwrv3lqwv11', 'gh_22b87fa7cb3c', 'officialaccounts', 'notification_messages', 'wxid_novlwrv3lqwv11', 'gh_22b87fa7cb3c', 'wxitil', 'userexperience_alarm', 'notification_messages']
-        self.TimeOut = 20  # 同步最短时间间隔（单位：秒）
         self.media_count = -1
 
         self.cookie = http.cookiejar.CookieJar()
@@ -265,6 +264,15 @@ class WebWeixin(object):
         }
         return True
 
+    def logout(self):
+        request = urllib.request.Request(
+            url='https://wx.qq.com/cgi-bin/mmwebwx-bin/webwxlogout?redirect=0&type=1&skey=%s' % self.skey,
+            data=urllib.parse.urlencode({
+                'sid': self.sid,
+                'uin': self.uin,
+            }).encode(encoding='utf-8'))
+        urllib.request.urlopen(request)
+
     def webwxinit(self):
         url = self.base_uri + '/webwxinit?pass_ticket=%s&skey=%s&r=%s' % (
             self.pass_ticket, self.skey, int(time.time()))
@@ -406,7 +414,7 @@ class WebWeixin(object):
             '_': int(time.time()),
         }
         url = 'https://' + self.syncHost + '/cgi-bin/mmwebwx-bin/synccheck?' + urllib.parse.urlencode(params)
-        data = self._get(url, timeout=5)
+        data = self._get(url, timeout=30)
         if data == '':
             return [-1,-1]
 
@@ -893,24 +901,18 @@ class WebWeixin(object):
                 logging.debug('[*] 你在其他地方登录了 WEB 版微信，债见')
                 break
             elif retcode == '0':
-                if selector == '2':
+                shouldSleep = True
+                if selector != '0':
+                    # 有需要同步的消息
                     r = self.webwxsync()
-                    if r is not None:
+                    if r is not None and r['AddMsgCount'] > 0:
+                        shouldSleep = False
                         self.handleMsg(r)
-                elif selector == '6':
+                if shouldSleep and time.time() - self.lastCheckTs <= 5:
-                    # TODO
+                    # 微信有bug，发送某些消息后轮询会立马返回并报告有消息，但是调用同步接口后发现根本没有新消息，当收到其它消息后又会恢复
-                    redEnvelope += 1
+                    # 这个现象一般出现在发送表情包之后。可以在浏览器打开控制台，会发现在手机发送表情后轮询接口被疯狂调用
-                    print('[*] 收到疑似红包消息 %d 次' % redEnvelope)
+                    # 这里是防止轮询接口调用太频繁而实际上根本没有消息，浪费带宽
-                    logging.debug('[*] 收到疑似红包消息 %d 次' % redEnvelope)
+                    time.sleep(5)
-                elif selector == '7':
-                    playWeChat += 1
-                    print('[*] 你在手机上玩微信被我发现了 %d 次' % playWeChat)
-                    logging.debug('[*] 你在手机上玩微信被我发现了 %d 次' % playWeChat)
-                    r = self.webwxsync()
-                elif selector == '0':
-                    time.sleep(1)
-            if (time.time() - self.lastCheckTs) <= 20:
-                time.sleep(time.time() - self.lastCheckTs)
 
     def sendMsg(self, name, word, isfile=False):
         id = self.getUSerID(name)
@@ -1020,6 +1022,7 @@ class WebWeixin(object):
                 listenProcess.terminate()
                 print('[*] 退出微信')
                 logging.debug('[*] 退出微信')
+                self.logout()
                 exit()
             elif text[:2] == '->':
                 [name, word] = text[2:].split(':')

wxbot.pdman.json

@@ -1,8 +1,8 @@
 {
   "modules": [
     {
-      "name": "core",
+      "name": "wxbot",
-      "chnname": "NOTA核心",
+      "chnname": "wxbot",
       "entities": [
         {
           "title": "msg",
@@ -37,12 +37,6 @@
               "remark": "",
               "chnname": "消息内容"
             },
-            {
-              "name": "from_id",
-              "type": "TEXT_ID",
-              "remark": "",
-              "chnname": "发送者ID"
-            },
             {
               "name": "from_name",
               "type": "TEXT",
@@ -55,12 +49,6 @@
               "remark": "昵称",
               "chnname": "发送者昵称"
             },
-            {
-              "name": "to_id",
-              "type": "TEXT_ID",
-              "remark": "",
-              "chnname": "接收者ID"
-            },
             {
               "name": "to_name",
               "type": "TEXT",
@@ -73,12 +61,6 @@
               "remark": "",
               "chnname": "接收者昵称"
             },
-            {
-              "name": "group_id",
-              "type": "TEXT_ID",
-              "remark": "",
-              "chnname": "群组ID"
-            },
             {
               "name": "group_name",
               "type": "TEXT",