Process - workflow 工作流

infoSetting.py

当前维度抓新/补充爬虫

click

content

时间调度任务

Logger.py

调度表

ApiService

GlobalValue.py

base spiders

\"\"\"# todo: font color=\"#f44336\

Command.csv

否

Logger

是

数仓维度负责人

DB Interface

env.yaml

main.py

\"\"\

分割线

http://127.0.0.1:8888/schedule_center/push_requests

APP DB

getDeviceInfo

发起单次调度任务

PexpectTelnetClient

self.host = hostself.child = pexpect.spawn('telnet %s' % (self.host))

connection(self)rootLogin(self)...manyspan style=\"font-size: inherit;\

SSHClient

self.ssh = paramiko.SSHClient()self.ssh._transport = transportself.channel = self.ssh._transport.open_session()

大致时间

getDbConf.py

...

m.s.realtime

接口

数据调度任务

setup.py

device_conn_inc

topic: crawler:spider:realtime

通用非实时的配置

禅道需求任务短会沟通

Connection.py

....

webname

TaskList

env.py

无调度

Client

device_conn_dec

Device.py

数据维度负责人

getConnInfo.py

调度任务发起工作流程图

config.py

getopt.getopt

外来数据

testCelery.py

url

enterprise_report:spider_msg:realtime

等待已有插队调度

全量爬虫/基类爬虫

Sharde/Node

Item(pretty_json)

Producer/Consumer

调度传参条件确定

device_status

中间服务数据调度

远程设备device通过main函数调度流程/优化/重构

Connector.py

开发

recruitment_spiders:spider_msg:realtime

msg

msg api

config.ini

get_ip_dict()readDevice()filterDev()zip_ya()devConn()decreaseConn()Task()usage()getMaxProcessspan style=\

继承

Scheduler

http://127.0.0.1:8888/schedule/get_general_data_supply

删除/合并老的调度

key: recruitment_info(same as general spider)

运营、产品

\"\"\"# todo: opt the import# todo: opt the necessary for some package?# todo: consider the path or variable. why does the result path reference the command file path? why command path using logPath?# todo: opt the logic of try block and if block.Author: Peter\"\"\"

应用端异步实时消息流程图

msg names

UtilsPackage

ParameterParse

contact_spiders:spider_msg:realtime

futures.ThreadPoolExecutor

Device