canal数据同步流程图

监听binlog

留学产品推荐顾问相关表

问题3：之前原子库基本上curd操作，业务聚合放到es了，es被当作内存库使用

留学产品推荐顾问相关宽表

留学产品楼层floor 

留学产品product

留学产品楼层宽表 

问题2：flink内有业务字段状态判断，偶合业务，不利于后期开发维护

flink反查数据库

方案1：业务层不改变原子表，加业务表，如果是宽表可以直接同步到es

留学产品宽表

kafka

问题1：宽表依赖的外部数据表binlog日志有变化，flink需反查组内其他表有无变化

ES用到相关表留学产品楼层floor: dxz_lan.dxz_system_recommend_place_content dxz_lan.dxz_system_recommend_place dxz_lan.dxz_system_floor_column dxz_biz_abroad.dxz_abroad_product dxz_m_abroad.dxz_valuation_price dxz_biz_abroad.dxz_abroad_product_sale dxz_lan.dxz_statistic_resource_total dxz_lan.dxz_order_detail

canal

es

数据库

将宽表数据同步ES

将监听数据到推送fkala

ES用到相关表留学产品product: dxz_biz_abroad.dxz_abroad_institute_recommend dxz_biz_abroad.dxz_abroad_member_recommend dxz_biz_abroad.dxz_abroad_pack_service dxz_biz_abroad.dxz_abroad_pack_service_details dxz_biz_abroad.dxz_abroad_product dxz_m_abroad.dxz_valuation_price dxz_biz_abroad.dxz_abroad_product_sale dxz_lan.dxz_statistic_resource_total dxz_lan.dxz_order_detail

flink

监听kafka消息

ES用到相关表留学产品推荐顾问相关表：dxz_lan.dxz_institute_sale_authoritydxz_lan.dxz_userdxz_lan.dxz_statistic_user_totaldxz_biz_abroad.dxz_abroad_productdxz_biz_abroad.dxz_abroad_member_recommenddxz_biz_abroad.dxz_abroad_product_analysis_hisdxz_biz_abroad.dxz_abroad_product_saledxz_biz_abroad.dxz_abroad_pack_servicedxz_biz_abroad.dxz_abroad_pack_service_details