canal数据同步流程图
2020-11-24 17:27:49 0 举报
flink实时聚合宽表同步es
作者其他创作
大纲/内容
监听binlog
留学产品推荐顾问相关表
问题3:之前原子库基本上curd操作,业务聚合放到es了,es被当作内存库使用
留学产品推荐顾问相关宽表
留学产品楼层floor
留学产品product
留学产品楼层宽表
问题2:flink内有业务字段状态判断,偶合业务,不利于后期开发维护
flink反查数据库
方案1:业务层不改变原子表,加业务表,如果是宽表可以直接同步到es
留学产品宽表
kafka
问题1:宽表依赖的外部数据表binlog日志有变化,flink需反查组内其他表有无变化
ES用到相关表留学产品楼层floor: dxz_lan.dxz_system_recommend_place_content dxz_lan.dxz_system_recommend_place dxz_lan.dxz_system_floor_column dxz_biz_abroad.dxz_abroad_product dxz_m_abroad.dxz_valuation_price dxz_biz_abroad.dxz_abroad_product_sale dxz_lan.dxz_statistic_resource_total dxz_lan.dxz_order_detail
canal
es
数据库
将宽表数据同步ES
将监听数据到推送fkala
ES用到相关表留学产品product: dxz_biz_abroad.dxz_abroad_institute_recommend dxz_biz_abroad.dxz_abroad_member_recommend dxz_biz_abroad.dxz_abroad_pack_service dxz_biz_abroad.dxz_abroad_pack_service_details dxz_biz_abroad.dxz_abroad_product dxz_m_abroad.dxz_valuation_price dxz_biz_abroad.dxz_abroad_product_sale dxz_lan.dxz_statistic_resource_total dxz_lan.dxz_order_detail
flink
监听kafka消息
ES用到相关表留学产品推荐顾问相关表:dxz_lan.dxz_institute_sale_authoritydxz_lan.dxz_userdxz_lan.dxz_statistic_user_totaldxz_biz_abroad.dxz_abroad_productdxz_biz_abroad.dxz_abroad_member_recommenddxz_biz_abroad.dxz_abroad_product_analysis_hisdxz_biz_abroad.dxz_abroad_product_saledxz_biz_abroad.dxz_abroad_pack_servicedxz_biz_abroad.dxz_abroad_pack_service_details
收藏
0 条评论
下一页