{"id":245,"date":"2022-02-14T11:15:50","date_gmt":"2022-02-14T11:15:50","guid":{"rendered":"https:\/\/liangqi.org\/?p=245"},"modified":"2022-02-14T11:15:50","modified_gmt":"2022-02-14T11:15:50","slug":"data-connect%e7%9a%84%e6%b6%88%e6%81%af%e5%a4%84%e7%90%86%e6%9c%ba%e5%88%b6","status":"publish","type":"post","link":"https:\/\/liangqi.org\/?p=245","title":{"rendered":"Data Connect\u7684\u6d88\u606f\u5904\u7406\u673a\u5236"},"content":{"rendered":"\n<p>Data Connect Service\u63d0\u4f9b\u540c\u6b65\u6570\u636e\u5230\u7b2c\u4e09\u65b9\u3002\u6570\u636e\u7684change\u662f\u6765\u81ea\u6570\u636e\u5e93binlog\u751f\u6210\u7684CDC (Change Data Capture)\u3002<\/p>\n\n\n\n<p>\u5728\u7cfb\u7edf\u8bbe\u8ba1\u4e2d\u6709\u51e0\u4e2a\u6311\u6218<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>Ordering\u7684\u95ee\u9898\u3002\u524d\u7aef\u7684service\u5e76\u4e0d\u4fdd\u8bc1\u987a\u5e8f\u3002\u90a3Data Connect\u5982\u4f55\u4fdd\u8bc1\u987a\u5e8f\u5462\uff1f\u6280\u672f\u4e0a\u6765\u8bb2\u662f\u4e0d\u53ef\u80fd\u7684\u3002\u4f46\u662f\u6211\u4eec\u53ef\u4ee5\u505a\u5230\u5728\u4e0d\u4fdd\u8bc1\u987a\u5e8f\u7684\u60c5\u51b5\u4e0b\u4f9d\u7136\u80fd\u6b63\u786e\u5904\u7406\u3002\u65b9\u6cd5\u662f\u6211\u4eec\u4ece\u6700\u5f00\u59cb\u7684\u670d\u52a1\u5f00\u59cb\u5f15\u5165\u4e86Versioning\u7684\u6982\u5ff5\u3002\u5982\u679cdata connect\u62ff\u5230\u7684\u5bf9\u4e8e\u67d0\u4e00\u6761\u8bb0\u5f55\u7684event\u65e9\u4e8e\u4e4b\u524d\u7684event\uff0c\u6211\u4eec\u5c31\u4e22\u5f03\u3002\u8fd9\u4e2a\u903b\u8f91\u6211\u4eec\u53ebstale checker\u3002\u8fd9\u4e2a\u5728sync\u7684\u903b\u8f91\u91cc\u662f\u53ef\u884c\u7684\u3002\u6bd4\u5982\u8bf4\u6211\u4eec\u6709\u201cCreate->Update&#8221;,\u5176\u5b9e\u5bf9\u4e8esync\uff0ccreate\u548cupdate\u90fd\u662f\u57fa\u4e8eId\u7684upsert\uff0c\u5982\u679c\u6211\u4eec\u62ff\u5230create\u665a\u4e00\u4e9b\uff0c\u5c31\u76f4\u63a5\u4e22\u5f03\u3002\u540c\u6837\u7684\u201cupdate->update&#8221;\u4e5f\u6ca1\u95ee\u9898\uff0c\u5f53\u7136\u524d\u63d0\u662f\u6bcf\u6b21\u6211\u4eec\u90fd\u662f\u62ff\u5230\u6574\u4e2a\u8bb0\u5f55\u7684full payload\uff0c\u4e0d\u7136\u5c31\u4f1a\u6709\u95ee\u9898\u3002\u6211\u4eec\u5e76\u6ca1\u6709field level\u7684versioning\u3002<\/li><li>Duplication\u3002Duplication\u5176\u5b9e\u6211\u4eec\u5e76\u4e0d\u592a\u5728\u610f\uff0c\u56e0\u4e3a\u6211\u4eec\u7684\u670d\u52a1\u57fa\u672c\u4e0a\u662fIdempotent\u7684\uff0c\u4e5f\u5c31\u662f\u8bf4\u540c\u6837\u7684\u6d88\u606f\u53ef\u4ee5\u88ab\u5904\u7406\u4e24\u904d\uff0c\u6ca1\u95ee\u9898\u3002\u539f\u56e0\u662fupdate\u672c\u6765\u5c31\u53ef\u4ee5\u91cd\u65b0\u6267\u884c\uff0ccreate\u4e5f\u53ef\u4ee5\uff0c\u56e0\u4e3a\u6211\u4eec\u662f\u6839\u636eid\u6765\u7684\u3002\u8fd8\u6709\u5c31\u662f\u6211\u4eec\u6709stale checker\u5176\u5b9e\u5c31\u80fd\u89e3\u51b3\u5f88\u591a\u7684\u95ee\u9898\u3002<\/li><li>Partition. \u6700\u5f00\u59cb\u7684\u7248\u672c\u6ca1\u6709separate topic\u3002\u6211\u4eecpartition by billing account id\u3002 \u597d\u5904\u6709\u51e0\u4e2a<ul><li>\u51cf\u5c11\u4e00\u4e2atenant\u5360\u7528\u6240\u6709\u7684consumer\u7684\u95ee\u9898\u3002\u5f53\u7136side effect\u662f\u67d0\u4e00\u4e9bconsumer\u4f1a\u6bd4\u8f83\u6162\uff0c\u5982\u679c\u4e00\u4e2atenant traffic\u6bd4\u8f83\u591a\u3002<\/li><li>\u51cf\u5c11row locking\u3002<\/li><li>\u63a7\u5236\u5e76\u53d1\u3002\u5176\u5b9e\u5e76\u53d1\u7684\u63a7\u5236\u6211\u4eec\u5176\u5b9e\u53ef\u4ee5\u7528<a href=\"https:\/\/liangqi.org\/?p=222\">Resillience4j \u2013 \u8fdc\u7a0b\u65b9\u6cd5\u8c03\u7528\u7684\u5de5\u5177\u7bb1<\/a> \u91cc\u9762\u63d0\u5230\u7684\u540c\u6b65\u63a7\u5236\u673a\u5236\u3002\u4f46\u662f\u4e5f\u4f1a\u6709\u8d1f\u9762\u5f71\u54cd\u3002<\/li><\/ul><\/li><\/ul>\n\n\n\n<p>\u6f5c\u5728\u7684enhance<\/p>\n\n\n\n<ul class=\"wp-block-list\"><li>workers\u3002\u6211\u4eec\u4e0d\u5e94\u8be5consumer thread\u91cc\u9762\u5904\u7406\u6240\u6709\u7684\u4e1c\u897f\u3002\u5e94\u8be5\u7528workers\uff0c\u8fd9\u6837\u5c31\u66f4\u5bb9\u6613scale out\u3002\u95ee\u9898\u662f\u6211\u4eec\u8fd8\u662f\u8981\u4fdd\u8bc1worker\u7684\u540c\u6b65\u95ee\u9898\u3002\u5982\u679c\u6709\u4e24\u4e2aupdate\u5bf9\u4e8e\u540c\u4e00\u4e2abilling account\uff0c\u6211\u4eec\u8981\u4fdd\u8bc1scale checker\u4f9d\u7136\u53ef\u4ee5\u5de5\u4f5c\u3002<\/li><li>\u51cf\u5c11tenant\u4e4b\u95f4\u7684\u5f71\u54cd\u3002\u5e94\u8be5\u53ef\u4ee5\u7528dedicated topic\u6765\u505a\u3002\u5f53\u7136\u673a\u5236\u53ef\u80fd\u4f1a\u6bd4\u8f83\u590d\u6742\u3002\u6211\u4eec\u5e94\u8be5\u8fd8\u662f\u9700\u8981common pool\u6765\u51fa\u6765ad-hoc traffic<\/li><\/ul>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Data Connect Service\u63d0\u4f9b\u540c\u6b65\u6570\u636e\u5230\u7b2c\u4e09\u65b9\u3002\u6570\u636e\u7684change\u662f\u6765\u81ea\u6570\u636e\u5e93binlog\u751f\u6210\u7684CDC (Change Data Capture)\u3002 \u5728\u7cfb\u7edf\u8bbe\u8ba1\u4e2d\u6709\u51e0\u4e2a\u6311\u6218 Ordering\u7684\u95ee\u9898\u3002\u524d\u7aef\u7684service\u5e76\u4e0d\u4fdd\u8bc1\u987a\u5e8f\u3002\u90a3Data Connect\u5982\u4f55\u4fdd\u8bc1\u987a\u5e8f\u5462\uff1f\u6280\u672f\u4e0a\u6765\u8bb2\u662f\u4e0d\u53ef\u80fd\u7684\u3002\u4f46\u662f\u6211\u4eec\u53ef\u4ee5\u505a\u5230\u5728\u4e0d\u4fdd\u8bc1\u987a\u5e8f\u7684\u60c5\u51b5\u4e0b\u4f9d\u7136\u80fd\u6b63\u786e\u5904\u7406\u3002\u65b9\u6cd5\u662f\u6211\u4eec\u4ece\u6700\u5f00\u59cb\u7684\u670d\u52a1\u5f00\u59cb\u5f15\u5165\u4e86Versioning\u7684\u6982\u5ff5\u3002\u5982\u679cdata connect\u62ff\u5230\u7684\u5bf9\u4e8e\u67d0\u4e00\u6761\u8bb0\u5f55\u7684event\u65e9\u4e8e\u4e4b\u524d\u7684event\uff0c\u6211\u4eec\u5c31\u4e22\u5f03\u3002\u8fd9\u4e2a\u903b\u8f91\u6211\u4eec\u53ebstale checker\u3002\u8fd9\u4e2a\u5728sync\u7684\u903b\u8f91\u91cc\u662f\u53ef\u884c\u7684\u3002\u6bd4\u5982\u8bf4\u6211\u4eec\u6709\u201cCreate->Update&#8221;,\u5176\u5b9e\u5bf9\u4e8esync\uff0ccreate\u548cupdate\u90fd\u662f\u57fa\u4e8eId\u7684upsert\uff0c\u5982\u679c\u6211\u4eec\u62ff\u5230create\u665a\u4e00\u4e9b\uff0c\u5c31\u76f4\u63a5\u4e22\u5f03\u3002\u540c\u6837\u7684\u201cupdate->update&#8221;\u4e5f\u6ca1\u95ee\u9898\uff0c\u5f53\u7136\u524d\u63d0\u662f\u6bcf\u6b21\u6211\u4eec\u90fd\u662f\u62ff\u5230\u6574\u4e2a\u8bb0\u5f55\u7684full payload\uff0c\u4e0d\u7136\u5c31\u4f1a\u6709\u95ee\u9898\u3002\u6211\u4eec\u5e76\u6ca1\u6709field level\u7684versioning\u3002 Duplication\u3002Duplication\u5176\u5b9e\u6211\u4eec\u5e76\u4e0d\u592a\u5728\u610f\uff0c\u56e0\u4e3a\u6211\u4eec\u7684\u670d\u52a1\u57fa\u672c\u4e0a\u662fIdempotent\u7684\uff0c\u4e5f\u5c31\u662f\u8bf4\u540c\u6837\u7684\u6d88\u606f\u53ef\u4ee5\u88ab\u5904\u7406\u4e24\u904d\uff0c\u6ca1\u95ee\u9898\u3002\u539f\u56e0\u662fupdate\u672c\u6765\u5c31\u53ef\u4ee5\u91cd\u65b0\u6267\u884c\uff0ccreate\u4e5f\u53ef\u4ee5\uff0c\u56e0\u4e3a\u6211\u4eec\u662f\u6839\u636eid\u6765\u7684\u3002\u8fd8\u6709\u5c31\u662f\u6211\u4eec\u6709stale checker\u5176\u5b9e\u5c31\u80fd\u89e3\u51b3\u5f88\u591a\u7684\u95ee\u9898\u3002 Partition. \u6700\u5f00\u59cb\u7684\u7248\u672c\u6ca1\u6709separate topic\u3002\u6211\u4eecpartition by billing account id\u3002 \u597d\u5904\u6709\u51e0\u4e2a \u51cf\u5c11\u4e00\u4e2atenant\u5360\u7528\u6240\u6709\u7684consumer\u7684\u95ee\u9898\u3002\u5f53\u7136side effect\u662f\u67d0\u4e00\u4e9bconsumer\u4f1a\u6bd4\u8f83\u6162\uff0c\u5982\u679c\u4e00\u4e2atenant traffic\u6bd4\u8f83\u591a\u3002 \u51cf\u5c11row locking\u3002 \u63a7\u5236\u5e76\u53d1\u3002\u5176\u5b9e\u5e76\u53d1\u7684\u63a7\u5236\u6211\u4eec\u5176\u5b9e\u53ef\u4ee5\u7528Resillience4j \u2013 \u8fdc\u7a0b\u65b9\u6cd5\u8c03\u7528\u7684\u5de5\u5177\u7bb1 \u91cc\u9762\u63d0\u5230\u7684\u540c\u6b65\u63a7\u5236\u673a\u5236\u3002\u4f46\u662f\u4e5f\u4f1a\u6709\u8d1f\u9762\u5f71\u54cd\u3002 \u6f5c\u5728\u7684enhance workers\u3002\u6211\u4eec\u4e0d\u5e94\u8be5consumer thread\u91cc\u9762\u5904\u7406\u6240\u6709\u7684\u4e1c\u897f\u3002\u5e94\u8be5\u7528workers\uff0c\u8fd9\u6837\u5c31\u66f4\u5bb9\u6613scale out\u3002\u95ee\u9898\u662f\u6211\u4eec\u8fd8\u662f\u8981\u4fdd\u8bc1worker\u7684\u540c\u6b65\u95ee\u9898\u3002\u5982\u679c\u6709\u4e24\u4e2aupdate\u5bf9\u4e8e\u540c\u4e00\u4e2abilling account\uff0c\u6211\u4eec\u8981\u4fdd\u8bc1scale checker\u4f9d\u7136\u53ef\u4ee5\u5de5\u4f5c\u3002 \u51cf\u5c11tenant\u4e4b\u95f4\u7684\u5f71\u54cd\u3002\u5e94\u8be5\u53ef\u4ee5\u7528dedicated topic\u6765\u505a\u3002\u5f53\u7136\u673a\u5236\u53ef\u80fd\u4f1a\u6bd4\u8f83\u590d\u6742\u3002\u6211\u4eec\u5e94\u8be5\u8fd8\u662f\u9700\u8981common pool\u6765\u51fa\u6765ad-hoc traffic<\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[28,16],"tags":[],"class_list":["post-245","post","type-post","status-publish","format-standard","hentry","category-design","category-16"],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v21.7 - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Data Connect\u7684\u6d88\u606f\u5904\u7406\u673a\u5236 - Liangqi\u2018s Technical Journey<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/liangqi.org\/?p=245\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Data Connect\u7684\u6d88\u606f\u5904\u7406\u673a\u5236 - Liangqi\u2018s Technical Journey\" \/>\n<meta property=\"og:description\" content=\"Data Connect Service\u63d0\u4f9b\u540c\u6b65\u6570\u636e\u5230\u7b2c\u4e09\u65b9\u3002\u6570\u636e\u7684change\u662f\u6765\u81ea\u6570\u636e\u5e93binlog\u751f\u6210\u7684CDC (Change Data Capture)\u3002 \u5728\u7cfb\u7edf\u8bbe\u8ba1\u4e2d\u6709\u51e0\u4e2a\u6311\u6218 Ordering\u7684\u95ee\u9898\u3002\u524d\u7aef\u7684service\u5e76\u4e0d\u4fdd\u8bc1\u987a\u5e8f\u3002\u90a3Data Connect\u5982\u4f55\u4fdd\u8bc1\u987a\u5e8f\u5462\uff1f\u6280\u672f\u4e0a\u6765\u8bb2\u662f\u4e0d\u53ef\u80fd\u7684\u3002\u4f46\u662f\u6211\u4eec\u53ef\u4ee5\u505a\u5230\u5728\u4e0d\u4fdd\u8bc1\u987a\u5e8f\u7684\u60c5\u51b5\u4e0b\u4f9d\u7136\u80fd\u6b63\u786e\u5904\u7406\u3002\u65b9\u6cd5\u662f\u6211\u4eec\u4ece\u6700\u5f00\u59cb\u7684\u670d\u52a1\u5f00\u59cb\u5f15\u5165\u4e86Versioning\u7684\u6982\u5ff5\u3002\u5982\u679cdata connect\u62ff\u5230\u7684\u5bf9\u4e8e\u67d0\u4e00\u6761\u8bb0\u5f55\u7684event\u65e9\u4e8e\u4e4b\u524d\u7684event\uff0c\u6211\u4eec\u5c31\u4e22\u5f03\u3002\u8fd9\u4e2a\u903b\u8f91\u6211\u4eec\u53ebstale checker\u3002\u8fd9\u4e2a\u5728sync\u7684\u903b\u8f91\u91cc\u662f\u53ef\u884c\u7684\u3002\u6bd4\u5982\u8bf4\u6211\u4eec\u6709\u201cCreate-&gt;Update&#8221;,\u5176\u5b9e\u5bf9\u4e8esync\uff0ccreate\u548cupdate\u90fd\u662f\u57fa\u4e8eId\u7684upsert\uff0c\u5982\u679c\u6211\u4eec\u62ff\u5230create\u665a\u4e00\u4e9b\uff0c\u5c31\u76f4\u63a5\u4e22\u5f03\u3002\u540c\u6837\u7684\u201cupdate-&gt;update&#8221;\u4e5f\u6ca1\u95ee\u9898\uff0c\u5f53\u7136\u524d\u63d0\u662f\u6bcf\u6b21\u6211\u4eec\u90fd\u662f\u62ff\u5230\u6574\u4e2a\u8bb0\u5f55\u7684full payload\uff0c\u4e0d\u7136\u5c31\u4f1a\u6709\u95ee\u9898\u3002\u6211\u4eec\u5e76\u6ca1\u6709field level\u7684versioning\u3002 Duplication\u3002Duplication\u5176\u5b9e\u6211\u4eec\u5e76\u4e0d\u592a\u5728\u610f\uff0c\u56e0\u4e3a\u6211\u4eec\u7684\u670d\u52a1\u57fa\u672c\u4e0a\u662fIdempotent\u7684\uff0c\u4e5f\u5c31\u662f\u8bf4\u540c\u6837\u7684\u6d88\u606f\u53ef\u4ee5\u88ab\u5904\u7406\u4e24\u904d\uff0c\u6ca1\u95ee\u9898\u3002\u539f\u56e0\u662fupdate\u672c\u6765\u5c31\u53ef\u4ee5\u91cd\u65b0\u6267\u884c\uff0ccreate\u4e5f\u53ef\u4ee5\uff0c\u56e0\u4e3a\u6211\u4eec\u662f\u6839\u636eid\u6765\u7684\u3002\u8fd8\u6709\u5c31\u662f\u6211\u4eec\u6709stale checker\u5176\u5b9e\u5c31\u80fd\u89e3\u51b3\u5f88\u591a\u7684\u95ee\u9898\u3002 Partition. \u6700\u5f00\u59cb\u7684\u7248\u672c\u6ca1\u6709separate topic\u3002\u6211\u4eecpartition by billing account id\u3002 \u597d\u5904\u6709\u51e0\u4e2a \u51cf\u5c11\u4e00\u4e2atenant\u5360\u7528\u6240\u6709\u7684consumer\u7684\u95ee\u9898\u3002\u5f53\u7136side effect\u662f\u67d0\u4e00\u4e9bconsumer\u4f1a\u6bd4\u8f83\u6162\uff0c\u5982\u679c\u4e00\u4e2atenant traffic\u6bd4\u8f83\u591a\u3002 \u51cf\u5c11row locking\u3002 \u63a7\u5236\u5e76\u53d1\u3002\u5176\u5b9e\u5e76\u53d1\u7684\u63a7\u5236\u6211\u4eec\u5176\u5b9e\u53ef\u4ee5\u7528Resillience4j \u2013 \u8fdc\u7a0b\u65b9\u6cd5\u8c03\u7528\u7684\u5de5\u5177\u7bb1 \u91cc\u9762\u63d0\u5230\u7684\u540c\u6b65\u63a7\u5236\u673a\u5236\u3002\u4f46\u662f\u4e5f\u4f1a\u6709\u8d1f\u9762\u5f71\u54cd\u3002 \u6f5c\u5728\u7684enhance workers\u3002\u6211\u4eec\u4e0d\u5e94\u8be5consumer thread\u91cc\u9762\u5904\u7406\u6240\u6709\u7684\u4e1c\u897f\u3002\u5e94\u8be5\u7528workers\uff0c\u8fd9\u6837\u5c31\u66f4\u5bb9\u6613scale out\u3002\u95ee\u9898\u662f\u6211\u4eec\u8fd8\u662f\u8981\u4fdd\u8bc1worker\u7684\u540c\u6b65\u95ee\u9898\u3002\u5982\u679c\u6709\u4e24\u4e2aupdate\u5bf9\u4e8e\u540c\u4e00\u4e2abilling account\uff0c\u6211\u4eec\u8981\u4fdd\u8bc1scale checker\u4f9d\u7136\u53ef\u4ee5\u5de5\u4f5c\u3002 \u51cf\u5c11tenant\u4e4b\u95f4\u7684\u5f71\u54cd\u3002\u5e94\u8be5\u53ef\u4ee5\u7528dedicated topic\u6765\u505a\u3002\u5f53\u7136\u673a\u5236\u53ef\u80fd\u4f1a\u6bd4\u8f83\u590d\u6742\u3002\u6211\u4eec\u5e94\u8be5\u8fd8\u662f\u9700\u8981common pool\u6765\u51fa\u6765ad-hoc traffic\" \/>\n<meta property=\"og:url\" content=\"https:\/\/liangqi.org\/?p=245\" \/>\n<meta property=\"og:site_name\" content=\"Liangqi\u2018s Technical Journey\" \/>\n<meta property=\"article:published_time\" content=\"2022-02-14T11:15:50+00:00\" \/>\n<meta name=\"author\" content=\"liangqi\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"liangqi\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/liangqi.org\/?p=245#article\",\"isPartOf\":{\"@id\":\"https:\/\/liangqi.org\/?p=245\"},\"author\":{\"name\":\"liangqi\",\"@id\":\"https:\/\/liangqi.org\/#\/schema\/person\/105c89d9b783fda67b62e3ce113d6cd3\"},\"headline\":\"Data Connect\u7684\u6d88\u606f\u5904\u7406\u673a\u5236\",\"datePublished\":\"2022-02-14T11:15:50+00:00\",\"dateModified\":\"2022-02-14T11:15:50+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/liangqi.org\/?p=245\"},\"wordCount\":85,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/liangqi.org\/#\/schema\/person\/105c89d9b783fda67b62e3ce113d6cd3\"},\"articleSection\":[\"Design\",\"\u6280\u672f\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\/\/liangqi.org\/?p=245#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/liangqi.org\/?p=245\",\"url\":\"https:\/\/liangqi.org\/?p=245\",\"name\":\"Data Connect\u7684\u6d88\u606f\u5904\u7406\u673a\u5236 - Liangqi\u2018s Technical Journey\",\"isPartOf\":{\"@id\":\"https:\/\/liangqi.org\/#website\"},\"datePublished\":\"2022-02-14T11:15:50+00:00\",\"dateModified\":\"2022-02-14T11:15:50+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/liangqi.org\/?p=245#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/liangqi.org\/?p=245\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/liangqi.org\/?p=245#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/liangqi.org\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Data Connect\u7684\u6d88\u606f\u5904\u7406\u673a\u5236\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/liangqi.org\/#website\",\"url\":\"https:\/\/liangqi.org\/\",\"name\":\"Liangqi\u2018s Technical Journey\",\"description\":\"Chasing Excellence; Enjoy life.\",\"publisher\":{\"@id\":\"https:\/\/liangqi.org\/#\/schema\/person\/105c89d9b783fda67b62e3ce113d6cd3\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/liangqi.org\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":[\"Person\",\"Organization\"],\"@id\":\"https:\/\/liangqi.org\/#\/schema\/person\/105c89d9b783fda67b62e3ce113d6cd3\",\"name\":\"liangqi\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/liangqi.org\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/liangqi.org\/wp-content\/uploads\/2022\/01\/P1100089-3-scaled.jpg\",\"contentUrl\":\"https:\/\/liangqi.org\/wp-content\/uploads\/2022\/01\/P1100089-3-scaled.jpg\",\"width\":2560,\"height\":1920,\"caption\":\"liangqi\"},\"logo\":{\"@id\":\"https:\/\/liangqi.org\/#\/schema\/person\/image\/\"},\"sameAs\":[\"https:\/\/liangqi.org\"],\"url\":\"https:\/\/liangqi.org\/?author=1\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Data Connect\u7684\u6d88\u606f\u5904\u7406\u673a\u5236 - Liangqi\u2018s Technical Journey","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/liangqi.org\/?p=245","og_locale":"en_US","og_type":"article","og_title":"Data Connect\u7684\u6d88\u606f\u5904\u7406\u673a\u5236 - Liangqi\u2018s Technical Journey","og_description":"Data Connect Service\u63d0\u4f9b\u540c\u6b65\u6570\u636e\u5230\u7b2c\u4e09\u65b9\u3002\u6570\u636e\u7684change\u662f\u6765\u81ea\u6570\u636e\u5e93binlog\u751f\u6210\u7684CDC (Change Data Capture)\u3002 \u5728\u7cfb\u7edf\u8bbe\u8ba1\u4e2d\u6709\u51e0\u4e2a\u6311\u6218 Ordering\u7684\u95ee\u9898\u3002\u524d\u7aef\u7684service\u5e76\u4e0d\u4fdd\u8bc1\u987a\u5e8f\u3002\u90a3Data Connect\u5982\u4f55\u4fdd\u8bc1\u987a\u5e8f\u5462\uff1f\u6280\u672f\u4e0a\u6765\u8bb2\u662f\u4e0d\u53ef\u80fd\u7684\u3002\u4f46\u662f\u6211\u4eec\u53ef\u4ee5\u505a\u5230\u5728\u4e0d\u4fdd\u8bc1\u987a\u5e8f\u7684\u60c5\u51b5\u4e0b\u4f9d\u7136\u80fd\u6b63\u786e\u5904\u7406\u3002\u65b9\u6cd5\u662f\u6211\u4eec\u4ece\u6700\u5f00\u59cb\u7684\u670d\u52a1\u5f00\u59cb\u5f15\u5165\u4e86Versioning\u7684\u6982\u5ff5\u3002\u5982\u679cdata connect\u62ff\u5230\u7684\u5bf9\u4e8e\u67d0\u4e00\u6761\u8bb0\u5f55\u7684event\u65e9\u4e8e\u4e4b\u524d\u7684event\uff0c\u6211\u4eec\u5c31\u4e22\u5f03\u3002\u8fd9\u4e2a\u903b\u8f91\u6211\u4eec\u53ebstale checker\u3002\u8fd9\u4e2a\u5728sync\u7684\u903b\u8f91\u91cc\u662f\u53ef\u884c\u7684\u3002\u6bd4\u5982\u8bf4\u6211\u4eec\u6709\u201cCreate->Update&#8221;,\u5176\u5b9e\u5bf9\u4e8esync\uff0ccreate\u548cupdate\u90fd\u662f\u57fa\u4e8eId\u7684upsert\uff0c\u5982\u679c\u6211\u4eec\u62ff\u5230create\u665a\u4e00\u4e9b\uff0c\u5c31\u76f4\u63a5\u4e22\u5f03\u3002\u540c\u6837\u7684\u201cupdate->update&#8221;\u4e5f\u6ca1\u95ee\u9898\uff0c\u5f53\u7136\u524d\u63d0\u662f\u6bcf\u6b21\u6211\u4eec\u90fd\u662f\u62ff\u5230\u6574\u4e2a\u8bb0\u5f55\u7684full payload\uff0c\u4e0d\u7136\u5c31\u4f1a\u6709\u95ee\u9898\u3002\u6211\u4eec\u5e76\u6ca1\u6709field level\u7684versioning\u3002 Duplication\u3002Duplication\u5176\u5b9e\u6211\u4eec\u5e76\u4e0d\u592a\u5728\u610f\uff0c\u56e0\u4e3a\u6211\u4eec\u7684\u670d\u52a1\u57fa\u672c\u4e0a\u662fIdempotent\u7684\uff0c\u4e5f\u5c31\u662f\u8bf4\u540c\u6837\u7684\u6d88\u606f\u53ef\u4ee5\u88ab\u5904\u7406\u4e24\u904d\uff0c\u6ca1\u95ee\u9898\u3002\u539f\u56e0\u662fupdate\u672c\u6765\u5c31\u53ef\u4ee5\u91cd\u65b0\u6267\u884c\uff0ccreate\u4e5f\u53ef\u4ee5\uff0c\u56e0\u4e3a\u6211\u4eec\u662f\u6839\u636eid\u6765\u7684\u3002\u8fd8\u6709\u5c31\u662f\u6211\u4eec\u6709stale checker\u5176\u5b9e\u5c31\u80fd\u89e3\u51b3\u5f88\u591a\u7684\u95ee\u9898\u3002 Partition. \u6700\u5f00\u59cb\u7684\u7248\u672c\u6ca1\u6709separate topic\u3002\u6211\u4eecpartition by billing account id\u3002 \u597d\u5904\u6709\u51e0\u4e2a \u51cf\u5c11\u4e00\u4e2atenant\u5360\u7528\u6240\u6709\u7684consumer\u7684\u95ee\u9898\u3002\u5f53\u7136side effect\u662f\u67d0\u4e00\u4e9bconsumer\u4f1a\u6bd4\u8f83\u6162\uff0c\u5982\u679c\u4e00\u4e2atenant traffic\u6bd4\u8f83\u591a\u3002 \u51cf\u5c11row locking\u3002 \u63a7\u5236\u5e76\u53d1\u3002\u5176\u5b9e\u5e76\u53d1\u7684\u63a7\u5236\u6211\u4eec\u5176\u5b9e\u53ef\u4ee5\u7528Resillience4j \u2013 \u8fdc\u7a0b\u65b9\u6cd5\u8c03\u7528\u7684\u5de5\u5177\u7bb1 \u91cc\u9762\u63d0\u5230\u7684\u540c\u6b65\u63a7\u5236\u673a\u5236\u3002\u4f46\u662f\u4e5f\u4f1a\u6709\u8d1f\u9762\u5f71\u54cd\u3002 \u6f5c\u5728\u7684enhance workers\u3002\u6211\u4eec\u4e0d\u5e94\u8be5consumer thread\u91cc\u9762\u5904\u7406\u6240\u6709\u7684\u4e1c\u897f\u3002\u5e94\u8be5\u7528workers\uff0c\u8fd9\u6837\u5c31\u66f4\u5bb9\u6613scale out\u3002\u95ee\u9898\u662f\u6211\u4eec\u8fd8\u662f\u8981\u4fdd\u8bc1worker\u7684\u540c\u6b65\u95ee\u9898\u3002\u5982\u679c\u6709\u4e24\u4e2aupdate\u5bf9\u4e8e\u540c\u4e00\u4e2abilling account\uff0c\u6211\u4eec\u8981\u4fdd\u8bc1scale checker\u4f9d\u7136\u53ef\u4ee5\u5de5\u4f5c\u3002 \u51cf\u5c11tenant\u4e4b\u95f4\u7684\u5f71\u54cd\u3002\u5e94\u8be5\u53ef\u4ee5\u7528dedicated topic\u6765\u505a\u3002\u5f53\u7136\u673a\u5236\u53ef\u80fd\u4f1a\u6bd4\u8f83\u590d\u6742\u3002\u6211\u4eec\u5e94\u8be5\u8fd8\u662f\u9700\u8981common pool\u6765\u51fa\u6765ad-hoc traffic","og_url":"https:\/\/liangqi.org\/?p=245","og_site_name":"Liangqi\u2018s Technical Journey","article_published_time":"2022-02-14T11:15:50+00:00","author":"liangqi","twitter_card":"summary_large_image","twitter_misc":{"Written by":"liangqi","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/liangqi.org\/?p=245#article","isPartOf":{"@id":"https:\/\/liangqi.org\/?p=245"},"author":{"name":"liangqi","@id":"https:\/\/liangqi.org\/#\/schema\/person\/105c89d9b783fda67b62e3ce113d6cd3"},"headline":"Data Connect\u7684\u6d88\u606f\u5904\u7406\u673a\u5236","datePublished":"2022-02-14T11:15:50+00:00","dateModified":"2022-02-14T11:15:50+00:00","mainEntityOfPage":{"@id":"https:\/\/liangqi.org\/?p=245"},"wordCount":85,"commentCount":0,"publisher":{"@id":"https:\/\/liangqi.org\/#\/schema\/person\/105c89d9b783fda67b62e3ce113d6cd3"},"articleSection":["Design","\u6280\u672f"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/liangqi.org\/?p=245#respond"]}]},{"@type":"WebPage","@id":"https:\/\/liangqi.org\/?p=245","url":"https:\/\/liangqi.org\/?p=245","name":"Data Connect\u7684\u6d88\u606f\u5904\u7406\u673a\u5236 - Liangqi\u2018s Technical Journey","isPartOf":{"@id":"https:\/\/liangqi.org\/#website"},"datePublished":"2022-02-14T11:15:50+00:00","dateModified":"2022-02-14T11:15:50+00:00","breadcrumb":{"@id":"https:\/\/liangqi.org\/?p=245#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/liangqi.org\/?p=245"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/liangqi.org\/?p=245#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/liangqi.org\/"},{"@type":"ListItem","position":2,"name":"Data Connect\u7684\u6d88\u606f\u5904\u7406\u673a\u5236"}]},{"@type":"WebSite","@id":"https:\/\/liangqi.org\/#website","url":"https:\/\/liangqi.org\/","name":"Liangqi\u2018s Technical Journey","description":"Chasing Excellence; Enjoy life.","publisher":{"@id":"https:\/\/liangqi.org\/#\/schema\/person\/105c89d9b783fda67b62e3ce113d6cd3"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/liangqi.org\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":["Person","Organization"],"@id":"https:\/\/liangqi.org\/#\/schema\/person\/105c89d9b783fda67b62e3ce113d6cd3","name":"liangqi","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/liangqi.org\/#\/schema\/person\/image\/","url":"https:\/\/liangqi.org\/wp-content\/uploads\/2022\/01\/P1100089-3-scaled.jpg","contentUrl":"https:\/\/liangqi.org\/wp-content\/uploads\/2022\/01\/P1100089-3-scaled.jpg","width":2560,"height":1920,"caption":"liangqi"},"logo":{"@id":"https:\/\/liangqi.org\/#\/schema\/person\/image\/"},"sameAs":["https:\/\/liangqi.org"],"url":"https:\/\/liangqi.org\/?author=1"}]}},"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"https:\/\/liangqi.org\/index.php?rest_route=\/wp\/v2\/posts\/245","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/liangqi.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/liangqi.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/liangqi.org\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/liangqi.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=245"}],"version-history":[{"count":1,"href":"https:\/\/liangqi.org\/index.php?rest_route=\/wp\/v2\/posts\/245\/revisions"}],"predecessor-version":[{"id":246,"href":"https:\/\/liangqi.org\/index.php?rest_route=\/wp\/v2\/posts\/245\/revisions\/246"}],"wp:attachment":[{"href":"https:\/\/liangqi.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=245"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/liangqi.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=245"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/liangqi.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=245"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}