{ "version": "https://jsonfeed.org/version/1", "title": "Big Data", "description": null, "home_page_url": "go/bigdata", "feed_url": "feed/bigdata.json", "items": [ { "author": { "url": "member/louhubiao", "name": "louhubiao", "avatar": "https://cdn.v2ex.com/gravatar/74b9b3ffc3a1bead85e0e18c07b47ba1?s=73&d=retro" }, "url": "t/1107283", "date_modified": "2025-01-23T03:13:26+00:00", "content_html": "

\u4e4b\u524d\u7528 datax \u914d\u7f6e\uff0cmysql \u540c\u6b65\u5230 hdfs \u8fd8\u662f\u5f88\u597d\u7528\u7684\uff0c\u4f46\u662f\u9700\u8981\u624b\u5199\u914d\u7f6e

\n

\u6709\u6ca1\u6709\u66f4\u52a0\u6613\u7528\u7684\uff0c\u6bd4\u5982\u672c\u5730\u8d77\u4e2a\u955c\u50cf\u5c31\u884c\uff0c\u4ee5\u53ca\u53ef\u89c6\u5316\u914d\u7f6e\u8868\u540c\u6b65\uff0c\u914d\u7f6e\u540c\u6b65\u54ea\u4e9b\u5b57\u6bb5

\n

\u6700\u597d\u662f\u80fd\u548c\u5176\u4ed6\u7cfb\u7edf\u8fdb\u884c\u4ea4\u4e92\uff0c\u4f8b\u5982\u8c03\u5ea6\u7cfb\u7edf\uff0c\u5728\u6570\u636e\u540c\u6b65\u5b8c\u6210\u540e\u518d\u6267\u884c\u540e\u7eed\u4efb\u52a1

\n", "date_published": "2025-01-23T03:12:47+00:00", "title": "\u5982\u4f55\u5c06\u8fdc\u7aef mysql \u53d8\u66f4\u540c\u6b65\u5230\u672c\u5730", "id": "t/1107283" }, { "author": { "url": "member/rockagen", "name": "rockagen", "avatar": "https://cdn.v2ex.com/gravatar/3fcdce3929a21588a7489b3787b8ae0d?s=73&d=retro" }, "url": "t/932299", "title": "\u5410\u69fd\u4e00\u4e0b\u56fd\u5185\u5f00\u6e90\u7684\u4e00\u4e9b\u4e1c\u897f", "id": "t/932299", "date_published": "2023-04-13T09:26:58+00:00", "content_html": "

\u7531\u4e8e\u5de5\u4f5c\u539f\u56e0\u9700\u8981\u540c\u6b65\u4e00\u4e9b\u6570\u636e\u5230 hive \uff0c\u5f53\u65f6\u9009\u578b\u6709 datax \u548c seatunnel \uff0cGoogle \u7814\u7a76\u4e86\u4e00\u4e0b\u90fd\u662f seatunnel \u5404\u79cd\u540a\u6253 datax \uff0c\u60f3\u7740 seatunnel \u6709 spark \u652f\u6301\u5c31\u91c7\u7528\u4e86\uff0c\u597d\uff0c\u4e07\u6076\u7684\u6e90\u5934\u5f00\u59cb\uff0cseatunnel \u8868\u9762\u770b\u8d77\u6765\u5404\u79cd\u9ad8\u5927\u4e0a\uff0c\u540e\u9762\u8fd8\u641e\u4e86\u81ea\u5df1\u7684\u5f15\u64ce\uff0c\u5b9e\u9645\u7528\u8d77\u6765\u554a\uff0c\u771f\u7684\u4e00\u5768\u72d7\u5c4e\uff0c\u57fa\u672c\u529f\u80fd\u4e0d\u597d\u597d\u641e\uff0c\u8f6f\u6587\u5230\u5904\u5439\uff0c\u5c31\u6700\u8fd1 v2.3.1 \u91cc\u7684 transform \u63d2\u4ef6\u7adf\u7136\u548c v2.3.0 \u4e0d\u517c\u5bb9\uff0cv2.1 \u5230 v2.3 \u7684\u4e0d\u517c\u5bb9\u6211\u5fcd\u4e86\uff0c0.01 \u7684\u5347\u7ea7\u80fd\u641e\u6210\u8fd9\u6837\u662f\u771f\u725b\u903c\u3002

\n

\u6587\u6863\u4f8b\u5b50\u4e0d\u597d\u597d\u5199\uff0c\u6bcf\u6b21\u51fa\u4e2a\u95ee\u9898\u8fd8\u5f97\u7ffb\u6e90\u7801\u770b\u600e\u4e48\u5904\u7406\uff0c\u6c38\u8fdc\u53ea\u4f1a\u5199\u4e2a hello world \u7ea7\u522b\u4f8b\u5b50\uff0c\u5c31\u8fd9\u7ea7\u522b\u4f8b\u5b50\u81ea\u5df1\u5199\u4e2a scala \u5c31\u641e\u5b9a\u4e86\u597d\u4f10\u3002

\n

\u5173\u952e\u662f\u8fd9\u7cdf\u7cd5\u7684\u9879\u76ee\u7adf\u7136\u8fd8\u80fd\u6df7\u8fdb Apache \uff0c\u674e\u5b89\u9707\u60ca\u8868\u793a\u4e0d\u7406\u89e3\u3002

\n

\u4e0a\u4e00\u6b21\u5403\u5c4e\u8fd8\u662f\u817e\u8baf\u5f00\u6e90\u7684\u6570\u636e\u5168\u5bb6\u6876\uff0c\u90a3\u662f\u771f\u6076\u5fc3\u3002

\n" }, { "author": { "url": "member/tonyjia87", "name": "tonyjia87", "avatar": "https://cdn.v2ex.com/gravatar/9eca2cae3995c6bab68bef70a015d404?s=73&d=retro" }, "url": "t/894551", "title": "ETL \u65b9\u6848\u6709\u54ea\u4e9b", "id": "t/894551", "date_published": "2022-11-11T10:07:13+00:00", "content_html": "\n" }, { "author": { "url": "member/miscnote", "name": "miscnote", "avatar": "https://cdn.v2ex.com/gravatar/fa3a0431ee232cdeeccd0608eb5aed21?s=73&d=retro" }, "url": "t/838166", "title": "\u4eca\u5929\u542c\u5230\u4e00\u4e2a\u7b11\u8bdd", "id": "t/838166", "date_published": "2022-03-05T05:17:09+00:00", "content_html": "

\u67d0\u516c\u53f8\uff08\u5e7f\u544a\u884c\u4e1a\uff09\u8fd0\u7ef4\u5c06 hadoop \u9ed8\u8ba4\u5b89\u88c5\u540e\u5c31\u6254\u7ed9\u6280\u672f\u7528\u3002\u9ed8\u8ba4\u5b89\u88c5\u7684 hdfs \u7684 data \u76ee\u5f55\u4f4d\u4e8e /tmp \u4e0b\u3002\u7136\u540e\u7cfb\u7edf\u4e00\u91cd\u542f\uff0c\u6240\u6709\u6570\u636e\u90fd\u6ca1\u4e86\u3002\u3002\u3002

\n" }, { "author": { "url": "member/tunzao", "name": "tunzao", "avatar": "https://cdn.v2ex.com/avatar/4ef4/2016/182412_large.png?m=1659852809" }, "url": "t/831157", "title": "\u4efb\u52a1\u8c03\u5ea6\u7cfb\u7edf\u7cfb\u5217\u4e4b Airflow", "id": "t/831157", "date_published": "2022-01-28T09:15:46+00:00", "content_html": "

http://tunzao.me/articles/airflow/

\n" }, { "author": { "url": "member/NeezerGu", "name": "NeezerGu", "avatar": "https://cdn.v2ex.com/gravatar/fa5df1ba58dbb71a937ed15596cfc1b5?s=73&d=retro" }, "url": "t/807595", "title": "100G \u5de6\u53f3\u6570\u636e\uff0c mysql \u7406\u8bba\u4e0a\u53ef\u80fd\u4f1a\u6bd4\u5927\u6570\u636e\u5f15\u64ce\u5feb\u5417\uff1f", "id": "t/807595", "date_published": "2021-10-13T09:16:52+00:00", "content_html": "

\u8ddf\u670b\u53cb\u95f2\u804a\u7a81\u7136\u60f3\u5230\uff0c\u56e0\u4e3a\u662f\u505a\u6570\u636e\u5206\u6790\u7684\uff0c\u5bf9\u5e95\u5c42\u5f15\u64ce\u5c31\u4e0d\u592a\u61c2\u5566\u3002

\n

\u6bd4\u5982\uff0c980pro 4 \u6761\u7ec4 raid 0 \uff08 YouTube \u4e0a\u6709\u89c6\u9891\uff0chttps://www.youtube.com/watch?v=OCGguruZyrw&t=413s\n\u751a\u81f3\u76f4\u63a5 256G \u5185\u5b58\uff0c\u62ff 120G \u5185\u5b58\u5f53\u786c\u76d8\uff08\u4e00\u65f6\u4e4b\u95f4\u5fd8\u4e86\u8fd9\u4e2a\u540d\u8bcd\uff09

\n

\u7528 mysql \u6216\u8005 clickhouse\uff0c\u5bf9\u6bd4 spark \u3001presto \u7b49\uff0c\u4f1a\u5b58\u5728\u524d\u8005\u66f4\u5feb\u7684\u53ef\u80fd\u5417\uff1f

\n" }, { "author": { "url": "member/zhangshaohan", "name": "zhangshaohan", "avatar": "https://cdn.v2ex.com/avatar/66d5/eebe/424937_large.png?m=1758694576" }, "url": "t/800736", "title": "\u4e0d\u5f71\u54cd\u4e1a\u52a1\u7cfb\u7edf\u7684\u60c5\u51b5\u4e0b\u5982\u4f55\u505a\u5b9e\u65f6\u5927\u5c4f\u5c55\u793a\u5462\uff1f", "id": "t/800736", "date_published": "2021-09-09T00:49:27+00:00", "content_html": "

\u76ee\u524d\u9700\u8981\u505a\u4e2a\u5927\u5c4f\u5c55\u793a\uff0c\u4e0d\u80fd\u5bf9\u4e1a\u52a1\u7cfb\u7edf\u6709\u4fb5\u5165\u6027\uff0c\u9700\u8981\u5b9e\u73b0\u7c7b\u4f3c\u4e8e\u65b0\u589e\u4e00\u4ef6\u5546\u54c1\u5927\u5c4f\u4e0a\u6570\u5b57\u5c31\u5b9e\u65f6\u52a0 1\uff0c\u6570\u636e\u5e93 MySQL

\n

\u76ee\u524d\u8003\u8651\u662f\u901a\u8fc7 canal \u901a\u8fc7 binlog \u8ba2\u9605\u67d0\u51e0\u5f20\u8868\u6765\u5b9e\u73b0\u65b0\u589e\u901a\u77e5\u673a\u5236\uff0c\u6709\u6ca1\u6709\u4ec0\u4e48\u66f4\u597d\u7684\u529e\u6cd5\u5462\uff1f\u662f\u4e2a\u5c0f\u5de5\u7a0b\uff0c\u5468\u671f\u5f88\u77ed\u3002

\n

\u611f\u8c22\u5927\u5bb6\uff01\uff01\uff01

\n" }, { "author": { "url": "member/iriddle911", "name": "iriddle911", "avatar": "https://cdn.v2ex.com/avatar/14fc/6679/290993_large.png?m=1548033213" }, "url": "t/795438", "title": "\u8feb\u4e8e\u538b\u529b\uff0c\u6c42\u4e2a\u597d\u7684\u5927\u6570\u636e\u8bfe\u7a0b", "id": "t/795438", "date_published": "2021-08-12T13:21:09+00:00", "content_html": "

\u672c\u4eba\u4ece\u4e8b to B \u552e\u524d\u5de5\u4f5c\uff0c\u6d89\u53ca\u5927\u6570\u636e\u9879\u76ee\u3002\u4f46\u5948\u4f55\u6ca1\u6709\u5927\u6570\u636e\u7684\u7814\u53d1\u3001\u9879\u76ee\u7ecf\u9a8c\uff0c\u81ea\u5b66\u5927\u6570\u636e\u603b\u4e0d\u5f97\u8981\u9886\u3002

\n

\u5728\u5404\u7f51\u7ad9\u627e\u5230\u7684\u8bfe\u7a0b\u57fa\u672c\u90fd\u662f\u9762\u5411\u7814\u53d1\u7684\uff0c\u4e0a\u6765\u5c31\u5f00\u59cb\u603c JAVA\uff0c\u5b66\u8d77\u6765\u592a\u96be\u4e86\u3002\u6545\u6c42\u5404\u4f4d\u5927\u4f6c\u63a8\u8350\uff0c\u80fd\u5feb\u901f\u4e86\u89e3\u5927\u6570\u636e\u6280\u672f\u751f\u6001\u7684\u4e66 /\u8bfe\u7a0b\uff0c\u5927\u81f4\u7c7b\u4f3c\u7ed9\u4ea7\u54c1\u7ecf\u7406\u8bb2\u6280\u672f\u8fd9\u6837\u7684\u7a0b\u5ea6\uff0c\u4e0d\u8981\u6c42\u638c\u63e1\u539f\u7406\uff0c\u77e5\u9053\u57fa\u672c\u5168\u666f\u8fc7\u7a0b\u5373\u53ef\u3002

\n

\u671f\u671b\uff1a\u80fd\u57fa\u672c\u4e86\u89e3\u5927\u6570\u636e /\u6570\u636e\u4e2d\u53f0\u7c7b\u4f3c\u9879\u76ee\u7684\u5168\u6d41\u7a0b\u6280\u672f\uff0c\u53ef\u4ee5\u548c RD \u8fdb\u884c\u5bf9\u8bdd\u3002

\n

\u672c\u4eba\u6280\u672f\u80cc\u666f\uff1a\u975e CS \u4e13\u4e1a\u51fa\u8eab\uff0c\u61c2\u4e00\u4e9b\u4e91\uff0c\u4f1a\u5199\u4e00\u4e9b\u524d\u7aef\uff0c\u77e5\u9053\u4f20\u7edf\u5de5\u7a0b\u5316\u7684\u7814\u53d1\u6d41\u7a0b\u3002\u6b63\u5728\u770b Hadoop \u7684\u5404\u79cd\u4e66\u3002

\n" }, { "author": { "url": "member/TalkWithKeyboard", "name": "TalkWithKeyboard", "avatar": "https://cdn.v2ex.com/avatar/28c2/7990/231971_large.png?m=1495468405" }, "url": "t/792769", "title": "\u7845\u8c37\u6700\u706b\u7684\u6570\u636e\u5206\u6790\u5de5\u5177 dbt \u7a76\u7adf\u662f\u4ec0\u4e48\uff1f", "id": "t/792769", "date_published": "2021-07-30T11:34:57+00:00", "content_html": "

dbt \u63d0\u4f9b\u4e86\u5f88\u591a\u8f6f\u4ef6\u5f00\u53d1\u7684\u601d\u60f3\u548c\u5de5\u5177\u5230\u6570\u636e\u5206\u6790\u9886\u57df\uff0c\u6709\u4e86 dbt \u4ee5\u540e\u66f4\u591a\u7684\u5206\u6790\u5e08\u4f1a\u4e60\u60ef\u6210\u4e3a\u8fd9\u4e9b\u529f\u80fd\u7684\u4f7f\u7528\u8005\uff0c\u800c\u4e0d\u662f\u5f00\u53d1\u8005\u3002\u8f6f\u4ef6\u5de5\u7a0b\u5f00\u53d1\u8005\u5df2\u7ecf\u4e60\u60ef\u5728\u9700\u8981\u5199\u4e00\u4e2a\u65b0\u7684\u9700\u6c42\u7684\u65f6\u5019\uff0c\u76f4\u63a5\u4f7f\u7528\u4e00\u4e2a\u73b0\u6210\u7684\u5de5\u5177\uff0c\u4ed6\u4eec\u53ea\u9700\u8981\u7b80\u5355\u7684\u6784\u5efa\u5c31\u53ef\u4ee5\u4f7f\u7528\uff0c\u7136\u540e\u5c06\u4ed6\u4eec\u7684\u5de5\u4f5c\u518d\u8d21\u732e\u56de\u793e\u533a\u3002\u5728 dbt \u4e2d\u5206\u6790\u5e08\u4e5f\u53ef\u4ee5\u505a\u540c\u6837\u7684\u4e8b\u60c5\u3002

\n

\u8fd9\u4e24\u5e74\u56fd\u5916\u7684\u6570\u636e\u4ea7\u54c1 dbt \u975e\u5e38\u7684\u706b\uff0c\u56fd\u5185\u597d\u50cf\u8ba8\u8bba\u5f88\u5c11\uff0c\u6700\u8fd1\u7ffb\u8bd1\u4e86\u4e00\u7bc7 dbt \u521b\u59cb\u4eba\u4ecb\u7ecd dbt \u7684\u6587\u7ae0\u4e5f\u5199\u4e86\u4e00\u7bc7\u4f7f\u7528 dbt \u7684\u6559\u7a0b\u5e2e\u52a9\u5927\u5bb6\u4e86\u89e3 dbt\uff1a\u6307\u8def

\n" }, { "author": { "url": "member/codeboy18", "name": "codeboy18", "avatar": "https://cdn.v2ex.com/avatar/3f57/60dc/280585_large.png?m=1614165693" }, "url": "t/757053", "title": "\u4e00\u6587\u8bfb\u61c2 clickhouse \u96c6\u7fa4\u76d1\u63a7", "id": "t/757053", "date_published": "2021-02-28T13:49:32+00:00", "content_html": "
\n

\u66f4\u591a\u7cbe\u5f69\u5185\u5bb9\uff0c\u8bf7\u5173\u6ce8\u5fae\u4fe1\u516c\u4f17\u53f7\uff1a\u540e\u7aef\u6280\u672f\u5c0f\u5c4b

\n
\n

\u4e00\u6587\u8bfb\u61c2 clickhouse \u96c6\u7fa4\u76d1\u63a7

\n

\u5e38\u8a00\u9053\uff0c\u5175\u9a6c\u672a\u81f3\uff0c\u7cae\u8349\u5148\u884c\uff0c\u5728 clickhouse \u4e0a\u751f\u4ea7\u73af\u5883\u4e4b\u524d\uff0c\u6211\u4eec\u5c31\u5f97\u5236\u5b9a\u597d\u76f8\u5173\u7684\u76d1\u63a7\u65b9\u6848\uff0c\u5305\u62ec metric \u91c7\u96c6\u3001\u62a5\u8b66\u7b56\u7565\u3001\u56fe\u5f62\u5316\u62a5\u8868\u3002\u6709\u4e86\u5168\u9762\u6709\u6548\u7684\u76d1\u63a7\uff0c\u6211\u4eec\u5c31\u4eff\u4f5b\u62e5\u6709\u4e86\u5343\u91cc\u773c\u987a\u98ce\u8033\uff0c\u5bf9\u4e8e\u7ebf\u4e0a\u4efb\u4f55\u98ce\u5439\u8349\u52a8\u90fd\u80fd\u53ca\u65f6\u611f\u77e5\uff0c\u5728\u5fc5\u8981\u7684\u60c5\u51b5\u4e0b\u63d0\u524d\u4ecb\u5165\u4ee5\u907f\u514d\u7ebf\u4e0a\u6545\u969c\u3002

\n

\u4e1a\u754c\u5e38\u7528\u7684\u76d1\u63a7\u65b9\u6848\u4e00\u822c\u662f\u57fa\u4e8e prometheus + grafana \u751f\u6001\u3002\u672c\u6587\u5c06\u4ecb\u7ecd\u7531 clickhouse-exporter(node-exporter) + prometheus + grafana \u7ec4\u6210\u7684\u76d1\u63a7\u65b9\u6848\u3002

\n

\"clickhouse

\n

\u4ee5\u4e0a\u4e3a\u76d1\u63a7\u65b9\u6848\u793a\u610f\u56fe

\n\n

1 \u90e8\u7f72\u4e0e\u914d\u7f6e

\n

1.1 clickhouse-server

\n

\u6211\u4eec\u751f\u4ea7\u73af\u5883\u7248\u672c\u4e3a20.3.8\uff0c\u6309\u7167\u5b98\u65b9\u6587\u6863\u90e8\u7f72\u5373\u53ef\u3002

\n

1.2 clickhouse-exporter

\n

clickhouse-exporter \u4e00\u822c\u4e0e clickhouse-server \u540c\u673a\u90e8\u7f72\u3002

\n

\u9996\u5148\u4e0b\u8f7d\u6700\u65b0\u4ee3\u7801\u5e76\u7f16\u8bd1(\u9700\u9884\u5148\u5b89\u88c5 Go)

\n
git clone https://github.com/ClickHouse/clickhouse_exporter \ncd clickhouse_exporter \ngo mod init \ngo mod vendor \ngo build \nls ./clickhouse_exporter \n
\n

\u7136\u540e\u542f\u52a8

\n
export CLICKHOUSE_USER=\"user\" \nexport CLICKHOUSE_PASSWORD=\"password\" \nnohup ./-scrape_uri=http://localhost:port/ >nohup.log 2>&1 & \n
\n

\u6700\u540e\u68c0\u67e5\u6307\u6807\u662f\u5426\u88ab\u6b63\u5e38\u91c7\u96c6\uff1a

\n
> curl localhost:9116/metrics | head \n# TYPE clickhouse_arena_alloc_bytes_total counter \nclickhouse_arena_alloc_bytes_total 9.799096840192e+12 \n# HELP clickhouse_arena_alloc_chunks_total Number of ArenaAllocChunks total processed \n# TYPE clickhouse_arena_alloc_chunks_total counter \nclickhouse_arena_alloc_chunks_total 2.29782524e+08 \n# HELP clickhouse_background_move_pool_task Number of BackgroundMovePoolTask currently processed \n# TYPE clickhouse_background_move_pool_task gauge \nclickhouse_background_move_pool_task 0 \n# HELP clickhouse_background_pool_task Number of BackgroundPoolTask currently processed \n
\n

1.3 node-exporter

\n

node-exporter \u9700\u4e0e clickhouse-server \u540c\u673a\u90e8\u7f72

\n

\u9996\u5148\u4e0b\u8f7d\u6700\u65b0\u4ee3\u7801\u5e76\u7f16\u8bd1

\n
git clone https://github.com/prometheus/node_exporter \nmake build \nls ./node_exporter \n
\n

\u7136\u540e\u542f\u52a8

\n
nohup ./node_exporter > nohup.log 2>&1 & \n
\n

\u6700\u540e\u68c0\u67e5\u6307\u6807\u662f\u5426\u88ab\u6b63\u5e38\u91c7\u96c6

\n
> curl localhost:9100/metrics \n# HELP go_gc_duration_seconds A summary of the GC invocation durations. \n# TYPE go_gc_duration_seconds summary \ngo_gc_duration_seconds{quantile=\"0\"} 6.3563e-05 \ngo_gc_duration_seconds{quantile=\"0.25\"} 7.4746e-05 \ngo_gc_duration_seconds{quantile=\"0.5\"} 9.0556e-05 \ngo_gc_duration_seconds{quantile=\"0.75\"} 0.000110677 \ngo_gc_duration_seconds{quantile=\"1\"} 0.004362325 \ngo_gc_duration_seconds_sum 28.451282046 \ngo_gc_duration_seconds_count 223479 \n... \n
\n

1.4 prometheus

\n

\u4fee\u6539 prometheus \u914d\u7f6e\u6587\u4ef6\uff0c\u6dfb\u52a0 alertmanager \u5730\u5740\u3001clickhouse-exporter \u5730\u5740

\n

prometheus.yml \u793a\u4f8b\u5982\u4e0b\uff1a

\n
global: \n scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute. \n evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute. \n \n# Alertmanager configuration \nalerting: \n alertmanagers: \n - static_configs: \n - targets: \n - alertmanager:9093 \n \n# Load rules once and periodically evaluate them according to the global 'evaluation_interval'. \nrule_files: \n - ./rules/*.rules \n \n# A scrape configuration containing exactly one endpoint to scrape: \n# Here it's Prometheus itself. \nscrape_configs: \n # The job name is added as a label `job=<job_name>` to any timeseries scraped from this config. \n - job_name: 'clickhouse' \n \n # metrics_path defaults to '/metrics' \n # scheme defaults to 'http'. \n static_configs: \n - targets: ['clickhouseexporter1:9116', 'clickhouseexporter2:9116', ...] \n
\n

*.rules \u793a\u4f8b\u5982\u4e0b\uff1a

\n
groups: \n - name: qps_too_high \n rules: \n - alert: clickhouse qps \u8d85\u51fa\u9608\u503c \n expr: rate(clickhouse_query_total[1m]) > 100 \n for: 2m \n labels: \n job: clickhouse-server \n severity: critical \n alertname: clickhouse qps \u8d85\u51fa\u9608\u503c \n annotations: \n summary: \"clickhouse qps \u8d85\u51fa\u9608\u503c\" \n description: \"clickhouse qps \u8d85\u8fc7\u9608\u503c(100), qps: {{ $value }}\" \n
\n

\u542f\u52a8 promethus

\n
nohup ./prometheus --config.file=/path/to/config --storage.tsdb.path=/path/to/storage --web.external-url=prometheus --web.enable-admin-api --web.enable-lifecycle --log.level=warn >nohup.log 2>&1 & \n
\n

\u6d4f\u89c8\u5668\u8f93\u5165http://prometheus_ip:9090\u68c0\u67e5 prometheus \u72b6\u6001

\n

1.5 alert manager

\n

\u9996\u5148\u4fee\u6539\u914d\u7f6e\u6587\u4ef6

\n

\u914d\u7f6e\u6587\u4ef6\u793a\u4f8b\u5982\u4e0b\uff1a

\n
route: \n receiver: 'default' \n group_by: ['service','project'] \n \nreceivers: \n- name: \"\u7535\u8bdd\" \n webhook_configs: \n - url: <url> \n \n- name: \"\u4f01\u4e1a\u5fae\u4fe1\" \n webhook_configs: \n - url: <url> \n \n- name: \"\u90ae\u7bb1\" \n webhook_configs: \n - url: <url> \n
\n

\u7136\u540e\u542f\u52a8

\n
nohup ./alertmanager --config.file=/path/to/config --log.level=warn >nohup.log 2>&1 & \n
\n

1.6 grafana

\n

\u5173\u4e8e clickhouse \u7684 dashboard \u6a21\u677f\u5df2\u7ecf\u6709\u5f88\u591a\uff0c\u5728\u8fd9\u91cc\u63a8\u8350\uff1a https://grafana.com/grafana/dashboards/882 \u5c06\u5b83\u5bfc\u5165\u5230\u65b0\u5efa\u7684 grafana dashboard \u4e4b\u540e\uff0c\u5373\u53ef\u5f97\u5230\u6f02\u4eae\u7684 clickhouse \u96c6\u7fa4\u770b\u677f(\u53ef\u80fd\u9700\u8981\u5fae\u8c03)\u3002

\n

\u53e6\u5916\u5efa\u8bae\u5b89\u88c5 clickhouse datasource \u63d2\u4ef6\u3002\u6709\u4e86\u8fd9\u4e2a\u63d2\u4ef6\u4fbf\u80fd\u5728 grafana \u4e2d\u914d\u7f6e clickhouse \u6570\u636e\u6e90\uff0c\u5e76\u901a\u8fc7 Clickhouse SQL \u914d\u7f6e\u56fe\u8868\uff0c\u8be6\u7ec6\u6587\u6863\u89c1\uff1a https://grafana.com/grafana/plugins/vertamedia-clickhouse-datasource

\n

2 \u91cd\u8981\u6307\u6807\u548c\u76d1\u63a7

\n

\u6211\u4eec\u53ef\u4ee5\u770b\u5230\uff0c\u4e0d\u7ba1\u662f node-exporter \u8fd8\u662f clickhouse-exporter\uff0c\u5b83\u4eec\u7684\u6307\u6807\u79cd\u7c7b\u5f88\u591a\uff0c\u5927\u6982\u6709\u51e0\u767e\u4e2a\u3002\u6211\u4eec\u7684\u7b56\u7565\u662f\u6293\u5927\u653e\u5c0f\uff0c\u5bf9\u4e8e\u91cd\u8981\u7684\u6307\u6807\u624d\u8bbe\u7f6e\u62a5\u8b66\u7b56\u7565\u5e76\u521b\u5efa\u770b\u677f\u3002

\n

\u4e0b\u9762\u5217\u4e3e\u4e00\u4e9b\u4e2a\u4eba\u89c9\u5f97\u6bd4\u8f83\u91cd\u8981\u7684\u6307\u6807

\n

2.1 \u7cfb\u7edf\u6307\u6807

\n

\u7cfb\u7edf\u6307\u6807\u7531 node-exporter \u91c7\u96c6

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\u6307\u6807\u540d\u6307\u6807\u542b\u4e49\u62a5\u8b66\u7b56\u7565\u7b56\u7565\u542b\u4e49
node_cpu_seconds_total\u673a\u5668\u7d2f\u8ba1 cpu \u65f6\u95f4(\u5355\u4f4d s)100 * sum without (cpu) (rate(node_cpu_seconds_total{mode='user'}[5m])) / count without (cpu) (node_cpu_seconds_total{mode='user'}) > 80\u7528\u6237\u6001 cpu \u5229\u7528\u7387\u5927\u4e8e 80%\u5219\u62a5\u8b66
node_filesystem_size_bytes/node_filesystem_avail_bytes\u673a\u5668\u4e0a\u4e2a\u6587\u4ef6\u5206\u533a\u5bb9\u91cf /\u53ef\u7528\u5bb9\u91cf100 * (node_filesystem_size_bytes{mountpoint=\"/data\"} - node_filesystem_avail_bytes{mountpoint=\"/data\"}) / node_filesystem_size_bytes{mountpoint=\"/data\"} > 80/data \u76d8\u5360\u7528\u8d85\u8fc7 80%\u5219\u62a5\u8b66
node_load55 \u5206\u949f load \u503cnode_load5 > 605 \u5206\u949f load \u503c\u8d85\u8fc7 60 \u5219\u62a5\u8b66(\u53ef\u6839\u636e\u5177\u4f53\u60c5\u51b5\u8bbe\u7f6e\u9608\u503c)
node_disk_reads_completed_total\u7d2f\u8ba1\u8bfb\u78c1\u76d8\u8bf7\u6c42\u6b21\u6570rate(node_disk_reads_completed_total[5m]) > 200read iops \u8d85\u8fc7 200 \u5219\u62a5\u8b66

2.2 clickhouse \u6307\u6807

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
\u6307\u6807\u540d\u6307\u6807\u542b\u4e49\u62a5\u8b66\u7b56\u7565\u7b56\u7565\u542b\u4e49
clickhouse_exporter_scrape_failures_totalprometheus \u6293\u53d6 exporter \u5931\u8d25\u603b\u6b21\u6570increase(clickhouse_exporter_scrape_failures_total[5m]) > 10prometheus \u6293\u53d6 export \u5931\u8d25\u6b21\u6570\u8d85\u8fc7\u9608\u503c\u5219\u62a5\u8b66\uff0c\u8bf4\u660e\u6b64\u65f6 ch \u670d\u52a1\u5668\u53ef\u80fd\u53d1\u751f\u5b95\u673a
promhttp_metric_handler_requests_totalexporter \u8bf7\u6c42 clickhouse \u5931\u8d25\u603b\u6b21\u6570increase(promhttp_metric_handler_requests_total{code=\"200\"}[2m]) == 02 \u5206\u949f\u5185\u67e5\u8be2 clickhouse \u6210\u529f\u6b21\u6570\u4e3a\u96f6\u5219\u62a5\u8b66\uff0c\u8bf4\u660e\u6b64\u65f6\u67d0\u4e2a ch \u5b9e\u4f8b\u53ef\u80fd\u4e0d\u53ef\u7528
clickhouse_readonly_replicach \u5b9e\u4f8b\u4e2d\u5904\u4e8e\u53ea\u8bfb\u72b6\u6001\u7684\u8868\u4e2a\u6570clickhouse_readonly_replica > 5ch \u4e2d\u53ea\u8bfb\u8868\u8d85\u8fc7 5 \u5219\u62a5\u8b66\uff0c\u8bf4\u660e\u6b64\u65f6 ch \u4e0e zk \u8fde\u63a5\u53ef\u80fd\u53d1\u751f\u5f02\u5e38
clickhouse_query_totalch \u5df2\u5904\u7406\u7684 query \u603b\u6570rate(clickhouse_query_total[1m]) > 30\u5355\u5b9e\u4f8b qps \u8d85\u8fc7 30 \u5219\u62a5\u8b66
clickhouse_querych \u4e2d\u6b63\u5728\u8fd0\u884c\u7684 query \u4e2a\u6570clickhouse_query > 30\u5355\u5b9e\u4f8b\u5e76\u53d1 query \u6570\u8d85\u8fc7\u9608\u503c\u5219\u62a5\u8b66
clickhouse_tcp_connectionch \u7684 TCP \u8fde\u63a5\u6570clickhouse_tcp_connection\t> XXX\u7565
clickhouse_http_connectionch \u7684 HTTP \u8fde\u63a5\u6570clickhouse_http_connection > XXX\u7565
clickhouse_zoo_keeper_requestch \u4e2d\u6b63\u5728\u8fd0\u884c\u7684 zk \u8bf7\u6c42\u6570clickhouse_zoo_keeper_request > XXX\u7565
clickhouse_replicas_max_queue_sizech \u4e2d zk \u526f\u672c\u540c\u6b65\u961f\u5217\u7684\u957f\u5ea6clickhouse_replicas_max_queue_size > 100zk \u526f\u672c\u540c\u6b65\u961f\u5217\u957f\u5ea6\u8d85\u8fc7\u9608\u503c\u5219\u62a5\u8b66\uff0c\u8bf4\u660e\u6b64\u65f6\u526f\u672c\u540c\u6b65\u961f\u5217\u51fa\u73b0\u5806\u79ef

2.3 \u5176\u4ed6\u5e38\u7528 SQL

\n

\u5728 clickhouse \u4e2d\uff0c\u6240\u6709\u88ab\u6267\u884c\u7684 Query \u90fd\u4f1a\u8bb0\u5f55\u5230system.query_log\u8868\u4e2d\u3002\u56e0\u6b64\u6211\u4eec\u53ef\u901a\u8fc7\u8be5\u8868\u76d1\u63a7\u96c6\u7fa4\u7684\u67e5\u8be2\u60c5\u51b5\u3002\u4ee5\u4e0b\u5217\u4e3e\u51e0\u79cd\u7528\u4e8e\u76d1\u63a7\u7684\u5e38\u7528 SQL \u3002\u4e3a\u4e86\u66f4\u65b9\u4fbf\u7684\u67e5\u770b\uff0c\u53ef\u6dfb\u52a0\u5230 grafana \u770b\u677f\u4e2d\u3002

\n

\u6700\u8fd1\u67e5\u8be2

\n
SELECT \n event_time, \n user, \n query_id AS query, \n read_rows, \n read_bytes, \n result_rows, \n result_bytes, \n memory_usage, \n exception \nFROM clusterAllReplicas('cluster_name', system, query_log) \nWHERE (event_date = today()) AND (event_time >= (now() - 60)) AND (is_initial_query = 1) AND (query NOT LIKE 'INSERT INTO%') \nORDER BY event_time DESC \nLIMIT 100 \n
\n

\u6162\u67e5\u8be2

\n
SELECT \n event_time, \n user, \n query_id AS query, \n read_rows, \n read_bytes, \n result_rows, \n result_bytes, \n memory_usage, \n exception \nFROM clusterAllReplicas('cluster_name', system, query_log) \nWHERE (event_date = yesterday()) AND query_duration_ms > 30000 AND (is_initial_query = 1) AND (query NOT LIKE 'INSERT INTO%') \nORDER BY query_duration_ms desc \nLIMIT 100 \n
\n

Top10 \u5927\u8868

\n
SELECT \n database, \n table, \n sum(bytes_on_disk) AS bytes_on_disk \nFROM clusterAllReplicas('cluster_name', system, parts) \nWHERE active AND (database != 'system') \nGROUP BY \n database, \n table \nORDER BY bytes_on_disk DESC \nLIMIT 10 \n
\n

Top10 \u67e5\u8be2\u7528\u6237

\n
SELECT \n user, \n count(1) AS query_times, \n sum(read_bytes) AS query_bytes, \n sum(read_rows) AS query_rows \nFROM clusterAllReplicas('cluster_name', system, query_log) \nWHERE (event_date = yesterday()) AND (is_initial_query = 1) AND (query NOT LIKE 'INSERT INTO%') \nGROUP BY user \nORDER BY query_times DESC \nLIMIT 10 \n
\n
\n

\u66f4\u591a\u7cbe\u5f69\u5185\u5bb9\uff0c\u8bf7\u626b\u7801\u5173\u6ce8\u5fae\u4fe1\u516c\u4f17\u53f7\uff1a\u540e\u7aef\u6280\u672f\u5c0f\u5c4b\u3002\u5982\u679c\u89c9\u5f97\u6587\u7ae0\u5bf9\u4f60\u6709\u5e2e\u52a9\u7684\u8bdd\uff0c\u8bf7\u591a\u591a\u5206\u4eab\u3001\u8f6c\u53d1\u3001\u5728\u770b\u3002

\n
\n\n" }, { "author": { "url": "member/xiaomaisui", "name": "xiaomaisui", "avatar": "https://cdn.v2ex.com/gravatar/711df1b43b91206122fde9ed100dc0d0?s=73&d=retro" }, "url": "t/725158", "title": "intellij \u8c03\u8bd5 spark \u7a0b\u5e8f\u53ef\u4ee5\u67e5\u770b\u5230 rdd \u6570\u636e\u5417", "id": "t/725158", "date_published": "2020-11-14T04:45:19+00:00", "content_html": "

\u5728\u7f16\u5199 spark \u7a0b\u5e8f\u7684\u65f6\u5019\u60f3\u901a\u8fc7 debug \u770b\u770b rdd \u7684\u8f6c\u6362\u64cd\u4f5c\u6b63\u786e\u4e0d\u6b63\u786e\uff0c\u4f46\u662f debug \u4ee5\u540e\u8c8c\u4f3c\u6ca1\u6709\u627e\u5230\uff0c\u6709\u6ca1\u6709\u5927\u4f6c\u77e5\u9053\u8fd9\u4e2a\u745e\u548c debug \u51fa\u6765 ?

\n" }, { "author": { "url": "member/oahebky", "name": "oahebky", "avatar": "https://cdn.v2ex.com/avatar/2e9e/c57d/429363_large.png?m=1587021641" }, "url": "t/715520", "title": "talend \u5728\u4e1a\u5185\uff08\u6570\u636e\u5206\u6790\u3001ETL\u3001BI\uff09\u662f\u4ec0\u4e48\u60c5\u51b5\uff1f\u597d\u7528\uff1f Java \u7684\u5b9e\u8d28\u6027\u6807\u51c6\uff1f\u6210\u4e3a\u5751\u4e86\uff0c\u88ab\u55e4\u4e4b\u4ee5\u9f3b\uff1f etc...???", "id": "t/715520", "date_published": "2020-10-16T02:49:54+00:00", "content_html": "

[ talend \u7b80\u4ecb]

\n

1. \u4ec0\u4e48\u662f Talend

\n

Talend Open Studio \u662f Talend \uff08\u62d3\u84dd\uff09\u516c\u53f8\u5f00\u53d1\u7684\u4e00\u4e2a\u6570\u636e\u96c6\u6210\u7684\u6570\u636e ETL \u8f6f\u4ef6\uff0c\u53ef\u4ee5\u7b80\u5316\u6570\u636e\u5904\u7406\u6d41\u7a0b\uff0c\u964d\u4f4e\u5165\u95e8\u95e8\u69db\uff0c\u4e0d\u9700\u8981\u638c\u63e1\u4e13\u4e1a\u7684 ETL \u77e5\u8bc6\uff0c\u4ec5\u4ec5\u901a\u8fc7 web \u754c\u9762\u548c\u7b80\u5355\u7684\u7ec4\u4ef6\u62d6\u62fd\u5c31\u53ef\u5b9e\u73b0\u6570\u636e\u5904\u7406\u3002\u53ef\u4ee5\u534f\u52a9\u4f01\u4e1a\u5229\u7528\u66f4\u591a\u6570\u636e\uff0c\u4e0d\u65ad\u63d0\u9ad8\u5176\u6570\u636e\u7684\u53ef\u7528\u6027\u3001\u53ef\u9760\u6027\u4ee5\u53ca\u6709\u7528\u6027\u3002BI \u5de5\u5177 Talend Open Studio \u529f\u80fd\u5f3a\u5927\uff0c\u53ef\u4ee5\u540c\u6b65\u591a\u79cd\u6570\u636e\u5e93\uff0c\u53ef\u4ee5\u6e05\u6d17\u3001\u7b5b\u9009\u3001java \u4ee3\u7801\u5904\u7406\u6570\u636e\u3001\u6570\u636e\u5bfc\u5165\u5bfc\u51fa\uff0c\u5185\u8054\u67e5\u8be2\u591a\u79cd\u6570\u636e\u5e93\uff0c\u4ee5\u4e0b\u7b80\u79f0 talend \u3002

\n

\u6982\u51b5\u6765\u8bf4\uff0ctalend \u7279\u70b9\u4e3b\u8981\u6709\u4ee5\u4e0b\u51e0\u70b9\uff1a

\n

\u6570\u636e\u6e90\uff1a\u5404\u79cd\u5e38\u7528\u6570\u636e\u5e93\uff08 mysql oracle hive \uff09\uff0c\u6587\u4ef6\u7b49\u3002

\n\n
\n

\u7b49\u5728\u4f7f\u7528 talend \u7684\u670b\u53cb\u8bf4\u8bf4\u770b\u3002

\n" }, { "author": { "url": "member/zealinux", "name": "zealinux", "avatar": "https://cdn.v2ex.com/gravatar/97cec5ffbda3f47b2cd453cf10b78244?s=73&d=retro" }, "url": "t/657850", "date_modified": "2020-03-31T02:51:59+00:00", "content_html": "

\u4e0d\u662f\u81ea\u5df1\u7528\uff0c\uff08\u4e0d\u8981\u7f51\u76d8\uff09\n\u60f3\u79c1\u6709\u5316\u90e8\u7f72\u516c\u53f8\u7684\u670d\u52a1\u3002\n\u5176\u4e2d\u6587\u4ef6\u5b58\u50a8\u662f\u7528\u7684 s3\uff0c\n\u73b0\u5728\u60f3\u627e\u4e2a\u5f00\u6e90\u7684\u7c7b\u4f3c\u7684\u670d\u52a1\uff0c\u81ea\u5df1\u642d\u5efa

\n

\u4e3b\u8981\u662f\u4e0a\u4f20 API \u63a5\u53e3\uff0c\u6709 bucket\uff0c\n\u4e0a\u4f20\u4e2a\u56fe\u7247\uff0c\u7ed9\u51fa\u4e2a\u516c\u5f00 http \u5730\u5740\uff0c\n\u5982\u679c\u6709\u6743\u9650\u8ba4\u8bc1\u5c31\u66f4\u597d\u4e86\u3002

\n
\n

\u611f\u89c9 nginx \u5e94\u8be5\u4e5f\u662f\u53ef\u4ee5\u505a\u8fd9\u6837\u7684\u4e8b\u60c5\uff0c\n\u4f46\u662f\u8981\u8003\u8651\u5230\u78c1\u76d8\u53ef\u80fd\u4f1a\u7206\u6389\uff0c\n\u4f46\u5982\u679c\u4e00\u5f00\u59cb\u5c31\u7533\u8bf7\u5f88\u5927\u7684\u78c1\u76d8\uff0c\u53c8\u5f88\u6d6a\u8d39\u3002\n\uff08\u542c\u522b\u4eba\u8bf4\uff0c\u8981\u7528 HDFS \u8fd9\u7c7b\u7684\u4e1c\u897f\u3002\uff09

\n

\u5927\u5bb6\u6709\u4ec0\u4e48\u53ef\u4ee5\u63a8\u8350\u7684\uff1f\n\uff08\u4e00\u5b9a\u8981\u652f\u6301 Docker \u90e8\u7f72\uff09

\n", "date_published": "2020-03-31T02:51:39+00:00", "title": "\u6709\u4ec0\u4e48\u81ea\u5efa [\u6587\u4ef6\u670d\u52a1] \u63a8\u8350\uff0c\u7528\u6765\u81ea\u5efa\u66ff\u4ee3 S3 \u670d\u52a1\uff1f", "id": "t/657850" }, { "author": { "url": "member/JohnXu20151211", "name": "JohnXu20151211", "avatar": "https://cdn.v2ex.com/gravatar/58b49211768add9b479f99100faba14f?s=73&d=retro" }, "url": "t/652310", "title": "\u600e\u4e48\u52a0\u6df1\u5927\u6570\u636e\u6280\u80fd\uff0c\u8fdb\u5165\u5927\u5382\u3002\u597d\u8ff7\u832b,\u6c42\u6307\u70b9", "id": "t/652310", "date_published": "2020-03-12T13:28:58+00:00", "content_html": "

\u505a\u5f00\u53d1 4 \u5e74\u4e86\u5427\uff0c\u73b0\u5728\u662f\u505a\u5927\u6570\u636e\u76f8\u5173\u7684\uff0c\u516c\u53f8\u662f\u505a\u4e09\u65b9\u652f\u4ed8\u7684\u3002\u5728\u8fd9\u5bb6\u516c\u53f8\u5df2\u7ecf\u5446\u4e86\u6709 1 \u5e74\u534a\u4e86\uff0c\u4ece\u4e00\u5f00\u59cb\u6765\u7684\u65f6\u5019\u5e76\u4e0d\u662f\u505a\u5927\u6570\u636e\u505a\u7684 java\uff0c\u540e\u6765\u5f00\u59cb\u505a\u4e86\u5927\u6570\u636e\u628a\u516c\u53f8\u4e4b\u524d\u7684\u5927\u6570\u636e\u5e73\u53f0\u91cd\u6784\u505a\u4e86\u4e00\u7248\uff0c\u4f46\u662f\u73b0\u5728\u8d8a\u5446\u7740\u8d8a\u53d1\u73b0\u5b66\u4e0d\u5230\u4ec0\u4e48\u4e1c\u897f\uff0c\u56e0\u4e3a\u5176\u5b9e\u516c\u53f8\u6ca1\u6709\u51e0\u4eba\u771f\u6b63\u61c2\u5927\u6570\u636e\u7684\uff0c\u5305\u62ec\u6211\u4eec\u5c0f\u7ec4\u4e5f\u4e00\u6837\u3002\u73b0\u5728\u516c\u53f8\u662f\u60f3\u505a\u5173\u4e8e\u5b9e\u65f6\u7684\u4e1c\u897f\uff0c\u4f46\u5176\u5b9e\u6240\u6709\u4eba\u90fd\u6ca1\u6709\u4e00\u4e2a\u601d\u8def\u8be5\u600e\u4e48\u505a\uff0c\u505a\u4ec0\u4e48\uff0c\u6bd5\u7adf\u6211\u4eec\u6570\u636e\u6bd4\u8f83\u5c40\u9650\u3002\u6211\u5176\u5b9e\u7279\u522b\u60f3\u80fd\u8d81\u5e74\u8f7b\u8fdb\u5927\u5382\u5386\u7ec3\u5386\u7ec3\uff0c\u5b66\u4e9b\u6280\u672f\u3002\u4f46\u662f\u672c\u8eab\u5b66\u5386\u4e0d\u9ad8\uff0c\u518d\u52a0\u4e0a\u73b0\u5728\u516c\u53f8\u7684\u5927\u6570\u636e\u6ca1\u6709\u4ec0\u4e48\u6280\u672f\u542b\u91cf\uff0c\u6211\u8ba4\u4e3a\u5927\u6570\u636e\u7684\u771f\u6b63\u7684\u6280\u672f\u5e94\u8be5\u5c31\u662f\u6570\u636e\u4e0b\u94bb\u3001\u5b9e\u65f6\u3001\u6316\u6398\u3001\u5206\u6790\u3001\u673a\u5668\u5b66\u4e60\u7b49\u7b49\u8fd9\u4e9b\u3002\u522b\u4eba\u4e00\u95ee\u4ec0\u4e48\u5b9e\u65f6\u3001\u6570\u636e\u4e0b\u94bb\u3001\u6570\u636e\u6316\u6398\u4ec0\u4e48\u7684\u6211\u4eec\u90fd\u6ca1\u6709\u505a\uff0c\u5176\u5b9e\u5c31\u76f8\u5f53\u4e8e\u4e2a\u67e5\u8be2\u3002\u6240\u4ee5\u60f3\u53bb\u5927\u5382\u53c8\u6ca1\u6709\u62ff\u5f97\u51fa\u624b\u7684\u9879\u76ee\uff0c\u73b0\u5728\u53c8\u5b66\u4e0d\u5230\u4ec0\u4e48\uff0c\u56e0\u4e3a\u6211\u611f\u89c9\u5c31\u6211\u4eec\u73b0\u5728\u80fd\u529b\u4e0d\u592a\u53ef\u80fd\u505a\u51fa\u76f8\u5bf9\u6709\u4e9b\u6280\u672f\u542b\u91cf\u7684\u4e1c\u897f\u3002\u81ea\u5df1\u6709\u65f6\u95f4\u5c31\u5404\u79cd\u5237\u7b97\u6cd5\u9898\uff0c\u770b\u5927\u6570\u636e\u7684\u89c6\u9891\u3002\u4f46\u662f\u50cf spark\u3001flink \u7b49\u7b49\u8fd9\u4e9b\u8fd8\u662f\u8981\u5b9e\u8df5\uff0c\u6211\u81ea\u5df1\u4e5f\u642d\u4e86\u4e2a\u5b66\u4e60\u7684\u96c6\u7fa4\uff0c\u4f46\u662f\u4e5f\u4e0d\u77e5\u9053\u81ea\u5df1\u80fd\u7528\u8fd9\u4e9b\u505a\u4e9b\u4ec0\u4e48\u80fd\u9762\u8bd5\u7684\u4e1c\u897f\u3002 \u6073\u8bf7\u5404\u4f4d\u5927\u4f6c\u8bf4\u51fa\u4f60\u4eec\u7684\u770b\u6cd5\uff0c\u63d0\u51fa\u4f60\u4eec\u7684\u5efa\u8bae\u3002

\n" }, { "author": { "url": "member/zealinux", "name": "zealinux", "avatar": "https://cdn.v2ex.com/gravatar/97cec5ffbda3f47b2cd453cf10b78244?s=73&d=retro" }, "url": "t/646403", "date_modified": "2020-02-21T08:39:58+00:00", "content_html": "

\u5e7f\u544a\u7cfb\u7edf\u4f1a\u8ddf\u8e2a\u4e00\u4e9b\u6570\u636e\uff08\u7c7b\u4f3c log \uff09\uff0c\u8fd9\u4e9b\u6570\u636e\u4e13\u95e8\u7528\u6765\u505a\u7edf\u8ba1\u7684\u3002

\n

\u4f46\u662f\u7531\u4e8e\u4ee5\u540e\u53ef\u80fd\u91cf\u6bd4\u8f83\u5927\uff0c\u4e0d\u60f3\u7740\u5b58\u6570\u636e\u5e93\uff08 MySQL \uff09\u3002

\n

\u73b0\u5728\u4e00\u822c\u662f\u600e\u4e48\u597d\u505a\u6cd5\uff1f

\n
\n

\u6bd4\u5982\uff0c\u73b0\u5728\u60f3\u5230\u7684\u6bd4\u8f83\u7b28\u7684\u65b9\u6cd5\uff1a

\n

\u7cfb\u7edf\u4f1a\u6709\u9891\u7e41\u8ddf\u8e2a\u6570\u636e\u4f20\u8fc7\u6765\uff0c\n\u90a3\u4e48\u6211\u5c31\u5728 DB \u91cc [\u8ddf\u8e2a\u65e5\u5fd7\u8868] \u91cc\u65b0\u5efa\u4e00\u6761\u8bb0\u5f55\u3002

\n

\u7136\u540e\u5b9a\u65f6\u6bcf 10 \u5206\u949f\u6c47\u603b\u4e00\u4e9b\u3002\u8bb0\u5f55\u5728 [\u5206\u5c0f\u65f6\u7edf\u8ba1\u8868] \u91cc\u3002\n\u7136\u540e\u518d\u5b9a\u65f6\u4ece [\u5206\u5c0f\u65f6\u7edf\u8ba1\u8868] \u7b97\u51fa\u603b\u91cf\uff0c\u5b58\u653e [\u603b\u91cf\u8868] \u7684\u3002

\n

\u73b0\u5728\u662f 3 \u5f20\u8868\u3002

\n
A - \u8ddf\u8e2a\u8bb0\u5f55\u8868\nB - \u5206\u5c0f\u65f6\u7edf\u8ba1\u8868\nC - \u6bcf\u4e2a\u5e7f\u544a\u7684\u7edf\u8ba1\u603b\u91cf\u8868\n
\n
\n

\u611f\u89c9\u8fd9\u79cd\u65b9\u6cd5\u4e0d\u597d\uff0c\nA \u8868\u5e94\u8be5\u7b97\u662f\u975e\u4e3b\u4e1a\u52a1\u7684\uff0c\u4e0d\u5e94\u8be5\u4e0e\u4e1a\u52a1\u8868\u4eec\u653e\u5728\u4e00\u8d77\u3002\nA \u8868\u4e5f\u4f1a\u8d8a\u6765\u8d8a\u5e9e\u5927\u3002\uff08\u5927\u8868\u505a\u7edf\u8ba1\u7684\u8bdd\uff0c\u53ef\u80fd\u4f1a\u628a MySQL \u641e\u5d29\u6e83\u6389\uff09\n\u5176\u4f59\u4e24\u4e2a\u8868\u662f\u7531\u4e8e A \u6765\u7b97\u51fa\u6765\u7684\u3002

\n

\u5e94\u8be5\u4f1a\u6709\u66f4\u597d\u7684\u89e3\u51b3\u65b9\u6cd5\uff0c\u6c42\u6559

\n", "date_published": "2020-02-21T08:39:46+00:00", "title": "[\u7cfb\u7edf\u8bbe\u8ba1] \u5e7f\u544a\u7cfb\u7edf\u7684\u7edf\u8ba1\u6570\u636e\u8be5\u600e\u4e48\u8bbe\u8ba1\uff1f", "id": "t/646403" }, { "author": { "url": "member/PingCAP", "name": "PingCAP", "avatar": "https://cdn.v2ex.com/avatar/daac/3644/162717_large.png?m=1536739465" }, "url": "t/634414", "title": "\u590d\u6742\u5206\u5e03\u5f0f\u67b6\u6784\u4e0b\u7684\u8ba1\u7b97\u6cbb\u7406\u4e4b\u8def", "id": "t/634414", "date_published": "2020-01-02T05:27:09+00:00", "content_html": "
\n

\u5206\u5e03\u5f0f\u6280\u672f\u7684\u53d1\u5c55\uff0c\u6df1\u523b\u5730\u6539\u53d8\u4e86\u6211\u4eec\u7f16\u7a0b\u7684\u6a21\u5f0f\u548c\u601d\u8003\u8f6f\u4ef6\u7684\u6a21\u5f0f\u3002\u503c 2019 \u5c81\u672b\uff0cPingCAP \u8054\u5408 InfoQ \u5171\u540c\u7b56\u5212\u51fa\u54c1\u201c\u5206\u5e03\u5f0f\u7cfb\u7edf\u524d\u6cbf\u6280\u672f \u201d\u4e13\u9898\uff0c \u9080\u8bf7\u4f17\u591a\u6280\u672f\u56e2\u961f\u5171\u540c\u53c2\u4e0e\uff0c\u4e00\u8d77\u63a2\u7d22\u8fd9\u4e2a\u53e4\u8001\u9886\u57df\u7684\u65b0\u751f\u673a\u3002\u672c\u6587\u51fa\u81ea\u5fae\u4f17\u94f6\u884c\u5927\u6570\u636e\u5e73\u53f0\u8d1f\u8d23\u4eba\u90b8\u5e05\u3002

\n
\n

\u5728\u5f53\u524d\u7684\u590d\u6742\u5206\u5e03\u5f0f\u67b6\u6784\u73af\u5883\u4e0b\uff0c\u670d\u52a1\u6cbb\u7406\u5df2\u7ecf\u5927\u884c\u5176\u9053\u3002\u4f46\u76ee\u5149\u5f80\u4e0b\u4e00\u5c42\uff0c\u4ece\u4e0a\u5c42 APP\u3001Service\uff0c\u5230\u5e95\u5c42\u8ba1\u7b97\u5f15\u64ce\u8fd9\u4e00\u5c42\u9762\uff0c\u5374\u8fd8\u662f\u5404\u4e2a\u5f15\u64ce\u5404\u81ea\u4e3a\u653f\uff0cClient-Server \u6a21\u5f0f\u7d27\u8026\u5408\u6ee1\u5929\u98de\u7684\u60c5\u51b5\u3002\u5982\u4f55\u505a\u597d\u201c\u8ba1\u7b97\u6cbb\u7406\u201d\uff0c\u8ba9\u590d\u6742\u73af\u5883\u4e0b\u5404\u79cd\u7c7b\u578b\u7684\u5927\u91cf\u8ba1\u7b97\u4efb\u52a1\uff0c\u90fd\u80fd\u66f4\u7b80\u6d01\u3001\u7075\u6d3b\u3001\u6709\u5e8f\u3001\u53ef\u63a7\u7684\u63d0\u4ea4\u6267\u884c\uff0c\u548c\u4fdd\u969c\u6210\u529f\u8fd4\u56de\u7ed3\u679c\uff1f\u8ba1\u7b97\u4e2d\u95f4\u4ef6 Linkis \u5c31\u662f\u4e0a\u8ff0\u95ee\u9898\u7684\u6700\u4f73\u5b9e\u8df5\u3002

\n

\u590d\u6742\u5206\u5e03\u5f0f\u67b6\u6784\u73af\u5883\u4e0b\u7684\u8ba1\u7b97\u6cbb\u7406\u6709\u4ec0\u4e48\u95ee\u9898\uff1f

\n

\u4ec0\u4e48\u662f\u590d\u6742\u5206\u5e03\u5f0f\u67b6\u6784\u73af\u5883\uff1f

\n

\u5206\u5e03\u5f0f\u67b6\u6784\uff0c\u6307\u7684\u662f\u7cfb\u7edf\u7684\u7ec4\u4ef6\u5206\u5e03\u5728\u901a\u8fc7\u7f51\u7edc\u76f8\u8fde\u7684\u4e0d\u540c\u8ba1\u7b97\u673a\u4e0a\uff0c\u7ec4\u4ef6\u4e4b\u95f4\u901a\u8fc7\u7f51\u7edc\u4f20\u9012\u6d88\u606f\u8fdb\u884c\u901a\u4fe1\u548c\u534f\u8c03\uff0c\u534f\u540c\u5b8c\u6210\u67d0\u4e00\u76ee\u6807\u3002\u4e00\u822c\u6765\u8bf4\u6709\u6c34\u5e73\uff08\u96c6\u7fa4\u5316\uff09\u548c\u5782\u76f4\uff08\u529f\u80fd\u6a21\u5757\u5207\u5206\uff09\u4e24\u4e2a\u62c6\u5206\u65b9\u5411\uff0c\u4ee5\u89e3\u51b3\u9ad8\u5185\u805a\u4f4e\u8026\u5408\u3001\u9ad8\u5e76\u53d1\u3001\u9ad8\u53ef\u7528\u7b49\u65b9\u9762\u95ee\u9898\u3002

\n

\u591a\u4e2a\u5206\u5e03\u5f0f\u67b6\u6784\u7684\u7cfb\u7edf\uff0c\u7ec4\u6210\u5206\u5e03\u5f0f\u7cfb\u7edf\u7fa4\uff0c\u5c31\u5f62\u6210\u4e86\u4e00\u4e2a\u76f8\u5bf9\u590d\u6742\u7684\u5206\u5e03\u5f0f\u67b6\u6784\u73af\u5883\u3002\u901a\u5e38\u5305\u542b\u591a\u79cd\u4e0a\u5c42\u5e94\u7528\u670d\u52a1\uff0c\u591a\u79cd\u5e95\u5c42\u57fa\u7840\u8ba1\u7b97\u5b58\u50a8\u5f15\u64ce\u3002\u5982\u4e0b\u56fe 1 \u6240\u793a\uff1a

\n

\"\"

\n<center>\u56fe 1</center>\n

\u4ec0\u4e48\u662f\u8ba1\u7b97\u6cbb\u7406\uff1f

\n

\u5c31\u50cf\u300a\u5fae\u670d\u52a1\u8bbe\u8ba1\u300b\u4e00\u4e66\u4e2d\u63d0\u5230\u7684\uff0c\u5982\u540c\u57ce\u5e02\u89c4\u5212\u5e08\u5728\u9762\u5bf9\u4e00\u5ea7\u5e9e\u5927\u3001\u590d\u6742\u4e14\u4e0d\u65ad\u53d8\u5316\u7684\u57ce\u5e02\u65f6\uff0c\u6240\u9700\u8981\u505a\u7684\u89c4\u5212\u3001\u8bbe\u8ba1\u548c\u6cbb\u7406\u4e00\u6837\uff0c\u5e9e\u5927\u590d\u6742\u7684\u8f6f\u4ef6\u7cfb\u7edf\u73af\u5883\u4e2d\u7684\u5404\u79cd\u533a\u57df\u3001\u5143\u7d20\u3001\u89d2\u8272\u548c\u5173\u7cfb\uff0c\u4e5f\u9700\u8981\u6574\u6cbb\u548c\u7ba1\u7406\uff0c\u4ee5\u4f7f\u5176\u4ee5\u4e00\u79cd\u66f4\u7b80\u6d01\u3001\u4f18\u96c5\u3001\u6709\u5e8f\u3001\u53ef\u63a7\u7684\u65b9\u5f0f\u534f\u540c\u8fd0\u4f5c\uff0c\u800c\u4e0d\u662f\u53d8\u6210\u4e00\u56e2\u4e71\u9ebb\u3002

\n

\u5728\u5f53\u524d\u7684\u590d\u6742\u5206\u5e03\u5f0f\u67b6\u6784\u73af\u5883\u4e0b\uff0c\u5927\u91cf APP\u3001Service \u95f4\u7684\u901a\u4fe1\u3001\u534f\u8c03\u548c\u7ba1\u7406\uff0c\u5df2\u7ecf\u6709\u4e86\u4ece SOA \uff08 Service-Oriented Architecture \uff09\u5230\u5fae\u670d\u52a1\u7684\u6210\u719f\u7406\u5ff5\uff0c\u53ca\u4ece ESB \u5230 Service Mesh \u7684\u4f17\u591a\u5b9e\u8df5\uff0c\u6765\u5b9e\u73b0\u5176\u4ece\u670d\u52a1\u6ce8\u518c\u53d1\u73b0\u3001\u914d\u7f6e\u7ba1\u7406\u3001\u7f51\u5173\u8def\u7531\uff0c\u5230\u6d41\u63a7\u7194\u65ad\u3001\u65e5\u5fd7\u76d1\u63a7\u7b49\u4e00\u7cfb\u5217\u5b8c\u6574\u7684\u670d\u52a1\u6cbb\u7406\u529f\u80fd\u3002\u670d\u52a1\u6cbb\u7406\u6846\u67b6\u7684\u201c\u4e2d\u95f4\u4ef6\u201d\u5c42\u8bbe\u8ba1\uff0c\u53ef\u4ee5\u5f88\u597d\u7684\u5b9e\u73b0\u670d\u52a1\u95f4\u7684\u89e3\u8026\u3001\u5f02\u6784\u5c4f\u853d\u548c\u4e92\u64cd\u4f5c\uff0c\u5e76\u63d0\u4f9b\u8def\u7531\u3001\u6d41\u63a7\u3001\u72b6\u6001\u7ba1\u7406\u3001\u76d1\u63a7\u7b49\u6cbb\u7406\u7279\u6027\u7684\u5171\u6027\u63d0\u70bc\u548c\u590d\u7528\uff0c\u589e\u5f3a\u6574\u4e2a\u67b6\u6784\u7684\u7075\u6d3b\u6027\u3001\u7ba1\u63a7\u80fd\u529b\u3001\u53ef\u6269\u5c55\u6027\u548c\u53ef\u7ef4\u62a4\u6027\u3002

\n

\u4f46\u76ee\u5149\u5f80\u4e0b\u4e00\u5c42\uff0c\u4f60\u4f1a\u53d1\u73b0\u5728\u4ece APP\u3001Service\uff0c\u5230\u540e\u53f0\u5f15\u64ce\u8fd9\u4e00\u5c42\u9762\uff0c\u5374\u8fd8\u662f\u5404\u4e2a\u5f15\u64ce\u5404\u81ea\u4e3a\u653f\uff0cClient-Server \u6a21\u5f0f\u7d27\u8026\u5408\u6ee1\u5929\u98de\u7684\u60c5\u51b5\u3002\u5728\u5927\u91cf\u7684\u4e0a\u5c42\u5e94\u7528\uff0c\u548c\u5927\u91cf\u7684\u5e95\u5c42\u5f15\u64ce\u4e4b\u95f4\uff0c\u7f3a\u4e4f\u4e00\u5c42\u901a\u7528\u7684\u201c\u4e2d\u95f4\u4ef6\u201d\u6846\u67b6\u8bbe\u8ba1\u3002\u7c7b\u4f3c\u4e0b\u56fe 2 \u7684\u7f51\u72b6\u3002

\n

\"\"

\n<center>\u56fe 2</center>\n

\u8ba1\u7b97\u6cbb\u7406\uff0c\u5173\u6ce8\u7684\u6b63\u662f\u4e0a\u5c42\u5e94\u7528\u548c\u5e95\u5c42\u8ba1\u7b97\uff08\u5b58\u50a8\uff09\u5f15\u64ce\u4e4b\u95f4\uff0c\u4ece Client \u5230 Server \u7684\u8fde\u63a5\u5c42\u8303\u56f4\uff0c\u6240\u5b58\u5728\u7684\u7d27\u8026\u5408\u3001\u7075\u6d3b\u6027\u548c\u7ba1\u63a7\u80fd\u529b\u6b20\u7f3a\u3001\u7f3a\u4e4f\u590d\u7528\u80fd\u529b\u3001\u53ef\u6269\u5c55\u6027\u3001\u53ef\u7ef4\u62a4\u6027\u5dee\u7b49\u95ee\u9898\u3002\u8981\u8ba9\u590d\u6742\u5206\u5e03\u5f0f\u67b6\u6784\u73af\u5883\u4e0b\u5404\u79cd\u7c7b\u578b\u7684\u8ba1\u7b97\u4efb\u52a1\uff0c\u90fd\u80fd\u66f4\u7b80\u6d01\u3001\u7075\u6d3b\u3001\u6709\u5e8f\u3001\u53ef\u63a7\u7684\u63d0\u4ea4\u6267\u884c\uff0c\u548c\u6210\u529f\u8fd4\u56de\u7ed3\u679c\u3002\u5982\u4e0b\u56fe 3 \u6240\u793a\uff1a

\n

\"\"

\n<center>\u56fe 3</center>\n

\u8ba1\u7b97\u6cbb\u7406\u95ee\u9898\u63cf\u8ff0

\n

\u66f4\u8be6\u7ec6\u7684\u6765\u770b\u8ba1\u7b97\u6cbb\u7406\u7684\u95ee\u9898\uff0c\u53ef\u4ee5\u5206\u4e3a\u5982\u4e0b\u6cbb\uff08 architecture\uff0c\u67b6\u6784\u5c42\u9762\uff09\u548c\u7406\uff08 insight\uff0c\u7ec6\u5316\u7279\u6027\uff09\u4e24\u4e2a\u5c42\u9762\u3002

\n

\u8ba1\u7b97\u6cbb\u7406\u4e4b\u6cbb\uff08 architecture \uff09-\u67b6\u6784\u5c42\u9762\u95ee\u9898

\n

\u7d27\u8026\u5408\u95ee\u9898\uff0c\u4e0a\u5c42\u5e94\u7528\u548c\u5e95\u5c42\u8ba1\u7b97\u5b58\u50a8\u5f15\u64ce\u95f4\u7684 CS \u8fde\u63a5\u6a21\u5f0f\u3002

\n

\u6240\u6709 APP & Service \u548c\u5e95\u5c42\u8ba1\u7b97\u5b58\u50a8\u5f15\u64ce\uff0c\u90fd\u662f\u901a\u8fc7 Client-Server \u6a21\u5f0f\u76f8\u8fde\uff0c\u5904\u4e8e\u7d27\u8026\u5408\u72b6\u6001\u3002\u4ee5 Analytics Engine \u7684 Spark \u4e3a\u4f8b\uff0c\u5982\u4e0b\u56fe 4\uff1a

\n

\"\"

\n<center>\u56fe 4</center>\n

\u8fd9\u79cd\u72b6\u6001\u4f1a\u5e26\u6765\u5982\u4e0b\u95ee\u9898\uff1a

\n\n

\u91cd\u590d\u9020\u8f6e\u5b50\u95ee\u9898\uff0c\u6bcf\u4e2a\u4e0a\u5c42\u5e94\u7528\u5de5\u5177\u7cfb\u7edf\u90fd\u8981\u91cd\u590d\u89e3\u51b3\u8ba1\u7b97\u6cbb\u7406\u95ee\u9898\u3002

\n

\u6bcf\u4e2a\u4e0a\u5c42\u5e94\u7528\u90fd\u8981\u91cd\u590d\u7684\u53bb\u96c6\u6210\u5404\u79cd client\uff0c\u521b\u5efa\u548c\u7ba1\u7406 client \u5230\u5f15\u64ce\u7684\u8fde\u63a5\u53ca\u5176\u72b6\u6001\uff0c\u5305\u62ec\u5e95\u5c42\u5f15\u64ce\u5143\u6570\u636e\u7684\u83b7\u53d6\u4e0e\u7ba1\u7406\u3002\u5728\u5e76\u53d1\u4f7f\u7528\u7684\u7528\u6237\u9010\u6e10\u53d8\u591a\u3001\u5e76\u53d1\u8ba1\u7b97\u4efb\u52a1\u91cf\u9010\u6e10\u53d8\u5927\u65f6\uff0c\u6bcf\u4e2a\u4e0a\u5c42\u5e94\u7528\u8fd8\u8981\u91cd\u590d\u7684\u53bb\u89e3\u51b3\u591a\u4e2a\u7528\u6237\u95f4\u5728 client \u7aef\u7684\u8d44\u6e90\u4e89\u7528\u3001\u6743\u9650\u9694\u79bb\uff0c\u8ba1\u7b97\u4efb\u52a1\u7684\u8d85\u65f6\u7ba1\u7406\u3001\u5931\u8d25\u91cd\u8bd5\u7b49\u7b49\u8ba1\u7b97\u6cbb\u7406\u95ee\u9898\u3002

\n

\"\"

\n<center>\u56fe 5</center>\n

\u60f3\u8c61\u4f60\u6709 10 \u4e2a\u5e76\u53d1\u4efb\u52a1\u6570\u8fc7\u767e\u7684\u4e0a\u5c42\u5e94\u7528\uff0c\u4e0d\u7ba1\u662f\u57fa\u4e8e Web \u7684 IDE \u5f00\u53d1\u73af\u5883\u3001\u53ef\u89c6\u5316 BI \u7cfb\u7edf\uff0c\u8fd8\u662f\u62a5\u8868\u7cfb\u7edf\u3001\u5de5\u4f5c\u6d41\u8c03\u5ea6\u7cfb\u7edf\u7b49\uff0c\u6bcf\u4e2a\u63a5\u5165 3 \u4e2a\u5e95\u5c42\u8ba1\u7b97\u5f15\u64ce\u3002\u4e0a\u8ff0\u7684\u8ba1\u7b97\u6cbb\u7406\u95ee\u9898\uff0c\u4f60\u53ef\u80fd\u5f97\u9010\u4e00\u91cd\u590d\u7684\u53bb\u89e3\u51b3 10*3=30 \u904d\uff0c\u800c\u8fd9\u6b63\u662f\u5f53\u524d\u5728\u5404\u4e2a\u516c\u53f8\u4e0d\u65ad\u53d1\u751f\u7684\u73b0\u5b9e\u60c5\u51b5\uff0c\u5176\u9020\u6210\u7684\u4eba\u529b\u6d6a\u8d39\u4e0d\u53ef\u5c0f\u89d1\u3002

\n

\u6269\u5c55\u96be\u95ee\u9898\uff0c\u4e0a\u5c42\u5e94\u7528\u65b0\u589e\u5bf9\u63a5\u5e95\u5c42\u8ba1\u7b97\u5f15\u64ce\uff0c\u7ef4\u62a4\u6210\u672c\u9ad8\uff0c\u6539\u52a8\u5927\u3002

\n

\u5728 CS \u7684\u7d27\u8026\u5408\u6a21\u5f0f\u4e0b\uff0c\u4e0a\u5c42\u5e94\u7528\u6bcf\u65b0\u589e\u5bf9\u63a5\u4e00\u4e2a\u5e95\u5c42\u8ba1\u7b97\u5f15\u64ce\uff0c\u90fd\u9700\u8981\u6709\u8f83\u5927\u6539\u52a8\u3002

\n

\u4ee5\u5bf9\u63a5 Spark \u4e3a\u4f8b\uff0c\u5728\u4e0a\u5c42\u5e94\u7528\u7cfb\u7edf\u4e2d\u7684\u6bcf\u4e00\u53f0\u9700\u8981\u63d0\u4ea4 Spark \u4f5c\u4e1a\u7684\u673a\u5668\uff0c\u90fd\u9700\u8981\u90e8\u7f72\u548c\u7ef4\u62a4\u597d Java \u548c Scala \u8fd0\u884c\u65f6\u73af\u5883\u548c\u53d8\u91cf\uff0c\u4e0b\u8f7d\u548c\u90e8\u7f72 Spark Client \u5305\uff0c\u4e14\u914d\u7f6e\u5e76\u7ef4\u62a4 Spark \u76f8\u5173\u7684\u73af\u5883\u53d8\u91cf\u3002\u5982\u679c\u8981\u4f7f\u7528 Spark on YARN \u6a21\u5f0f\uff0c\u90a3\u4e48\u4f60\u8fd8\u9700\u8981\u5728\u6bcf\u4e00\u53f0\u9700\u8981\u63d0\u4ea4 Spark \u4f5c\u4e1a\u7684\u673a\u5668\u4e0a\uff0c\u53bb\u90e8\u7f72\u548c\u7ef4\u62a4 Hadoop \u76f8\u5173\u7684 jar \u5305\u548c\u73af\u5883\u53d8\u91cf\u3002\u518d\u5982\u679c\u4f60\u7684 Hadoop \u96c6\u7fa4\u9700\u8981\u542f\u7528 Kerberos \u7684\uff0c\u90a3\u4e48\u5f88\u4e0d\u5e78\uff0c\u4f60\u8fd8\u9700\u8981\u5728\u4e0a\u8ff0\u7684\u6bcf\u53f0\u673a\u5668\u53bb\u7ef4\u62a4\u548c\u8c03\u8bd5 keytab\u3001principal \u7b49\u4e00\u5806 Kerberos \u76f8\u5173\u914d\u7f6e\u3002

\n

\"\"

\n<center>\u56fe 6</center>\n

\u8fd9\u8fd8\u4ec5\u4ec5\u662f\u5bf9\u63a5 Spark \u4e00\u4e2a\u5e95\u5c42\u5f15\u64ce\u3002\u968f\u7740\u4e0a\u5c42\u5e94\u7528\u7cfb\u7edf\u548c\u5e95\u5c42\u5f15\u64ce\u7684\u6570\u91cf\u589e\u591a\uff0c\u9700\u8981\u7ef4\u62a4\u7684\u5173\u7cfb\u4f1a\u662f\u4e2a\u7b1b\u5361\u5c14\u79ef\u5f0f\u7684\u589e\u957f\uff0c\u5149 Client \u548c\u914d\u7f6e\u7684\u90e8\u7f72\u7ef4\u62a4\uff0c\u5c31\u4f1a\u6210\u4e3a\u4e00\u4ef6\u5f88\u4ee4\u4eba\u5934\u75bc\u7684\u4e8b\u60c5\u3002

\n

\u5e94\u7528\u5b64\u5c9b\u95ee\u9898\uff0c\u8de8\u4e0d\u540c\u5e94\u7528\u5de5\u5177\u3001\u4e0d\u540c\u8ba1\u7b97\u4efb\u52a1\u95f4\u7684\u4e92\u901a\u95ee\u9898\u3002

\n

\u591a\u4e2a\u76f8\u4e92\u6709\u5173\u8054\u7684\u4e0a\u5c42\u5e94\u7528\uff0c\u5411\u540e\u53f0\u5f15\u64ce\u63d0\u4ea4\u6267\u884c\u7684\u4e0d\u540c\u8ba1\u7b97\u4efb\u52a1\u4e4b\u95f4\uff0c\u5f80\u5f80\u662f\u6709\u6240\u5173\u8054\u548c\u5171\u6027\u7684\uff0c\u6bd4\u5982\u9700\u8981\u5171\u4eab\u4e00\u4e9b\u7528\u6237\u5b9a\u4e49\u7684\u8fd0\u884c\u65f6\u73af\u5883\u53d8\u91cf\u3001\u51fd\u6570\u3001\u7a0b\u5e8f\u5305\u3001\u6570\u636e\u6587\u4ef6\u7b49\u3002\u5f53\u524d\u60c5\u51b5\u5f80\u5f80\u662f\u4e00\u4e2a\u4e2a\u5e94\u7528\u7cfb\u7edf\u5c31\u50cf\u4e00\u5ea7\u5ea7\u5b64\u5c9b\uff0c\u76f8\u5173\u4fe1\u606f\u548c\u8d44\u6e90\u65e0\u6cd5\u76f4\u63a5\u5171\u4eab\uff0c\u9700\u8981\u624b\u52a8\u5728\u4e0d\u540c\u5e94\u7528\u7cfb\u7edf\u91cc\u91cd\u590d\u5b9a\u4e49\u548c\u7ef4\u62a4\u3002

\n

\u5178\u578b\u4f8b\u5b50\u662f\u5728\u6570\u636e\u6279\u5904\u7406\u7a0b\u5e8f\u5f00\u53d1\u8fc7\u7a0b\u4e2d\uff0c\u7528\u6237\u5728\u6570\u636e\u63a2\u7d22\u5f00\u53d1 IDE \u7cfb\u7edf\u4e2d\u5b9a\u4e49\u7684\u4e00\u7cfb\u5217\u53d8\u91cf\u3001\u51fd\u6570\uff0c\u5230\u4e86\u6570\u636e\u53ef\u89c6\u5316\u7cfb\u7edf\u91cc\u5f80\u5f80\u53c8\u8981\u91cd\u65b0\u5b9a\u4e49\u4e00\u904d\uff1b IDE \u7cfb\u7edf\u8fd0\u884c\u751f\u6210\u7684\u6570\u636e\u6587\u4ef6\u4f4d\u7f6e\u548c\u540d\u79f0\uff0c\u4e0d\u80fd\u76f4\u63a5\u65b9\u4fbf\u7684\u4f20\u9012\u7ed9\u53ef\u89c6\u5316\u7cfb\u7edf\uff1b\u4f9d\u8d56\u7684\u7a0b\u5e8f\u5305\u4e5f\u9700\u8981\u4ece IDE \u7cfb\u7edf\u4e0b\u8f7d\u3001\u91cd\u65b0\u4e0a\u4f20\u5230\u53ef\u89c6\u5316\u7cfb\u7edf\uff1b\u5230\u4e86\u5de5\u4f5c\u6d41\u8c03\u5ea6\u7cfb\u7edf\uff0c\u8fd9\u4e2a\u8fc7\u7a0b\u8fd8\u8981\u518d\u91cd\u590d\u4e00\u904d\u3002\u4e0d\u540c\u4e0a\u5c42\u5e94\u7528\u95f4\uff0c\u8ba1\u7b97\u4efb\u52a1\u7684\u8fd0\u884c\u4f9d\u8d56\u7f3a\u4e4f\u4e92\u901a\u3001\u590d\u7528\u80fd\u529b\u3002

\n

\"\"

\n<center>\u56fe 7</center>\n

\u8ba1\u7b97\u6cbb\u7406\u4e4b\u7406\uff08 insight \uff09- \u7ec6\u5316\u7279\u6027\u95ee\u9898

\n

\u9664\u4e86\u4e0a\u8ff0\u7684\u67b6\u6784\u5c42\u9762\u95ee\u9898\uff0c\u8981\u60f3\u8ba9\u590d\u6742\u5206\u5e03\u5f0f\u67b6\u6784\u73af\u5883\u4e0b\uff0c\u5404\u79cd\u7c7b\u578b\u7684\u8ba1\u7b97\u4efb\u52a1\uff0c\u90fd\u80fd\u66f4\u7b80\u6d01\u3001\u7075\u6d3b\u3001\u6709\u5e8f\u3001\u53ef\u63a7\u7684\u63d0\u4ea4\u6267\u884c\uff0c\u548c\u6210\u529f\u8fd4\u56de\u7ed3\u679c\uff0c\u8ba1\u7b97\u6cbb\u7406\u8fd8\u9700\u5173\u6ce8\u9ad8\u5e76\u53d1\uff0c\u9ad8\u53ef\u7528\uff0c\u591a\u79df\u6237\u9694\u79bb\uff0c\u8d44\u6e90\u7ba1\u63a7\uff0c\u5b89\u5168\u589e\u5f3a\uff0c\u8ba1\u7b97\u7b56\u7565\u7b49\u7b49\u7ec6\u5316\u7279\u6027\u95ee\u9898\u3002\u8fd9\u4e9b\u95ee\u9898\u90fd\u6bd4\u8f83\u76f4\u767d\u6613\u61c2\uff0c\u8fd9\u91cc\u5c31\u4e0d\u4e00\u4e00\u5c55\u5f00\u8bba\u8ff0\u4e86\u3002

\n

\u57fa\u4e8e\u8ba1\u7b97\u4e2d\u95f4\u4ef6 Linkis \u7684\u8ba1\u7b97\u6cbb\u7406 - \u6cbb\u4e4b\u8def\uff08 Architecture \uff09

\n

Linkis \u67b6\u6784\u8bbe\u8ba1\u4ecb\u7ecd

\n

\u6838\u5fc3\u529f\u80fd\u6a21\u5757\u4e0e\u6d41\u7a0b

\n

\u8ba1\u7b97\u4e2d\u95f4\u4ef6 Linkis\uff0c\u662f\u5fae\u4f17\u94f6\u884c\u4e13\u95e8\u8bbe\u8ba1\u7528\u6765\u89e3\u51b3\u4e0a\u8ff0\u7d27\u8026\u5408\u3001\u91cd\u590d\u9020\u8f6e\u5b50\u3001\u6269\u5c55\u96be\u3001\u5e94\u7528\u5b64\u5c9b\u7b49\u8ba1\u7b97\u6cbb\u7406\u95ee\u9898\u7684\u3002\u5f53\u524d\u4e3b\u8981\u89e3\u51b3\u7684\u662f\u590d\u6742\u5206\u5e03\u5f0f\u67b6\u6784\u7684\u5178\u578b\u573a\u666f-\u6570\u636e\u5e73\u53f0\u73af\u5883\u4e0b\u7684\u8ba1\u7b97\u6cbb\u7406\u95ee\u9898\u3002

\n

Linkis \u4f5c\u4e3a\u8ba1\u7b97\u4e2d\u95f4\u4ef6\uff0c\u5728\u4e0a\u5c42\u5e94\u7528\u548c\u5e95\u5c42\u5f15\u64ce\u4e4b\u95f4\uff0c\u6784\u5efa\u4e86\u4e00\u5c42\u4e2d\u95f4\u5c42\u3002\u80fd\u591f\u5e2e\u52a9\u4e0a\u5c42\u5e94\u7528\uff0c\u901a\u8fc7\u5176\u5bf9\u5916\u63d0\u4f9b\u7684\u6807\u51c6\u5316\u63a5\u53e3\uff08\u5982 HTTP, JDBC, Java \u2026\uff09\uff0c\u5feb\u901f\u7684\u8fde\u63a5\u5230\u591a\u79cd\u5e95\u5c42\u8ba1\u7b97\u5b58\u50a8\u5f15\u64ce\uff08\u5982 Spark\u3001Hive\u3001TiSpark\u3001MySQL\u3001Python \u7b49\uff09\uff0c\u63d0\u4ea4\u6267\u884c\u5404\u79cd\u7c7b\u578b\u7684\u8ba1\u7b97\u4efb\u52a1\uff0c\u5e76\u5b9e\u73b0\u8de8\u4e0a\u5c42\u5e94\u7528\u95f4\u7684\u8ba1\u7b97\u4efb\u52a1\u8fd0\u884c\u65f6\u4e0a\u4e0b\u6587\u548c\u4f9d\u8d56\u7684\u4e92\u901a\u548c\u5171\u4eab\u3002\u4e14\u901a\u8fc7\u63d0\u4f9b\u591a\u79df\u6237\u3001\u9ad8\u5e76\u53d1\u3001\u4efb\u52a1\u5206\u53d1\u548c\u7ba1\u7406\u7b56\u7565\u3001\u8d44\u6e90\u7ba1\u63a7\u7b49\u7279\u6027\u652f\u6301\uff0c\u4f7f\u5f97\u5404\u79cd\u8ba1\u7b97\u4efb\u52a1\u66f4\u7075\u6d3b\u3001\u53ef\u9760\u3001\u53ef\u63a7\u7684\u63d0\u4ea4\u6267\u884c\uff0c\u6210\u529f\u8fd4\u56de\u7ed3\u679c\uff0c\u5927\u5927\u964d\u4f4e\u4e86\u4e0a\u5c42\u5e94\u7528\u5728\u8ba1\u7b97\u6cbb\u7406\u5c42\u7684\u5f00\u53d1\u548c\u8fd0\u7ef4\u6210\u672c\u3001\u4e0e\u6574\u4e2a\u73af\u5883\u7684\u67b6\u6784\u590d\u6742\u5ea6\uff0c\u586b\u8865\u4e86\u901a\u7528\u8ba1\u7b97\u6cbb\u7406\u8f6f\u4ef6\u7684\u7a7a\u767d\u3002\uff08\u56fe 8\u30019 \uff09

\n

\"\"

\n<center>\u56fe 8</center>\n

\"\"

\n<center>\u56fe 9</center>\n

\u8981\u66f4\u8be6\u7ec6\u7684\u4e86\u89e3\u8ba1\u7b97\u4efb\u52a1\u901a\u8fc7 Linkis \u7684\u63d0\u4ea4\u6267\u884c\u8fc7\u7a0b\uff0c\u6211\u4eec\u5148\u6765\u770b\u770b Linkis \u6838\u5fc3\u7684\u201c\u8ba1\u7b97\u6cbb\u7406\u670d\u52a1\u201d\u90e8\u5206\u7684\u5185\u90e8\u67b6\u6784\u548c\u6d41\u7a0b\u3002\u5982\u4e0b\u56fe 10\uff1a

\n

\"\"

\n<center>\u56fe 10</center>\n

\u8ba1\u7b97\u6cbb\u7406\u670d\u52a1\uff1a\u8ba1\u7b97\u4e2d\u95f4\u4ef6\u7684\u6838\u5fc3\u8ba1\u7b97\u6846\u67b6\uff0c\u4e3b\u8981\u8d1f\u8d23\u4f5c\u4e1a\u8c03\u5ea6\u548c\u751f\u547d\u5468\u671f\u7ba1\u7406\u3001\u8ba1\u7b97\u8d44\u6e90\u7ba1\u7406\uff0c\u4ee5\u53ca\u5f15\u64ce\u8fde\u63a5\u5668\u7684\u751f\u547d\u5468\u671f\u7ba1\u7406\u3002

\n

\u516c\u5171\u589e\u5f3a\u670d\u52a1\uff1a\u901a\u7528\u516c\u5171\u670d\u52a1\uff0c\u63d0\u4f9b\u57fa\u7840\u516c\u5171\u529f\u80fd\uff0c\u53ef\u670d\u52a1\u4e8e Linkis \u5404\u79cd\u670d\u52a1\u53ca\u4e0a\u5c42\u5e94\u7528\u7cfb\u7edf\u3002

\n

\u5176\u4e2d\u8ba1\u7b97\u6cbb\u7406\u670d\u52a1\u7684\u4e3b\u8981\u6a21\u5757\u5982\u4e0b\uff1a

\n\n

\u5982\u56fe 10 \u6240\u793a\uff0c\u4e00\u4e2a\u4f5c\u4e1a\u7684\u63d0\u4ea4\u6267\u884c\u4e3b\u8981\u5206\u4e3a\u4ee5\u4e0b 11 \u6b65\uff1a

\n

1. \u4e0a\u5c42\u5e94\u7528\u5411\u8ba1\u7b97\u4e2d\u95f4\u4ef6\u63d0\u4ea4\u4f5c\u4e1a\uff0c\u5fae\u670d\u52a1\u7f51\u5173 SpringCloud Gateway \u63a5\u6536\u4f5c\u4e1a\u5e76\u8f6c\u53d1\u7ed9 Entrance\u3002

\n

2. Entrance \u6d88\u8d39\u4f5c\u4e1a\uff0c\u4e3a\u4f5c\u4e1a\u5411 AppManager \u7533\u8bf7\u53ef\u7528 EngineConn\u3002

\n

3. \u5982\u679c\u4e0d\u5b58\u5728\u53ef\u590d\u7528\u7684 Engine\uff0cAppManager \u5c1d\u8bd5\u5411 ResourceManager \u7533\u8bf7\u8d44\u6e90\uff0c\u4e3a\u4f5c\u4e1a\u542f\u52a8\u4e00\u4e2a\u65b0 EngineConn\u3002

\n
    \n
  1. \u7533\u8bf7\u5230\u8d44\u6e90\uff0c\u8981\u6c42 EngineConnManager \u4f9d\u7167\u8d44\u6e90\u542f\u52a8\u65b0 EngineConn
    2. \n
\n

5. EngineConnManager \u542f\u52a8\u65b0 EngineConn\uff0c\u5e76\u4e3b\u52a8\u56de\u63a8\u65b0 EngineConn \u4fe1\u606f\u3002

\n

6. AppManager \u5c06\u65b0 EngineConn \u5206\u914d\u7ed9 Entrance\uff0cEntrance \u5c06 EngineConn \u5206\u914d\u7ed9\u7528\u6237\u4f5c\u4e1a\uff0c\u4f5c\u4e1a\u5f00\u59cb\u6267\u884c\uff0c\u5c06\u8ba1\u7b97\u4efb\u52a1\u63d0\u4ea4\u7ed9 EngineConn\u3002

\n

7. EngineConn \u5c06\u8ba1\u7b97\u4efb\u52a1\u63d0\u4ea4\u7ed9\u5e95\u5c42\u8ba1\u7b97\u5f15\u64ce\u3002

\n

8. EngineConn \u5b9e\u65f6\u76d1\u542c\u5e95\u5c42\u5f15\u64ce\u6267\u884c\u60c5\u51b5\uff0c\u56de\u63a8\u76f8\u5173\u65e5\u5fd7\u3001\u8fdb\u5ea6\u548c\u72b6\u6001\u7ed9 Entrance\uff0cEntrance \u901a\u8fc7 WebSocket\uff0c\u4e3b\u52a8\u56de\u63a8 EngineConn \u4f20\u8fc7\u6765\u7684\u65e5\u5fd7\u3001\u8fdb\u5ea6\u548c\u72b6\u6001\u7ed9\u4e0a\u5c42\u5e94\u7528\u7cfb\u7edf\u3002

\n

9. EngineConn \u6267\u884c\u5b8c\u6210\u540e\uff0c\u56de\u63a8\u8ba1\u7b97\u4efb\u52a1\u7684\u72b6\u6001\u548c\u7ed3\u679c\u96c6\u4fe1\u606f\uff0cEntrance \u5c06\u4f5c\u4e1a\u548c\u7ed3\u679c\u96c6\u4fe1\u606f\u66f4\u65b0\u5230 JobHistory\uff0c\u5e76\u901a\u77e5\u4e0a\u5c42\u5e94\u7528\u7cfb\u7edf\u3002

\n

10. \u4e0a\u5c42\u5e94\u7528\u7cfb\u7edf\u8bbf\u95ee JobHistory\uff0c\u62ff\u5230\u4f5c\u4e1a\u548c\u7ed3\u679c\u96c6\u4fe1\u606f\u3002

\n

11. \u4e0a\u5c42\u5e94\u7528\u7cfb\u7edf\u8bbf\u95ee Storage\uff0c\u8bf7\u6c42\u4f5c\u4e1a\u7ed3\u679c\u96c6\u3002

\n

\u8ba1\u7b97\u4efb\u52a1\u7ba1\u7406\u7b56\u7565\u652f\u6301

\n

\u5728\u590d\u6742\u5206\u5e03\u5f0f\u73af\u5883\u4e0b\uff0c\u4e00\u4e2a\u8ba1\u7b97\u4efb\u52a1\u5f80\u5f80\u4e0d\u5355\u4f1a\u662f\u7b80\u5355\u7684\u63d0\u4ea4\u6267\u884c\u548c\u8fd4\u56de\u7ed3\u679c\uff0c\u8fd8\u53ef\u80fd\u9700\u8981\u9762\u5bf9\u63d0\u4ea4\u5931\u8d25\u3001\u6267\u884c\u5931\u8d25\u3001hang \u4f4f\u7b49\u95ee\u9898\uff0c\u4e14\u5728\u5927\u91cf\u5e76\u53d1\u573a\u666f\u4e0b\u8fd8\u9700\u901a\u8fc7\u8ba1\u7b97\u4efb\u52a1\u7684\u8c03\u5ea6\u5206\u53d1\uff0c\u89e3\u51b3\u79df\u6237\u95f4\u4e92\u76f8\u5f71\u54cd\u3001\u8d1f\u8f7d\u5747\u8861\u7b49\u95ee\u9898\u3002

\n

Linkis \u901a\u8fc7\u5bf9\u8ba1\u7b97\u4efb\u52a1\u7684\u6807\u7b7e\u5316\uff0c\u5b9e\u73b0\u4e86\u5728\u4efb\u52a1\u8c03\u5ea6\u3001\u5206\u53d1\u3001\u8def\u7531\u7b49\u65b9\u9762\u8ba1\u7b97\u4efb\u52a1\u7ba1\u7406\u7b56\u7565\u7684\u652f\u6301\uff0c\u5e76\u53ef\u6309\u9700\u914d\u7f6e\u8d85\u65f6\u3001\u81ea\u52a8\u91cd\u8bd5\uff0c\u53ca\u7070\u5ea6\u3001\u591a\u6d3b\u7b49\u7b56\u7565\u652f\u6301\u3002\u5982\u4e0b\u56fe 11\u3002

\n

\"\"

\n<center>\u56fe 11</center>\n

\u57fa\u4e8e Spring Cloud \u5fae\u670d\u52a1\u6846\u67b6

\n

\u8bf4\u5b8c\u4e86\u4e1a\u52a1\u67b6\u6784\uff0c\u6211\u4eec\u73b0\u5728\u6765\u804a\u804a\u6280\u672f\u67b6\u6784\u3002\u5728\u8ba1\u7b97\u6cbb\u7406\u5c42\u73af\u5883\u4e0b\uff0c\u5f88\u591a\u7c7b\u578b\u7684\u8ba1\u7b97\u4efb\u52a1\u5177\u6709\u751f\u547d\u5468\u671f\u8f83\u77ed\u7684\u7279\u5f81\uff0c\u5982\u4e00\u4e2a Spark job \u53ef\u80fd\u51e0\u5341\u79d2\u5230\u51e0\u5206\u949f\u5c31\u6267\u884c\u5b8c\uff0cEngineConn \uff08 EnginConnector \uff09\u4f1a\u662f\u5927\u91cf\u52a8\u6001\u542f\u505c\u7684\u72b6\u6001\u3002\u524d\u7aef\u7528\u6237\u548c Linkis \u4e2d\u5176\u4ed6\u7ba1\u7406\u89d2\u8272\u7684\u670d\u52a1\uff0c\u9700\u8981\u80fd\u591f\u53ca\u65f6\u52a8\u6001\u53d1\u73b0\u76f8\u5173\u670d\u52a1\u5b9e\u4f8b\u7684\u72b6\u6001\u53d8\u5316\uff0c\u5e76\u83b7\u53d6\u6700\u65b0\u7684\u670d\u52a1\u5b9e\u4f8b\u8bbf\u95ee\u5730\u5740\u4fe1\u606f\u3002\u540c\u65f6\u9700\u8981\u8003\u8651\uff0c\u5404\u6a21\u5757\u95f4\u7684\u901a\u4fe1\u3001\u8def\u7531\u3001\u534f\u8c03\uff0c\u53ca\u5404\u6a21\u5757\u7684\u6a2a\u5411\u6269\u5c55\u3001\u8d1f\u8f7d\u5747\u8861\u3001\u9ad8\u53ef\u7528\u7b49\u80fd\u529b\u3002

\n

\u57fa\u4e8e\u4ee5\u4e0a\u9700\u6c42\uff0cLinkis \u5b9e\u9645\u662f\u57fa\u4e8e Spring Cloud \u5fae\u670d\u52a1\u6846\u67b6\u6280\u672f\uff0c\u5c06\u4e0a\u8ff0\u7684\u6bcf\u4e00\u4e2a\u6a21\u5757 /\u89d2\u8272\uff0c\u90fd\u5c01\u88c5\u6210\u4e86\u4e00\u4e2a\u5fae\u670d\u52a1\uff0c\u6784\u5efa\u4e86\u591a\u4e2a\u5fae\u670d\u52a1\u7ec4\uff0c\u6574\u5408\u5f62\u6210\u4e86 Linkis \u7684\u5b8c\u6574\u8ba1\u7b97\u4e2d\u95f4\u4ef6\u80fd\u529b\u3002\u5982\u4e0b\u56fe 12\uff1a

\n

\"\"

\n<center>\u56fe 12</center>\n

\u4ece\u591a\u79df\u6237\u7ba1\u7406\u89d2\u5ea6\uff0c\u4e0a\u8ff0\u670d\u52a1\u53ef\u533a\u5206\u4e3a\u79df\u6237\u76f8\u5173\u670d\u52a1\uff0c\u548c\u79df\u6237\u65e0\u5173\u670d\u52a1\u4e24\u79cd\u7c7b\u578b\u3002\u79df\u6237\u76f8\u5173\u670d\u52a1\uff0c\u662f\u6307\u4e00\u4e9b\u4efb\u52a1\u903b\u8f91\u5904\u7406\u8d1f\u8377\u91cd\u3001\u8d44\u6e90\u6d88\u8017\u9ad8\uff0c\u6216\u9700\u8981\u6839\u636e\u5177\u4f53\u79df\u6237\u3001\u7528\u6237\u3001\u7269\u7406\u673a\u5668\u7b49\uff0c\u505a\u9694\u79bb\u5212\u5206\u3001\u907f\u514d\u76f8\u4e92\u5f71\u54cd\u7684\u670d\u52a1\uff0c\u5982 Entrance\u3001EnginConn \uff08 EnginConnector \uff09 Manager\u3001EnginConn \uff1b\u5176\u4ed6\u5982 App Manger\u3001Resource Manager\u3001Context Service \u7b49\u670d\u52a1\uff0c\u90fd\u662f\u79df\u6237\u65e0\u5173\u7684\u3002

\n

Eureka \u627f\u62c5\u4e86\u5fae\u670d\u52a1\u52a8\u6001\u6ce8\u518c\u4e0e\u53d1\u73b0\u4e2d\u5fc3\uff0c\u53ca\u6240\u6709\u79df\u6237\u65e0\u5173\u670d\u52a1\u7684\u8d1f\u8f7d\u5747\u8861\u3001\u6545\u969c\u8f6c\u79fb\u529f\u80fd\u3002

\n

Eureka \u6709\u4e2a\u5c40\u9650\uff0c\u5c31\u662f\u5728\u5176\u5ba2\u6237\u7aef\uff0c\u5bf9\u540e\u7aef\u5fae\u670d\u52a1\u5b9e\u4f8b\u7684\u53d1\u73b0\u4e0e\u72b6\u6001\u5237\u65b0\u673a\u5236\uff0c\u662f\u5ba2\u6237\u7aef\u4e3b\u52a8\u8f6e\u8be2\u5237\u65b0\uff0c\u6700\u5feb\u53ef\u8bbe 1 \u79d2 1 \u6b21\uff08\u5b9e\u9645\u8981\u51e0\u79d2\u624d\u80fd\u5b8c\u6210\u5237\u65b0\uff09\u3002\u8fd9\u6837\u5728 Linkis \u8fd9\u79cd\u9700\u8981\u5feb\u901f\u5237\u65b0\u5927\u91cf\u540e\u7aef EnginConn \u7b49\u670d\u52a1\u7684\u72b6\u6001\u7684\u573a\u666f\u4e0b\uff0c\u65f6\u6548\u5f97\u4e0d\u5230\u6ee1\u8db3\uff0c\u4e14\u5b9a\u65f6\u8f6e\u8be2\u5237\u65b0\u5bf9 Eureka server\u3001\u5bf9\u540e\u7aef\u5fae\u670d\u52a1\u5b9e\u4f8b\u7684\u6210\u672c\u90fd\u5f88\u9ad8\u3002

\n

\u4e3a\u6b64\u6211\u4eec\u5bf9 Spring Cloud Ribbon \u505a\u4e86\u6539\u9020\uff0c\u5728\u5176\u4e2d\u5c01\u88c5\u4e86 Eureka client \u7684\u5fae\u670d\u52a1\u5b9e\u4f8b\u72b6\u6001\u5237\u65b0\u65b9\u6cd5\uff0c\u5e76\u628a\u5b83\u505a\u6210\u6ee1\u8db3\u6761\u4ef6\u4e3b\u52a8\u8bf7\u6c42\u5237\u65b0\uff0c\u800c\u4e0d\u4f1a\u518d\u9891\u7e41\u7684\u5b9a\u671f\u8f6e\u8be2\u3002\u4ece\u800c\u5728\u6ee1\u8db3\u65f6\u6548\u7684\u540c\u65f6\uff0c\u5927\u5927\u964d\u4f4e\u4e86\u72b6\u6001\u83b7\u53d6\u7684\u6210\u672c\u3002\u5982\u4e0b\u56fe 13\uff1a

\n

\"\"

\n<center>\u56fe 13</center>\n

Spring Cloud Gateway \u627f\u62c5\u4e86\u5916\u90e8\u8bf7\u6c42 Linkis \u7684\u5165\u53e3\u7f51\u5173\u7684\u89d2\u8272\uff0c\u5e2e\u52a9\u5728\u670d\u52a1\u5b9e\u4f8b\u4e0d\u65ad\u53d1\u751f\u53d8\u5316\u7684\u60c5\u51b5\u4e0b\uff0c\u7b80\u5316\u524d\u7aef\u7528\u6237\u7684\u8c03\u7528\u903b\u8f91\uff0c\u5feb\u901f\u65b9\u4fbf\u7684\u83b7\u53d6\u6700\u65b0\u7684\u670d\u52a1\u5b9e\u4f8b\u8bbf\u95ee\u5730\u5740\u4fe1\u606f\u3002

\n

Spring Cloud Gateway \u6709\u4e2a\u5c40\u9650\uff0c\u5c31\u662f\u4e00\u4e2a WebSocket \u5ba2\u6237\u7aef\u53ea\u80fd\u5c06\u8bf7\u6c42\u8f6c\u53d1\u7ed9\u4e00\u4e2a\u7279\u5b9a\u7684\u540e\u53f0\u670d\u52a1\uff0c\u65e0\u6cd5\u5b8c\u6210\u4e00\u4e2a WebSocket \u5ba2\u6237\u7aef\u901a\u8fc7\u7f51\u5173 API \u5bf9\u63a5\u540e\u53f0\u591a\u4e2a WebSocket \u5fae\u670d\u52a1\uff0c\u800c\u8fd9\u5728\u6211\u4eec\u7684 Entrance HA \u7b49\u573a\u666f\u9700\u8981\u7528\u5230\u3002

\n

\u4e3a\u6b64 Linkis \u5bf9 Spring Cloud Gateway \u505a\u4e86\u76f8\u5e94\u6539\u9020\uff0c\u5728 Gateway \u4e2d\u5b9e\u73b0\u4e86 WebSocket \u8def\u7531\u8f6c\u53d1\u5668\uff0c\u7528\u4e8e\u4e0e\u5ba2\u6237\u7aef\u5efa\u7acb WebSocket \u8fde\u63a5\u3002\u5efa\u7acb\u8fde\u63a5\u6210\u529f\u540e\uff0c\u4f1a\u81ea\u52a8\u5206\u6790\u5ba2\u6237\u7aef\u7684 WebSocket \u8bf7\u6c42\uff0c\u901a\u8fc7\u89c4\u5219\u5224\u65ad\u51fa\u8bf7\u6c42\u8be5\u8f6c\u53d1\u7ed9\u54ea\u4e2a\u540e\u7aef\u5fae\u670d\u52a1\uff0c\u7136\u540e\u5c06 WebSocket \u8bf7\u6c42\u8f6c\u53d1\u7ed9\u5bf9\u5e94\u7684\u540e\u7aef\u5fae\u670d\u52a1\u5b9e\u4f8b\u3002\u8be6\u89c1 Github \u4e0a Linkis \u7684 Wiki \u4e2d\uff0c\u201cGateway \u7684\u591a WebSocket \u8bf7\u6c42\u8f6c\u53d1\u5b9e\u73b0\u201d\u4e00\u6587\u3002

\n

\"\"

\n<center>\u56fe 14</center>\n

**Spring Cloud OpenFeign ** \u63d0\u4f9b\u7684 HTTP \u8bf7\u6c42\u8c03\u7528\u63a5\u53e3\u5316\u3001\u89e3\u6790\u6a21\u677f\u5316\u80fd\u529b\uff0c\u5e2e\u52a9 Linkis \u6784\u5efa\u4e86\u5e95\u5c42 RPC \u901a\u4fe1\u6846\u67b6\u3002\u4f46\u57fa\u4e8e Feign \u7684\u5fae\u670d\u52a1\u4e4b\u95f4 HTTP \u63a5\u53e3\u7684\u8c03\u7528\uff0c\u53ea\u80fd\u6ee1\u8db3\u7b80\u5355\u7684 A \u5fae\u670d\u52a1\u5b9e\u4f8b\u6839\u636e\u7b80\u5355\u7684\u89c4\u5219\u968f\u673a\u9009\u62e9 B \u5fae\u670d\u52a1\u4e4b\u4e2d\u7684\u67d0\u4e2a\u670d\u52a1\u5b9e\u4f8b\uff0c\u800c\u8fd9\u4e2a B \u5fae\u670d\u52a1\u5b9e\u4f8b\u5982\u679c\u60f3\u5f02\u6b65\u56de\u4f20\u4fe1\u606f\u7ed9\u8c03\u7528\u65b9\uff0c\u662f\u65e0\u6cd5\u5b9e\u73b0\u7684\u3002\u540c\u65f6\uff0c\u7531\u4e8e Feign \u53ea\u652f\u6301\u7b80\u5355\u7684\u670d\u52a1\u9009\u53d6\u89c4\u5219\uff0c\u65e0\u6cd5\u505a\u5230\u5c06\u8bf7\u6c42\u8f6c\u53d1\u7ed9\u6307\u5b9a\u7684\u5fae\u670d\u52a1\u5b9e\u4f8b\uff0c\u65e0\u6cd5\u505a\u5230\u5c06\u4e00\u4e2a\u8bf7\u6c42\u5e7f\u64ad\u7ed9\u63a5\u6536\u65b9\u5fae\u670d\u52a1\u7684\u6240\u6709\u5b9e\u4f8b\u3002Linkis \u57fa\u4e8e Feign \u5b9e\u73b0\u4e86\u4e00\u5957\u81ea\u5df1\u7684\u5e95\u5c42 RPC \u901a\u4fe1\u65b9\u6848\uff0c\u96c6\u6210\u5230\u4e86\u6240\u6709 Linkis \u7684\u5fae\u670d\u52a1\u4e4b\u4e2d\u3002\u4e00\u4e2a\u5fae\u670d\u52a1\u65e2\u53ef\u4ee5\u4f5c\u4e3a\u8bf7\u6c42\u8c03\u7528\u65b9\uff0c\u4e5f\u53ef\u4ee5\u4f5c\u4e3a\u8bf7\u6c42\u63a5\u6536\u65b9\u3002\u4f5c\u4e3a\u8bf7\u6c42\u8c03\u7528\u65b9\u65f6\uff0c\u5c06\u901a\u8fc7 Sender \u8bf7\u6c42\u76ee\u6807\u63a5\u6536\u65b9\u5fae\u670d\u52a1\u7684 Receiver \uff1b\u4f5c\u4e3a\u8bf7\u6c42\u63a5\u6536\u65b9\u65f6\uff0c\u5c06\u63d0\u4f9b Receiver \u7528\u6765\u5904\u7406\u8bf7\u6c42\u63a5\u6536\u65b9 Sender \u53d1\u9001\u8fc7\u6765\u7684\u8bf7\u6c42\uff0c\u4ee5\u4fbf\u5b8c\u6210\u540c\u6b65\u54cd\u5e94\u6216\u5f02\u6b65\u54cd\u5e94\u3002\u5982\u4e0b\u56fe\u793a\u610f\u3002\u8be6\u89c1 GitHub \u4e0a Linkis \u7684 Wiki \u4e2d\uff0c\u201cLinkis RPC \u67b6\u6784\u4ecb\u7ecd\u201d\u4e00\u6587\u3002

\n

\"\"

\n<center>\u56fe 15</center>\n

\u81f3\u6b64\uff0cLinkis \u5bf9\u4e0a\u5c42\u5e94\u7528\u548c\u5e95\u5c42\u5f15\u64ce\u7684\u89e3\u8026\u539f\u7406\uff0c\u5176\u6838\u5fc3\u67b6\u6784\u4e0e\u6d41\u7a0b\u8bbe\u8ba1\uff0c\u53ca\u57fa\u4e8e Spring Cloud \u5fae\u670d\u52a1\u6846\u67b6\u5b9e\u73b0\u7684\uff0c\u5404\u6a21\u5757\u5fae\u670d\u52a1\u5316\u52a8\u6001\u7ba1\u7406\u3001\u901a\u4fe1\u8def\u7531\u3001\u6a2a\u5411\u6269\u5c55\u80fd\u529b\u4ecb\u7ecd\u5b8c\u6bd5\u3002

\n

\u89e3\u8026\uff1aLinkis \u5982\u4f55\u89e3\u8026\u4e0a\u5c42\u5e94\u7528\u548c\u5e95\u5c42\u5f15\u64ce

\n

Linkis \u4f5c\u4e3a\u8ba1\u7b97\u4e2d\u95f4\u4ef6\uff0c\u5728\u4e0a\u5c42\u5e94\u7528\u548c\u5e95\u5c42\u5f15\u64ce\u4e4b\u95f4\uff0c\u6784\u5efa\u4e86\u4e00\u5c42\u4e2d\u95f4\u5c42\u3002\u4e0a\u5c42\u5e94\u7528\u6240\u6709\u8ba1\u7b97\u4efb\u52a1\uff0c\u5148\u901a\u8fc7 HTTP\u3001WebSocket\u3001Java \u7b49\u63a5\u53e3\u65b9\u5f0f\u63d0\u4ea4\u7ed9 Linkis\uff0c\u518d\u7531 Linkis \u8f6c\u4ea4\u7ed9\u5e95\u5c42\u5f15\u64ce\u3002\u539f\u6709\u7684\u4e0a\u5c42\u5e94\u7528\u4ee5 CS \u6a21\u5f0f\u76f4\u8fde\u5e95\u5c42\u5f15\u64ce\u7684\u7d27\u8026\u5408\u5f97\u4ee5\u89e3\u9664\uff0c\u56e0\u6b64\u5b9e\u73b0\u4e86\u89e3\u8026\u3002\u5982\u4e0b\u56fe 16 \u6240\u793a\uff1a

\n

\"\"

\n<center>\u56fe 16</center>\n

\u901a\u8fc7\u89e3\u8026\uff0c\u5e95\u5c42\u5f15\u64ce\u7684\u53d8\u52a8\u6709\u4e86 Linkis \u8fd9\u5c42\u4e2d\u95f4\u4ef6\u7f13\u51b2\uff0c\u5982\u5f15\u64ce client \u7684\u7248\u672c\u5347\u7ea7\uff0c\u65e0\u9700\u518d\u5bf9\u6bcf\u4e00\u4e2a\u5bf9\u63a5\u7684\u4e0a\u5c42\u5e94\u7528\u505a\u9010\u4e2a\u6539\u52a8\uff0c\u53ef\u5728 Linkis \u5c42\u7edf\u4e00\u5b8c\u6210\u3002\u5e76\u80fd\u5728 Linkis \u5c42\uff0c\u5b9e\u73b0\u5bf9\u4e0a\u5c42\u5e94\u7528\u66f4\u52a0\u900f\u660e\u548c\u53cb\u597d\u7684\u5347\u7ea7\u7b56\u7565\uff0c\u5982\u7070\u5ea6\u5207\u6362\u3001\u591a\u6d3b\u7b49\u7b56\u7565\u652f\u6301\u3002\u4e14\u5373\u4f7f\u540e\u7ee7\u63a5\u5165\u66f4\u591a\u4e0a\u5c42\u5e94\u7528\u548c\u5e95\u5c42\u5f15\u64ce\uff0c\u6574\u4e2a\u73af\u5883\u590d\u6742\u5ea6\u4e5f\u4e0d\u4f1a\u6709\u5927\u7684\u53d8\u5316\uff0c\u5927\u5927\u964d\u4f4e\u4e86\u5f00\u53d1\u8fd0\u7ef4\u5de5\u4f5c\u8d1f\u62c5\u3002

\n

\u590d\u7528\uff1a\u5bf9\u4e8e\u4e0a\u5c42\u5e94\u7528\uff0cLinkis \u5982\u4f55\u51dd\u7ec3\u8ba1\u7b97\u6cbb\u7406\u6a21\u5757\u4f9b\u590d\u7528\uff0c\u907f\u514d\u91cd\u590d\u5f00\u53d1

\n

\u4e0a\u5c42\u5e94\u7528\u590d\u7528 Linkis \u793a\u4f8b\uff08 Scriptis \uff09

\n

\u6709\u4e86 Linkis\uff0c\u4e0a\u5c42\u5e94\u7528\u53ef\u4ee5\u57fa\u4e8e Linkis\uff0c\u5feb\u901f\u5b9e\u73b0\u5bf9\u591a\u79cd\u540e\u53f0\u8ba1\u7b97\u5b58\u50a8\u5f15\u64ce\u7684\u5bf9\u63a5\u652f\u6301\uff0c\u53ca\u53d8\u91cf\u3001\u51fd\u6570\u7b49\u81ea\u5b9a\u4e49\u4e0e\u7ba1\u7406\u3001\u8d44\u6e90\u7ba1\u63a7\u3001\u591a\u79df\u6237\u3001\u667a\u80fd\u8bca\u65ad\u7b49\u8ba1\u7b97\u6cbb\u7406\u7279\u6027\u3002

\n

\u4f18\u70b9

\n

\u4ee5\u5fae\u4f17\u94f6\u884c\u4e0e Linkis \u540c\u65f6\u5f00\u6e90\u7684\uff0c\u4ea4\u4e92\u5f0f\u6570\u636e\u5f00\u53d1\u63a2\u7d22\u5de5\u5177 Scriptis \u4e3a\u4f8b\uff0cScriptis \u7684\u5f00\u53d1\u4eba\u5458\u53ea\u9700\u5173\u6ce8 Web UI\u3001\u591a\u79cd\u6570\u636e\u5f00\u53d1\u8bed\u8a00\u652f\u6301\u3001\u811a\u672c\u7f16\u8f91\u529f\u80fd\u7b49\u7eaf\u524d\u7aef\u529f\u80fd\u5b9e\u73b0\uff0cLinkis \u5305\u529e\u4e86\u5176\u4ece\u5b58\u50a8\u8bfb\u5199\u3001\u8ba1\u7b97\u4efb\u52a1\u63d0\u4ea4\u6267\u884c\u3001\u4f5c\u4e1a\u72b6\u6001\u65e5\u5fd7\u66f4\u65b0\u3001\u8d44\u6e90\u7ba1\u63a7\u7b49\u7b49\u51e0\u4e4e\u6240\u6709\u540e\u53f0\u529f\u80fd\u3002\u57fa\u4e8e Linkis \u7684\u5927\u91cf\u8ba1\u7b97\u6cbb\u7406\u5c42\u80fd\u529b\u7684\u590d\u7528\uff0c\u5927\u5927\u964d\u4f4e\u4e86 Scriptis \u9879\u76ee\u7684\u5f00\u53d1\u6210\u672c\uff0c\u4f7f\u5f97 Scritpis \u76ee\u524d\u53ea\u9700\u8981\u6709\u9650\u7684\u524d\u7aef\u4eba\u5458\uff0c\u5373\u53ef\u5b8c\u6210\u7ef4\u62a4\u548c\u7248\u672c\u8fed\u4ee3\u5de5\u4f5c\u3002

\n

\u5982\u4e0b\u56fe 17\uff0cScriptis \u9879\u76ee 99.5% \u7684\u4ee3\u7801\uff0c\u90fd\u662f\u524d\u7aef\u7684 JS\u3001CSS \u4ee3\u7801\u3002\u540e\u53f0\u57fa\u672c\u5b8c\u5168\u590d\u7528 Linkis\u3002

\n

\"\"

\n<center>\u56fe 17</center>\n

\u5feb\u901f\u6269\u5c55\uff1a\u5bf9\u4e8e\u5e95\u5c42\u5f15\u64ce\uff0cLinkis \u5982\u4f55\u4ee5\u5f88\u5c0f\u7684\u5f00\u53d1\u91cf\uff0c\u5b9e\u73b0\u65b0\u5e95\u5c42\u5f15\u64ce\u5feb\u901f\u5bf9\u63a5

\n

\u6a21\u5757\u5316\u53ef\u63d2\u62d4\u7684\u8ba1\u7b97\u5f15\u64ce\u63a5\u5165\u8bbe\u8ba1\uff0c\u65b0\u5f15\u64ce\u63a5\u5165\u7b80\u5355\u5feb\u901f\u3002

\n

\u5bf9\u4e8e\u5178\u578b\u4ea4\u4e92\u5f0f\u6a21\u5f0f\u8ba1\u7b97\u5f15\u64ce\uff08\u63d0\u4ea4\u4efb\u52a1\uff0c\u6267\u884c\uff0c\u8fd4\u56de\u7ed3\u679c\uff09\uff0c\u7528\u6237\u53ea\u9700\u8981 buildApplication \u548c executeLine \u8fd9 2 \u4e2a\u65b9\u6cd5\uff0c\u5c31\u53ef\u4ee5\u5b8c\u6210\u4e00\u4e2a\u65b0\u7684\u8ba1\u7b97\u5f15\u64ce\u63a5\u5165 Linkis\uff0c\u4ee3\u7801\u91cf\u6781\u5c11\u3002\u793a\u4f8b\u5982\u4e0b\u3002

\n

(1) AppManager \u90e8\u5206\uff1a\u7528\u6237\u5fc5\u987b\u5b9e\u73b0\u7684\u63a5\u53e3\u662f ApplicationBuilder\uff0c\u7528\u6765\u5c01\u88c5\u65b0\u5f15\u64ce\u8fde\u63a5\u5668\u5b9e\u4f8b\u542f\u52a8\u547d\u4ee4\u3002

\n
1. //\u7528\u6237\u5fc5\u987b\u5b9e\u73b0\u7684\u65b9\u6cd5: \u7528\u4e8e\u5c01\u88c5\u65b0\u5f15\u64ce\u8fde\u63a5\u5668\u5b9e\u4f8b\u542f\u52a8\u547d\u4ee4\n2. def buildApplication(protocol:Protocol):ApplicationRequest\n
\n

(2) EngineConn \u90e8\u5206\uff1a\u7528\u6237\u53ea\u9700\u5b9e\u73b0 executeLine \u65b9\u6cd5\uff0c\u5411\u65b0\u5f15\u64ce\u63d0\u4ea4\u6267\u884c\u8ba1\u7b97\u4efb\u52a1\uff1a

\n
1. //\u7528\u6237\u5fc5\u987b\u5b9e\u73b0\u7684\u65b9\u6cd5\uff1a\u7528\u4e8e\u8c03\u7528\u5e95\u5c42\u5f15\u64ce\u63d0\u4ea4\u6267\u884c\u8ba1\u7b97\u4efb\u52a1\n2. def executeLine(context: EngineConnContext,code: String): ExecuteResponse\n
\n

\u5f15\u64ce\u76f8\u5173\u5176\u4ed6\u529f\u80fd /\u65b9\u6cd5\u90fd\u5df2\u6709\u9ed8\u8ba4\u5b9e\u73b0\uff0c\u65e0\u5b9a\u5236\u5316\u9700\u6c42\u53ef\u76f4\u63a5\u590d\u7528\u3002

\n

\u8fde\u901a\uff1aLinkis \u5982\u4f55\u6253\u901a\u5e94\u7528\u5b64\u5c9b

\n

\u901a\u8fc7 Linkis \u63d0\u4f9b\u7684\u4e0a\u4e0b\u6587\u670d\u52a1\uff0c\u548c\u5b58\u50a8\u3001\u7269\u6599\u5e93\u670d\u52a1\uff0c\u63a5\u5165\u7684\u591a\u4e2a\u4e0a\u5c42\u5e94\u7528\u4e4b\u95f4\uff0c\u53ef\u8f7b\u677e\u5b9e\u73b0\u73af\u5883\u53d8\u91cf\u3001\u51fd\u6570\u3001\u7a0b\u5e8f\u5305\u3001\u6570\u636e\u6587\u4ef6\u7b49\uff0c\u76f8\u5173\u4fe1\u606f\u548c\u8d44\u6e90\u7684\u5171\u4eab\u548c\u590d\u7528\uff0c\u6253\u901a\u5e94\u7528\u5b64\u5c9b\u3002

\n

\"\"

\n<center>\u56fe 18</center>\n

Context Service \u4e0a\u4e0b\u6587\u670d\u52a1\u4ecb\u7ecd

\n

Context Service \uff08 CS \uff09\u4e3a\u4e0d\u540c\u4e0a\u5c42\u5e94\u7528\u7cfb\u7edf\uff0c\u4e0d\u540c\u8ba1\u7b97\u4efb\u52a1\uff0c\u63d0\u4f9b\u4e86\u7edf\u4e00\u7684\u4e0a\u4e0b\u6587\u7ba1\u7406\u670d\u52a1\uff0c\u53ef\u5b9e\u73b0\u4e0a\u4e0b\u6587\u7684\u81ea\u5b9a\u4e49\u548c\u5171\u4eab\u3002\u5728 Linkis \u4e2d\uff0cCS \u9700\u8981\u7ba1\u7406\u7684\u4e0a\u4e0b\u6587\u5185\u5bb9\uff0c\u53ef\u5206\u4e3a\u5143\u6570\u636e\u4e0a\u4e0b\u6587\u3001\u6570\u636e\u4e0a\u4e0b\u6587\u548c\u8d44\u6e90\u4e0a\u4e0b\u6587 3 \u90e8\u5206\u3002

\n

\"\"

\n<center>\u56fe 19</center>\n

\u5143\u6570\u636e\u4e0a\u4e0b\u6587\uff0c\u5b9a\u4e49\u4e86\u8ba1\u7b97\u4efb\u52a1\u4e2d\u5e95\u5c42\u5f15\u64ce\u5143\u6570\u636e\u7684\u8bbf\u95ee\u548c\u4f7f\u7528\u89c4\u8303\uff0c\u4e3b\u8981\u529f\u80fd\u5982\u4e0b\uff1a

\n\n

\u57fa\u4e8e\u8ba1\u7b97\u4e2d\u95f4\u4ef6 Linkis \u7684\u8ba1\u7b97\u6cbb\u7406 - \u7406\u4e4b\u8def\uff08 Insight \uff09

\n

Linkis \u8ba1\u7b97\u6cbb\u7406\u7ec6\u5316\u7279\u6027\u8bbe\u8ba1\u4e0e\u5b9e\u73b0\u4ecb\u7ecd\uff0c\u5728\u9ad8\u5e76\u53d1\u3001\u9ad8\u53ef\u7528\u3001\u591a\u79df\u6237\u9694\u79bb\u3001\u8d44\u6e90\u7ba1\u63a7\u3001\u8ba1\u7b97\u4efb\u52a1\u7ba1\u7406\u7b56\u7565\u7b49\u65b9\u9762\uff0c\u505a\u4e86\u5927\u91cf\u7ec6\u5316\u8003\u91cf\u548c\u5b9e\u73b0\uff0c\u4fdd\u969c\u8ba1\u7b97\u4efb\u52a1\u5728\u590d\u6742\u6761\u4ef6\u4e0b\u6210\u529f\u6267\u884c\u3002

\n

\u8ba1\u7b97\u4efb\u52a1\u7684\u9ad8\u5e76\u53d1\u652f\u6301

\n

Linkis \u7684 Job \u57fa\u4e8e\u591a\u7ea7\u5f02\u6b65\u8bbe\u8ba1\u6a21\u5f0f\uff0c\u670d\u52a1\u95f4\u901a\u8fc7\u9ad8\u6548\u7684 RPC \u548c\u6d88\u606f\u961f\u5217\u6a21\u5f0f\u8fdb\u884c\u5feb\u901f\u901a\u4fe1\uff0c\u5e76\u53ef\u4ee5\u901a\u8fc7\u7ed9 Job \u6253\u4e0a\u521b\u5efa\u8005\u3001\u7528\u6237\u7b49\u591a\u79cd\u7c7b\u578b\u7684\u6807\u7b7e\u8fdb\u884c\u4efb\u52a1\u7684\u8f6c\u53d1\u548c\u9694\u79bb\u6765\u63d0\u9ad8 Job \u7684\u5e76\u53d1\u80fd\u529b\u3002\u901a\u8fc7 Linkis \u53ef\u4ee5\u505a\u5230 1 \u4e2a\u5165\u53e3\u670d\u52a1\uff08 Entrance \uff09\u540c\u65f6\u627f\u63a5\u8d85 1 \u4e07+ \u5728\u7ebf\u7684 Job \u8bf7\u6c42\u3002

\n

\u591a\u7ea7\u5f02\u6b65\u7684\u8bbe\u8ba1\u67b6\u6784\u56fe\u5982\u4e0b\uff1a

\n

\"\"

\n<center>\u56fe 20</center>\n

\u5982\u4e0a\u56fe\u6240\u793a Job \u4ece GateWay \u5230 Entrance \u540e\uff0cJob \u4ece\u751f\u6210\u5230\u6267\u884c\uff0c\u5230\u4fe1\u606f\u63a8\u9001\u7ecf\u5386\u4e86\u591a\u4e2a\u7ebf\u7a0b\u6c60\uff0c\u6bcf\u4e2a\u73af\u8282\u90fd\u901a\u8fc7\u5f02\u6b65\u7684\u8bbe\u8ba1\u6a21\u5f0f\uff0c\u6bcf\u4e00\u4e2a\u7ebf\u7a0b\u6c60\u4e2d\u7684\u7ebf\u7a0b\u90fd\u91c7\u7528\u8fd0\u884c\u4e00\u6b21\u5373\u7ed3\u675f\u7684\u65b9\u5f0f\uff0c\u964d\u4f4e\u7ebf\u7a0b\u5f00\u9500\u3002\u6574\u4e2a Job \u4ece\u8bf7\u6c42\u2014\u6267\u884c\u2014\u5230\u4fe1\u606f\u63a8\u9001\u5168\u90fd\u5f02\u6b65\u5b8c\u6210\uff0c\u663e\u8457\u7684\u63d0\u9ad8\u4e86 Job \u7684\u5e76\u53d1\u80fd\u529b\u3002

\n

\u8fd9\u91cc\u9488\u5bf9\u8ba1\u7b97\u4efb\u52a1\u6700\u5173\u952e\u7684\u4e00\u73af Job \u8c03\u5ea6\u5c42\u8fdb\u884c\u8bf4\u660e\uff0c\u6d77\u91cf\u7528\u6237\u6210\u5343\u4e0a\u4e07\u7684\u5e76\u53d1\u4efb\u52a1\u7684\u538b\u529b\uff0c\u5728 Job \u8c03\u5ea6\u5c42\u4e2d\u662f\u5982\u4f55\u8fdb\u884c\u5b9e\u73b0\u7684\u5462\uff1f

\n

\u5728\u8bf7\u6c42\u63a5\u6536\u5c42\uff0c\u8bf7\u6c42\u63a5\u6536\u961f\u5217\u4e2d\uff0c\u4f1a\u7f13\u5b58\u524d\u7aef\u7528\u6237\u63d0\u4ea4\u8fc7\u6765\u7684\u6210\u5343\u4e0a\u4e07\u8ba1\u7b97\u4efb\u52a1\uff0c\u5e76\u6309\u7cfb\u7edf /\u7528\u6237\u5c42\u7ea7\u5212\u5206\u7684\u8c03\u5ea6\u7ec4\uff0c\u5206\u53d1\u5230\u4e0b\u6e38 Job \u8c03\u5ea6\u6c60\u4e2d\u7684\u5404\u4e2a\u8c03\u5ea6\u961f\u5217\uff1b\u5230 Job \u8c03\u5ea6\u5c42\uff0c\u591a\u4e2a\u8c03\u5ea6\u7ec4\u5bf9\u5e94\u7684\u8c03\u5ea6\u5668\uff0c\u4f1a\u540c\u65f6\u6d88\u8d39\u5bf9\u5e94\u7684\u8c03\u5ea6\u961f\u5217\uff0c\u83b7\u53d6 Job \u5e76\u63d0\u4ea4\u7ed9 Job \u6267\u884c\u6c60\u8fdb\u884c\u6267\u884c\u3002\u8fc7\u7a0b\u4e2d\u5927\u91cf\u4f7f\u7528\u4e86\u591a\u7ebf\u7a0b\u3001\u591a\u7ea7\u5f02\u6b65\u8c03\u5ea6\u6267\u884c\u7b49\u6280\u672f\u3002\u793a\u610f\u5982\u4e0b\u56fe 21\uff1a

\n

\"\"

\n<center>\u56fe 21</center>\n

\u5176\u4ed6\u7ec6\u5316\u7279\u6027

\n

Linkis \u8fd8\u5728\u9ad8\u53ef\u7528\u3001\u591a\u79df\u6237\u9694\u79bb\u3001\u8d44\u6e90\u7ba1\u63a7\u3001\u8ba1\u7b97\u4efb\u52a1\u7ba1\u7406\u7b56\u7565\u7b49\u65b9\u9762\uff0c\u505a\u4e86\u5f88\u591a\u7ec6\u5316\u8003\u91cf\u548c\u5b9e\u73b0\u3002\u7bc7\u5e45\u6709\u9650\uff0c\u5728\u8fd9\u91cc\u4e0d\u518d\u8be6\u8ff0\u6bcf\u4e2a\u7ec6\u5316\u7279\u6027\u7684\u5b9e\u73b0\uff0c\u53ef\u53c2\u89c1 Github \u4e0a Linkis \u7684 Wiki\u3002\u540e\u7ee7\u6211\u4eec\u4f1a\u9488\u5bf9 Linkis \u7684\u8ba1\u7b97\u6cbb\u7406-\u7406\u4e4b\u8def\uff08 Insight \uff09\u7684\u7ec6\u5316\u7279\u6027\u76f8\u5173\u5185\u5bb9\uff0c\u518d\u505a\u4e13\u9898\u4ecb\u7ecd\u3002

\n

\u7ed3\u8bed

\n

\u57fa\u4e8e\u5982\u4e0a\u89e3\u8026\u3001\u590d\u7528\u3001\u5feb\u901f\u6269\u5c55\u3001\u8fde\u901a\u7b49\u67b6\u6784\u8bbe\u8ba1\u4f18\u70b9\uff0c\u53ca\u9ad8\u5e76\u53d1\u3001\u9ad8\u53ef\u7528\u3001\u591a\u79df\u6237\u9694\u79bb\u3001\u8d44\u6e90\u7ba1\u63a7\u7b49\u7ec6\u5316\u7279\u6027\u5b9e\u73b0\uff0c\u8ba1\u7b97\u4e2d\u95f4\u4ef6 Linkis \u5728\u5fae\u4f17\u751f\u4ea7\u73af\u5883\u7684\u5e94\u7528\u6548\u679c\u663e\u8457\u3002\u6781\u5927\u7684\u52a9\u529b\u4e86\u5fae\u4f17\u94f6\u884c\u4e00\u7ad9\u5f0f\u5927\u6570\u636e\u5e73\u53f0\u5957\u4ef6 WeDataSphere \u7684\u5feb\u901f\u6784\u5efa\uff0c\u4e14\u6784\u6210\u4e86 WeDataSphere \u5168\u8fde\u901a\u3001\u591a\u79df\u6237\u3001\u8d44\u6e90\u7ba1\u63a7\u7b49\u4f01\u4e1a\u7ea7\u7279\u6027\u7684\u57fa\u77f3\u3002

\n

Linkis \u5728\u5fae\u4f17\u5e94\u7528\u60c5\u51b5\u5982\u56fe 22\uff1a

\n

\"\"

\n<center>\u56fe 22</center>\n

\u6211\u4eec\u5df2\u5c06 Linkis \u5f00\u6e90\uff0cGithub repo \u5730\u5740\uff1ahttps://github.com/WeBankFinTech/Linkis\u3002

\n

\u6b22\u8fce\u5bf9\u7c7b\u4f3c\u8ba1\u7b97\u6cbb\u7406\u95ee\u9898\u611f\u5174\u8da3\u7684\u540c\u5b66\uff0c\u53c2\u4e0e\u5230\u8ba1\u7b97\u4e2d\u95f4\u4ef6 Linkis \u7684\u793e\u533a\u534f\u4f5c\u4e2d\uff0c\u5171\u540c\u628a Linkis \u5efa\u8bbe\u5f97\u66f4\u52a0\u5b8c\u5584\u548c\u6613\u7528\u3002

\n
\n

\u4f5c\u8005\u4ecb\u7ecd\uff1a\u90b8\u5e05\uff0c\u5fae\u4f17\u94f6\u884c\u5927\u6570\u636e\u5e73\u53f0\u8d1f\u8d23\u4eba\uff0c\u4e3b\u5bfc\u5fae\u4f17\u94f6\u884c WeDataSphere \u5927\u6570\u636e\u5e73\u53f0\u5957\u4ef6\u7684\u5efa\u8bbe\u8fd0\u8425\u4e0e\u5f00\u6e90\uff0c\u5177\u5907\u4e30\u5bcc\u7684\u5927\u6570\u636e\u5e73\u53f0\u5f00\u53d1\u5efa\u8bbe\u5b9e\u8df5\u7ecf\u9a8c\u3002

\n
\n

\u672c\u6587\u662f\u300c\u5206\u5e03\u5f0f\u7cfb\u7edf\u524d\u6cbf\u6280\u672f\u300d\u4e13\u9898\u6587\u7ae0\uff0c\u76ee\u524d\u8be5\u4e13\u9898\u5728\u6301\u7eed\u66f4\u65b0\u4e2d\uff0c\u6b22\u8fce\u5927\u5bb6\u4fdd\u6301\u5173\u6ce8\ud83d\udc47

\n

\"\"

\n" }, { "author": { "url": "member/gancl", "name": "gancl", "avatar": "https://cdn.v2ex.com/avatar/757f/9aa4/53616_large.png?m=1401100020" }, "url": "t/577518", "date_modified": "2019-06-26T01:30:45+00:00", "content_html": "", "date_published": "2019-06-26T01:30:19+00:00", "title": "\u6296\u97f3\u4e0a\u5f88\u706b\u7684\u6570\u636e\u53ef\u89c6\u5316\u89c6\u9891\u91d1\u5341\u6570\u9e3d\u7684\u5236\u4f5c\u65b9\u6cd5,\u662f\u7528 unity3d \u505a\u7684\u8fd8\u662f moviepy \u8fd8\u662f\u7528\u52a8\u753b\u5236\u4f5c\u8f6f\u4ef6\u624b\u52a8\u505a\u7684?", "id": "t/577518" }, { "author": { "url": "member/ANEYE", "name": "ANEYE", "avatar": "https://cdn.v2ex.com/avatar/b1b9/4e88/386264_large.png?m=1568262469" }, "url": "t/557976", "title": "\u8bfe\u7a0b\u9879\u76ee\u6709\u4ec0\u4e48\u4e91\u5e94\u7528\u5927\u6570\u636e\u9879\u76ee\u63a8\u8350\u5417", "id": "t/557976", "date_published": "2019-04-23T08:49:31+00:00", "content_html": "

\u5927\u4e09\u5728\u8bfb\uff0c\u672c\u5b66\u671f\u4e24\u95e8\u8bfe\u4e00\u95e8\u8bfe\u8981\u6c42\u505a\u5927\u6570\u636e\u9879\u76ee\u6bd4\u5982\u63a8\u8350\u7cfb\u7edf\u9884\u6d4b\u5206\u6790\uff0c\u53e6\u4e00\u95e8\u4e5f\u6709\u4e91\u5e94\u7528\u5927\u6570\u636e\u76f8\u5173\u9879\u76ee\uff0c\u6216\u8005\u5c0f\u7a0b\u5e8f\u4ec0\u4e48\u7684\u3002\u4e0d\u77e5\u9053\u505a\u5565\u80fd\u591f\u51fa\u5f69\u53c8\u4e0d\u592a\u96be\uff0c\u5404\u4f4d\u5e26\u54e5\u6709\u4ec0\u4e48\u63a8\u8350\u7684\u5417\uff1f

\n" }, { "author": { "url": "member/taurenshaman", "name": "taurenshaman", "avatar": "https://cdn.v2ex.com/avatar/6a0f/7ec1/40604_large.png?m=1371187588" }, "url": "t/287263", "title": "\u8111\u7f51\u7edc\u7ec4\u56fe\u8c31\uff08 Brainnetome Atlas\uff09", "id": "t/287263", "date_published": "2016-06-21T03:47:56+00:00", "content_html": "\u6700\u65b0\u7248\uff1a http://lore.chuci.info/taurenshaman/json/b607bbab7e0b4130a1cde86940f08466\r
\u89e3\u6790\u81ea\uff1a http://atlas.brainnetome.org/\r
\r
\u6570\u636e\u5305\u62ec 6 \u4e2a\u5206\u7ec4\uff0c\uff1a\r
1. /js/bnatlasviewer/bnatlas_tree.json\r
2. /images/imglist.txt\r
3. \u4ece js \uff08/js/bnatlasviewer/bnatlasviewer.js \uff09\u4e2d\u89e3\u6790\u51fa\u7684 SVG \u56fe\u7247\r
4. /js/bnatlasviewer/data_centers.json\r
5. /js/bnatlasviewer/BDf_FDR05.json\r
6. /js/bnatlasviewer/PCf_FDR05.json\r
\r
\r
{\r
"title": "\u8111\u7f51\u7edc\u7ec4\u56fe\u8c31\uff08 Brainnetome Atlas \uff09",\r
"url": "http://atlas.brainnetome.org",\r
"groups": [\r
{\r
"url": "http://atlas.brainnetome.org/js/bnatlasviewer/data_centers.json",\r
"remark": "converted to json format",\r
"data": [ /** .... **/ ]\r
},\r
/** .... **/\r
]\r
}" }, { "author": { "url": "member/Kai", "name": "Kai", "avatar": "https://cdn.v2ex.com/avatar/021b/bc7e/1024_large.png?m=1657391813" }, "url": "t/165115", "title": "All of you working with Big Data, what is your Data?", "id": "t/165115", "date_published": "2015-01-24T12:53:57+00:00", "content_html": "https://news.ycombinator.com/item?id=8939328" }, { "author": { "url": "member/meteor2013", "name": "meteor2013", "avatar": "https://cdn.v2ex.com/avatar/e9b0/9450/30450_large.png?m=1355010412" }, "url": "t/98165", "title": "\u8bf7\u95ee\u54ea\u91cc\u53ef\u4ee5\u67e5\u5230\u4e16\u754c\u5404\u56fd\u7684\u666e\u901a\u4eba\u4eec\u7684\u8dd1\u6b65\u6570\u636e\uff0c\u7761\u7720\u6570\u636e\uff0c\u8109\u640f\u6570\u636e\u7b49\u5065\u5eb7\u4fe1\u606f\uff1f\u6709\u8fd9\u6837\u7684\u7edf\u8ba1\u673a\u6784\u5b58\u5728\u5417\uff1f", "id": "t/98165", "date_published": "2014-01-23T20:32:42+00:00", "content_html": "" }, { "author": { "url": "member/meteor2013", "name": "meteor2013", "avatar": "https://cdn.v2ex.com/avatar/e9b0/9450/30450_large.png?m=1355010412" }, "url": "t/97330", "date_modified": "2014-01-16T19:08:08+00:00", "content_html": "\u5c0f\u516c\u53f8\uff0c\u4e2a\u4eba\u5c31\u6ca1\u6709\u529e\u6cd5\u73a9\u4e86\u5417\uff1f\u611f\u89c9\u81f3\u5c11\u6536\u96c6\u6570\u636e\u5c31\u6ca1\u529e\u6cd5\u3002
\u90a3\u4e48\uff0c\u5c0f\u516c\u53f8\uff0c\u4e2a\u4eba\u5728\u5927\u6570\u636e\u65f6\u4ee3\u80fd\u505a\u4e9b\u4ec0\u4e48\u5462\uff1f

\u4f7f\u7528\u5927\u516c\u53f8\u63d0\u4f9b\u7684\u6570\u636e\u505a\u70b9\u5c0f\u7684\u5e94\u7528\uff1f", "date_published": "2014-01-16T17:58:23+00:00", "title": "\u7a81\u7136\u5728\u60f3\uff1a \u5927\u6570\u636e\u662f\u4e0d\u662f\u53ea\u6709\u50cf\u963f\u91cc\u5df4\u5df4\u8fd9\u6837\u7684\u5927\u4f01\u4e1a\u624d\u80fd\u73a9\uff1f\u5c0f\u516c\u53f8\uff0c\u4e2a\u4eba\u5728\u5927\u6570\u636e\u65f6\u4ee3\u80fd\u505a\u4e9b\u4ec0\u4e48\u5462\uff1f", "id": "t/97330" }, { "author": { "url": "member/ihacku", "name": "ihacku", "avatar": "https://cdn.v2ex.com/avatar/c8cd/63e1/3073_large.png?m=1675398764" }, "url": "t/88400", "title": "Presto-Distributed SQL query engine for big data", "id": "t/88400", "date_published": "2013-11-07T06:24:11+00:00", "content_html": "http://prestodb.io/" }, { "author": { "url": "member/Livid", "name": "Livid", "avatar": "https://cdn.v2ex.com/avatar/c4ca/4238/1_large.png?m=1776858751" }, "url": "t/49035", "title": "Treasure Data \u83b7\u5f97 Matz \u6295\u8d44...", "id": "t/49035", "date_published": "2012-09-30T02:25:04+00:00", "content_html": "http://techcrunch.com/2012/09/28/treasure-data-launches-cloud-based-data-warehouse-with-investment-from-ruby-creator-yukihiro-matz-matsumoto/" }, { "author": { "url": "member/Livid", "name": "Livid", "avatar": "https://cdn.v2ex.com/avatar/c4ca/4238/1_large.png?m=1776858751" }, "url": "t/48356", "title": "\u6709\u4eba\u7528\u8fc7 Treasure Data \u4e48\uff1f", "id": "t/48356", "date_published": "2012-09-20T17:23:22+00:00", "content_html": "http://www.treasure-data.com/

\u4e00\u4e2a\u770b\u8d77\u6765\u5f88\u4e0d\u9519\u7684 MapReduce \u4e91\u65b9\u6848\uff0c\u76ee\u524d\u5728 500GB \u4e4b\u5185\u90fd\u662f\u514d\u8d39\uff0c\u770b\u8d77\u6765\u4ed6\u4eec\u6253\u7b97\u5728 9.27 \u7684\u65f6\u5019\u516c\u5e03\u6b63\u5f0f\u7684\u5b9a\u4ef7\u3002" }, { "author": { "url": "member/Livid", "name": "Livid", "avatar": "https://cdn.v2ex.com/avatar/c4ca/4238/1_large.png?m=1776858751" }, "url": "t/48332", "title": "Treasure Data", "id": "t/48332", "date_published": "2012-09-20T13:02:21+00:00", "content_html": "http://www.treasure-data.com/

A promising Hadoop cloud." } ] } ubao msn snddm index pchome yahoo rakuten mypaper meadowduck bidyahoo youbao zxmzxm asda bnvcg cvbfg dfscv mmhjk xxddc yybgb zznbn ccubao uaitu acv GXCV ET GDG YH FG BCVB FJFH CBRE CBC GDG ET54 WRWR RWER WREW WRWER RWER SDG EW SF DSFSF fbbs ubao fhd dfg ewr dg df ewwr ewwr et ruyut utut dfg fgd gdfgt etg dfgt dfgd ert4 gd fgg wr 235 wer3 we vsdf sdf gdf ert xcv sdf rwer hfd dfg cvb rwf afb dfh jgh bmn lgh rty gfds cxv xcv xcs vdas fdf fgd cv sdf tert sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf sdf shasha9178 shasha9178 shasha9178 shasha9178 shasha9178 liflif2 liflif2 liflif2 liflif2 liflif2 liblib3 liblib3 liblib3 liblib3 liblib3 zhazha444 zhazha444 zhazha444 zhazha444 zhazha444 dende5 dende denden denden2 denden21 fenfen9 fenf619 fen619 fenfe9 fe619 sdf sdf sdf sdf sdf zhazh90 zhazh0 zhaa50 zha90 zh590 zho zhoz zhozh zhozho zhozho2 lislis lls95 lili95 lils5 liss9 sdf0ty987 sdft876 sdft9876 sdf09876 sd0t9876 sdf0ty98 sdf0976 sdf0ty986 sdf0ty96 sdf0t76 sdf0876 df0ty98 sf0t876 sd0ty76 sdy76 sdf76 sdf0t76 sdf0ty9 sdf0ty98 sdf0ty987 sdf0ty98 sdf6676 sdf876 sd876 sd876 sdf6 sdf6 sdf9876 sdf0t sdf06 sdf0ty9776 sdf0ty9776 sdf0ty76 sdf8876 sdf0t sd6 sdf06 s688876 sd688 sdf86