大数据平台

2019-08-16 08:49:25   460  举报





AI智能生成

大数据平台是一种基于云计算的数据处理和分析工具，它能够处理海量的数据并提供实时的数据分析和决策支持。大数据平台通常包括数据采集、存储、处理和分析等多个模块，可以帮助企业快速地获取有价值的信息并做出正确的决策。此外，大数据平台还具有高可靠性、高可扩展性和低成本等特点，能够满足不同规模企业的需求。总之，大数据平台是现代企业不可或缺的一部分，它能够帮助企业更好地利用数据资源，提高竞争力和创新能力。

大数据开源组件

作者其他创作

大纲/内容

存储

分布式文件系统

HDFS

KFS

GFS

Ceph

GlusterFS

Alluxio

数据库

MPP

Green Plum

NoSQL

Cassandra

HBase

二级索引

Phonex

elastic search

云厂商

https://yq.aliyun.com/articles/703153?spm=a2c4e.11155435.0.0.22e14394zqsnXc

Pharos

https://mp.weixin.qq.com/s/iH-RpgdwldNXAeBdrdNp9A

hindex 华为

https://github.com/Huawei-Hadoop/hindex

https://juejin.im/post/5b69698c6fb9a04f92446e62

CockroachDB

Hypertable

asterixdb

https://github.com/apache/asterixdb

GridDB

https://github.com/griddb/griddb_nosql

ScyllaDB

https://github.com/scylladb/scylla

NewSQl

HTAP

开源

TBase

TiDB

Snappydata

Splice machine

Kudu

Delta Lake

iceberg

Hudi

SequoiaDB

https://gitee.com/wangzhonnew/SequoiaDB

Greenplum

商用

Google F1/Spinner

Oracle Exadata

SAP Hana

MemSQL

Vertica

ApsaraDB for HBase2.0

列式存储

Clickhouse

LevelDB

RocksDB

Vertica

行存

Voldemort

Apache Accumulo

Redis

Hibari

图数据

Neo4j

Dgraph

JanusGraph

Apache TinkerPop Gremlin

Google开源

cayley

https://github.com/cayleygraph/cayley

欧若数网

nebula

https://github.com/vesoft-inc/nebula

tigergraph

创邻科技

galaxybase

百度

BGraph

https://ai.baidu.com/tech/kg/bgraph

HugeGraph

https://github.com/hugegraph

阿里

GDB

https://www.aliyun.com/product/gdb?spm=5176.12825654.eofdhaal5.61.23952c4aScJs23&aly_as=7w41-L_h

蚂蚁金服

GeaBase

https://tech.antfin.com/products/GEABASE

费马科技

LightGraph

PandaGraph

北大研究所

gStore

https://github.com/pkumod/gStore

星环

StellarDB

明略科技

字节跳动

https://zhuanlan.zhihu.com/p/79366551

华为

Titan

AWS

Neptune

微软

Graph Engine

Grakn

HGraphDB

https://github.com/rayokota/hgraphdb

妙盈科技

Miotech

阿里

Graphscope

https://github.com/alibaba/GraphScope

多模式

ArangoDB

OrientDB

存储格式

Avro

Parquet

Arrow

CarbonData

ORC

IndexR

https://github.com/shunfei/indexr

内存数据库

开源

Alluxio

Ignite

Terracotta

Apache Geode

商用

Gemfire

子主题

文档型

MongoDB

CouchDB

Couchbase

时序数据库

Open TSDB

TDegine

Influxdb

Prometheus

Graphite

Druid

https://github.com/m3db/m3

iotdb

https://github.com/apache/iotdb

beringei

https://github.com/facebookarchive/beringei

timescaledb

https://github.com/timescale/timescaledb

KairosDB

https://github.com/kairosdb/kairosdb

Crate

https://github.com/crate/crate

questdb

https://github.com/questdb/questdb

对象存储

开源

minio

https://github.com/minio/minio

子主题

Apache Ozone

https://hadoop.apache.org/ozone/

商用

aws S3

阿里 OSS

云原生

PORLARDB

CynosDB

HashDB

位置数据库

cartodb

https://github.com/CartoDB/cartodb

查询

Hive

Drill

Phoenix

https://www.cnblogs.com/yulu080808/p/8749056.html

Stinger/Tez

Pig

Shark

SparkSQL

Apache Tajo

Kylin

Surus

Trafodion

https://github.com/apache/trafodion

StreamCQL

联邦查询

Quicksql

跨数据源查询引擎

moonbox

https://github.com/edp963/moonbox

xsql

https://github.com/Qihoo360/XSQL

XSQL can be regard as presto implemented by Spark. XSQL is easier to deploy on yarn, more friendly to Spark Programmer

Dremio

https://github.com/dremio/dremio-oss

http://www.imooc.com/article/details/id/289835

Presto

QuickSQL与Presto区别

子主https://mp.weixin.qq.com/s?__biz=MjM5MDE0Mjc4MA==&mid=2651019031&idx=2&sn=4cd5490a14e1ed5fb5b9bd735f939591&chksm=bdbeaf448ac92652bc994a5b3ec13b5af3470031ff2226184a9e736c0c9262250c660a3f4c6d&scene=27#wechat_redirect题

drill

openLooKeng

https://openlookeng.io/

octosql

https://github.com/cube2222/octosql

apache metamodel

https://github.com/apache/metamodel

Linkis

https://github.com/WeBankFinTech/Linkis

MPP

Presto

Impala

HAWQ

http://hdb.docs.pivotal.io/230/hawq/overview/HAWQOverview.html

OpenCog

TensorFlow

Keras

mxnet

Caffe

pytorch

管理

部署

Linkis

Livy

协调

Zookeeper

资源

Yarn

Mesos

子主题

展示

Lumify

Davinci

https://github.com/edp963/davinci

Surperset

Metabase

https://github.com/metabase/metabase

datart

https://github.com/running-elephant/datart

dataease

https://dataease.io/

Hue

Zeppelin

绘图

Carbon

https://github.com/dawnlabs/carbon

D3.js

echart

InMaps

GraphX

Graph

Keylines

gephi

Graphviz

社交网络图

https://en.wikipedia.org/wiki/Social_network_analysis_software

Luigi

GraphBuilder

如何挑选前段可视化框架和平台

https://zhuanlan.zhihu.com/p/149398216

Kibana

Grafana

Nanocubes

CBoard

https://www.oschina.net/p/cboard

Scriptis

https://github.com/WeBankFinTech/Scriptis

RDB

PageNow

https://www.oschina.net/p/pagenow

metatron

https://github.com/metatron-app/metatron-discovery

RDP

DataGear

https://gitee.com/datagear/datagear

ChartFun

https://github.com/ddiu8081/ChartFun

datawrapper

https://github.com/datawrapper/datawrapper

子主题

https://data.avuejs.com/

https://www.datawrapper.de/

http://datacolour.cn/

https://gitee.com/DataColour/DashboardClient

报表设计器

https://xinglie.github.io/report-designer/

监控

ELK

Grafana+Prometos

Nagios

Zabbix

Open falcon

Ganglia

运维

平台

Ambari

Cloudra Manager

EasyHadoop

工具

Ansible

平台

CDH

HDP

FusionInsight

MapR

Streamsets

https://github.com/streamsets

CDAP

https://github.com/cdapio/cdap

databand

https://gitee.com/475660/databand

搜索

ElasticSearch

Solr

Lucene

Nutch

Sphinx

SenseiDB

Katta

https://gitee.com/yiidata/katta

跟踪

Dapper

Zipkin

Pinpoint

SkyWalking

中台

开源

weban datasphere studio

dinky

http://www.dlink.top/

metatron discovery

dbt

https://github.com/fishtown-analytics/dbt

streampack

https://github.com/apache/incubator-streampark

商业

dataiku

dataworks

dataphin

袋鼠云

数栖云

华为大禹

dbt

https://www.getdbt.com/

domnio

datarobot

Serverless/Faas

KNative

https://github.com/knative

OpenFaas

https://github.com/openfaas/faas

Nuclio

https://github.com/nuclio/nuclio

https://github.com/fnproject/fn

OpenWhisk

https://github.com/apache/openwhisk

PAAS

https://github.com/dokku/dokku

https://github.com/dreamfactorysoftware/dreamfactory

经分

商用

维客数据

分析云

经营归因分析系统

客户云

营销云

开源

调优

https://github.com/linkedin/dr-elephant

博客与书籍

康凯森

https://www.bcmeng.com/

黄华超

https://huachaohuang.com/about/

刘未鹏

http://mindhacks.cn/

王垠

http://www.yinwang.org/

周志明

http://icyfenix.cn/

子主题

物联网接入平台

EMQX

https://www.emqx.com/zh/products/emqx

TDEngine

https://www.taosdata.com/

流程挖掘

Uipath

望繁信

https://zhuanlan.zhihu.com/p/452145669

容智信息

https://www.infodator.com/

采集

爬虫

八爪鱼

后羿采集器

Scrapy

汇总：https://www.cnblogs.com/cy163/p/3869175.html

finndycloud

httrack

同步

Logstash

Flinkx

Cloudera Flume

DataX

源码：https://github.com/alibaba/DataX

Facebook Scribe

Debezium

Canal

源码：https://github.com/alibaba/canal

Sqoop

Maxwell

源码：https://github.com/zendesk/maxwell

Chukwa

宜信Dbus

Datalink

https://github.com/ucarGroup/DataLink

DataBus

airbyte

https://github.com/airbytehq/airbyte/

OGG

开源

Kettle

Apache hop

inlong

NiFi

Streamsets

https://streamsets.com/

https://github.com/streamsets/datacollector-oss

airbyte

https://airbyte.com/

https://github.com/airbytehq/airbyte

waterdrop

https://github.com/InterestingLab/waterdrop

https://github.com/apache/incubator-seatunnel

Wormhole

https://github.com/edp963/wormhole

Piflow

https://github.com/cas-bigdatalab/piflow

Marmaray

https://github.com/uber/marmaray

Apache Gobblin

https://github.com/apache/incubator-gobblin

meltano

https://github.com/aroder/meltano

https://gitlab.com/meltano/meltano

PipelineWise

https://github.com/transferwise/pipelinewise

singer

https://www.singer.io/

pipelinewise

https://github.com/transferwise/pipelinewise

gazette

https://github.com/gazette/core

flow

https://github.com/estuary/flow

Porter

https://gitee.com/sxfad/porter

子主题

bitsail

商用

Datastage

Informatica

tapdata

https://tapdata.net/

fivetran

https://www.fivetran.com/

datapipline

https://www.datapipeline.com/

clougence

https://www.clougence.com/

roma

eipaas

计算

批处理

Spark

DataTorrent

Vespa

https://github.com/vespa-engine

Cubert

https://github.com/linkedin/Cubert

流处理

Flink/Blink

Storm

Facebook Puma

Twitter Rainbird

Spark Streaming/Spark Struted Streaming

宜信Wormhole

https://github.com/cas-bigdatalab/piflow

编程框架

Apache Beam

Apache Apex

迭代计算

Twister

Apache Giraph

Apache Hama

Guagua

https://github.com/ShifuML/guagua

网格计算

量子计算

RapidMiner

Weka

H2O

Apache MADlib

一站式机器学习平台

submarine

https://github.com/apache/submarine

Kubeflow

MLFlow

SQLFlow

Acumos AI

https://www.acumos.org/platform/

Polyaxon

kedro

PaddlePaddle

推荐参考文件

https://neptune.ai/blog/the-best-kubeflow-alternatives

https://neptune.ai/blog/the-best-mlflow-alternatives

字节跳动开源

klever

https://github.com/kleveross/klever

webank开源

Prophecis

https://github.com/WeBankFinTech/Prophecis

爱奇艺

Deepthought

滴滴开源

DLFlow

https://github.com/didi/dlflow

广州泰迪开源

https://github.com/GZTipDM/TipDM

九章云

https://github.com/DataCanvasIO

https://baijiahao.baidu.com/s?id=1687330658782863165&wfr=spider&for=pc

Intel

https://github.com/intel-analytics/analytics-zoo

安全

权限

Apache Ranger

Apache Sentry

Apache Eagle

认证

Kerberos

Knox

LDAP

脱敏

https://github.com/armenak/DataDefender

https://github.com/arx-deidentifier/arx

https://github.com/microsoft/presidio

https://github.com/datanymizer/datanymizer

https://github.com/DivanteLtd/anonymizer

https://github.com/Allurx/desensitization

https://github.com/haozhang-x/spring-boot-data-desensitize

https://github.com/aws/aws-encryption-sdk-java

https://github.com/adorsys/datasafe

治理

元数据管理

开源

Apache Atlas

WhereHows

https://github.com/akhandoker/WhereHows

https://github.com/linkedin/datahub

Apache griffin

NetFlix metacat

https://github.com/Netflix/metacat

Lyft Amundsen

https://github.com/lyft/amundsen

openrefine

https://github.com/OpenRefine/OpenRefine

marquez

https://github.com/MarquezProject/marquez

https://blog.csdn.net/xiangwang2206/article/details/109634852

odd-platform

https://github.com/opendatadiscovery/odd-platform

商用

Uber Databook

AirBnd Dataportal

Google Catalog

亿信华辰

https://www.edq.com/

安全

cloudra sentry

hdp ranger

血缘分析

主数据管理

https://github.com/datacleaner/DataCleaner

数据质量

Apache griffin

awslabs/deequ

https://gitee.com/dawsongzhao/deequ

datacleaner/DataCleaner

https://github.com/rhiever/datacleaner

https://github.com/datacleaner/DataCleaner

great-expectations/great_expectations

https://github.com/great-expectations/great_expectations

OpenRefine/OpenRefine

pyeve/cerberus

ResidentMario/missingno

WeBankFinTech/Qualitis

mobydq

https://github.com/ubisoft/mobydq

子主题

https://github.com/rich-iannone/pointblank

https://github.com/sodadata/soda-sql

https://github.com/topics/data-quality

数据协同

zepplin

测试

参考文档

工具

Berkeley BigDataBench

Hadoop基准测试工具

TestDFSIO

mrbench

nnbench

数据生成

BDGS

微型负载

Hadoop GridMix

TeraSort

YCSB

LinkBench

综合测试

Hibench

BigDataBenchmark

http://prof.ict.ac.cn/publications/

端到端测试

Bigbench

论文：Bigbench:Towards an industry standard benchmark for big data analytics

标准：

TPC-DS

数据生成

terrapin

https://github.com/pinterest/terrapin

调度

Oozie

Azkaban

EasyScheduler

源码地址：

介绍博客：

Luigi

Airflow

Conductor

选型分析：

EasyScheduler VS Azkaban VS Airflow

博文地址：

TASKCTL

https://sourceforge.net/projects/taskctl-source/?source=navbar

zeus

xxl-job

https://github.com/xuxueli/xxl-job

架构

Lamda

Kappa

基设

裸金属

云平台

DC/OS

源码：https://github.com/dcos/dcos

虚拟机

缓冲

Kafka

RocketMQ

ActiveMQ

RabbitMQ

ZeroMQ

Pulsar

挖掘

Mahout

KNIME

云编辑器

https://github.com/eclipse-theia/theia

https://github.com/cdr/code-server

无码/低代码开发

良码

https://godspen.ymm56.com/

https://github.com/appsmithorg/appsmith

json开发界面

https://github.com/CareyToboo/ovine

node-red

https://github.com/node-red/node-red

system-designer

https://github.com/design-first/system-designer

华炎魔方

https://github.com/steedos/steedos-platform/

红讯

http://www.redxun.cn/jsaas.jsp

https://github.com/steedos/awesome-low-code

https://github.com/odoo/odoo

前端界面

https://github.com/baidu/amis

https://github.com/didi/epage

https://github.com/MrXujiang/h5-Dooring

https://gitee.com/jabdp/jabdp

https://zhuanlan.zhihu.com/p/92524268

https://gitee.com/dibo_software/diboot?utm_source=alading&utm_campaign=repo

https://github.com/dibo-software/diboot

https://gitee.com/jepaas/jepaas-opensource

https://gitee.com/agile-bpm/agile-bpm-basic

https://gitee.com/jeecg/jeecg-boot

https://www.zhihu.com/question/363277641/answer/1181250156

https://juejin.cn/post/6844904118591422472

https://github.com/formio/formio

https://www.wavemaker.com/

https://www.xplenty.com/blog/the-best-low-code-development-platforms/

https://gitee.com/smallc/SpringBlade

https://github.com/zhangdaiscott/jeecg-boot

拖拽组件

https://github.com/brick-design/react-visual-editor

https://go.decisions.com/business-process-management/

https://www.domo.com/

无远开发平台

https://wuyuan.io/

支持离线部署

https://www.infoq.cn/article/7mfbrc6r56t6upwsq8si

https://juejin.cn/post/6844903608614387720

阿里开源

https://lowcode-engine.cn/

飞冰

宜搭

formily

https://v2.formilyjs.org/

https://mp.weixin.qq.com/s/0ZXNIJSXU3gEZnnuDlyr0A

腾讯

tmagic-editor

Oteam

百度

爱搭

amis

ivx

https://www.ivx.cn/

全部搜录

https://github.com/xioxu/awesome-lowcode

低代码框架

https://github.com/ToolJet/ToolJet

监控

zabbix

CAT

open-falcon

promothus+grafana

RPA

https://camunda.com/download/modeler/

https://github.com/camunda

影刀

Uipath

容智

来也科技

Power Automate

前端

组件仓库

https://github.com/teambit/bit

可视化编辑器

https://github.com/premieroctet/openchakra

可视化编辑器

https://github.com/brick-design/react-visual-editor

子主题

https://www.reddit.com/r/reactjs/comments/fcc8la/are_there_any_visual_editors_for_building_reactjs/

原型工具

https://reactstudio.com/

https://www.framer.com/

https://github.com/koenbok/Framer

https://froala.com/

https://www.progress.com/kinvey

拖拽组件

https://github.com/cerner/kaiju

图表编辑器

https://github.com/plotly/react-chart-editor

可视化大屏

https://github.com/jackchen0120/vueDataV

https://github.com/lizhensheng/vue-data-view

https://github.com/ddiu8081/ChartFun/tree/master

https://gitee.com/DataColour/DashboardClient

https://gitee.com/jman325_admin/pagenow_open

https://redash.io/

https://github.com/getredash/redash

https://gitee.com/smartchart/smartchart

https://github.com/getredash/redash

https://github.com/apache/superset

https://github.com/metabase/metabase

数据可视化

https://medium.com/dailyjs/data-visualization-libraries-for-react-developers-in-2019-a2b9c01262f8

商业可视化免费工具

https://www.pcmag.com/news/10-free-data-visualization-tools

https://www.toptal.com/designers/data-visualization/data-visualization-tools

https://softwareconnect.com/dashboard/

https://www.scoro.com/blog/best-kpi-dashboard-software-tools-reviewed/

https://www.predictiveanalyticstoday.com/open-source-dashboard-software/

组件

表格

https://github.com/nhn/tui.grid

https://github.com/davidguttman/react-pivot

图

https://github.com/awslabs/diagram-maker

RUST

基础工具

编辑器：https://github.com/lapce/lapce

https://github.com/jhspetersson/fselect

数据处理

全文检索搜索引擎

https://github.com/meilisearch/meilisearch

https://github.com/valeriansaliou/sonic

https://github.com/kaedroho/rusticsearch

https://github.com/lnx-search/lnx

KV数据库

https://github.com/engula/engula

https://github.com/datafuselabs/openkv

https://github.com/PumpkinDB/PumpkinDB

https://github.com/tikv/tikv

https://github.com/andresilva/cask

https://github.com/mrxiaozhuox/dorea

https://github.com/MrCroxx/RunKV

子主题

实时数据库

https://github.com/mateusfreira/nun-db

risingwave

数据集成

https://github.com/vectordotdev/vector

https://github.com/awslabs/flowgger

云数仓

https://github.com/datafuselabs/databend

流数据库

https://github.com/MaterializeInc/materialize

https://github.com/skytable/skytable

数据处理库

https://github.com/rayon-rs/rayon

https://github.com/pola-rs/polars

并行计算

https://github.com/TimelyDataflow/timely-dataflow

https://github.com/apache/arrow-datafusion/tree/master/ballista

SQL增强

https://github.com/max-sixty/prql

https://github.com/sqlparser-rs/sqlparser-rs

https://github.com/SeaQL/sea-query

https://github.com/zombodb/postgres-parser

数据生成

https://github.com/getsynth/synth

流数据平台（NiFi）

https://github.com/infinyon/fluvio

数据仓库

https://github.com/tensorbase/tensorbase

https://github.com/cswinter/LocustDB

查询引擎

https://github.com/apache/arrow-datafusion

数据湖

https://github.com/delta-io/delta-rs

FAAS @ Serverless

https://github.com/flock-lab/flock

https://github.com/cloudfuse-io/buzz-rust

数据接口

https://github.com/roapi/roapi

https://github.com/datafuselabs/opendal

数据库引擎

https://github.com/gluesql/gluesql

https://github.com/alex-dukhno/isomorphicdb

时序数据库

https://github.com/influxdata/influxdb_iox

https://github.com/njaard/sonnerie

图数据库

https://github.com/indradb/indradb

https://github.com/dyedgreen/cqlite

https://github.com/jjacobson93/graphael

https://github.com/0xSiO/bolt-rs

数据库学习

https://github.com/erikgrinaker/toydb

SQLITE

https://github.com/joaoh82/rust_sqlite

https://cstack.github.io/db_tutorial/

OLAP数据库

https://github.com/risinglightdb/risinglight

https://github.com/cswinter/LocustDB

https://github.com/KuiBaDB/KuiBaDB

文档数据库

https://github.com/khonsulabs/bonsaidb

https://github.com/vincentdchan/PoloDB

计算

安全计算

https://github.com/apache/incubator-teaclave

区块链

https://github.com/hypercube-lab/hypercube

分布式协议

https://github.com/datafuselabs/openraft

数据库相关技术

https://github.com/skyzh/type-exercise-in-rust

SQL迁移工具

https://github.com/rust-db/barrel

数据库管理工具

https://github.com/TaKO8Ki/gobang

文件存储

https://github.com/TheNeikos/rustbreak

存储引擎

https://github.com/Fullstop000/wickdb

数据安全

https://github.com/Qovery/replibyte

https://github.com/rustsec/advisory-db

访问控制

https://github.com/poonai/inspektor

子主题

数据加载

https://github.com/sfu-db/connector-x

学习资源

https://github.com/pingcap/talent-plan

https://cstack.github.io/db_tutorial/

https://github.com/KOBA789/relly

https://github.com/nukep/llamadb

https://github.com/spacejam/sled

https://github.com/datanymizer/datanymizer

https://github.com/nimrodshn/btree

https://github.com/OsnaCS/uosql-server

https://blog.hidva.com/tags/#KuiBaDB

rust基础工具

https://github.com/rust-lang/rust-clippy

子主题

WEb

https://github.com/tauri-apps/tauri

数字化转型

如何衡量评估数字化转型
（https://mp.weixin.qq.com/s/yscT9UfT25I5Gbmtfy0vNA）

企业数据质量成熟度模型

CMMI DMM 数据管理能力成熟度模型

DCMM数据管理能力成熟度模型再解读

DCAM 数据管理能力成熟度模型

MD3M 主数据管理成熟度模型

DataFlux 主数据管理成熟度模型

IBM 数据治理能力成熟度模型

DSMM 数据安全能力成熟度模型

数字化转型成熟度模型

信通院：企业IT数字化能力和运营效果成熟度模型（IOMM）

中信联：团体标准—数字化转型新型能力体系建设指南

毕马威&阿里：消费品生态全链路数智化转型框架

普华永道：企业数字化成熟度评估架构

华为：开放数字化成熟度模型ODMM

智能制造成熟度模型

有哪些数字化转型的方法和技术？

华为数字化转型建议

https://mp.weixin.qq.com/s/oXTHKFpinD2cKRdNw4fbrw

企业如何成功实现数字化转型？

数字化转型如何避免踩坑

数字化转型有没有可以借鉴的经验和学习的成功案例？

数字化转型应该从哪里开始？

如何制定数字化转型战略？

谈数据：深入聊一聊企业数字化转型这个事儿

IT如何在数字化转型中赋能？

数字化转型的浪潮，对“IT”来说是前所未有的机遇，也是前所未有的挑战，转型能否成功关键在于企业能否“战略性”使用“IT”。

这需要一个过程，这个过程不仅需要改变“老板”对“IT”的认知，更需要“IT”自身作出改变，努力地提升自己的专业技能，深入地学习业务知识，并将技术与业务相结合，为企业创造价值。唯有如此，才能不辜负时代给我们这代“IT人”的使命

企业数字化转型：IT部门的未来！

数字化转型与智慧工厂的关系？

数字化转型企业

华为

海尔

富士康

美的

格力

数字化转型与工业4.0

数字化转型与数据治理的关系？

数字化转型与智能制造的关系？

数字化转型与灯塔工厂的关系？