[人工智能]实用机器学习中文版-1.2数据获取

1. flow chart for data acquisition 数据采集流程图

2. discover what data is available 发现可用的数据

• identify existing datasets 识别现有数据集
• find benchmark datasets to evaluate a new idea 寻找基准数据集来评估新想法
  （1） a diverse set of small to medium datasets for a new hyper-parameter tuning algorithm一种新的超参数调优算法的不同的中小数据集
  （2）large scale datasets for a very big deep neural network 一个非常大的深度神经网络的大规模数据集
• collect new data 收集新的数据
  （1）driving videos covering different driving scenarity涵盖不同驾驶场景的驾驶视频

3. Popular ML datasets 常见的机器学习数据集

• MNIST: digits written by employees of the US Census Bureau
• ImageNet: millions of images from image search engines
• AudioSet: YouTube sound clips for sound classification
• LibriSpeech: 1000 hours of English speech from audiobook
• Kinetics: YouTube videos clips for human actions classification
• KITTI: traffic scenarios recorded by cameras and other sensors
• Amazon Review: customer reviews and from Amazon online shopping
• SQuAD: question-answer pairs derived from Wikipedia

4. Where to Find Datasets 哪里去找数据集

• Paperswithcodes Datasets: academic datasets with leaderboard
• Kaggle Datasets: ML datasets uploaded by data scientists
• Google Dataset search: search datasets in the Web
• Various toolkits datasets: tensorflow, huggingface
• Various conference/company ML competitions
• Open Data on AWS: 100+ large-scale raw data
• Data lakes in your own organization

5. Datasets Comparison 数据集对比

• You often need to deal with raw data in industrial settings 经常需要在工业环境中处理原始数据
• Data curation can be a big projection involving multiple teams.Processing pipeline, storage, legal issue, privacy,…
  数据管理可能是一个涉及多个团队的大型项目。处理管道、存储、法律问题、隐私……

6. Data Integration数据融合

• Combine data from multiple sources into a coherent dataset 将来自多个数据源的数据组合成一个一致的数据集
• Product data is often stored in multiple tables 产品数据通常存储在多个表中
• a table for house information, a table for sales, a table for listing agents 用于房屋信息的表,销售表,清单代理的表格
• Join tables by keys, which are often entity IDs；通过键连接表，键通常是实体id;
• Key issues: identify IDs, missing rows, redundant columns, value conflicts 关键问题:标识id，缺少行，冗余列，值冲突

7. Generate Synthetic Data 生成合成数据

use GAN 用GAN网络生成合成数据

8. Summary 小结

• Finding the right data is challenging 寻找正确的数据是一项挑战
• Raw data in industrial settings VS academic datasets 工业环境中的原始数据VS学术数据集
• Data integration combines data from multiple sources 结合数据集成来自多个数据源的数据
• Data augmentation a common practice 数据增加一个常见的做法
• Synthesizing data is getting popular 合成数据变得越来越流行