[C++知识库]Benchmark tool library for c++ code

Benchmark是什么？

Wikipedia解释

如下几类：

• Business and economics（商业和经济）

  • Benchmarking, evaluating performance within organizations（基准测试是将业务流程和绩效指标与其他公司的行业最佳和最佳实践进行比较的做法。 通常测量的维度是质量、时间和成本）
  • Benchmark price (基准价格 (BP) 是国际市场特定部分中每单位数量的价格，由在伦敦金属交易所等市场上一贯出口最大数量或数量的国家或生产者组织设定。 [1] [2] 这个价格是定期设定的，通常是每月一次，作为国际贸易的指导方针。)
  • Benchmark (crude oil), oil-specific practices (基准原油或标记原油是用作原油买卖双方参考价格的原油。有三个主要基准，即西德克萨斯中质原油 (WTI)、布伦特混合原油和迪拜原油。)
  • Benchmark, an investment performance attribution （投资业绩归因是一组技术，业绩分析师用来解释为什么投资组合的业绩与基准不同）

• Science and technology (科学与技术)

  • Benchmark (surveying), a point of known elevation marked for the purpose of surveying （基准、基准或测量基准这一术语起源于测量员在石头结构中凿出的水平标记）
  • Benchmarking (geolocating), an activity involving finding benchmarks （是参与者寻找基准（也称为调查标记或大地控制点）的业余爱好活动）
  • Benchmark (computing), the result of running a computer program to assess performance （在计算中，基准是运行一个计算机程序、一组程序或其他操作的行为，以评估一个对象的相对性能，通常是通过对它运行许多标准测试和试验。 [1] 术语基准也通常用于精心设计的基准测试程序本身。）
  • Benchmark, a best-performing, or gold standard test in medicine and statistics （医学和统计学中表现最佳或黄金标准的测试）

  和我们相关的是Science and technology中的Benchmark (computing)，主要通过考察一个计算机程序、一组程序、其他操作行为，然后对一个对象进行性能评估。

Benchmark (computing)

• Benchmark原则

  1. Relevance: Benchmarks should measure relatively vital features.（基准应该衡量相对重要的特征）
  2. Representativeness: Benchmark performance metrics should be broadly accepted by industry and academia.（代表性：基准性能指标应被业界和学术界广泛接受）
  3. Equity: All systems should be fairly compared.（公平：所有系统都应该被公平地比较）
  4. Repeatability: Benchmark results can be verified.（重复性：可以验证基准测试结果）
  5. Cost-effectiveness: Benchmark tests are economical.（成本效益：基准测试是经济的）
  6. Scalability: Benchmark tests should work across systems possessing a range of resources from low to high.（可扩展性：基准测试应该适用于拥有从低到高的一系列资源的系统）
  7. Transparency: Benchmark metrics should be easy to understand.（透明度：基准指标应该易于理解）

• Benchmark类型

  1. Real program
     • word processing software
     • tool software of CAD
     • user’s application software (i.e.: MIS)
  2. Component Benchmark / Microbenchmark
     • core routine consists of a relatively small and specific piece of code.
     • measure performance of a computer’s basic components
     • may be used for automatic detection of computer’s hardware parameters like number of registers, cache size, memory latency, etc.
  3. Kernel
     • contains key codes
     • normally abstracted from actual program
     • popular kernel: Livermore loop
     • linpack benchmark (contains basic linear algebra subroutine written in FORTRAN language)
     • results are represented in Mflop/s.
  4. Synthetic Benchmark
     • Procedure for programming synthetic benchmark:
       • take statistics of all types of operations from many application programs
       • get proportion of each operation
       • write program based on the proportion above
     • Types of Synthetic Benchmark are:
       • Whetstone
       • Dhrystone
     • These were the first general purpose industry standard computer benchmarks. They do not necessarily obtain high scores on modern pipelined computers.
  5. I/O benchmarks
  6. Database benchmarks
     • measure the throughput and response times of database management systems (DBMS)
  7. Parallel benchmarks
     • used on machines with multiple cores and/or processors, or systems consisting of multiple machines