OpenBox: Generalized and Efficient Blackbox Optimization System

OpenBox is an efficient open-source system designed for solving generalized black-box optimization (BBO) problems, such as automatic hyper-parameter tuning, automatic A/B testing, experimental design, database knob tuning, processor architecture and circuit design, resource allocation, automatic chemical design, etc.

The design of OpenBox follows the philosophy of providing “BBO as a service” - we opt to implement OpenBox as a distributed, fault-tolerant, scalable, and efficient service, with a wide range of application scope, stable performance across problems and advantages such as ease of use, portability, and zero maintenance.

There are two ways to use OpenBox: Standalone python package and Online BBO service.

OpenBox GitHub: https://github.com/PKU-DAIR/open-box

---

News

• OpenBox based solutions achieved the First Place of ACM CIKM 2021 AnalyticCup (Track - Automated Hyperparameter Optimization of Recommendation System).

• OpenBox team won the Top Prize (special prize) in the open-source innovation competition at 2021 CCF ChinaSoft conference.

• Pasca, which adopts Openbox to support neural architecture search functionality, won the Best Student Paper Award at WWW’22.

---

Who should consider using OpenBox

• Those who want to tune hyper-parameters for their ML tasks automatically.

• Those who want to find the optimal configuration for their configuration search tasks (e.g., database knob tuning).

• Data platform owners who want to provide BBO service in their platform.

• Researchers and data scientists who want to solve generalized BBO problems easily.

---

OpenBox capabilities

OpenBox has a wide range of functionality scope, which includes:

1. BBO with multiple objectives and constraints.

2. BBO with transfer learning.

3. BBO with distributed parallelization.

4. BBO with multi-fidelity acceleration.

5. BBO with early stops.

In the following, we provide a taxonomy of existing BBO systems:

System/Package	FIOC	Multi-obj.	Constraint	History	Distributed
Hyperopt	√	×	×	×	√
Spearmint	×	×	√	×	×
SMAC3	√	√	×	×	×
BoTorch	×	√	√	×	×
Ax	√	√	√	×	√
Optuna	√	√	√	×	√
GPflowOPT	×	√	√	×	×
Vizier	√	×	△	△	√
HyperMapper	√	√	√	×	×
HpBandSter	√	×	×	×	√
OpenBox	√	√	√	√	√

• FIOC: Support different input variable types, including Float, Integer, Ordinal and Categorical.

• Multi-obj.: Support optimizing multiple objectives.

• Constraint: Support inequality constraints.

• History: Support injecting prior knowledge from previous tasks into the current search (i.e. transfer learning).

• Distributed: Support parallel evaluations in a distributed environment.

• △ means the system cannot support it for general cases or requires additional dependencies.

---

Table of Contents

• Overview
• Installation
• Quick Start
• Examples
  • Single-Objective Black-box Optimization
  • Single-Objective Black-box Optimization with Constraints
  • Multi-Objective Black-box Optimization
  • Multi-Objective Black-box Optimization with Constraints
  • Ask-and-tell Interface
• Visualization (New!)
• Advanced Usage
  • Problem Definition with Complex Search Space
  • Parallel Evaluation
  • Automatic Algorithm Selection
  • Early Stopping
  • Transfer Learning
• OpenBox as Service
  • Service Introduction
  • Service Deployment
  • Service Tutorial
• Developer's Guide
  • Extend Algorithms in OpenBox
  • Contributing to OpenBox
• Maintainer's Guide
  • Git / GitHub
  • Docs
  • GitHub Actions
  • Test
  • Releases
• Research and Publications
• Change Logs