benchmarks

Performance Benchmarks (Beta version)

1. ResNet-56 / CIFAR-10 / 2.00x - 2.55x

Method	Base (%)	Pruned (%)	$\Delta$ Acc (%)	Speed Up
NIPS [1]	-	-	-0.03	1.76x
Geometric [2]	93.59	93.26	-0.33	1.70x
Polar [3]	93.80	93.83	+0.03	1.88x
CP [4]	92.80	91.80	-1.00	2.00x
AMC [5]	92.80	91.90	-0.90	2.00x
HRank [6]	93.26	92.17	-0.09	2.00x
SFP [7]	93.59	93.36	-0.23	2.11x
ResRep [8]	93.71	93.71	+0.00	2.12x
Group-L1	93.53	92.93	-0.60	2.12x
Group-BN	93.53	93.29	-0.24	2.12x
Group-GReg	93.53	93.55	+0.02	2.12x
Ours w/o SL	93.53	93.46	-0.07	2.11x
Ours	93.53	93.77	+0.38	2.13x
GBN [9]	93.10	92.77	-0.33	2.51x
AFP [10]	93.93	92.94	-0.99	2.56x
C-SGD [11]	93.39	93.44	+0.05	2.55x
GReg-1 [12]	93.36	93.18	-0.18	2.55x
GReg-2 [12]	93.36	93.36	-0.00	2.55x
Ours w/o SL	93.53	93.36	-0.17	2.51x
Ours	93.53	93.64	+0.11	2.57x

Note 1: $\text{speed up} = \frac{\text{Base MACs}}{\text{Pruned MACs}}$

Note 2: Baseline methods are not implemented in this repo, because they may require additional modifications to the standard models and training scripts. We are working to support more algorithms.

Note 3: Donwload pretrained resnet-56 from Dropbox or Github Release

Note 4: Training logs are available at run/.

Note 5: "w/o SL" = "without sparse learning"

1.1 Download pre-trained models for reproducibility

wget https://github.com/VainF/Torch-Pruning/releases/download/v1.1.4/cifar10_resnet56.pth

or train a new model:

python main.py --mode pretrain --dataset cifar10 --model resnet56 --lr 0.1 --total-epochs 200 --lr-decay-milestones 120,150,180 

1.2 CIFAR-10 Pruning

- L1-Norm Pruner (Group-L1)

A group-level pruner adapted from Pruning Filters for Efficient ConvNets

# 2.11x
python main.py --mode prune --model resnet56 --batch-size 128 --restore </path/to/pretrained/model> --dataset cifar10  --method l1 --speed-up 2.11 --global-pruning

- BN Pruner (Group-BN)

A group-level pruner adapted from Learning Efficient Convolutional Networks through Network Slimming

# 2.11x
python main.py --mode prune --model resnet56 --batch-size 128 --restore </path/to/pretrained/model> --dataset cifar10  --method slim --speed-up 2.11 --global-pruning --reg 1e-5

- Growing Regularization (Group-GReg)

A group-level pruner adapted from Neural Pruning via Growing Regularization

# 2.11x
python main.py --mode prune --model resnet56 --batch-size 128 --restore </path/to/pretrained/model> --dataset cifar10  --method growing_reg --speed-up 2.11 --global-pruning --reg 1e-4 --delta_reg 1e-5

- Group Pruner (This Work)

# 2.11x without sparse learning (Ours w/o SL)
python main.py --mode prune --model resnet56 --batch-size 128 --restore </path/to/pretrained/model> --dataset cifar10  --method group_norm --speed-up 2.11 --global-pruning

# 2.55x without sparse learning (Ours w/o SL)
python main.py --mode prune --model resnet56 --batch-size 128 --restore </path/to/pretrained/model> --dataset cifar10  --method group_norm --speed-up 2.55 --global-pruning

# 2.11x (Ours)
python main.py --mode prune --model resnet56 --batch-size 128 --restore </path/to/pretrained/model> --dataset cifar10  --method group_sl --speed-up 2.11 --global-pruning --reg 5e-4

# 2.55x (Ours)
python main.py --mode prune --model resnet56 --batch-size 128 --restore </path/to/pretrained/model> --dataset cifar10  --method group_sl --speed-up 2.55 --global-pruning --reg 5e-4

2. VGG-19 / CIFAR-100 / 8.8x

Method	Base (%)	Pruned (%)	$\Delta$ Acc (%)	Speed Up
OBD [13]	73.34	60.70	-12.64	5.73x
OBD [13]	73.34	60.66	-12.68	6.09x
EigenD [13]	73.34	65.18	-8.16	8.80×
GReg-1 [12]	74.02	67.55	-6.67	8.84×
GReg-2 [12]	74.02	67.75	-6.27	8.84×
Ours w/o SL	73.50	67.60	-5.44	8.87x
Ours	73.50	70.39	-3.11	8.92×

2.1 Download pre-trained models for reproducibility

wget https://github.com/VainF/Torch-Pruning/releases/download/v1.1.4/cifar100_vgg19.pth

or train a new model:

python main.py --mode pretrain --dataset cifar100 --model vgg19 --lr 0.1 --total-epochs 200 --lr-decay-milestones 120,150,180 

2.2 CIFAR-100 Pruning

- Group Pruner (This Work)

# 8.84x without sparse learning (Ours w/o SL)
python main.py --mode prune --model vgg19 --batch-size 128 --restore </path/to/pretrained/model> --dataset cifar100  --method group_norm --speed-up 8.84 --global-pruning

# 8.84x (Ours)
python main.py --mode prune --model vgg19 --batch-size 128 --restore </path/to/pretrained/model> --dataset cifar100  --method group_sl --speed-up 8.84 --global-pruning --reg 5e-4

3. ResNet50 / ImageNet / 2.00 GMACs

- Group L1-Norm Pruner (without Sparse Learning)

python -m torch.distributed.launch --nproc_per_node=4 --master_port 18119 --use_env main_imagenet.py --model resnet50 --epochs 90 --batch-size 64 --lr-step-size 30 --lr 0.01 --prune --method l1 --pretrained --output-dir run/imagenet/resnet50_sl --target-flops 2.00 --cache-dataset --print-freq 100 --workers 16 --data-path PATH_TO_IMAGENET --output-dir PATH_TO_OUTPUT_DIR # &> output.log

For the latest results on Vision Transformers, please refer to examples/transformers.

More results will be released soon!

4. Latency Test (ResNet-50, Batch Size 64)

python benchmark_latency.py

[Iter 0]        Pruning ratio: 0.00,         MACs: 4.12 G,   Params: 25.56 M,        Latency: 45.22 ms +- 0.03 ms
[Iter 1]        Pruning ratio: 0.05,         MACs: 3.68 G,   Params: 22.97 M,        Latency: 46.53 ms +- 0.06 ms
[Iter 2]        Pruning ratio: 0.10,         MACs: 3.31 G,   Params: 20.63 M,        Latency: 43.85 ms +- 0.08 ms
[Iter 3]        Pruning ratio: 0.15,         MACs: 2.97 G,   Params: 18.36 M,        Latency: 41.22 ms +- 0.10 ms
[Iter 4]        Pruning ratio: 0.20,         MACs: 2.63 G,   Params: 16.27 M,        Latency: 39.28 ms +- 0.20 ms
[Iter 5]        Pruning ratio: 0.25,         MACs: 2.35 G,   Params: 14.39 M,        Latency: 34.60 ms +- 0.19 ms
[Iter 6]        Pruning ratio: 0.30,         MACs: 2.02 G,   Params: 12.46 M,        Latency: 33.38 ms +- 0.27 ms
[Iter 7]        Pruning ratio: 0.35,         MACs: 1.74 G,   Params: 10.75 M,        Latency: 31.46 ms +- 0.20 ms
[Iter 8]        Pruning ratio: 0.40,         MACs: 1.50 G,   Params: 9.14 M,         Latency: 29.04 ms +- 0.19 ms
[Iter 9]        Pruning ratio: 0.45,         MACs: 1.26 G,   Params: 7.68 M,         Latency: 27.47 ms +- 0.28 ms
[Iter 10]       Pruning ratio: 0.50,         MACs: 1.07 G,   Params: 6.41 M,         Latency: 20.68 ms +- 0.13 ms
[Iter 11]       Pruning ratio: 0.55,         MACs: 0.85 G,   Params: 5.14 M,         Latency: 20.48 ms +- 0.21 ms
[Iter 12]       Pruning ratio: 0.60,         MACs: 0.67 G,   Params: 4.07 M,         Latency: 18.12 ms +- 0.15 ms
[Iter 13]       Pruning ratio: 0.65,         MACs: 0.53 G,   Params: 3.10 M,         Latency: 15.19 ms +- 0.01 ms
[Iter 14]       Pruning ratio: 0.70,         MACs: 0.39 G,   Params: 2.28 M,         Latency: 13.47 ms +- 0.01 ms
[Iter 15]       Pruning ratio: 0.75,         MACs: 0.29 G,   Params: 1.61 M,         Latency: 10.07 ms +- 0.01 ms
[Iter 16]       Pruning ratio: 0.80,         MACs: 0.18 G,   Params: 1.01 M,         Latency: 8.96 ms +- 0.02 ms
[Iter 17]       Pruning ratio: 0.85,         MACs: 0.10 G,   Params: 0.57 M,         Latency: 7.03 ms +- 0.04 ms
[Iter 18]       Pruning ratio: 0.90,         MACs: 0.05 G,   Params: 0.25 M,         Latency: 5.81 ms +- 0.03 ms
[Iter 19]       Pruning ratio: 0.95,         MACs: 0.01 G,   Params: 0.06 M,         Latency: 5.70 ms +- 0.03 ms
[Iter 20]       Pruning ratio: 1.00,         MACs: 0.01 G,   Params: 0.06 M,         Latency: 5.71 ms +- 0.03 ms

Benchmark of Importance Criteria

ResNet50 pre-trained on ImageNet-1K, local pruning without fine-tuning.

python benchmark_importance_criteria.py

Single-layer means group_reduction='first', which only leverages the first layer of a group for importance estimation.

References

[1] Nisp: Pruning networks using neuron importance score propagation.

[2] Filter pruning via geometric median for deep convolutional neural networks acceleration.

[3] Neuron-level structured pruning using polarization regularizer.

[4] Pruning Filters for Efficient ConvNets.

[5] Amc: Automl for model compression and acceleration on mobile devices.

[6] Hrank: Filter pruning using high-rank feature map.

[7] Soft filter pruning for accelerating deep convolutional neural networks

[8] Resrep: Lossless cnn pruning via decoupling remembering and forgetting.

[9] Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks.

[10] Auto-balanced filter pruning for efficient convolutional neural networks.

[11] Centripetal sgd for pruning very deep convolutional networks with complicated structure

[12] Neural pruning via growing regularization

[13] Eigendamage: Structured pruning in the kroneckerfactored eigenbasis.