[not for land] testing torchao coverage on torchbench/dynamo models

[HDCharles]

Stack from ghstack (oldest at bottom):

• [not for land] other torchbench torchao testing stuff #2155
• -> [not for land] testing torchao coverage on torchbench/dynamo models #2075

Summary:

Test Plan:

Reviewers:

Subscribers:

Tasks:

Tags:

[xuzhao9]

We could possibly make this a userbenchmark to better make this a continuous CI and get dashboard, bisection support, etc.

[HDCharles]

	batchsize	int8dynamic		int8weightonly		int4weightonly		baseline		int8weightonly-bs1		int4weightonly-bs1		baseline-bs1		int8dynamic-bs32		baseline-bs32
model name	default	time	% speedup	time	speedup	time	speedup	time		time	speedup	time	speedup	time		time	% speedup	time
torchrec_dlrm	0	0.00	err	0.00	err	0.00	err	0.00		0.00	err	0.00	err	0.00		0.00	err	0.00
BERT_pytorch	16	4.24	-24.80%	4.23	-24.59%	27.88	-88.57%	3.19		2.22	-44.43%	2.98	-58.71%	1.23		6.19	-11.40%	5.48
Background_Matting	1	7.55	0.35%	7.53	0.55%	7.55	0.36%	7.58		7.57	-0.33%	7.53	0.24%	7.55		7.54	0.44%	7.57
DALLE2_pytorch	1	39.43	8.76%	47.01	-8.77%	0.00	err	42.89		47.97	-4.17%	0.00	err	45.97		161.05	-4.09%	154.47
LearningToPaint	96	1.60	-9.61%	1.53	-5.85%	1.51	-4.67%	1.44		0.62	-14.17%	0.60	-10.59%	0.53		1.04	-19.50%	0.84
Super_SloMo	6	20.15	-0.34%	20.07	0.03%	20.12	-0.22%	20.08		4.16	-0.19%	4.20	-1.24%	4.15		29.72	0.09%	29.74
alexnet	128	2.54	-2.17%	2.78	-10.69%	3.23	-23.31%	2.48		0.71	-44.23%	0.39	0.93%	0.39		0.92	-4.95%	0.88
basic_gnn_edgecnn	1	1.83	-0.90%	1.81	0.11%	1.82	-0.31%	1.81		1.81	-0.20%	1.82	-0.59%	1.81		1.83	4.43%	1.91
basic_gnn_gcn	1	3.70	0.07%	3.54	4.61%	3.53	4.98%	3.70		3.54	-0.34%	3.57	-1.21%	3.52		3.55	-3.87%	3.41
basic_gnn_gin	1	1.15	-0.04%	1.15	-0.03%	1.15	-0.31%	1.15		1.16	-0.34%	1.16	-0.40%	1.16		1.17	-1.53%	1.15
basic_gnn_sage	1	2.42	10.78%	2.43	10.24%	2.68	0.12%	2.68		2.41	1.13%	2.68	-8.98%	2.44		2.41	1.38%	2.44
cm3leon_generate	1	0.00	err	0.00	err	0.00	err	221.86		0.00	err	0.00	err	222.89		0.00	err	686.92
dcgan	256	0.75	0.37%	0.76	-2.05%	0.75	-0.13%	0.75		0.28	0.71%	0.31	-7.55%	0.28		0.35	5.57%	0.37
demucs	8	21.09	0.38%	21.04	0.59%	21.68	-2.39%	21.17		21.12	-0.21%	21.76	-3.15%	21.08		21.01	0.60%	21.13
densenet121	64	9.71	-3.78%	9.72	-3.87%	9.73	-4.05%	9.34		2.57	-15.54%	2.58	-16.09%	2.17		5.91	-4.65%	5.63
detectron2_fasterrcnn_r_101_c4	1	27.33	-0.95%	27.24	-0.62%	27.11	-0.15%	27.07		27.27	-0.79%	27.18	-0.46%	27.05		0.00	err	0.00
detectron2_fasterrcnn_r_101_fpn	1	14.42	-0.23%	14.36	0.24%	15.61	-7.79%	14.39		14.53	1.16%	15.61	-5.82%	14.70		523.09	-13.72%	451.34
detectron2_fasterrcnn_r_50_dc5	1	17.78	-5.05%	17.77	-4.96%	27.75	-39.16%	16.89		18.57	-8.27%	27.56	-38.19%	17.03		0.00	err	402.53
detectron2_fasterrcnn_r_50_fpn	1	12.55	-0.29%	12.68	-1.36%	14.06	-11.02%	12.51		12.71	-1.24%	14.04	-10.61%	12.55		501.90	-20.92%	396.91
detectron2_fcos_r_50_fpn	1	26.11	-26.03%	27.03	-28.54%	25.64	-24.66%	19.31		26.42	-29.38%	25.83	-27.77%	18.65		494.60	0.09%	495.03
detectron2_maskrcnn_r_101_c4	1	29.75	-1.45%	29.62	-1.03%	29.44	-0.42%	29.32		29.61	-1.46%	29.70	-1.76%	29.18		0.00	err	0.00
detectron2_maskrcnn_r_101_fpn	1	17.61	5.63%	16.78	10.87%	18.36	1.31%	18.60		16.98	2.69%	18.69	-6.67%	17.44		587.27	-10.96%	522.87
detectron2_maskrcnn_r_50_c4	1	26.67	0.76%	26.85	0.11%	27.13	-0.92%	26.88		26.85	0.06%	26.99	-0.45%	26.87		0.00	err	0.00
detectron2_maskrcnn_r_50_fpn	1	15.20	8.61%	15.44	6.96%	16.80	-1.72%	16.51		15.29	0.43%	16.65	-7.79%	15.35		517.43	-14.88%	440.43
dlrm	2048	1.64	-12.92%	1.61	-11.34%	2.42	-40.93%	1.43		0.59	-11.35%	0.61	-14.18%	0.52		0.70	-21.70%	0.55
doctr_det_predictor	0	0.00	err	0.00	err	0.00	err	0.00		0.00	err	0.00	err	0.00		0.00	err	0.00
doctr_reco_predictor	1	1.40	-1.38%	1.41	-2.28%	1.60	-13.58%	1.38		1.46	-5.03%	1.40	-0.73%	1.39		2.75	-6.95%	2.56
drq	1	0.44	-31.76%	0.37	-19.50%	0.34	-11.78%	0.30		0.38	-24.09%	0.35	-17.58%	0.29		0.45	-33.25%	0.30
fastNLP_Bert	1	5.20	-28.71%	5.10	-27.40%	9.61	-61.46%	3.70		4.73	-20.16%	9.67	-60.89%	3.78		52.42	-5.79%	49.38
functorch_dp_cifar10	64	0.71	-6.31%	0.71	-7.14%	0.71	-6.84%	0.66		0.61	-8.76%	0.61	-9.00%	0.55		0.70	-6.35%	0.65
functorch_maml_omniglot	1	0.25	-11.73%	0.26	-15.18%	0.25	-12.31%	0.22		0.26	-13.87%	0.25	-12.30%	0.22		0.25	-13.74%	0.22
hf_Albert	1	1.96	-18.19%	2.29	-30.00%	9.38	-82.95%	1.60		2.28	-29.99%	9.52	-83.21%	1.60		21.65	-7.56%	20.01
hf_Bart	1	3.82	-31.60%	4.38	-40.39%	12.80	-79.60%	2.61		4.38	-40.48%	12.77	-79.58%	2.61		0.00	err	35.63
hf_BigBird	1	20.79	-0.41%	23.53	-12.01%	86.71	-76.12%	20.70		23.50	-13.85%	86.74	-76.66%	20.24		0.00	err	err
hf_DistilBert	1	1.79	-36.33%	1.84	-38.19%	6.07	-81.23%	1.14		1.85	-38.05%	6.08	-81.16%	1.14		0.00	err	16.82
hf_GPT2	1	3.58	-10.63%	3.47	-7.79%	8.75	-63.42%	3.20		3.47	-7.87%	8.74	-63.46%	3.19		0.00	err	77.21
hf_GPT2_large	1	16.42	-1.16%	16.44	-1.27%	27.08	-40.07%	16.23		16.46	-2.81%	27.09	-40.94%	16.00		0.00	err	err
hf_Longformer	1	21.42	-12.71%	20.18	-7.35%	90.41	-79.32%	18.70		20.25	-9.13%	90.47	-79.66%	18.40		0.00	err	err
hf_Reformer	1	3.30	-11.21%	3.33	-12.06%	12.74	-77.00%	2.93		3.21	-13.17%	12.70	-78.08%	2.78		50.45	-6.95%	46.94
hf_T5	1	10.21	-8.29%	10.91	-14.15%	35.47	-73.60%	9.36		10.85	-13.88%	35.40	-73.60%	9.34		0.00	err	err
hf_T5_base	1	28.88	-6.70%	29.77	-9.49%	93.20	-71.09%	26.94		29.82	-9.76%	93.15	-71.11%	26.91		0.00	err	err
hf_T5_generate	0	0.00	err	0.00	err	0.00	err	0.00		0.00	err	0.00	err	0.00		0.00	err	0.00
hf_T5_large	1	14.79	-21.73%	18.70	-38.08%	53.23	-78.25%	11.58		18.75	-38.50%	53.31	-78.37%	11.53		0.00	err	err
hf_Whisper	8	3.79	-25.89%	3.21	-12.45%	29.15	-90.36%	2.81		1.50	-34.27%	4.51	-78.16%	0.98		12.28	-16.65%	10.23
lennard_jones	1000	0.20	-0.54%	0.25	-20.23%	0.25	-19.77%	0.20		0.25	-22.56%	0.25	-21.50%	0.20		0.21	-5.89%	0.20
llama	32	2.86	-28.29%	2.93	-30.00%	8.59	-76.10%	2.05		2.53	-36.12%	2.28	-29.07%	1.62		2.86	-27.62%	2.07
llama_v2_7b_16h	1	14.80	39.12%	35.66	-42.27%	191.29	-89.24%	20.59		35.67	-42.56%	191.28	-89.29%	20.49		0.00	err	err
maml_omniglot	5	0.27	-19.64%	0.25	-10.10%	0.29	-24.54%	0.22		0.26	-13.06%	0.27	-17.04%	0.23		0.27	-9.48%	0.24
mnasnet1_0	32	1.90	-8.73%	1.91	-9.20%	1.90	-8.97%	1.73		0.88	-20.21%	0.87	-18.84%	0.70		1.91	-8.07%	1.75
mobilenet_v2	16	1.52	-20.42%	1.35	-10.72%	1.35	-10.27%	1.21		0.82	-20.37%	0.84	-22.39%	0.65		1.94	-8.09%	1.79
mobilenet_v3_large	32	1.81	-9.55%	1.85	-11.48%	1.85	-11.35%	1.64		0.94	-18.71%	0.92	-16.72%	0.76		1.81	-9.58%	1.64
moco	32	0.00	err	0.00	err	0.00	err	10.22		0.00	err	0.00	err	7.54		0.00	err	10.38
nanogpt	1	1.72	-46.69%	1.91	-51.85%	1.97	-53.36%	0.92		1.91	-52.53%	1.95	-53.31%	0.91		3.84	-23.61%	2.94
nvidia_deeprecommender	256	2.33	-0.18%	2.33	-0.06%	2.33	-0.21%	2.33		0.54	-1.11%	0.54	0.24%	0.54		0.70	0.64%	0.70
opacus_cifar10	64	0.83	-5.72%	0.83	-6.59%	0.82	-4.93%	0.78		0.73	-8.24%	0.73	-7.79%	0.67		0.83	-7.17%	0.77
phi_1_5	1	11.51	-6.67%	18.26	-41.15%	78.47	-86.31%	10.74		18.16	-40.82%	78.15	-86.25%	10.75		0.00	err	err
phlippe_densenet	128	1.80	-11.82%	1.77	-10.34%	1.78	-11.12%	1.58		0.86	-20.62%	0.83	-17.81%	0.69		1.07	-15.41%	0.90
phlippe_resnet	128	0.78	-11.94%	0.78	-12.20%	0.77	-11.39%	0.68		0.49	-16.90%	0.48	-16.56%	0.40		0.58	-13.55%	0.50
pyhpc_equation_of_state	1048576	0.23	13.25%	0.23	15.08%	0.27	-0.41%	0.27		0.22	13.53%	0.22	13.23%	0.25		0.25	2.22%	0.25
pyhpc_isoneutral_mixing	1048576	1.57	-2.45%	1.57	-2.63%	1.56	-1.99%	1.53		0.26	6.82%	0.27	5.44%	0.28		0.99	-1.87%	0.97
pyhpc_turbulent_kinetic_energy	1048576	2.13	0.09%	2.14	-0.45%	2.15	-0.50%	2.13		2.13	1.28%	2.15	0.41%	2.16		2.13	-0.24%	2.13
pytorch_CycleGAN_and_pix2pix	1	1.80	0.17%	1.82	-1.14%	1.83	-1.24%	1.80		1.81	-0.48%	1.83	-1.29%	1.80		1.83	-0.13%	1.83
pytorch_stargan	16	2.68	0.27%	2.70	-0.62%	2.71	-0.77%	2.69		2.68	0.25%	2.70	-0.44%	2.69		2.69	0.00%	2.69
pytorch_unet	1	7.55	-0.03%	7.55	0.09%	7.66	-1.37%	7.55		7.55	0.52%	7.53	0.83%	7.59		234.94	-0.23%	234.40
resnet152	32	8.69	-5.30%	9.00	-8.58%	8.63	-4.67%	8.23		2.99	-14.27%	3.01	-14.92%	2.56		8.66	-3.73%	8.33
resnet18	8	0.95	-16.77%	0.88	-9.93%	0.86	-8.27%	0.79		0.65	-12.79%	0.63	-10.85%	0.56		1.68	-2.70%	1.64
resnet50	32	4.06	-3.51%	4.08	-3.97%	4.10	-4.32%	3.92		1.18	-11.69%	1.21	-13.55%	1.05		4.08	-0.53%	4.06
resnext50_32x4d	8	2.07	-15.04%	1.95	-9.46%	2.00	-11.88%	1.76		1.16	-15.27%	1.20	-17.87%	0.98		4.76	-3.80%	4.58
sam	32	52.23	5.91%	60.19	-8.09%	0.00	err	55.32		60.09	-7.83%	0.00	err	55.38		52.37	6.09%	55.56
shufflenet_v2_x1_0	64	2.22	-7.45%	2.24	-8.43%	2.25	-8.65%	2.06		0.00	err	0.00	err	0.00		1.60	-11.71%	1.41
soft_actor_critic	256	0.40	-37.08%	0.31	-19.96%	0.29	-12.98%	0.25		0.30	-15.92%	0.30	-16.04%	0.25		0.36	-29.61%	0.25
speech_transformer	1	10.55	0.08%	12.93	-18.36%	16.56	-36.24%	10.56		12.58	-18.50%	16.97	-39.58%	10.25		12.19	-20.13%	9.74
squeezenet1_1	16	0.67	-1.17%	0.67	-1.37%	0.67	-0.95%	0.66		0.40	-1.12%	0.40	0.60%	0.40		1.04	0.28%	1.05
stable_diffusion_text_encoder	1	3.90	-43.25%	4.35	-49.17%	2.59	-14.62%	2.21		4.43	-52.06%	2.65	-19.84%	2.12		3.91	-44.24%	2.18
stable_diffusion_unet	1	0.00	err	38.14	-9.40%	148.70	-76.76%	34.56		38.34	-9.56%	148.78	-76.69%	34.68		38.64	-9.45%	34.98
timm_efficientdet	32	107.14	0.91%	104.77	3.19%	102.21	5.78%	108.12		5.54	3.68%	5.36	7.18%	5.75		0.00	err	104.46
timm_efficientnet	64	5.01	-3.29%	5.04	-3.82%	5.02	-3.55%	4.84		1.11	-14.33%	1.11	-13.88%	0.95		3.10	-7.31%	2.87
timm_nfnet	128	22.59	0.76%	22.83	-0.31%	22.68	0.37%	22.76		2.27	-6.47%	2.25	-5.78%	2.12		6.99	-2.25%	6.83
timm_regnet	32	12.20	-1.10%	12.28	-1.72%	12.25	-1.47%	12.07		2.67	-9.12%	2.63	-7.85%	2.42		12.40	-1.95%	12.15
timm_resnest	32	3.37	-2.51%	3.38	-2.77%	3.36	-2.13%	3.29		0.79	-11.04%	0.80	-12.29%	0.70		3.41	-3.97%	3.28
timm_vision_transformer	32	4.52	-26.88%	4.02	-17.81%	37.92	-91.29%	3.30		1.54	-41.41%	2.37	-61.95%	0.90		4.60	-28.08%	3.31
timm_vision_transformer_large	32	85.80	17.96%	109.05	-7.19%	1208.51	-91.63%	101.21		12.88	-48.66%	42.51	-84.45%	6.61		85.88	19.18%	102.35
timm_vovnet	32	3.94	-1.42%	3.97	-1.97%	3.99	-2.53%	3.89		1.11	-10.93%	1.10	-9.83%	0.99		4.00	-2.23%	3.91
torch_multimodal_clip	32	8.20	-13.32%	10.15	-30.02%	31.12	-77.17%	7.11		12.99	-39.27%	18.89	-58.25%	7.89		8.23	-7.57%	7.60
tts_angular	64	2.95	-9.14%	2.95	-9.20%	3.04	-11.78%	2.68		4.59	-25.46%	3.68	-6.93%	3.42		2.88	-9.75%	2.60
vgg16	4	1.48	-18.35%	1.81	-32.91%	1.20	0.76%	1.21		1.35	-44.25%	0.71	6.69%	0.75		6.50	1.20%	6.58
vision_maskrcnn	0	0.00	err	0.00	err	0.00	err	0.00		0.00	err	0.00	err	0.00		0.00	err	err
yolov3	8	6.15	0.37%	6.18	-0.06%	6.21	-0.52%	6.18		1.97	0.04%	1.96	0.39%	1.97		21.65	err	err
								84						83				72
coverage		96.43%		97.62%		95.24%				97.59%		95.18%				91.67%
faster		24.69%		15.85%		8.75%				16.05%		13.92%				19.70%
any faster		55.95%

[xuzhao9]

#2116 should be a good starting point better integration of torch_ao with torchbench