add index blocked with error "not found instance” after injection pd leader io delay #53282
Labels
component/ddl
This issue is related to DDL of TiDB.
may-affects-5.4
This bug maybe affects 5.4.x versions.
may-affects-6.1
may-affects-6.5
may-affects-7.1
may-affects-7.5
may-affects-8.1
severity/major
type/bug
This issue is a bug.
Bug Report
Please answer these questions before submitting your issue. Thanks!
1. Minimal reproduce step (Required)
1、enable tidb_enable_dist_task and global sort
2、run sysbench
3、add index for sbtest1 and sbtest2 concurrently
4、inject pd leader io delay 5s
operator logs:
the status of ddl job is not synced after 1h0m0s (now: 2024-05-14 21:05:56, jobId: 546, job type: add index /* ingest cloud /, state: running)
operatorLogs:
[2024-05-14 20:05:50] ###### start adding index
ALTER TABLE
sbtest2
ADD INDEXindex_test_1715688350713
(c
)[2024-05-14 20:05:50] ###### wait for ddl job finish
[2024-05-14 21:05:56] ###### wait for ddl job finish timeout(1h0m0s)
select job_id, job_type, state from information_schema.ddl_jobs where query = 'ALTER TABLE
sbtest2
ADD INDEXindex_test_1715688350713
(c
)'jobId: 546, job type: add index / ingest cloud */, state: running
2. What did you expect to see? (Required)
add index can success after fault recover
3. What did you see instead (Required)
add index blocked with error "not found instance”
[2024/05/14 23:31:22.997 +08:00] [INFO] [scheduler.go:264] ["schedule task meet err, reschedule it"] [task-id=78] [task-type=backfill] [allocated-slots=true] [error="not found instance"]
4. What is your TiDB version? (Required)
./tidb-server -V
Release Version: v8.1.0-alpha
Edition: Community
Git Commit Hash: 48a3626
Git Branch: heads/refs/tags/v8.1.0-alpha
UTC Build Time: 2024-05-14 07:09:37
GoVersion: go1.21.6
Race Enabled: false
Check Table Before Drop: false
Store: unistore
2024-05-14T15:24:02.701+0800
The text was updated successfully, but these errors were encountered: