profile
viewpoint

bb7133/tispark 1

TiSpark is built for running Apache Spark on top of TiDB/TiKV

bb7133/bigflow 0

Baidu Bigflow is a interface that allows for writing distributed computing programs and provides lots of simple, flexible, powerful APIs. Using Bigflow, you can easily handle data of any scale.

bb7133/chronus 0

distributed time series database base on influxdb

bb7133/community 0

TiDB community content

bb7133/ddl-bench 0

Simple benchmark scripts for MySQL DDL statements

bb7133/docs 0

TiDB/TiKV/PD documents.

bb7133/docs-cn 0

TiDB/TiKV/PD documents in Chinese.

bb7133/failpoint 0

An implementation of failpoints for Golang.

bb7133/flink 0

Mirror of Apache Flink

issue commentpingcap/tidb

COLLATE causes an incorrect result in the presence of an index

Hi @mrigger, thanks for reporting.

The explicit COLLATE syntax is supported recently and probably this is the cause of this bug. We'll fix it soon.

mrigger

comment created time in 13 hours

issue commentpingcap/br

BR cannot restore to `alter-primary-key = true` server

Please notice that with alter-primary-key=true, the table is created with PKIsHandle=false and the primary key is encoded the same way as unique index.

kennytm

comment created time in a day

issue commentpingcap/br

BR cannot restore to `alter-primary-key = true` server

Maybe it's because the embedded TiDB doesn't handle alter-primary-key flag correctly.

kennytm

comment created time in a day

pull request commentpingcap/tidb

gc_worker: Disable Green GC by default (#15764)

/merge

sre-bot

comment created time in a day

pull request commentpingcap/tidb

*: support more system variables in statement summary (#15508)

/merge

djshow832

comment created time in 3 days

PR opened pingcap/tidb

log: refine logs for debug status/DNM

Refine terror.Log logs, DNM

+139 -146

0 comment

48 changed files

pr created time in 3 days

create barnchbb7133/tidb

branch : bb7133/terror_log

created branch time in 3 days

pull request commentpingcap/tidb

expression: fix use sequence with wrong object name

/run-all-tests

AilinKid

comment created time in 4 days

pull request commentpingcap/tidb

ddl: fix flashback tiflash table

Please hold it DNM before we get everything clear.

crazycs520

comment created time in 4 days

issue openedpingcap/mysql-tester

*: Different expected error behavior when recording the test output.

For example, given an test file, say t/some_test.test:

sh> cat t/some_test.test -n
     1	drop database if exists tdb;
     2	create database tdb;
     3	use tdb;
     4	
     5	create table t(a int primary key);
     6	insert into t values (1);
     7	# an incorrect expected error
     8	--Error 1062
     9	insert into t values (2);

Notice that in line 8, an incorrect expected error is given, and when we try to record the result, MySQL test framework gives an error like the followings:

sh>mysqltest -u root < t/mysql_replace.test
drop database if exists tdb;
create database tdb;
use tdb;
create table t(a int primary key);
insert into t values (1);
mysqltest: At line 9: Query 'insert into t values (2)' succeeded, should have failed with error '1062'
not ok

But in this framework, recording is done successfully:

sh>run-tests.sh -s tidb-server -r some_test -b n
skip building tidb-server, using existing binary: ./tidb-server
skip building mysqltest, using existing binary: ./mysql_test
start tidb-server, log file: ./mysql-test.out
tidb-server(PID: 10971) started
record result for case: "some_test"
./t/some_test.test: ok! 2 test cases passed, take time 0.023170004 s

Great, All tests passed
mysqltest end
tidb-server(PID: 10971) stopped

created time in 5 days

PR closed pingcap/tidb

Reviewers
*: rebase master to 4.0 branch component/planner

Rebase master for 4.0.0-rc

+9977 -7639

0 comment

185 changed files

bb7133

pr closed time in 5 days

PR opened pingcap/tidb

*: rebase master to 4.0 branch

Rebase master for 4.0.0-rc

+9977 -7639

0 comment

185 changed files

pr created time in 5 days

create barnchbb7133/tidb

branch : release-4.0

created branch time in 5 days

pull request commentpingcap/tidb

*: consolidate some settings related to the collation

Is it too late? :(

Not exactly, see my comment above :).

bb7133

comment created time in 5 days

Pull request review commentpingcap/tidb

*: consolidate some settings related to the collation

 func GetCollator(collate string) Collator { 	if atomic.LoadInt32(&newCollationEnabled) == 1 { 		ctor, ok := newCollatorMap[collate] 		if !ok {+			logutil.BgLogger().Warn(+				"Unable to get collator by name, use binCollator instead.",+				zap.String("name", collate),+				zap.Stack("stack")) 			return newCollatorMap["utf8mb4_bin"] 		} 		return ctor 	}-	ctor, ok := collatorMap[collate]-	if !ok {-		return collatorMap["utf8mb4_bin"]-	}-	return ctor+	return binCollatorInstance }  // GetCollatorByID get the collator according to id, it will return the binary collator if the corresponding collator doesn't exist. func GetCollatorByID(id int) Collator { 	if atomic.LoadInt32(&newCollationEnabled) == 1 { 		ctor, ok := newCollatorIDMap[id] 		if !ok {+			logutil.BgLogger().Warn(+				"Unable to get collator by ID, use binCollator instead.",+				zap.Int("ID", id),+				zap.Stack("stack")) 			return newCollatorMap["utf8mb4_bin"] 		} 		return ctor 	}-	ctor, ok := collatorIDMap[id]-	if !ok {-		return collatorMap["utf8mb4_bin"]-	}-	return ctor-}--type binCollator struct {-}--// Compare implement Collator interface.-func (bc *binCollator) Compare(a, b string) int {-	return strings.Compare(a, b)-}--// Key implement Collator interface.-func (bc *binCollator) Key(str string) []byte {-	return []byte(str)-}--// Pattern implements Collator interface.-func (bc *binCollator) Pattern() WildcardPattern {-	return &binPattern{}+	return binCollatorInstance }  // CollationID2Name return the collation name by the given id. // If the id is not found in the map, the default collation is returned. func CollationID2Name(id int32) string { 	name, ok := mysql.Collations[uint8(id)] 	if !ok {+		// TODO(bb7133): fix repeating logs when the following code is uncommented.+		//logutil.BgLogger().Warn(+		//	"Unable to get collation name from ID, use default collation instead.",+		//	zap.Int32("ID", id),+		//	zap.Stack("stack")) 		return mysql.DefaultCollationName 	} 	return name } -type binPaddingCollator struct {+// GetCollationByName wraps charset.GetCollationByName, it checks the collation.+func GetCollationByName(name string) (coll *charset.Collation, err error) {+	if coll, err = charset.GetCollationByName(name); err != nil {+		return nil, errors.Trace(err)+	}+	if atomic.LoadInt32(&newCollationEnabled) == 1 {

Got your point, but this code style is commonly used by collate.go, I think we can get another PR for them all.

bb7133

comment created time in 5 days

pull request commentpingcap/tidb

*: consolidate some settings related to the collation

/merge

bb7133

comment created time in 5 days

push eventbb7133/tidb

cfzjywxk

commit sha 5282d089c7692ce89cae363b54c29386b70c33c8

executor: support generated column on point get and batch point get (#15544)

view details

Lynn

commit sha b1ccb30a7595122430c5c80d5069da0a1fda5f7c

*: fix updating the column value when the column is dropping and in WriteOnly state (#15539)

view details

bb7133

commit sha 8f380b5b704ccd2b43dfa85b564d5a248b0b184b

ddl: forbid changing column collation at some cases. (#15562) * ddl: forbid changing column collation in some cases. When the new collation framework is enabled, the index of string columns is collation-aware. Changing the collation of such columns require the rewrite of the index, which is not supported yet, so we forbid such cases for now.

view details

reafans

commit sha 3f71bbfd3c6e215ecaed5c9217446d706340bf1c

executor: change some sys table's status_addr filed to instance. (#15578)

view details

crazycs

commit sha aca6838500c76ba580078ccadac19df22972d574

executor/inspect: fix tikv threshold cpu check diagnose rule (#15538)

view details

Jack Yu

commit sha a019c83032cca85a246171a8ec422f60c9789d1c

.github: add the auto-assignment for project UT Coverage (#15579)

view details

bb7133

commit sha cc0e7a38a1c42506a1354bcf67630e1a563b4b40

*: refine collation-related settings

view details

bb7133

commit sha b905347d1c2d14d7bfeca41a3e4e6feeb1949a04

fix lint and address comment

view details

push time in 5 days

push eventpingcap/tidb

bb7133

commit sha 8f380b5b704ccd2b43dfa85b564d5a248b0b184b

ddl: forbid changing column collation at some cases. (#15562) * ddl: forbid changing column collation in some cases. When the new collation framework is enabled, the index of string columns is collation-aware. Changing the collation of such columns require the rewrite of the index, which is not supported yet, so we forbid such cases for now.

view details

push time in 5 days

pull request commentpingcap/tidb

ddl: forbid changing column collation at some cases.

/run-integration-common-test

bb7133

comment created time in 5 days

pull request commentpingcap/tidb

ddl: forbid changing column collation at some cases.

/merge

bb7133

comment created time in 5 days

pull request commentpingcap/tidb

*: consolidate some settings related to the collation

/run-unit-test

bb7133

comment created time in 5 days

pull request commentpingcap/tidb

*: consolidate some settings related to the collation

/run-unit-test

bb7133

comment created time in 5 days

Pull request review commentpingcap/tidb

ddl: forbid changing column collation at some cases.

 func modifiableCharsetAndCollation(toCharset, toCollate, origCharset, origCollat 	return nil } -// modifiable checks if the 'origin' type can be modified to 'to' type with out the need to+// checkModifyTypes checks if the 'origin' type can be modified to 'to' type with out the need to // change or check existing data in the table. // It returns true if the two types has the same Charset and Collation, the same sign, both are // integer types or string types, and new Flen and Decimal must be greater than or equal to origin.

Done, thanks.

bb7133

comment created time in 5 days

push eventbb7133/tidb

bb7133

commit sha 487e2b0b230a4eca9d4a29f973e64c9bf32a17bb

update comment

view details

push time in 5 days

pull request commentpingcap/tidb

*: consolidate some settings related to the collation

/run-all-tests

bb7133

comment created time in 5 days

push eventbb7133/tidb

bb7133

commit sha 12296b1dc589bbd4b17e3a8b12872c5aaaf31159

address some of comments

view details

push time in 6 days

push eventbb7133/tidb

Song Gao

commit sha 565ec5d0476bf0c5b83174e99247d9e9bd096195

planner: Add explain info for ClusterLogTableExtractor (#15542)

view details

crazycs

commit sha 3906d5a4daa245aa5d46622090d6abe50b1d17bd

store: convert request error to warning when cop request to tidb (#15410)

view details

mmyj

commit sha bbac2b2fe4d0fb452b9c1b6e319421643fab4e54

executor: Improve the performance of `aggFuncSum` by using sliding window (#14887)

view details

Lingyu Song

commit sha d60567f76525306b92b25b33be4b1912ed144abf

privilege: fix select current_role() error (#15534)

view details

bb7133

commit sha 8b47bd9a914e057cb3289f5c4175d7a2cbc4d079

Merge branch 'master' into bb7133/unsupported_collations

view details

push time in 6 days

push eventbb7133/tidb

Mingcong Han

commit sha adaf8d22c4616e21f16eb96f25e55b2b25ffa19e

planner: don't reset optFlag when build DataSource from View (#15547)

view details

Jack Yu

commit sha 7ceb49f5bcdc09dcf9158e60c7709f5223e10a23

*: do not add extra locks for optimistic transactions (#15558)

view details

张某

commit sha a498f0d7e9d54929ccfc1eceeaee77d06630f780

executor, infoschema: Migrate `dataForTableTiFlashReplica` to the executor package. (#15078)

view details

QingFu Wei

commit sha fd0235d030b492330e692e14398bd8876cc10434

util/memory: fix misleading log information when spilling to disk (#15467)

view details

Song Gao

commit sha 565ec5d0476bf0c5b83174e99247d9e9bd096195

planner: Add explain info for ClusterLogTableExtractor (#15542)

view details

crazycs

commit sha 3906d5a4daa245aa5d46622090d6abe50b1d17bd

store: convert request error to warning when cop request to tidb (#15410)

view details

mmyj

commit sha bbac2b2fe4d0fb452b9c1b6e319421643fab4e54

executor: Improve the performance of `aggFuncSum` by using sliding window (#14887)

view details

Lingyu Song

commit sha d60567f76525306b92b25b33be4b1912ed144abf

privilege: fix select current_role() error (#15534)

view details

bb7133

commit sha 8c4c232d2d34c54be3ea4b48cb1e0b858944cea2

ddl: forbid changing column collation at some cases. When the new collation framework is enabled, the index of string columns is collation-aware. Changing the collation of such columns require the rewrite of the index, which is not supported yet, so we forbid such cases for now.

view details

bb7133

commit sha 6f647c127cf97309a7e7f1b29350cb5d58629c38

address some of comments

view details

push time in 6 days

Pull request review commentpingcap/tidb

ddl: forbid changing column collation at some cases.

 func modifiableCharsetAndCollation(toCharset, toCollate, origCharset, origCollat // change or check existing data in the table. // It returns true if the two types has the same Charset and Collation, the same sign, both are // integer types or string types, and new Flen and Decimal must be greater than or equal to origin.-func modifiable(origin *types.FieldType, to *types.FieldType) error {+func modifiable(origin *types.FieldType, to *types.FieldType, needRewriteCollationData bool) error {

Addressed.

bb7133

comment created time in 6 days

Pull request review commentpingcap/tidb

ddl: forbid changing column collation at some cases.

 func (d *ddl) getModifiableColumnJob(ctx sessionctx.Context, ident ast.Ident, or 		return nil, errors.Trace(err) 	} -	if err = modifiable(&col.FieldType, &newCol.FieldType); err != nil {+	if err = modifiable(&col.FieldType, &newCol.FieldType, isColumnWithIndex(col.Name.L, t.Meta().Indices)); err != nil {+		if strings.Contains(err.Error(), "Unsupported modifying collation") {+			colErrMsg := "Unsupported modifying collation of column '%s' from '%s' to '%s' when index is defined on it"

Addressed.

bb7133

comment created time in 6 days

Pull request review commentpingcap/tidb

ddl: forbid changing column collation at some cases.

 func checkAlterTableCharset(tblInfo *model.TableInfo, dbInfo *model.DBInfo, toCh 		if len(col.Charset) == 0 { 			continue 		}-		if err = modifiableCharsetAndCollation(toCharset, toCollate, col.Charset, col.Collate); err != nil {+		if err = modifiableCharsetAndCollation(toCharset, toCollate, col.Charset, col.Collate, isColumnWithIndex(col.Name.L, tblInfo.Indices)); err != nil {+			if strings.Contains(err.Error(), "Unsupported modifying collation") {+				colErrMsg := "Unsupported converting collation of column '%s' from '%s' to '%s' when index is defined on it"

Addressed.

bb7133

comment created time in 6 days

Pull request review commentpingcap/tidb

ddl: forbid changing column collation at some cases.

 func (d *ddl) DropColumn(ctx sessionctx.Context, ti ast.Ident, spec *ast.AlterTa }  // modifiableCharsetAndCollation returns error when the charset or collation is not modifiable.-func modifiableCharsetAndCollation(toCharset, toCollate, origCharset, origCollate string) error {+// needRewriteCollationData is used when trying to modify the collation of a column, it is true when the column is with+// index because index of a string column is collation-aware.+func modifiableCharsetAndCollation(toCharset, toCollate, origCharset, origCollate string, needRewriteCollationData bool) error { 	if !charset.ValidCharsetAndCollation(toCharset, toCollate) { 		return ErrUnknownCharacterSet.GenWithStack("Unknown character set: '%s', collation: '%s'", toCharset, toCollate) 	}++	if needRewriteCollationData && collate.NewCollationEnabled() && !collate.CompatibleCollate(origCollate, toCollate) {

No... Please notice that L2547 returned without more checks. For example, the following statements will be allowed:

# change from utf8mb4_bin to utf8mb4_general_ci
tidb> alter table t modify c varchar(10) collate utf8mb4_general_ci;
bb7133

comment created time in 6 days

pull request commentpingcap/tidb

*: consolidate some settings related to the collation

/run-common-test /run-integration-common-test

bb7133

comment created time in 6 days

pull request commentpingcap/tidb

ddl: forbid changing column collation at some cases.

/run-unit-test

bb7133

comment created time in 6 days

pull request commentpingcap/tidb

ddl: forbid changing column collation at some cases.

/run-all-tests

bb7133

comment created time in 6 days

pull request commentpingcap/tidb

*: consolidate some settings related to the collation

/run-all-tests

bb7133

comment created time in 6 days

push eventbb7133/tidb

bb7133

commit sha 05ab1ff579309dba6a6bd1d9f40c8ef8abf85f27

fix lint and address comment

view details

push time in 6 days

Pull request review commentpingcap/tidb

ddl: forbid changing column collation at some cases.

 func (d *ddl) DropColumn(ctx sessionctx.Context, ti ast.Ident, spec *ast.AlterTa }  // modifiableCharsetAndCollation returns error when the charset or collation is not modifiable.-func modifiableCharsetAndCollation(toCharset, toCollate, origCharset, origCollate string) error {+// needRewriteCollationData is used when trying to modify the collation of a column, it is true when the column is with+// index because index of a string column is collation-aware.+func modifiableCharsetAndCollation(toCharset, toCollate, origCharset, origCollate string, needRewriteCollationData bool) error {

Sure.

bb7133

comment created time in 6 days

Pull request review commentpingcap/tidb

ddl: forbid changing column collation at some cases.

 func (d *ddl) getModifiableColumnJob(ctx sessionctx.Context, ident ast.Ident, or 		return nil, errors.Trace(err) 	} -	if err = modifiable(&col.FieldType, &newCol.FieldType); err != nil {+	if err = modifiable(&col.FieldType, &newCol.FieldType, isColumnWithIndex(col.Name.L, t.Meta().Indices)); err != nil {+		if strings.Contains(err.Error(), "Unsupported modifying collation") {

I used this code before, but found that errUnsupportedModify is equal to errUnsupportedModifyCharset because they share the same error code and class.

Any more suggestion?

bb7133

comment created time in 6 days

push eventbb7133/tidb

Mingcong Han

commit sha adaf8d22c4616e21f16eb96f25e55b2b25ffa19e

planner: don't reset optFlag when build DataSource from View (#15547)

view details

Jack Yu

commit sha 7ceb49f5bcdc09dcf9158e60c7709f5223e10a23

*: do not add extra locks for optimistic transactions (#15558)

view details

张某

commit sha a498f0d7e9d54929ccfc1eceeaee77d06630f780

executor, infoschema: Migrate `dataForTableTiFlashReplica` to the executor package. (#15078)

view details

QingFu Wei

commit sha fd0235d030b492330e692e14398bd8876cc10434

util/memory: fix misleading log information when spilling to disk (#15467)

view details

bb7133

commit sha 942b13054af435329d9f7652ab281259e4b3318a

*: refine collation-related settings

view details

bb7133

commit sha 7a0dd415a0b06b5588c8f4577886e6b08147a14f

fix lint and address comment

view details

push time in 6 days

Pull request review commentpingcap/tidb

*: consolidate some settings related to the collation

 func (s *testSerialSuite) TestAutoRandom(c *C) { 	config.GetGlobalConfig().Experimental.AllowAutoRandom = false 	assertExperimentDisabled("create table auto_random_table (a int primary key auto_random(3))") }++func (s *testSerialSuite) TestForbidUnsupportedCollations(c *C) {+	collate.SetNewCollationEnabledForTest(true)+	defer collate.SetNewCollationEnabledForTest(false)+	tk := testkit.NewTestKit(c, s.store)++	mustGetUnsupportedCollation := func(sql string, coll string) {+		tk.MustGetErrMsg(sql, fmt.Sprintf("[ddl:1273]Unsupported collation when new collation is enabled: '%s'", coll))+	}+	// Test default collation of database.+	mustGetUnsupportedCollation("create database ucd charset utf8mb4 collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")+	mustGetUnsupportedCollation("create database ucd charset utf8 collate utf8_unicode_ci", "utf8_unicode_ci")+	tk.MustExec("create database ucd")+	tk.MustExec("alter database ucd charset utf8mb4 collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")+	tk.MustExec("alter database collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")++	// Test default collation of table.+	tk.MustExec("use ucd")+	mustGetUnsupportedCollation("create table t(a varchar(20)) charset utf8mb4 collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")+	mustGetUnsupportedCollation("create table t(a varchar(20)) collate utf8_unicode_ci", "utf8_unicode_ci")+	tk.MustExec("create table t(a varchar(20)) collate utf8mb4_general_ci")+	mustGetUnsupportedCollation("alter table t default collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")+	mustGetUnsupportedCollation("alter table t convert to charset utf8mb4 collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")++	// Test collation of columns.+	mustGetUnsupportedCollation("create table t1(a varchar(20) collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")+	mustGetUnsupportedCollation("create table t1(a varchar(20) charset utf8 collate utf8_unicode_ci", "utf8_unicode_ci")+	mustGetUnsupportedCollation("alter table t1 modify a varchar(20) collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")+	mustGetUnsupportedCollation("alter table t1 modify a varchar(20) charset utf8 collate utf8_unicode_ci", "utf8_unicode_ci")+	mustGetUnsupportedCollation("alter table t1 modify a varchar(20) charset utf8 collate utf8_unicode_ci", "utf8_unicode_ci")++	// TODO(bb7133): fix the following cases by setting charset from collate firstly.

It's just about the 'correct' error message.

bb7133

comment created time in 6 days

Pull request review commentpingcap/tidb

*: consolidate some settings related to the collation

 func (s *testSerialSuite) TestAutoRandom(c *C) { 	config.GetGlobalConfig().Experimental.AllowAutoRandom = false 	assertExperimentDisabled("create table auto_random_table (a int primary key auto_random(3))") }++func (s *testSerialSuite) TestForbidUnsupportedCollations(c *C) {+	collate.SetNewCollationEnabledForTest(true)+	defer collate.SetNewCollationEnabledForTest(false)+	tk := testkit.NewTestKit(c, s.store)++	mustGetUnsupportedCollation := func(sql string, coll string) {+		tk.MustGetErrMsg(sql, fmt.Sprintf("[ddl:1273]Unsupported collation when new collation is enabled: '%s'", coll))+	}+	// Test default collation of database.+	mustGetUnsupportedCollation("create database ucd charset utf8mb4 collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")+	mustGetUnsupportedCollation("create database ucd charset utf8 collate utf8_unicode_ci", "utf8_unicode_ci")+	tk.MustExec("create database ucd")+	tk.MustExec("alter database ucd charset utf8mb4 collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")+	tk.MustExec("alter database collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")++	// Test default collation of table.+	tk.MustExec("use ucd")+	mustGetUnsupportedCollation("create table t(a varchar(20)) charset utf8mb4 collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")+	mustGetUnsupportedCollation("create table t(a varchar(20)) collate utf8_unicode_ci", "utf8_unicode_ci")+	tk.MustExec("create table t(a varchar(20)) collate utf8mb4_general_ci")+	mustGetUnsupportedCollation("alter table t default collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")+	mustGetUnsupportedCollation("alter table t convert to charset utf8mb4 collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")++	// Test collation of columns.+	mustGetUnsupportedCollation("create table t1(a varchar(20) collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")+	mustGetUnsupportedCollation("create table t1(a varchar(20) charset utf8 collate utf8_unicode_ci", "utf8_unicode_ci")+	mustGetUnsupportedCollation("alter table t1 modify a varchar(20) collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")+	mustGetUnsupportedCollation("alter table t1 modify a varchar(20) charset utf8 collate utf8_unicode_ci", "utf8_unicode_ci")+	mustGetUnsupportedCollation("alter table t1 modify a varchar(20) charset utf8 collate utf8_unicode_ci", "utf8_unicode_ci")++	// TODO(bb7133): fix the following cases by setting charset from collate firstly.

No, it's easy. But I think we should fix it in another PR.

bb7133

comment created time in 6 days

Pull request review commentpingcap/tidb

*: fix updating the column value when the column is dropping and in WriteOnly state

 func (b *PlanBuilder) buildUpdate(ctx context.Context, update *ast.UpdateStmt) ( 		tblID2table[id], _ = b.is.TableByID(id) 	} 	updt.TblColPosInfos, err = buildColumns2Handle(updt.OutputNames(), tblID2Handle, tblID2table, true)+	if err == nil {
	if err != nil {

Better to use this style.

zimulala

comment created time in 6 days

Pull request review commentpingcap/tidb

*: consolidate some settings related to the collation

 func (s *testSerialSuite) TestAutoRandom(c *C) { 	config.GetGlobalConfig().Experimental.AllowAutoRandom = false 	assertExperimentDisabled("create table auto_random_table (a int primary key auto_random(3))") }++func (s *testSerialSuite) TestForbidUnsupportedCollations(c *C) {+	collate.SetNewCollationEnabledForTest(true)+	defer collate.SetNewCollationEnabledForTest(false)+	tk := testkit.NewTestKit(c, s.store)++	mustGetUnsupportedCollation := func(sql string, coll string) {+		tk.MustGetErrMsg(sql, fmt.Sprintf("[ddl:1273]Unsupported collation when new collation is enabled: '%s'", coll))+	}+	// Test default collation of database.+	mustGetUnsupportedCollation("create database ucd charset utf8mb4 collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")+	mustGetUnsupportedCollation("create database ucd charset utf8 collate utf8_unicode_ci", "utf8_unicode_ci")+	tk.MustExec("create database ucd")+	tk.MustExec("alter database ucd charset utf8mb4 collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")+	tk.MustExec("alter database collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")++	// Test default collation of table.+	tk.MustExec("use ucd")+	mustGetUnsupportedCollation("create table t(a varchar(20)) charset utf8mb4 collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")+	mustGetUnsupportedCollation("create table t(a varchar(20)) collate utf8_unicode_ci", "utf8_unicode_ci")+	tk.MustExec("create table t(a varchar(20)) collate utf8mb4_general_ci")+	mustGetUnsupportedCollation("alter table t default collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")+	mustGetUnsupportedCollation("alter table t convert to charset utf8mb4 collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")++	// Test collation of columns.+	mustGetUnsupportedCollation("create table t1(a varchar(20) collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")+	mustGetUnsupportedCollation("create table t1(a varchar(20) charset utf8 collate utf8_unicode_ci", "utf8_unicode_ci")+	mustGetUnsupportedCollation("alter table t1 modify a varchar(20) collate utf8mb4_unicode_ci", "utf8mb4_unicode_ci")+	mustGetUnsupportedCollation("alter table t1 modify a varchar(20) charset utf8 collate utf8_unicode_ci", "utf8_unicode_ci")+	mustGetUnsupportedCollation("alter table t1 modify a varchar(20) charset utf8 collate utf8_unicode_ci", "utf8_unicode_ci")++	// TODO(bb7133): fix the following cases by setting charset from collate firstly.

No, it is easy...but it should be fixed in another PR.

bb7133

comment created time in 6 days

PR opened pingcap/tidb

ddl: forbid changing column collation at some cases. component/DDL

What problem does this PR solve?

When the new collation framework is enabled, the index of string columns is collation-aware. Changing the collation of such columns require the rewrite of the index, which is not supported yet, so we forbid such cases for now.

What is changed and how it works?

Added more checks.

Related changes

  • NA

Check List <!--REMOVE the items that are not applicable-->

Tests <!-- At least one of them must be included. -->

  • Integration test

Side effects

  • NA

Release note <!-- bugfixes or new feature need a release note -->

  • NA
+72 -29

0 comment

4 changed files

pr created time in 6 days

create barnchbb7133/tidb

branch : bb7133/collation_index

created branch time in 6 days

pull request commentpingcap/tidb

*: consolidate some settings related to the collation

PTAL @wjhuang2016 @qw4990

bb7133

comment created time in 7 days

PR opened pingcap/tidb

*: consolidate some settings related to the collation compatibility-breaker component/charset priority/release-blocker

What is changed and how it works?

What's Changed: This PR aims to update the following collation-related features:

  1. Restrict the settings of collation-related system variables collation_connection, collation_server and collation_database. Before this PR, they can be set to anything but now only valid collations are permitted, for example:

    Before this PR:

    tidb> set @@collation_connection='some_nonexist_collation';
    Query OK, 0 rows affected (0.00 sec)
    tidb> set names utf8mb4 collation some_nonexist_collation;
    Query OK, 0 rows affected (0.00 sec)
    

    After this PR(same with MySQL):

    tidb> set @@collation_connection='some_nonexist_collation';
    ERROR 1273 (HY000): Unknown collation: 'some_nonexist_collation'
    tidb> set names utf8mb4 collate some_nonexist_collation;
    ERROR 1273 (HY000): Unknown collation: 'some_nonexist_collation'
    

    NOTICE: This potentially breaks compatibility with old TiDB, but it should be fine since it means nothing if such system variables are set to an invalid value.

  2. Forbid unsupported collations when new collation is enabled(see #14734 for more details about the switch): Before this PR when new collation is enabled:

    tidb> set names utf8mb4 collate utf8mb4_unicode_ci;
    Query OK, 0 rows affected (0.00 sec)
    

    After this PR when new collation is enabled:

    tidb> set names utf8mb4 collate utf8mb4_unicode_ci;
    ERROR 1273 (HY000): Unsupported collation when new collation is enabled: 'utf8mb4_unicode_ci'
    
  3. Update the result of information_schema.COLLATIONS when new collation is enabled: utf8mb4_general_ci and utf8_general_ci are added since they're supported now:

    tidb> show collation;
    +--------------------+---------+------+---------+----------+---------+
    | Collation          | Charset | Id   | Default | Compiled | Sortlen |
    +--------------------+---------+------+---------+----------+---------+
    | binary             | binary  |   63 | Yes     | Yes      |       1 |
    | ascii_bin          | ascii   |   65 | Yes     | Yes      |       1 |
    | latin1_bin         | latin1  |   47 | Yes     | Yes      |       1 |
    | utf8mb4_bin        | utf8mb4 |   46 | Yes     | Yes      |       1 |
    | utf8_bin           | utf8    |   83 | Yes     | Yes      |       1 |
    | utf8mb4_general_ci | utf8mb4 |   45 |         | Yes      |       1 |
    | utf8_general_ci    | utf8    |   33 |         | Yes      |       1 |
    +--------------------+---------+------+---------+----------+---------+
    7 rows in set (0.00 sec)
    

Related changes

  • NA

Check List <!--REMOVE the items that are not applicable-->

Tests <!-- At least one of them must be included. -->

  • Integration test
  • Manual test (add detailed scripts or steps below) See above.

Side effects

  • Breaking backward compatibility

Release note

  • When the new collation framework is enabled, collations not supported are now forbidden.
  • Validate the value of system variables collation_connection, collation_server and collation_database, Unknown collation will be returned if they're not valid.
+320 -118

0 comment

14 changed files

pr created time in 7 days

create barnchbb7133/tidb

branch : bb7133/unsupported_collations

created branch time in 7 days

push eventpingcap/tidb

bb7133

commit sha 5e8706a1f673355187364f43d794795e81d29c63

util/collate: remove `CollatorOption` (#15472) `CollatorOption` was used to specify the `PadLen` which aims to support "PADDING" attribute of the collations. But now it is not needed anymore, since TiDB actually implements "PADDING" by removing trailing spaces.

view details

push time in 9 days

PR merged pingcap/tidb

Reviewers
util/collate: remove `CollatorOption` component/expression component/util status/LGT2 status/can-merge

What problem does this PR solve?

Problem Summary:

CollatorOption was used to specify the PadLen which aims to support "PADDING" attribute of the collations. But now it is not needed anymore, since TiDB actually implements "PADDING" by removing trailing spaces.

What is changed and how it works?

What's Changed:

As stated above, simply remove CollatorOption, no public change.

How it Works:

As stated above, simply remove CollatorOption.

Related changes

Not needed.

Check List <!--REMOVE the items that are not applicable-->

Tests <!-- At least one of them must be included. -->

  • Unit test
  • Integration test

Side effects

  • NA

Release note <!-- bugfixes or new feature need a release note -->

  • NA
+18 -29

6 comments

7 changed files

bb7133

pr closed time in 9 days

pull request commentpingcap/tidb

util/collate: remove `CollatorOption`

/merge

bb7133

comment created time in 9 days

push eventpingcap/tidb

crazycs

commit sha 875036363e1e2385c66ca0ca7685861cb0eb20d2

model: add schema name, table name to ddl job. (#11561) (#15450)

view details

push time in 9 days

PR merged pingcap/tidb

model: add schema name, table name to ddl job. (#11561) component/DDL component/executor status/can-merge type/3.1-cherry-pick

<!-- Thank you for contributing to TiDB!

PR Title Format:

  1. pkg [, pkg2, pkg3]: what's changed
  2. *: what's changed

-->

What problem does this PR solve?

cherry-pick https://github.com/pingcap/tidb/pull/11561

+94 -6

7 comments

4 changed files

crazycs520

pr closed time in 9 days

pull request commentpingcap/tidb

docs/design: update proposal of invisible index

/merge

Deardrops

comment created time in 10 days

pull request commentpingcap/tidb

model: add schema name, table name to ddl job. (#11561)

/merge

crazycs520

comment created time in 10 days

push eventpingcap/tidb

crazycs

commit sha 796e623d2be3c7eba12728067c5f7f04c86e52a7

ddl: support recover truncate table (#15398) (#15458)

view details

push time in 10 days

PR merged pingcap/tidb

ddl: support recover truncate table (#15398) component/DDL component/executor type/3.0-cherry-pick

<!-- Thank you for contributing to TiDB!

PR Title Format:

  1. pkg [, pkg2, pkg3]: what's changed
  2. *: what's changed

-->

What problem does this PR solve?

cherry-pickhttps://github.com/pingcap/tidb/pull/15398/files

Release note <!-- bugfixes or new feature need a release note -->

  • support recover truncate table.
+34 -20

1 comment

2 changed files

crazycs520

pr closed time in 10 days

pull request commentpingcap/tidb

ddl: support recover truncate table (#15398)

/merge

crazycs520

comment created time in 10 days

Pull request review commentpingcap/tidb

docs/design: add proposal for invisible index

+# Proposal: Invisible index++- Author(s):     [Deardrops](https://github.com/Deardrops), [wjhuang2016](https://github.com/wjhuang2016)+- Last updated:  Mar. 12, 2020+- Discussion at: https://github.com/pingcap/tidb/issues/9246++## Abstract++MySQL supports [Invisible indexes](https://dev.mysql.com/doc/refman/8.0/en/invisible-indexes.html); that is, indexes that are not used by the optimizer. ++This is a useful feature for dropping an index in a safe way. Invisible indexes make it possible to test the effect of removing an index on query performance, without making a destructive change that must be undone should the index turn out to be required. Dropping and re-adding an index can be expensive for a large table, whereas making it invisible and visible are fast, in-place operations.++Support the option of `VISIBLE | INVISIBLE`.+```+CREATE [...] INDEX index_name+    [index_type]+    ON tbl_name (key_part,...)+    [index_option]++index_option:+    {VISIBLE | INVISIBLE}+```++Also, consider the following:++- Display information about invisible indexes in the output of `INFORMATION_SCHEMA.STATISTICS` or `SHOW INDEX`.+- Index hints need to report errors when used in invisible indexes.+- The primary key cannot be set to an invisible index.++## Background++Index has a great influence on the performance of the database. Whether there is an index or whether the optimizer chooses the right index determines the performance of reading and writing the database to a great extent. In some cases, we would like to know the impact of deleting an index on the read-write performance of the database.  At present, our method is to tell the optimizer to ignore the index through Index Hint.  Although this method can achieve the goal, it is not practical to modify all SQL statements.++## Proposal++Adding an option (visible or not) to the index. If it is not visible, it's called **Invisible Index**. Invisible Index cannot be used by the optimizer (with the `use_invisible_indexes` switch on), but the index is maintained during DML operations. For a query statement, invisible index has the same effect as ignoring the index through Index Hint.++The option `INVISIBLE` of an index can be changed by using the following DDL statement:++```+ALTER TABLE table_name ALTER INDEX index_name { INVISIBLE | VISIBLE };+```++Or set option `INVISIBLE` when creating an index:++```+CREATE INDEX index_name ON table_name(key) [ INVISIBLE | VISIBLE ];+```++`INVISIBLE`, `VISIBLE` is as part of the index option, they can also be set when creating a table.++In order to know whether an index is invisible, it can be read from the `INFORMATION_SCHEMA.STATISTICS` table or through the `SHOW INDEX` command.++In addition, add a new flag `use_invisible_indexes` in system variable `optimizer_switch`, which determine whether the option `INVISIBLE` takes effect. If `use_invite_indexes` is on, the optimizer can still use invisible index.++A table with no explicit primary key may still have an effective implicit primary key if it has any UNIQUE indexes on NOT NULL columns. In this case, the first such index places the same constraint on table rows as an explicit primary key and that index cannot be made invisible.++## Rationale++Another solution for implement invisible indexes is: Indicate invisibility by DDL state `WriteOnly`. This solution has the following problems:++- The logic of schema change needs to be changed, which is relatively complicated to implement.+- Cannot distinguish between indexes that are currently in WriteOnly and invisible indexes.+- Handling the switch is troublesome.++## Compatibility and Migration Plan++This the a new feature and it's absolutly compatible with old TiDB versions, also, it's not impact any data migration.+The syntax and functions are basically compatible with MySQL. Expect one:++	When use invisible index in `SQL Hint`, and set `use_invisible_indexes = false`, MySQL allow use the invisible index.+	But in TiDB, It's **not allowed**, and will throw a `Unresolved name` error, Because the behavior is more reasonable.++## Implementation++- Add syntax support in parser+- Add a new column `IS_VISIBLE` in `information_schema.statistics`+- Add a new column `VISIBLE` in `SHOW INDEX FROM table` statement+- Show invisiable column infomations in `SHOW CREATE TABLE` statement+- Add `use_invisible_indexes` in system variable `@@optimizer_switch`+- Add new error message `ERROR 3522 (HY000): A primary key index cannot be invisible`+- Ignore invisible index in optimizer and add unit tests++## Testing Plan++- Unit test+- Port All mysql test related to invisible index
- Learn from MySQL test cases related to invisible indexes
Deardrops

comment created time in 10 days

Pull request review commentpingcap/tidb

docs/design: add proposal for invisible index

+# Proposal: Invisible index++- Author(s):     [Deardrops](https://github.com/Deardrops), [wjhuang2016](https://github.com/wjhuang2016)+- Last updated:  Mar. 12, 2020+- Discussion at: https://github.com/pingcap/tidb/issues/9246++## Abstract++MySQL supports [Invisible indexes](https://dev.mysql.com/doc/refman/8.0/en/invisible-indexes.html); that is, indexes that are not used by the optimizer. ++This is a useful feature for dropping an index in a safe way. Invisible indexes make it possible to test the effect of removing an index on query performance, without making a destructive change that must be undone should the index turn out to be required. Dropping and re-adding an index can be expensive for a large table, whereas making it invisible and visible are fast, in-place operations.++Support the option of `VISIBLE | INVISIBLE`.+```+CREATE [...] INDEX index_name+    [index_type]+    ON tbl_name (key_part,...)+    [index_option]++index_option:+    {VISIBLE | INVISIBLE}+```++Also, consider the following:++- Display information about invisible indexes in the output of `INFORMATION_SCHEMA.STATISTICS` or `SHOW INDEX`.+- Index hints need to report errors when used in invisible indexes.+- The primary key cannot be set to an invisible index.++## Background++Index has a great influence on the performance of the database. Whether there is an index or whether the optimizer chooses the right index determines the performance of reading and writing the database to a great extent. In some cases, we would like to know the impact of deleting an index on the read-write performance of the database.  At present, our method is to tell the optimizer to ignore the index through Index Hint.  Although this method can achieve the goal, it is not practical to modify all SQL statements.++## Proposal++Adding an option (visible or not) to the index. If it is not visible, it's called **Invisible Index**. Invisible Index cannot be used by the optimizer (with the `use_invisible_indexes` switch on), but the index is maintained during DML operations. For a query statement, invisible index has the same effect as ignoring the index through Index Hint.++The option `INVISIBLE` of an index can be changed by using the following DDL statement:++```+ALTER TABLE table_name ALTER INDEX index_name { INVISIBLE | VISIBLE };+```++Or set option `INVISIBLE` when creating an index:++```+CREATE INDEX index_name ON table_name(key) [ INVISIBLE | VISIBLE ];+```++`INVISIBLE`, `VISIBLE` is as part of the index option, they can also be set when creating a table.++In order to know whether an index is invisible, it can be read from the `INFORMATION_SCHEMA.STATISTICS` table or through the `SHOW INDEX` command.++In addition, add a new flag `use_invisible_indexes` in system variable `optimizer_switch`, which determine whether the option `INVISIBLE` takes effect. If `use_invite_indexes` is on, the optimizer can still use invisible index.++A table with no explicit primary key may still have an effective implicit primary key if it has any UNIQUE indexes on NOT NULL columns. In this case, the first such index places the same constraint on table rows as an explicit primary key and that index cannot be made invisible.++## Rationale++Another solution for implement invisible indexes is: Indicate invisibility by DDL state `WriteOnly`. This solution has the following problems:++- The logic of schema change needs to be changed, which is relatively complicated to implement.+- Cannot distinguish between indexes that are currently in WriteOnly and invisible indexes.+- Handling the switch is troublesome.++## Compatibility and Migration Plan++This the a new feature and it's absolutly compatible with old TiDB versions, also, it's not impact any data migration.+The syntax and functions are basically compatible with MySQL. Expect one:++	When use invisible index in `SQL Hint`, and set `use_invisible_indexes = false`, MySQL allow use the invisible index.+	But in TiDB, It's **not allowed**, and will throw a `Unresolved name` error, Because the behavior is more reasonable.
	But in TiDB, It's **not allowed** and an `Unresolved name` error will be thrown.
Deardrops

comment created time in 10 days

Pull request review commentpingcap/tidb

docs/design: add proposal for invisible index

+# Proposal: Invisible index++- Author(s):     [Deardrops](https://github.com/Deardrops), [wjhuang2016](https://github.com/wjhuang2016)+- Last updated:  Mar. 12, 2020+- Discussion at: https://github.com/pingcap/tidb/issues/9246++## Abstract++MySQL supports [Invisible indexes](https://dev.mysql.com/doc/refman/8.0/en/invisible-indexes.html); that is, indexes that are not used by the optimizer. ++This is a useful feature for dropping an index in a safe way. Invisible indexes make it possible to test the effect of removing an index on query performance, without making a destructive change that must be undone should the index turn out to be required. Dropping and re-adding an index can be expensive for a large table, whereas making it invisible and visible are fast, in-place operations.++Support the option of `VISIBLE | INVISIBLE`.+```+CREATE [...] INDEX index_name+    [index_type]+    ON tbl_name (key_part,...)+    [index_option]++index_option:+    {VISIBLE | INVISIBLE}+```++Also, consider the following:++- Display information about invisible indexes in the output of `INFORMATION_SCHEMA.STATISTICS` or `SHOW INDEX`.+- Index hints need to report errors when used in invisible indexes.+- The primary key cannot be set to an invisible index.++## Background++Index has a great influence on the performance of the database. Whether there is an index or whether the optimizer chooses the right index determines the performance of reading and writing the database to a great extent. In some cases, we would like to know the impact of deleting an index on the read-write performance of the database.  At present, our method is to tell the optimizer to ignore the index through Index Hint.  Although this method can achieve the goal, it is not practical to modify all SQL statements.++## Proposal++Adding an option (visible or not) to the index. If it is not visible, it's called **Invisible Index**. Invisible Index cannot be used by the optimizer (with the `use_invisible_indexes` switch on), but the index is maintained during DML operations. For a query statement, invisible index has the same effect as ignoring the index through Index Hint.++The option `INVISIBLE` of an index can be changed by using the following DDL statement:++```+ALTER TABLE table_name ALTER INDEX index_name { INVISIBLE | VISIBLE };+```++Or set option `INVISIBLE` when creating an index:++```+CREATE INDEX index_name ON table_name(key) [ INVISIBLE | VISIBLE ];+```++`INVISIBLE`, `VISIBLE` is as part of the index option, they can also be set when creating a table.++In order to know whether an index is invisible, it can be read from the `INFORMATION_SCHEMA.STATISTICS` table or through the `SHOW INDEX` command.++In addition, add a new flag `use_invisible_indexes` in system variable `optimizer_switch`, which determine whether the option `INVISIBLE` takes effect. If `use_invite_indexes` is on, the optimizer can still use invisible index.++A table with no explicit primary key may still have an effective implicit primary key if it has any UNIQUE indexes on NOT NULL columns. In this case, the first such index places the same constraint on table rows as an explicit primary key and that index cannot be made invisible.++## Rationale++Another solution for implement invisible indexes is: Indicate invisibility by DDL state `WriteOnly`. This solution has the following problems:++- The logic of schema change needs to be changed, which is relatively complicated to implement.+- Cannot distinguish between indexes that are currently in WriteOnly and invisible indexes.+- Handling the switch is troublesome.++## Compatibility and Migration Plan++This the a new feature and it's absolutly compatible with old TiDB versions, also, it's not impact any data migration.+The syntax and functions are basically compatible with MySQL. Expect one:
The syntax and functions are basically compatible with MySQL expect:
Deardrops

comment created time in 10 days

Pull request review commentpingcap/tidb

docs/design: add proposal for invisible index

+# Proposal: Invisible index++- Author(s):     [Deardrops](https://github.com/Deardrops), [wjhuang2016](https://github.com/wjhuang2016)+- Last updated:  Mar. 12, 2020+- Discussion at: https://github.com/pingcap/tidb/issues/9246++## Abstract++MySQL supports [Invisible indexes](https://dev.mysql.com/doc/refman/8.0/en/invisible-indexes.html); that is, indexes that are not used by the optimizer. ++This is a useful feature for dropping an index in a safe way. Invisible indexes make it possible to test the effect of removing an index on query performance, without making a destructive change that must be undone should the index turn out to be required. Dropping and re-adding an index can be expensive for a large table, whereas making it invisible and visible are fast, in-place operations.++Support the option of `VISIBLE | INVISIBLE`.+```+CREATE [...] INDEX index_name+    [index_type]+    ON tbl_name (key_part,...)+    [index_option]++index_option:+    {VISIBLE | INVISIBLE}+```++Also, consider the following:++- Display information about invisible indexes in the output of `INFORMATION_SCHEMA.STATISTICS` or `SHOW INDEX`.+- Index hints need to report errors when used in invisible indexes.+- The primary key cannot be set to an invisible index.++## Background++Index has a great influence on the performance of the database. Whether there is an index or whether the optimizer chooses the right index determines the performance of reading and writing the database to a great extent. In some cases, we would like to know the impact of deleting an index on the read-write performance of the database.  At present, our method is to tell the optimizer to ignore the index through Index Hint.  Although this method can achieve the goal, it is not practical to modify all SQL statements.++## Proposal++Adding an option (visible or not) to the index. If it is not visible, it's called **Invisible Index**. Invisible Index cannot be used by the optimizer (with the `use_invisible_indexes` switch on), but the index is maintained during DML operations. For a query statement, invisible index has the same effect as ignoring the index through Index Hint.++The option `INVISIBLE` of an index can be changed by using the following DDL statement:++```+ALTER TABLE table_name ALTER INDEX index_name { INVISIBLE | VISIBLE };+```++Or set option `INVISIBLE` when creating an index:++```+CREATE INDEX index_name ON table_name(key) [ INVISIBLE | VISIBLE ];+```++`INVISIBLE`, `VISIBLE` is as part of the index option, they can also be set when creating a table.++In order to know whether an index is invisible, it can be read from the `INFORMATION_SCHEMA.STATISTICS` table or through the `SHOW INDEX` command.++In addition, add a new flag `use_invisible_indexes` in system variable `optimizer_switch`, which determine whether the option `INVISIBLE` takes effect. If `use_invite_indexes` is on, the optimizer can still use invisible index.++A table with no explicit primary key may still have an effective implicit primary key if it has any UNIQUE indexes on NOT NULL columns. In this case, the first such index places the same constraint on table rows as an explicit primary key and that index cannot be made invisible.++## Rationale++Another solution for implement invisible indexes is: Indicate invisibility by DDL state `WriteOnly`. This solution has the following problems:++- The logic of schema change needs to be changed, which is relatively complicated to implement.+- Cannot distinguish between indexes that are currently in WriteOnly and invisible indexes.+- Handling the switch is troublesome.++## Compatibility and Migration Plan++This the a new feature and it's absolutly compatible with old TiDB versions, also, it's not impact any data migration.
This the a new feature and it's absolutly compatible with old TiDB versions, also, it does not impact any data migration.
Deardrops

comment created time in 10 days

Pull request review commentpingcap/tidb

docs/design: add proposal for invisible index

+# Proposal: Invisible index++- Author(s):     [Deardrops](https://github.com/Deardrops), [wjhuang2016](https://github.com/wjhuang2016)+- Last updated:  Mar. 12, 2020+- Discussion at: https://github.com/pingcap/tidb/issues/9246++## Abstract++MySQL supports [Invisible indexes](https://dev.mysql.com/doc/refman/8.0/en/invisible-indexes.html); that is, indexes that are not used by the optimizer. ++This is a useful feature for dropping an index in a safe way. Invisible indexes make it possible to test the effect of removing an index on query performance, without making a destructive change that must be undone should the index turn out to be required. Dropping and re-adding an index can be expensive for a large table, whereas making it invisible and visible are fast, in-place operations.++Support the option of `VISIBLE | INVISIBLE`.+```+CREATE [...] INDEX index_name+    [index_type]+    ON tbl_name (key_part,...)+    [index_option]++index_option:+    {VISIBLE | INVISIBLE}+```++Also, consider the following:++- Display information about invisible indexes in the output of `INFORMATION_SCHEMA.STATISTICS` or `SHOW INDEX`.+- Index hints need to report errors when used in invisible indexes.+- The primary key cannot be set to an invisible index.++## Background++Index has a great influence on the performance of the database. Whether there is an index or whether the optimizer chooses the right index determines the performance of reading and writing the database to a great extent. In some cases, we would like to know the impact of deleting an index on the read-write performance of the database.  At present, our method is to tell the optimizer to ignore the index through Index Hint.  Although this method can achieve the goal, it is not practical to modify all SQL statements.++## Proposal++Adding an option (visible or not) to the index. If it is not visible, it's called **Invisible Index**. Invisible Index cannot be used by the optimizer (with the `use_invisible_indexes` switch on), but the index is maintained during DML operations. For a query statement, invisible index has the same effect as ignoring the index through Index Hint.++The option `INVISIBLE` of an index can be changed by using the following DDL statement:++```+ALTER TABLE table_name ALTER INDEX index_name { INVISIBLE | VISIBLE };+```++Or set option `INVISIBLE` when creating an index:
Or by setting option `INVISIBLE` when creating an index:
Deardrops

comment created time in 10 days

Pull request review commentpingcap/tidb

docs/design: add proposal for invisible index

+# Proposal: Invisible index++- Author(s):     [Deardrops](https://github.com/Deardrops), [wjhuang2016](https://github.com/wjhuang2016)+- Last updated:  Mar. 12, 2020+- Discussion at: https://github.com/pingcap/tidb/issues/9246++## Abstract++MySQL supports [Invisible indexes](https://dev.mysql.com/doc/refman/8.0/en/invisible-indexes.html); that is, indexes that are not used by the optimizer. 
MySQL supports [invisible indexes](https://dev.mysql.com/doc/refman/8.0/en/invisible-indexes.html); that is, indexes that are not used by the optimizer. 
Deardrops

comment created time in 10 days

Pull request review commentpingcap/tidb

docs/design: add proposal for invisible index

+# Proposal: Invisible index++- Author(s):     [Deardrops](https://github.com/Deardrops), [wjhuang2016](https://github.com/wjhuang2016)+- Last updated:  Mar. 12, 2020+- Discussion at: https://github.com/pingcap/tidb/issues/9246++## Abstract++MySQL supports [Invisible indexes](https://dev.mysql.com/doc/refman/8.0/en/invisible-indexes.html); that is, indexes that are not used by the optimizer. ++This is a useful feature for dropping an index in a safe way. Invisible indexes make it possible to test the effect of removing an index on query performance, without making a destructive change that must be undone should the index turn out to be required. Dropping and re-adding an index can be expensive for a large table, whereas making it invisible and visible are fast, in-place operations.++Support the option of `VISIBLE | INVISIBLE`.+```+CREATE [...] INDEX index_name+    [index_type]+    ON tbl_name (key_part,...)+    [index_option]++index_option:+    {VISIBLE | INVISIBLE}+```++Also, consider the following:
Also, the following behaviors need to be supported as well:
Deardrops

comment created time in 10 days

Pull request review commentpingcap/tidb

docs/design: add proposal for invisible index

+# Proposal: Invisible index++- Author(s):     [Deardrops](https://github.com/Deardrops), [wjhuang2016](https://github.com/wjhuang2016)+- Last updated:  Mar. 12, 2020+- Discussion at: https://github.com/pingcap/tidb/issues/9246++## Abstract++MySQL supports [Invisible indexes](https://dev.mysql.com/doc/refman/8.0/en/invisible-indexes.html); that is, indexes that are not used by the optimizer. ++This is a useful feature for dropping an index in a safe way. Invisible indexes make it possible to test the effect of removing an index on query performance, without making a destructive change that must be undone should the index turn out to be required. Dropping and re-adding an index can be expensive for a large table, whereas making it invisible and visible are fast, in-place operations.++Support the option of `VISIBLE | INVISIBLE`.+```+CREATE [...] INDEX index_name+    [index_type]+    ON tbl_name (key_part,...)+    [index_option]++index_option:+    {VISIBLE | INVISIBLE}+```++Also, consider the following:++- Display information about invisible indexes in the output of `INFORMATION_SCHEMA.STATISTICS` or `SHOW INDEX`.+- Index hints need to report errors when used in invisible indexes.+- The primary key cannot be set to an invisible index.++## Background++Index has a great influence on the performance of the database. Whether there is an index or whether the optimizer chooses the right index determines the performance of reading and writing the database to a great extent. In some cases, we would like to know the impact of deleting an index on the read-write performance of the database.  At present, our method is to tell the optimizer to ignore the index through Index Hint.  Although this method can achieve the goal, it is not practical to modify all SQL statements.++## Proposal++Adding an option (visible or not) to the index. If it is not visible, it's called **Invisible Index**. Invisible Index cannot be used by the optimizer (with the `use_invisible_indexes` switch on), but the index is maintained during DML operations. For a query statement, invisible index has the same effect as ignoring the index through Index Hint.
Adding an option (visible or not) to the index. If it is not visible, it's called **Invisible Index**. Invisible indexes cannot be used by the optimizer (with the `use_invisible_indexes` switch on), but the index is maintained during DML operations. For a query statement, invisible indexes have the same effect as ignoring the index through Index Hint.
Deardrops

comment created time in 10 days

PR opened pingcap/tidb

util/collate: remove `CollatorOption` component/util

What problem does this PR solve?

Problem Summary:

What is changed and how it works?

What's Changed: CollatorOption was used to specify the PadLen which aims to support "PADDING" attribute of the collations. But now it is not needed anymore, since TiDB actually implements "PADDING" by removing trailing spaces.

How it Works: Simply remove CollatorOption.

Related changes

Not needed.

Check List <!--REMOVE the items that are not applicable-->

Tests <!-- At least one of them must be included. -->

  • Unit test
  • Integration test

Side effects

  • NA

Release note <!-- bugfixes or new feature need a release note -->

  • NA
+18 -29

0 comment

7 changed files

pr created time in 10 days

create barnchbb7133/tidb

branch : bb7133/remove_pad_len

created branch time in 10 days

created tagpingcap/tidb

tagv4.0.0-beta.2

TiDB is an open source distributed HTAP database compatible with the MySQL protocol

created time in 10 days

release pingcap/tidb

v4.0.0-beta.2

released time in 10 days

push eventpingcap/tidb

Lynn

commit sha f8b2d9676492aabf5226cf44cb552298e5632c86

server: if status address already in use, return an error (#15177)

view details

push time in 10 days

PR merged pingcap/tidb

server: if status address already in use, return an error component/server needs-cherry-pick-3.0 status/LGT2 status/can-merge type/enhancement

<!-- Thank you for contributing to TiDB! Please read TiDB's CONTRIBUTING document BEFORE filing this PR. -->

What problem does this PR solve? <!--add issue link with summary if exists-->

If the status address already in use, we only log it. We don't know why the status server is unavailable except by looking at the logs. If we don't start the status server, we can set report-status to false.

What is changed and how it works?

If the status address already in use, we return an error and interrupt the service.

Check List <!--REMOVE the items that are not applicable-->

Tests <!-- At least one of them must be included. -->

  • Unit test

Related changes

  • Need to cherry-pick to the release branch

Release note

  • If the status address already in use, return an error
+63 -36

13 comments

3 changed files

zimulala

pr closed time in 10 days

pull request commentpingcap/tidb

server: if status address already in use, return an error

/run-unit-test /run-integration-common-test

zimulala

comment created time in 10 days

push eventpingcap/tidb

pingcap-github-bot

commit sha 19adfd797a67c8e4d5e301e54f83b9f22f1e5c54

privilege: add DDL and DML privilege check for system tables (#15095) (#15417) (#15445)

view details

push time in 10 days

PR merged pingcap/tidb

Reviewers
privilege: add DDL and DML privilege check for system tables (#15095) (#15417) compatibility-breaker component/infoschema component/privilege status/all-tests-passed type/3.0-cherry-pick type/3.1-cherry-pick

cherry-pick #15417 to release-3.1


cherry-pick #15095 to release-3.0 Conflicting files: tables.go privileges.go privileges_test.go


<!-- Thank you for contributing to TiDB! Please read TiDB's CONTRIBUTING document BEFORE filing this PR. -->

What problem does this PR solve? <!--add issue link with summary if exists-->

Add DDL and DML privilege check for system tables in performance_schema and metrics_schema.

What is changed and how it works?

Prevent users from executing ALTER, DROP, INDEX, INSERT, UPDATE, DELETE statements on predefined tables. Privileges of those tables which are defined in these schema by users themselves are kept untouched.

Check List <!--REMOVE the items that are not applicable-->

Tests <!-- At least one of them must be included. -->

  • Unit test
  • Manual test (add detailed scripts or steps below)
mysql> delete from events_statements_summary_by_digest;
ERROR 8121 (HY000): privilege check fail

Code changes

  • Has exported function/method change

Side effects

  • Breaking backward compatibility

Related changes

N/A

Release note

  • Forbid users to execute DDL and update/delete/insert predefined tables in performance_schema and metrics_schema.
  • Compatibility declaration: Executing DDL and update/delete/insert predefined tables in performance_schema and metrics_schema are not allowed any longer.
+56 -4

2 comments

4 changed files

sre-bot

pr closed time in 10 days

Pull request review commentpingcap/tidb

ddl: add syntax for setting the cache step of auto id explicitly.

 func onRebaseAutoID(store kv.Storage, t *meta.Meta, job *model.Job) (ver int64, 	return ver, nil } +func onModifyTableAutoIncCache(t *meta.Meta, job *model.Job) (ver int64, _ error) {
func onModifyTableAutoIDCache(t *meta.Meta, job *model.Job) (ver int64, _ error) {

Please refine the name.

AilinKid

comment created time in 11 days

Pull request review commentpingcap/tidb

ddl: add syntax for setting the cache step of auto id explicitly.

 func (d *ddl) AlterTableComment(ctx sessionctx.Context, ident ast.Ident, spec *a 	return errors.Trace(err) } +// AlterTableComment updates the table comment information.
// AlterTableComment updates the table comment information.

Please fix the comment.

AilinKid

comment created time in 11 days

pull request commentpingcap/parser

parser: add autoid cache option

LGTM, but let's hold it DNM until the TiDB PR is ready to merge.

AilinKid

comment created time in 11 days

pull request commentpingcap/parser

parser: add autoid cache option

Please address the comment and fix CI.

AilinKid

comment created time in 11 days

Pull request review commentpingcap/parser

parser: add autoid cache option

 UnReservedKeyword: |	"ADVISE" |	"ASCII" |	"AUTO_INCREMENT"+|	"AUTO_INCREMENT_CACHE"

Please rename it to AUTO_ID_CACHE, as you mentioned, it doesn't affect AutoIncrement column only.

AilinKid

comment created time in 11 days

push eventpingcap/tidb

pingcap-github-bot

commit sha 0885f6a67867de264a63aba8209569fab24c86e5

privilege: add DDL and DML privilege check for system tables (#15095) (#15417)

view details

push time in 11 days

PR merged pingcap/tidb

Reviewers
privilege: add DDL and DML privilege check for system tables (#15095) compatibility-breaker component/infoschema component/privilege status/LGT2 status/all-tests-passed type/3.0 cherry-pick

cherry-pick #15095 to release-3.0 Conflicting files: tables.go privileges.go privileges_test.go


<!-- Thank you for contributing to TiDB! Please read TiDB's CONTRIBUTING document BEFORE filing this PR. -->

What problem does this PR solve? <!--add issue link with summary if exists-->

Add DDL and DML privilege check for system tables in performance_schema and metrics_schema.

What is changed and how it works?

Prevent users from executing ALTER, DROP, INDEX, INSERT, UPDATE, DELETE statements on predefined tables. Privileges of those tables which are defined in these schema by users themselves are kept untouched.

Check List <!--REMOVE the items that are not applicable-->

Tests <!-- At least one of them must be included. -->

  • Unit test
  • Manual test (add detailed scripts or steps below)
mysql> delete from events_statements_summary_by_digest;
ERROR 8121 (HY000): privilege check fail

Code changes

  • Has exported function/method change

Side effects

  • Breaking backward compatibility

Related changes

N/A

Release note

  • Forbid users to execute DDL and update/delete/insert predefined tables in performance_schema and metrics_schema.
  • Compatibility declaration: Executing DDL and update/delete/insert predefined tables in performance_schema and metrics_schema are not allowed any longer.
+55 -4

4 comments

4 changed files

sre-bot

pr closed time in 11 days

push eventpingcap/tidb

Maxwell

commit sha 12aabbdc6729e0d7c9a196b89e75c1486f5b4b57

ddl: basic support for invisible index (#15366)

view details

push time in 11 days

PR merged pingcap/tidb

Reviewers
ddl: basic support for invisible index component/DDL component/executor status/LGT3 status/PTAL status/can-merge

<!-- Thank you for contributing to TiDB! Please read TiDB's CONTRIBUTING document BEFORE filing this PR. -->

What problem does this PR solve? <!--add issue link with summary if exists-->

Part of #9246

What is changed and how it works?

  1. Store Invisible Flag in these statements:

    • CREATE TABLE t (a INT, INDEX (a) INVISIBLE)
    • CREATE INDEX a on t(a) INVISIBLE
    • ALTER TABLE t ADD INDEX a INVISIBLE
  2. Add a new column Visible in SHOW INDEX:

mysql> show index from t;
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+------------+
| Table | Non_unique | Key_name | Seq_in_index | Column_name | Collation | Cardinality | Sub_part | Packed | Null | Index_type | Comment | Index_comment | Visible | Expression |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+------------+
| t     |          1 | a        |            1 | a           | A         |           0 |     NULL | NULL   | YES  | BTREE      |         |               | NO      | NULL       |
+-------+------------+----------+--------------+-------------+-----------+-------------+----------+--------+------+------------+---------+---------------+---------+------------+
1 row in set (0.00 sec)
  1. Add a new column IS_VISIBLE in table information_schema.statistics:
mysql> select table_name, index_name, is_visible from information_schema.statistics where table_name = "t";
+------------+------------+------------+
| table_name | index_name | is_visible |
+------------+------------+------------+
| t          | a          | NO         |
+------------+------------+------------+
1 row in set (0.00 sec)

  1. Show Invisible Index in SHOW CREATE TABLE
mysql> show create table t;
+-------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                                     |
+-------+--------------------------------------------------------------------------------------------------------------------------------------------------+
| t     | CREATE TABLE `t` (
  `a` int(11) DEFAULT NULL,
  KEY `a` (`a`) /*!80000 INVISIBLE */
) ENGINE=InnoDB DEFAULT CHARSET=utf8mb4 COLLATE=utf8mb4_bin |
+-------+--------------------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (0.00 sec)

Check List <!--REMOVE the items that are not applicable-->

Tests <!-- At least one of them must be included. -->

  • Unit test

Code changes

  • Has exported variable/fields change

Side effects

  • Increased code complexity

Related changes

  • None

Release note

  • None
+78 -22

6 comments

9 changed files

Deardrops

pr closed time in 11 days

push eventpingcap/tidb

Lonng

commit sha f7e694322c9d57ed43f5ec214c1b7b7705077b5a

Merge release 3.0 into release 3.1 (#15414)

view details

push time in 11 days

PR merged pingcap/tidb

Merge release 3.0 into release 3.1 component/DDL component/config component/executor component/planner

Unmerged paths: (use "git add <file>..." to mark resolution)

    both modified:   bindinfo/bind_test.go
    both modified:   cmd/explaintest/r/explain_easy.result
    both added:      cmd/explaintest/r/explain_union_scan.result
    both modified:   cmd/explaintest/r/partition_pruning.result
    both modified:   config/config.go
    both modified:   config/config.toml.example
    both modified:   config/config_test.go
    both modified:   ddl/ddl.go
    both modified:   ddl/ddl_api.go
    both modified:   executor/grant.go
    both modified:   executor/insert_test.go
    both modified:   executor/show.go
    both modified:   executor/simple.go
    both modified:   executor/table_reader.go
    both modified:   executor/write.go
    both added:      expression/testdata/partition_pruner_out.json
    both modified:   go.mod
    both modified:   go.sum
    both modified:   infoschema/builder.go
    both modified:   infoschema/infoschema.go
    both modified:   infoschema/perfschema/tables_test.go
    both modified:   infoschema/slow_log.go
    both modified:   infoschema/tables.go
    both modified:   infoschema/tables_test.go
    both modified:   meta/autoid/autoid.go
    both modified:   meta/autoid/autoid_test.go
    both modified:   meta/autoid/errors.go
    both modified:   planner/core/explain.go
    both modified:   planner/core/integration_test.go
    both modified:   planner/core/planbuilder.go
    both added:      planner/core/testdata/plan_normalized_suite_out.json
    both modified:   planner/core/testdata/plan_suite_out.json
    both modified:   privilege/privileges/cache.go
    both modified:   privilege/privileges/cache_test.go
    both modified:   server/http_handler.go
    both modified:   server/server.go
    both modified:   session/session_test.go
    both modified:   sessionctx/variable/varsutil.go
    both modified:   sessionctx/variable/varsutil_test.go
    both modified:   store/mockstore/mocktikv/mvcc_leveldb.go
    both modified:   store/tikv/2pc.go
    both modified:   store/tikv/region_request.go
    both modified:   store/tikv/txn.go
    both modified:   table/table.go
    both modified:   table/tables/partition_test.go
    both modified:   table/tables/tables.go
    both modified:   table/tables/tables_test.go
    both modified:   util/execdetails/execdetails.go
    both modified:   util/misc.go
    both modified:   util/mock/context.go
    both modified:   util/stmtsummary/statement_summary.go
+2233 -858

4 comments

104 changed files

lonng

pr closed time in 11 days

PR merged pingcap/kvproto

Merge release-3.0 into release-3.1

Unmerged paths: (use "git add <file>..." to mark resolution)

    both modified:   pkg/coprocessor/coprocessor.pb.go
    both modified:   pkg/import_sstpb/import_sstpb.pb.go
    both modified:   pkg/metapb/metapb.pb.go
    both modified:   pkg/pdpb/pdpb.pb.go
+1 -1

1 comment

1 changed file

lonng

pr closed time in 12 days

push eventpingcap/kvproto

Lonng

commit sha 2838e21ca2227d42b74f235a702669700e679765

Merge release-3.0 into release-3.1 (#578)

view details

push time in 12 days

pull request commentpingcap/kvproto

Merge release-3.0 into release-3.1

LGTM

lonng

comment created time in 12 days

push eventpingcap/parser

Lonng

commit sha 9ebea32e03a66dd580def63d793f24ac265752d2

Merge release-3.0 into release-3.1 (#776)

view details

push time in 12 days

PR merged pingcap/parser

Merge release-3.0 into release-3.1

Unmerged paths: (use "git add <file>..." to mark resolution)

    both modified:   ast/ddl.go
    both modified:   mysql/errcode.go
    both modified:   mysql/errname.go
    both modified:   parser.go
    both modified:   parser.y
    both modified:   parser_test.go
+7215 -7056

2 comments

7 changed files

lonng

pr closed time in 12 days

pull request commentpingcap/parser

Merge release-3.0 into release-3.1

LGTM

lonng

comment created time in 12 days

pull request commentpingcap/tidb

Merge release 3.0 into release 3.1

/run-unit-test /run-check_dev

lonng

comment created time in 12 days

create barnchbb7133/tidb

branch : bb7133/webank-3.0.5-hotfix

created branch time in 14 days

PR opened pingcap-incubator/tidb-in-action

session1/chapter7: update autorandom.md

update autorandom.md

+5 -10

0 comment

1 changed file

pr created time in 14 days

more