write_to_gpfdist_timeout controls timeout value (in seconds) for writing data to gpfdist server. Default value is 300, valid scope is [1, 7200]

Set CURLOPT_TIMEOUT as write_to_gpfdist_timeout
For any error, retry with double interval time, returns SQL ERROR if write_to_gpfdist_timeout is reached

Add regression test for GUC writable_external_table_timeout

(cherry picked from commit ab73713211e93a95e696429f4821e893ee3397de)

Here are some reminders before you submit the pull request

• [ ] Add tests for the change
• [ ] Document changes
• [ ] Communicate in the mailing list if needed
• [ ] Pass make installcheck
• [ ] Review a PR in return to support the community

The REFRESH MATERIALIZED used to count pgstat on QD. But it always result to 0 since the es_processed are not colleted from segments. In GPDB, we should count the pgstat on segments and union them on QD, so both the segments and coordinator will have pgstat for this relation. This fix issue: https://github.com/greenplum-db/gpdb/issues/11375

Here are some reminders before you submit the pull request

• [ ] Add tests for the change
• [ ] Document changes
• [ ] Communicate in the mailing list if needed
• [ ] Pass make installcheck
• [ ] Review a PR in return to support the community

CI failed with the patch (not flaky one). FYI.

That's because the fts_error test left an orphan temp table, vacuumdb didn't work when in this case, I created a PR to fix that first

In matview, intoClause->viewQuery is not null.

We have already considered IsBinaryUpgrade in GetNewOrPreassignedOid, so there is no need to deal with this variable in index_create.

Here are some reminders before you submit the pull request

• [ ] Add tests for the change
• [ ] Document changes
• [ ] Communicate in the mailing list if needed
• [ ] Pass make installcheck
• [ ] Review a PR in return to support the community

I think we can introduce a template engine to generate regress tests based on the regress test template. The content of regress test template as follow(based on Liquid):

select n_tup_ins, n_tup_upd, n_tup_del, n_tup_hot_upd, n_live_tup, n_dead_tup, n_mod_since_analyze from pg_stat_all_tables where relid = 'table_for_truncate_abort'::regclass;
{% if 'replicated' in amit_conf_distby %}
{% comment %}
ORCA can't be used in the replicated table, the update of table_for_iud will not be implemented by delete + insert
{% endcomment %}
1600|301|777|0|777|1124|777
{% else %}
1901|0|1078|0|777|1124|777
{% endif %}
{% if 'replicated' not in amit_conf_distby %}
{% comment %}
ERROR:  PARTITION BY clause cannot be used with DISTRIBUTED REPLICATED clause
So don't do the following test in DISTRIBUTED REPLICATED
{% endcomment %}
-- Test pgstat table stat for partition table on QD
CREATE TABLE rankpart (id int, rank int, product int)
with ({{amit_conf_opts}})
{{amit_conf_distby}} PARTITION BY RANGE (rank)
( START (1) END (10) EVERY (5),
  DEFAULT PARTITION extra );
...
{% endif %}

amit_conf_distby will be replaced by appendonly=false, appendonly=true,orientation=row, appendonly=true,orientation=column, and amit_conf_distby will be replaced by distributed randomly, distributed replicated, ''(an empty string which means distributed by). So that we can test pgstat_qd_tabstat on heap/ao_row/ao_column and/or distributed by/distributed replicated/distributed randomly.

test pgstat_qd_tabstat-heap_hash_dist ... ok (83.60 sec)  (diff:0.13 sec)
test pgstat_qd_tabstat-heap_rand_dist ... ok (83.65 sec)  (diff:0.14 sec)
test pgstat_qd_tabstat-heap_replicated ... ok (63.62 sec)  (diff:0.13 sec)
test pgstat_qd_tabstat-aoro_hash_dist ... ok (84.45 sec)  (diff:0.13 sec)
test pgstat_qd_tabstat-aoro_rand_dist ... ok (84.58 sec)  (diff:0.14 sec)
test pgstat_qd_tabstat-aoro_replicated ... ok (64.32 sec)  (diff:0.13 sec)
test pgstat_qd_tabstat-aoco_hash_dist ... ok (86.36 sec)  (diff:0.13 sec)
test pgstat_qd_tabstat-aoco_rand_dist ... ok (86.47 sec)  (diff:0.14 sec)
test pgstat_qd_tabstat-aoco_replicated ... or (65.70 sec)  (diff:0.25 sec)

Let's just merge the current fix for now. For the test template, feel free to open a new PR so we can have more people review it.

It doesn't need to drop, without this commit, it dumped as unknown::cstring::date, with this commit it dumps as unknown::text::date.

@asimrp We do the dispatch here and the es_processed is 0 before we fetch the total processed across segments. And pg don't do dispatch, it keeps increase es_processed.

updated

Since the test takes ~35 seconds, I added a "table driven" test.

I would like to ask the team to keep in mind the overhead of adding tests such as run time as we add more.

updated to return a new list called unreachable_hosts

As in 7X I think AI_PASSIVE should be removed here.

The L81/L123 braces are an unnecessary additional code block. Please remove them.

Instead of flipping the role based on some operation I feel its much cleaner expectation to remember, code and test. Coordinator has it in dispatch mode and segments in utility mode.

https://github.com/greenplum-db/gpdb/pull/11350/files#r562979054

remove the original socket creation.

see https://github.com/greenplum-db/gpdb/pull/11350/files#r562979054...the old socket creation code needs to be removed also.

(ignore for 7X) creating a comment here for reference in the #11355 issue.

This line should be:

	if (rc != 0)
		rc = setupListen("0.0.0.0", serverPort, AF_INET);

super nit: add a newline here.

Thinking more, if there is genuine inconsistency pg_trigger, pg_index or pg_rules table will complain about inconsistency across nodes. Hence, I feel we can safely avoid checking consistency for these variables for pg_class across coordinator and segments. Locally on each postgres instance consistency of pg_trigger with relhastriggers pg_class value should be performed anyways.

Why is gpcheckcat failing in PR pipeline? The failures look legit.

The reason for the failure is relhastriggers=true flag is not cleared from pg_class when triggers are dropped for the table, reference comment in RemoveTriggerById().

    /*                                                                                                                                   
     * We do not bother to try to determine whether any other triggers remain,                                                           
     * which would be needed in order to decide whether it's safe to clear the                                                           
     * relation's relhastriggers.  (In any case, there might be a concurrent                                                             
     * process adding new triggers.)  Instead, just force a relcache inval to                                                            
     * make other backends (and this one too!) rebuild their relcache entries.                                                           
     * There's no great harm in leaving relhastriggers true even if there are                                                            
     * no triggers left.                                                                                                                 
     */                                                                                                                                  

Instead, vac_update_relstats() invoked during vacuum update these fields in pg_class along with updating relfrozenxid to reflect the reality.

Hence, tables for which trigger was added and then removed, only on nodes vacuum is triggered relhastriggers will reflect the reality. As we are auto-vacuuming only content 0 and coordinator and not other contents, gpcheckcat reports miss-match. Though this is harmless miss-match as in-reality no triggers are present for the table.

relhasindex and relhasrules also fall in same bucket. True doesn't mean index, rule or trigger is for sure present they may be dropped. Only false means for sure these objects are not present for the table.

I am thinking ideally, gpcheckcat shouldn't be checking for these fields consistency across coordinator/segments as they can differ. Question is how to detect genuine miss-match. Maybe need to add little more sophisticated mechanism like if mismatch is found then check if really the underlying object is present on nodes which reported value as true, if not then its not really inconsistency. No idea how easy or hard it would be to add such sophistication to gpcheckcat. Or its okay to not check for these fields consistency and done.

Add support for ipv6 for running gpnetbenchClient and gpnetbenchServer

Here are some reminders before you submit the pull request

• [ ] Add tests for the change
• [ ] Document changes
• [ ] Communicate in the mailing list if needed
• [ ] Pass make installcheck
• [ ] Review a PR in return to support the community

Rather than hard coding DEFAULT_LOCALE_SETTING to en_US.utf8 on linux, which is not available on all platforms, use locale -a output.

Pipeline

What if instead we let initdb pick the default locale (as it already has logic for it) for coordinator and then use the same for all segment initdb's (this should reduce the intelligence in the management scripts) ?

That's a good suggestion, and we considered making a larger change like that, but at the moment we're trying to make minimally-impactful changes to the utilities so we can focus on gpupgrade, but we'll keep that in mind when we get to rewriting gpinitsystem as part of the cloud readiness effort.

This trips up Clang Tidy, both locally and in Travis CI, after greenplum-db/gpdb#11336 . To quickly turn Travis CI green on master, let's remove the 3 offending files. We can have a longer discussion about whether we should remove all tests that are currently not compiled, and whether we should fail unit tests when the test names themselves are invalid.

Question for reviewers:

1. Should I remove only the 3 files that offend clang-tidy, or all of the test .cpp files that are not built in CMake?
2. If "yes" to the previous question, should I also fix the defect where we let a test invocation "succeed" even if there's no such a test? c.f. #11403

Here are some reminders before you submit the pull request

• [ ] Add tests for the change
• [ ] Document changes
• [ ] Communicate in the mailing list if needed
• [ ] Pass make installcheck
• [ ] Review a PR in return to support the community