Gsutil error "Caught socket error, retrying: timed out "

Question

I am facing this problem since a couple of days now.
I have installed Gcloud sdk on a Fedora 34 VM, built with Vagrant.
I have executed gcloud init, gcloud auth login and gcloud config set project <my_project>.
In the last couple of days, I was not able to use gsutil properly. I have found bits of reference in this post: Gsutil Always Time Out, suggesting to use the option -D for debugging gsutil.

Before using gsutil, I set the bucket name variable and execute the command to authorize the service account "<service_account>@<my_project>.iam.gserviceaccount.com":

export GCS_BUCKET_URL="gs://<my_project>-<string>/<bucket_subname>/"
gcloud auth activate-service-account --key-file=/home/<myuser>/DEV/auth/credentials.json

When I execute "gsutil -D cp ./gcp-config "${GCS_BUCKET_URL}/data-${date_time}/gcp-config", I do see:

gsutil version: 5.4
checksum: <checksum> (OK)
boto version: 2.49.0
python version: 3.9.7 (default, Aug 30 2021, 00:00:00) [GCC 11.2.1 20210728 (Red Hat 11.2.1-1)]
OS: Linux 5.14.13-200.fc34.x86_64
multiprocessing available: True
using cloud sdk: True
pass cloud sdk credentials to gsutil: True
config path(s): /home/<myuser>/.boto, /home/<myuser>/.config/gcloud/legacy_credentials/<gserviceaccount.dir>/.boto
gsutil path: /home/<myuser>/google-cloud-sdk/bin/gsutil
compiled crcmod: False
installed via package manager: False
editable install: False
Command being run: /home/<myuser>/google-cloud-sdk/platform/gsutil/gsutil -o GSUtil:default_project_id=<my_project> -D cp ./gcp-config gs://<my_project>-<string>/<bucket_subname>/data-211027082549/gcp-config
config_file_list: ['/home/<myuser>/.boto', '/home/<myuser>/.config/gcloud/legacy_credentials/<service_account>@<my_project>.iam.gserviceaccount.com/.boto']
config: [('working_dir', '/mnt/pyami'), ('debug', '0'), ('https_validate_certificates', 'True'), ('working_dir', '/mnt/pyami'), ('debug', '0'), ('content_language', 'en'), ('default_api_version', '2'), ('default_project_id', '<my_project>')]
DEBUG 1027 08:35:42.782979 multiprocess_file_storage.py] Read credential file
DEBUG 1027 08:35:42.783419 multiprocess_file_storage.py] Read credential file
INFO 1027 08:35:42.784689 base_api.py] Calling method storage.objects.list with StorageObjectsListRequest: <StorageObjectsListRequest
 bucket: '<my_project>-<string>'
 delimiter: '/'
 maxResults: 1000
 prefix: '<bucket_subname>/data-211027082549/gcp-config'
 projection: ProjectionValueValuesEnum(noAcl, 1)>
INFO 1027 08:35:42.785718 base_api.py] Making http GET to https://storage.googleapis.com/storage/v1/b/<my_project>-<string>/o?alt=json&fields=nextPageToken%2Cprefixes%2Citems%2Fname&delimiter=%2F&maxResults=1000&prefix=<bucket_subname>%2Fdata-211027082549%2Fgcp-config&projection=noAcl
INFO 1027 08:35:42.786104 base_api.py] Headers: {'accept': 'application/json',
 'accept-encoding': 'gzip, deflate',
 'content-length': '0',
 'user-agent': 'apitools Python/3.9.7 gsutil/5.4 (linux) analytics/enabled '
               'interactive/True command/cp google-cloud-sdk/362.0.0'}
INFO 1027 08:35:42.786389 base_api.py] Body: (none)
INFO 1027 08:35:42.786540 transport.py] Attempting refresh to obtain initial access_token
DEBUG 1027 08:35:42.786998 multiprocess_file_storage.py] Read credential file
DEBUG 1027 08:35:42.787312 multiprocess_file_storage.py] Read credential file
DEBUG 1027 08:35:42.790359 crypt.py] [<crypt_data>]
INFO 1027 08:35:42.790598 client.py] Refreshing access_token
INFO 1027 08:36:42.926645 retry_util.py] Retrying request, attempt #1...
DEBUG 1027 08:36:42.927215 http_wrapper.py] Caught socket error, retrying: timed out
DEBUG 1027 08:36:42.927584 http_wrapper.py] Retrying request to url https://storage.googleapis.com/storage/v1/b/<my_project>-<string>/o?alt=json&fields=nextPageToken%2Cprefixes%2Citems%2Fname&delimiter=%2F&maxResults=1000&prefix=<bucket_subname>%2Fdata-211027082549%2Fgcp-config&projection=noAcl after exception timed out
INFO 1027 08:36:44.856812 transport.py] Attempting refresh to obtain initial access_token
DEBUG 1027 08:36:44.858911 multiprocess_file_storage.py] Read credential file
DEBUG 1027 08:36:44.859711 multiprocess_file_storage.py] Read credential file
DEBUG 1027 08:36:44.862151 crypt.py] [<crypt_data>]
INFO 1027 08:36:44.862644 client.py] Refreshing access_token
INFO 1027 08:37:44.899579 retry_util.py] Retrying request, attempt #2...
DEBUG 1027 08:37:44.903247 http_wrapper.py] Caught socket error, retrying: timed out
DEBUG 1027 08:37:44.905944 http_wrapper.py] Retrying request to url https://storage.googleapis.com/storage/v1/b/<my_project>-<string>/o?alt=json&fields=nextPageToken%2Cprefixes%2Citems%2Fname&delimiter=%2F&maxResults=1000&prefix=<bucket_subname>%2Fdata-211027082549%2Fgcp-config&projection=noAcl after exception timed out
INFO 1027 08:37:49.982855 transport.py] Attempting refresh to obtain initial access_token
DEBUG 1027 08:37:49.983732 multiprocess_file_storage.py] Read credential file
DEBUG 1027 08:37:49.984001 multiprocess_file_storage.py] Read credential file
DEBUG 1027 08:37:49.985360 crypt.py] [<crypt_data>]
INFO 1027 08:37:49.985699 client.py] Refreshing access_token
INFO 1027 08:38:50.090714 retry_util.py] Retrying request, attempt #3...

This is repeated for many retries.
I've checked the credentials files (.boto files) and they look good.
The service account "<service_account>@<my_project>.iam.gserviceaccount.com" has permissions "Security admin" and "Storage admin" on the bucket gs://<my_project>-.
I have tried to re-install Gcloud SDK, it didn't help the situation.
From this answer it seems gsutil on my vm is facing an issue with a proxy.
So I've also tried the unsafe option to change security level on my router:

1. Disabled Firewall
2. Enabled WAN ping (following this topic)
   I still have the same behaviour.

At GoogleDocs/GoogleSDK install I read: "Cloud SDK requires Python; supported versions are Python 3 (preferred, 3.5 to 3.8) and Python 2 (2.7.9 or higher)."
Since my python version is 3.9.7, I wonder if this is the problem.

Any suggestions would be helpful here. Thanks in advance!

UPDATE
I've changed python version, installing 3.6.15, but it didn't solve the problem.

Answer 1

Finally, this worked for me. With a Vagrant-managed vm, eth0 is the NAT interface that Vagrant use as device to perform remote commands. The second interface, which in my case is eth1, is for personal use. my eth1 is bridged. I've tried this:

1. eth1->settings->IPv4
2. changed IPv4 method from 'Automatic (DHCP)' to 'Manual'
3. set static ip with correct netmask and gateway
4. reload eth1 settings (switch off/switch on the eth1)
5. tried gsutil as described above.

Well, it worked fine.

I really do hope this answer can help others, I just spent approximately 8 working hours on this.

Cheers~!