Datastax DSBulk Utility giving errors on load CSV data to Astra

Question

I am migrating data from EC2 Cassandra Nodes to DataStax Astra (Premium Account) using DSBulk utility.

Command used: dsbulk load -url folder_created_during_unload -header true -k keyspace -t table -b "secure-connect-file.zip" -u username -p password

This command gives error after a few seconds. On checking the documentation, i found that i can add --executor.maxPerSecond in this command to limit the loading.

After this, the load command executed without any error. But if i enter a value over 15,000, the load command starts giving the error again.

Now, if a table has over 100M entries and 15,000 entries are migrated every second, it would hours and hours to complete the migration of one table. The complete database would take several days to migrate.

I want to understand what is causing this error and if there is a way to load the data at a higher speed.

Answer 1

To add to Aaron's response, you are hitting the default limit of 4K operations per second on your Astra DB.

We contacted you directly last week when we detected that you were hitting the limit but haven't heard back. I've reached out to you directly again today to let you know that I've logged a request on your behalf to increase the limit on your DB. Cheers!

Answer 2

What's happening here, is that DSBulk is running into the rate limit on the database. At the moment, it looks like the only way to increase that rate limit is to submit a ticket to support.

To submit a ticket, look for the "Other Resources" section of the Astra Dashboard's left nav. Click "Get Support" on the bottom.

When the "Help Center" pops up, click "Create Request" in the lower right corner.

On the next page, click the green/cyan "Submit a Ticket" button in the upper right corner. Describe the problem you're having (rate limit) along with what DSBulk outputs when set for more than 15k/sec.