Apache Iceberg fast forward fails

Question

I'm trying to fast_forward my iceberg branch but it fails with:

pyspark.errors.exceptions.captured.IllegalArgumentException: Cannot fast-forward: main is not an ancestor of mywap

1.First I'm creating mywap branch via

ALTER TABLE `test_table` CREATE OR REPLACE BRANCH `mywap` RETAIN 2 DAYS

And then I do fast-forward:

CALL dev.system.fast_forward(table => "test_table", branch => "main", to => "mywap")

And I get the aforementioned error.

2.As a workaround I've tried explicitly provide ancestor:

main_id = spark.sql(f'select snapshot_id from test_table.refs where name = "main" and type="BRANCH"').first()['snapshot_id']
spark.sql(f'ALTER TABLE test_table CREATE OR REPLACE BRANCH `mywap` as of VERSION {main_id} RETAIN 2 DAYS')

But the code fails with the same error. Any ideas would be appreciated)

I'm using pyspark with iceberg-spark-runtime-3.5_2.12-1.5.1

Answer 1

I had the same problem. My problem is because I inserted data into table then I created audit_branch. Now, a snapshot_id is like this

+------------+------+-------------------+-----------------------+---------------------+----------------------+
|        name|  type|        snapshot_id|max_reference_age_in_ms|min_snapshots_to_keep|max_snapshot_age_in_ms|
+------------+------+-------------------+-----------------------+---------------------+----------------------+
|audit_branch|BRANCH|4813474390545787817|                   NULL|                 NULL|                  NULL|
|        main|BRANCH|4813474390545787817|                   NULL|                 NULL|                  NULL|
+------------+------+-------------------+-----------------------+---------------------+----------------------+

After that, I truncated main branch of the table. The snapshot_id of main branch has changed to

+------------+------+-------------------+-----------------------+---------------------+----------------------+
|        name|  type|        snapshot_id|max_reference_age_in_ms|min_snapshots_to_keep|max_snapshot_age_in_ms|
+------------+------+-------------------+-----------------------+---------------------+----------------------+
|audit_branch|BRANCH|4813474390545787817|                   NULL|                 NULL|                  NULL|
|        main|BRANCH|7458611448033585182|                   NULL|                 NULL|                  NULL|
+------------+------+-------------------+-----------------------+---------------------+----------------------+

Saying that, data was still in audit_branch with an old snapshot_id but main branch has already updated to a new snapshot_id. That's why, in my case, it cannot fast forward audit_branch to main branch.

Answer 2

check all the branches for your table using SELECT * FROM <table_name>.refs;

if you see main branch appear after the branch you created then it not right. May be try to create the table and try to cut a branch out of it again.