CODEWITHSUNDEEP
airflow
nono
nono

Reputation: 2451

Maximum memory size for an XCOM in Airflow

I was wondering if there is any memory size limit for an XCOM in airflow ?

Upvotes: 14

Views: 18078

Answers (3)

Judy3 Yang
Judy3 Yang

Reputation: 101

Airflow is NOT a processing framework. It is not Spark, neither Flink. Airflow is an orchestrator, and it the best orchestrator. There is no optimisations to process big data in Airflow neither a way to distribute it (maybe with one executor, but this is another topic). If you try to exchange big data between your tasks, you will end up with a memory overflow error! Oh, and do you know the xcom limit size in Airflow?

It depends on the database you use:

  1. SQLite: 2 GB
  2. Postgres: 1 GB
  3. MySQL: 64 KB

Yes, 64 Kilobytes for MySQL! Again, use XComs only for sharing small amount of data.

ref: https://marclamberti.com/blog/airflow-xcom/

Upvotes: 10

Newt
Newt

Reputation: 887

According to the source code check this source code link, maximum XCOM size is 48KB.

Upvotes: 7

nono
nono

Reputation: 2451

After looking at the source code it looks there is none, the type is a large binary in SQLAlchemy. Code So according to the documentation is an unlengthed binary type for the target platform, such as BLOB on MySQL and BYTEA for PostgreSQL.

Upvotes: 6

Related Questions