CODEWITHSUNDEEP
postgresql
ajlind
ajlind

Reputation: 363

Postgres 12 - how to reduce size of wal-directory

I have upgraded postgres 9 to 12 with pgdump and restore it to new database. All seems to be ok. Afterwards I noticed, that size of ../12/main/ has grown a lot and most of it is in pg_wal-directory. Size of ./base is 27GB and size of ./pg_wal is 61GB. On postgres 9 values were 31GB / 1GB (pg_xlog).

After some checking, postgres.conf-file shows default values as:

# max_wal_size = 64GB 
min_wal_size = 5GB

I change value of max_wal_size to 4GB and removed comment

SHOW ALL;

told

max_wal_size | 4GB

But size of pg_wal stays on 61GB. Is there some other parameter, I should change?

Upvotes: 8

Views: 21456

Answers (3)

Oliver Crow
Oliver Crow

Reputation: 376

Following on from jjanes' answer ...

If you have a server with very low traffic, a development server for example, have done something like a data import which generates a lot of writes, and so have a large number of wal segment files, the amount of storage consumed by wal segments may grow up to max_wal_size. If you reduce max_wal_size in the server config, the actual number of wal segment files won't drop until the server cycles through those old files.

Here's a function which will force the server to cycle through all of the existing segment files. It does this by repeatedly calling pg_switch_wal() to switch to a new wal segment and then checkpoint to force writing a checkpoint on the new wal segment.

In the scenario described this will trigger postgres to reduce the number of segment files down to the current max_wal_size and possibly below.

This may be counterproductive on a live server, particularly if you are doing wal archiving, so please only use this if your situation matches the description above.

Depending on how many wal segments you have, this may take some time to complete.

create or replace function pg_wal_cycle_all() 
returns int language plpgsql 
as $$
declare
    wal_count int;
    wal_seg varchar;
begin 
    select count(*) - 1 
    into wal_count 
    from pg_ls_dir('pg_wal');

    for wal in 1..wal_count loop 
        select pg_walfile_name(pg_switch_wal()) into wal_seg;
        raise notice 'segment %', wal_seg;
        checkpoint;
    end loop;
    return wal_count;
end;$$;

Execute with:

select pg_wal_cycle_all();

Upvotes: 6

jjanes
jjanes

Reputation: 44137

It seems like at some point, the server must have actually been running with max_wal_size = 64GB, probably during the initial import. If it were undergoing intense activity, it could have recycled most of the wal files for future use, thinking it will need them very soon based on high recent activity. Once recycled, the files are never removed until after they are used. If the server has low activity once the initial import is done, it could take a very long time to use that amount of WAL files. You could check to see how many of the files are recycled for future use by a query like this:

select count(*) as all, 
    count(*) filter (where filename > pg_walfile_name(pg_current_wal_lsn())) as future 
from pg_ls_dir('pg_wal') as t(filename);

If you are desperate to get that space back, you could very very carefully delete the files with names which are the furthest in the future. Deleting the wrong ones though would thoroughly destroy your database. If you don't really need the space, it is safest to just let them alone, and they will slowly be re-used and deleted naturally over time.

Upvotes: 3

Laurenz Albe
Laurenz Albe

Reputation: 246083

The default value is 1GB – you must be using a modified version of PostgreSQL.

I would first check pg_stat_archiver and the log file if there are problems archiving WAL segments. Then I would check for stale replication slots. Finally, I would check the setting of wal_keep_segments.

If all three don't account for the size, you just have to wait: WAL segments are removed at every checkpoint. The more activity there is, the faster WAL segments will get filled and deleted.

Upvotes: 4

Related Questions