CODEWITHSUNDEEP
makefiletar
warem
warem

Reputation: 1683

tar: file changed as we read it

I am using make and tar to backup. When executing makefile, tar command shows file changed as we read it. In this case,

How can I stop the tar's warning to stop my backup following the warning?

Edit-2: it might be the reason

As I said above, the bash shell script worked well in my old computer. Comparing with the old computer, the msys version is different. So is the version of tar command. In the old computer, tar is 1.13.19 and it is 1.23 in the new computer. I copied the old tar command without copying its dependency msys-1.0.dll to the new computer and renamed it tar_old. And I also updated the tar command in the shell script and run the script. Then everything is ok. So, it seemed that the problem is the tar command. I am sure that there is no any file changed when taring. Is it a bug for tar command in new version? I don't know.

Edit-1: add more details

The backup is invoked by a bash shell script. It scans the target directory and builds makefile then invokes make to use tar command for backup. Followed is a typical makefile built by the bash shell script.

#--------------------------------------------
# backup VC
#--------------------------------------------
# the program for packing
PACK_TOOL=tar

# the option for packing tool
PACK_OPTION=cjvf

# M$: C driver
WIN_C_DIR=c:

# M$: D driver
WIN_D_DIR=d:

# M$: where the software is
WIN_PRG_DIR=wuyu/tools
# WIN_PRG_DIR=

# where to save the backup files
BAKDIR=/home/Wu.Y/MS_bak_MSYS

VC_FRAMEWORK=/home/Wu.Y/MS_bak_MSYS/tools/VC/VC_framework.tar.bz2
VC_2010=/home/Wu.Y/MS_bak_MSYS/tools/VC/VC_2010.tar.bz2

.PHONY: all

all: $(VC_FRAMEWORK) $(VC_2010)

$(VC_FRAMEWORK): $(WIN_C_DIR)/$(WIN_PRG_DIR)/VC/Framework/*
    @$(PACK_TOOL) $(PACK_OPTION) "$@" --ignore-failed-read /c/$(WIN_PRG_DIR)/VC/Framework
$(VC_2010): $(WIN_C_DIR)/$(WIN_PRG_DIR)/VC/VS2010/*
    @$(PACK_TOOL) $(PACK_OPTION) "$@" --ignore-failed-read /c/$(WIN_PRG_DIR)/VC/VS2010

As you can see, the tar package is stored in ~/MS_bak_MSYS/tools/VC/VC_2010.tar.bz2. I run the script in ~/qqaa. ~/MS_bak_MSYS is excluded from tar command. So, the tar file I am creating is not inside a directory I am trying to put into tar file. This is why I felt it strange that the warning came up.

Upvotes: 121

Views: 194691

Answers (13)

NetVicious
NetVicious

Reputation: 4037

If you're doing the tar from the same folder you will receive that message because the temp file the tar command it's creating it's placed on the path you're on the moment of running the command.

You can run the tar command from a different directory you want and place an absolute/partial path on the -C parameter, this will made the tar command to change to that path first and later it will compress the files you place, on this case the . will be the same /dir1/dir2/dir3/ path

tar cvjpf file.tar.bz2 -C /dir1/dir2/dir3/ .

Upvotes: 0

Bret Weinraub
Bret Weinraub

Reputation: 2283

Its important to check what version of tar you have

tar --version

Then read the relevant section below [esp. if you are at tar 1.34 or before].

tar 1.34 or before

These versions of tar use the "ctime" flag from stat, which is derived from inodes' ctime.

Note ctime changes anytime the inodes' metadata changes, and the metadata includes innocuous data such as "last accessed time" as well as "number of hard links to file".

Here is the block from 1.26's src/create.c



   if ((timespec_cmp (get_stat_ctime (&final_stat), original_ctime) != 0
      /* Original ctime will change if the file is a directory and
 --remove-files is given */
      && !(remove_files_option && is_dir))
     || original_size < final_stat.st_size)
   {
     WARNOPT (WARN_FILE_CHANGED,
      (0, 0, _("%s: file changed as we read it"),
quotearg_colon (p)));
     set_exit_status (TAREXIT_DIFFERS);
   }

And some links about what ctime is:

tar 1.35+

In these versions, tar uses "mtime" [aka modified time] from the inode, as reflected in the change log entry:

2022-06-10  Paul Eggert  <[email protected]>

    Warn “file changed as we read it” less often

    * src/create.c (dump_file0): Remove an fstatat call that is
    unnecessary because the file wasn’t read so we can treat
    the first
    fstatat as atomic.  Warn “file changed” when the file’s
    size,
    mtime, user ID, group ID, or mode changes, instead of when the
    file’s size or ctime changes.  Also, when such a change happens,
    do not change exit status if --ignore-failed-read.  Finally,
    don’t
    attempt to change atime back if it didn’t change.

And the code in question [still in src/create.c]

         Do not check atime which is saved only to replace it later.
         Do not check ctime where changes might be benign (e.g.,
         another process creates a hard link to the file).  */

      /* If the file's user ID, group ID or mode changed, tar may
         have output the wrong info for the file.  */
      ok &= st1.st_uid == st2.st_uid;
      ok &= st1.st_gid == st2.st_gid;
      ok &= st1.st_mode == st2.st_mode;

      /* Likewise for the file's mtime, but skip this check if it
         is a directory possibly updated by --remove-files.  */
      if (! (is_dir && remove_files_option))
        ok &= ! timespec_cmp (get_stat_mtime (&st1),
                  get_stat_mtime (&st2));

Therefore, if you are struggling with this error, I suggest trying tar 1.35 and see if that resolves the issue.

For me, other processes on the box were disrupting a tar 1.25 archive creation through no fault of their own.

Upvotes: 10

Pila
Pila

Reputation: 241

I have a source file that takes 1-2 seconds to be written. As immediate backup, I am taring the source before each run if the source was changed. If I try that too quickly, before write of the source is finished, tar complains. Not a single of the above answers is useful.

(( $(( $(date +%s) - $(date -r "source.sh" +%s 2>/dev/null) )) < 3 )) && sleep 3; tar -zcvf Source.tar.gz source.sh

I simply wait 3 seconds if the file is less than 3 seconds old, and then tar it.

I have it in an alias, add timestamps to the tar name and use --newer to compare against a reference tar if the source is changed.

Upvotes: 0

Alve
Alve

Reputation: 730

I got this same error when I was trying to create an archive of the same folder as where I put the file in. To solve this, I made my shell resolve the paths instead of passing the whole directory:

tar czf release.tar.gz ./*

This makes the tar command never see its own archive when creating it, which fixes the error

Upvotes: 0

sastorsl
sastorsl

Reputation: 2135

Exit codes for tar are restricted, so you don't get to much information. You can assume that ec=1 is safe to ignore, but it might trip - i.e. the gzip-example in other posts (exit code from external programs).

The reason for the file changed as we read it error/warning can be varying.

  • A log file inside the directory.
  • Writing to a tar file in the same directory you are trying to back up.
  • etc.

Possible workarounds can involve:

  • exclude known files (log files, tar-files, etc)
  • ensure log files are written to other directories

This can be quite involved, so you might want to still just run the tar command and preferably safely ignore some errors / warnings.

To do this you will have to:

  • Save the tar output.
  • Save the exit code
  • Check the output against known warnings and errors, not unlike tar's own ignore.
  • Conditionally pass another exit code to the next program in the pipe.

In OP's case this would have to be wrapped in a script and run as PACK_TOOL.

# List of errors and warnings from "tar" which we will safely ignore.
# Adapt to your findings and needs
IGNORE_ERROR="^tar:.*(Removing leading|socket ignored|file changed as we read it)"

# Save stderr from "tar"
RET=$(tar zcf $BACKUP --exclude Cache --exclude output.log --exclude "*cron*sysout*" $DIR 2>&1)
EC=$?  # Save "tar's" exit code
echo "$RET"
if [ $EC -ne 0 ]
then
  # Check the RET output, remove (grep -v) any errors / warning you wish to ignore
  REAL_ERRORS=$(echo "$RET" | grep "^tar: " | grep -Ev "${IGNORE_ERROR:?}")
  # If there is any output left you actually got an error to check
  if [ -n "$REAL_ERRORS" ]
  then
      echo "ERROR during backup of ${DIR:?} to ${BACKUP:?}"
  else
      echo "OK backup of (warnings ignored) ${DIR:?}"
      EC=0
  fi
else
  echo "OK backup of ${DIR:?}"
fi

Upvotes: 1

pprishchepa
pprishchepa

Reputation: 1107

I am not sure does it suit you but I noticed that tar does not fail on changed/deleted files in pipe mode. See what I mean.

Test script:

#!/usr/bin/env bash
set -ex
tar cpf - ./files | aws s3 cp - s3://my-bucket/files.tar
echo $?

Deleting random files manually...

Output:

+ aws s3 cp - s3://my-bucket/files.tar
+ tar cpf - ./files
tar: ./files/default_images: File removed before we read it
tar: ./files: file changed as we read it
+ echo 0
0

Upvotes: 0

ninohead
ninohead

Reputation: 364

It worked for me by adding a simple sleep timeout of 20 sec. This might happen if your source directory is still writing. Hence put a sleep so that the backup would finish and then tar should work fine. This also helped me in getting the right exit status.

sleep 20
tar -czf ${DB}.${DATE}.tgz ./${DB}.${DATE}

Upvotes: 0

Mohd Abdul Mujib
Mohd Abdul Mujib

Reputation: 13928

Simply using an outer directory for the output, solved the problem for me.

sudo tar czf ./../31OCT18.tar.gz ./

Upvotes: 5

Jeremy
Jeremy

Reputation: 71

To enhance Fabian's one-liner; let us say that we want to ignore only exit status 1 but to preserve the exit status if it is anything else:

tar -czf sample.tar.gz dir1 dir2 || ( export ret=$?; [[ $ret -eq 1 ]] || exit "$ret" )

This does everything sandeep's script does, on one line.

Upvotes: 7

Mohammad Azim
Mohammad Azim

Reputation: 2933

Although its very late but I recently had the same issue.

Issue is because dir . is changing as xyz.tar.gz is created after running the command. There are two solutions:

Solution 1: tar will not mind if the archive is created in any directory inside .. There can be reasons why can't create the archive outside the work space. Worked around it by creating a temporary directory for putting the archive as:

mkdir artefacts
tar -zcvf artefacts/archive.tar.gz --exclude=./artefacts .
echo $?
0

Solution 2: This one I like. create the archive file before running tar:

touch archive.tar.gz
tar --exclude=archive.tar.gz -zcvf archive.tar.gz .
echo $?
0

Upvotes: 106

Fabian Ritzmann
Fabian Ritzmann

Reputation: 1715

Here is a one-liner for ignoring the tar exit status if it is 1. There is no need to set +e as in sandeep's script. If the tar exit status is 0 or 1, this one-liner will return with exit status 0. Otherwise it will return with exit status 1. This is different from sandeep's script where the original exit status value is preserved if it is different from 1.

tar -czf sample.tar.gz dir1 dir2 || [[ $? -eq 1 ]]

Upvotes: 24

sandeep
sandeep

Reputation: 1319

I also encounter the tar messages "changed as we read it". For me these message occurred when I was making tar file of Linux file system in bitbake build environment. This error was sporadic.

For me this was not due to creating tar file from the same directory. I am assuming there is actually some file overwritten or changed during tar file creation.

The message is a warning and it still creates the tar file. We can still suppress these warning message by setting option

--warning=no-file-changed

(http://www.gnu.org/software/tar/manual/html_section/warnings.html )

Still the exit code return by the tar is "1" in warning message case: http://www.gnu.org/software/tar/manual/html_section/Synopsis.html

So if we are calling the tar file from some function in scripts, we can handle the exit code something like this:

set +e 
tar -czf sample.tar.gz dir1 dir2
exitcode=$?

if [ "$exitcode" != "1" ] && [ "$exitcode" != "0" ]; then
    exit $exitcode
fi
set -e

Upvotes: 116

MadScientist
MadScientist

Reputation: 100836

If you want help debugging a problem like this you need to provide the make rule or at least the tar command you invoked. How can we see what's wrong with the command if there's no command to see?

However, 99% of the time an error like this means that you're creating the tar file inside a directory that you're trying to put into the tar file. So, when tar tries to read the directory it finds the tar file as a member of the directory, starts to read it and write it out to the tar file, and so between the time it starts to read the tar file and when it finishes reading the tar file, the tar file has changed.

So for example something like:

tar cf ./foo.tar .

There's no way to "stop" this, because it's not wrong. Just put your tar file somewhere else when you create it, or find another way (using --exclude or whatever) to omit the tar file.

Upvotes: 45

Related Questions