How to rename large number of files

Question

I have a directory with files like this

a.JPG
b.JPG
c.JPG

I would like to perform something like this

git mv a.JPG a.jpg

I tried using xargs and other tools but nothing seems to work.

Answer 1

Note that Damien's answer (find . -name "*.JPG" | xargs -L1 bash -c 'git mv $0 ${0/JPG/jpg}') works even on Windows, with the new Git 2.48 (Q1 2025), batch 9: the MinGW compatibility layer has been taught to support POSIX semantics for atomic renames when other process(es) have a file opened at the destination path.

See commit 391bcea, commit a270cb1, commit b0b65ec (27 Oct 2024) by Patrick Steinhardt (pks-t).
^{(Merged by Junio C Hamano -- gitster -- in commit 183ea3e, 13 Nov 2024)}

compat/mingw: support POSIX semantics for atomic renames

^{Signed-off-by: Patrick Steinhardt
Signed-off-by: Taylor Blau
Reviewed-by: Johannes Sixt}

By default, Windows restricts access to files when those files have been opened by another process.
As explained in the preceding commits, these restrictions can be loosened such that reads, writes and/or deletes of files with open handles are allowed.

While we set up those sharing flags in most relevant code paths now, we still don't properly handle POSIX-style atomic renames in case the target path is open.
This is failure demonstrated by t0610, where one of our tests spawns concurrent writes in a reftable-enabled repository and expects all of them to succeed.
This test fails most of the time because the process that has acquired the "tables.list" lock is unable to rename it into place while other processes are busy reading that file.

Windows 10 has introduced the FILE_RENAME_FLAG_POSIX_SEMANTICS flag that allows us to fix this usecase.
When set, it is possible to rename a file over a preexisting file even when the target file still has handles open.
Those handles must have been opened with the FILE_SHARE_DELETE flag, which we have ensured in the preceding commits.

Careful readers might have noticed that previous usecase does not mention the above flag, but instead mentions FILE_RENAME_POSIX_SEMANTICS.
This flag is not for use with SetFileInformationByHandle() though, which is what we use.
And while the FILE_RENAME_FLAG_POSIX_SEMANTICS flag exists, it is not documented here or anywhere else as far as I can tell.

Unfortunately, we still support Windows systems older than Windows 10 that do not yet have this new flag.

(I do mention that in "Git 2.13 not working on Windows XP")

Our _WIN32_WINNT SDK version still targets 0x0600, which is Windows Vista and later.
And even though that Windows version is out-of-support, bumping the SDK version all the way to 0x0A00, which is Windows 10 and later, is not an option as it would make it impossible to compile on Windows 8.1, which is still supported.
Instead, we have to manually declare the relevant infrastructure to make this feature available and have fallback logic in place in case we run on a Windows version that does not yet have this flag.

On another note: mingw_rename() has a retry loop that is used in case deleting a file failed because it's still open in another process.
One might be pressed to not use this loop anymore when we can use POSIX semantics.
But unfortunately, we have to keep it around due to our dependence on the FILE_SHARE_DELETE flag.
While we know to set that sharing flag now, other applications may not do so and may thus still cause sharing violations when we try to rename a file.

(See also "Is an atomic file rename (with overwrite) possible on Windows?")

This fixes concurrent writes in the reftable backend as demonstrated in t0610, but may also end up fixing other usecases where Git wants to perform renames.

---

Note: Git 2.48 (Q1 2025), pre 2.48-rc1 (bis) updates the way the rename() emulation on Windows handles directories to correct an earlier attempt to do the same.

See commit b30404d (17 Dec 2024) by Johannes Schindelin (dscho).
^{(Merged by Junio C Hamano -- gitster -- in commit c4cc685, 23 Dec 2024)}

mingw_rename: do support directory renames

^{Signed-off-by: Johannes Schindelin}

In 391bcea ("compat/mingw: support POSIX semantics for atomic renames", 2024-10-27, Git v2.48.0-rc0 -- merge listed in batch #9), we taught the mingw_rename() function to respect POSIX semantics, but we did so only as a fallback after _wrename() fails.

This hid a bug in the implementation that was not caught by Git's test suite: The CreateFileW() function can open handles to directories, but not when asked to use the FILE_ATTRIBUTE_NORMAL flag, as that flag only is allowed for files.

Let's fix this by using the common FILE_FLAG_BACKUP_SEMANTICS flag that can be used for opening handles to directories, too.

Answer 2

To rename file extensions from .JPG to lowercase. This uses git mv, so that git history is preserved.

find . -name "*.JPG" | xargs -L1 bash -c 'git mv $0 ${0/JPG/jpg}'

Answer 3

The core of the solution will be to use a tool/method that will automate the bulk rename. You can either use mv in combination with git add or just git mv. In either case you may have to take extra steps if you are using a case insensitive filesystem. So before we tackle the bulk renaming, it may be useful to discuss how case is handled a bit.

Case Sensitivity

Some systems (or system+filesystem combinations—like the default variant of the HFS+ filesystem on Mac OS X^*) are case preserving, but case insensitive. On such systems, you may need to be careful when making renames that involve only changing the case of a name. The usual workaround is to use a temporary name that differs by more than just case as a “bridge” between the two names that differ by case alone (e.g. mv foo.JPG tmp && mv tmp foo.jpg).

^{* It is possible to use case sensitive file systems on Mac OS X (including a case sensitive variant of HFS+).}

From here on, I will assume a case insensitive filesystem.

The mv command on Mac OS X can handle case-change-only renames in a single step. It will give an “overwrite?” prompt if run with the -i option, and it will skip the rename if given the -n option. It is only succeeding through the “enough rope to hang your self” default operation of many parts of Unix-like systems.

The git mv command is a bit more paranoid about the situation. It refuses the operation (“destination exists” error) unless given -f/--force option.

# this will succeed, though it may fail/prompt if mv is aliased to use -n/-i
mv foo.JPG foo.jpg

# this will succeed
mv -f bar.JPG bar.jpg

# this will succeed but give a warning
git mv -f quux.JPG quux.jpg

Bulk Rename Options

Perl rename

The desired operation is simple enough to do with a bit of shell scripting, but you could get the Perl rename utility (the one Jordan Lewis mentions) if you needed to do something that was a lot more complicated. You could try the rename from Debian's perl package, or if you feel up to using CPAN, you could install File::Rename, which includes the rename program.

ksh, bash, zsh, dash

The -ef used below is not POSIX compatible. Likewise, while -e is specified in POSIX, it is not pure-Bourne compatible. Both of them are widely supported though.

for f in *.JPG; do
    ff="${f%.JPG}.jpg"
    test -e "$f" || continue        # possible when not using nullglob
    test "$f" != "$ff" || continue  # possible when using nocaseglob
    if test -e "$ff" &&
       ! test "$f" -ef "$ff"; then  # possible on a case sensitive filesystem
        echo "skipping <$f>: destination <$ff> exists and is distinct" 1>&2
        continue
    fi

    # "mv" with "git rm" and "git add"
    mv -f "$f" "$ff"     &&
    git rm --cached "$f" &&
    git add "$ff"
done

The last section (mv, git rm, git add) could be replaced with just git mv:

    # "git mv"
    git mv -f "$f" "$ff"

If you are very concerned with how the rename might fail on case insensitive systems, then you could use a temp name:

    # temp-based "mv" with "git rm" and "git add"
    t="$ff.tmp"; while test -e "$t"; do t="$t.tmp"; done
    mv -n "$f" "$t"      &&
    mv -n "$t" "$ff"     &&
    git rm --cached "$f" &&
    git add "$ff"

Or with git mv:

    # temp-based "git mv"
    t="$ff.tmp"; while test -e "$t"; do t="$t.tmp"; done
    git mv "$f" "$t"  &&
    git mv "$t" "$ff"

zsh/zmv

This one needs -f for both zmv and git mv.

zsh -c 'autoload zmv && $0 $@' zmv -fp git -o 'mv -f' '(*).JPG' '$1 x.jpg'

---

Now that you have them all renamed and updated in Git's index you can commit them.

But will other Git users using case sensitive filesystems be able to check them out?

git checkout After a Case-Only Rename

If there are other users of your history, they will probably still have the JPG files and when they eventually checkout (a descendent of) your commit with the jpg files. What will happen for them?

No matter what happens, there is no need for “rename to temp, commit, rename to final, commit”. git checkout does not apply commits in sequence when moving between commits. It really works by “merging” the index and working tree from HEAD to the new commit. This effectively means that it “jumps” directly to the new commit while dragging along non-conflicting changes found between HEAD and the index/working-tree.

Internally, Git views renames as a deletion and an addition. I did not find any documentation that described the behavior of git checkout with respect to the order of deletions and additions, so I looked at the source code. git checkout processes all deletions before any updates/additions (cmd_checkout -> switch_branches -> merge_working_tree (-> reset_tree) -> unpack_trees -> check_updates).

You can test this out right after your rename commit:

git checkout HEAD~ # note: detached HEAD
# verify that the original names are back in place
git checkout -     # back to your branch
# verify that the new names are in place again

The git blame on the file seemed to indicate a likely commit: Make unpack-tree update removed files before any updated files, which was first released in Git 1.5.6-rc0 (2008-06-18). So, though undocumented(?), this behavior was implemented specifically to support case insensitive filesystems.

Thanks, Linus!

Answer 4

Whether or not you can just change the case of the file will depend on your filesystem. Even if it works on your filesystem, you could cause problems for other people updating. You'll be best renaming them, committing them, then renaming them back. Change everything to *.tmp with the following bash script:

for i in *.JPG; do mv $i ${i%.JPG}.tmp; done

Then move them all in git. You can use a similar command, but I would recommend checking out guess-renames which will help with the move.

Then rename them all back to *.jpg with a similar process.

Answer 5

Use the standard Linux rename(1) utility. Once you've renamed the files, git add them.

Answer 6

Maybe rename them to *.somethingelse and then rename back to *.jpg