Turbo-charging git-diff and making reviews easier

Are you working with git repositories in which a lot of code churn is happening? For example, do you have a repository with Puppet hierarchical data and there are a lot of pull requests where code is just being moved around between layers of hieradata? diff(1) ordinarily only differentiates between added and removed lines with green and red colours (certainly the user controls what are actually those colours) however git-diff(1) recently, at the end of 2017, got a new functionality where text that is just moved around can be highlighted in different colours. This is my case and this feature was a godsend because I did not want to spend a lot of time reviewing the syntax side of things if the hieradata is just being moved. That is why I wanted to share it with you so that it could help you out and perhaps someone has some even more good tips on how to make the reviewing process more effective.

This is how it ordinarily looks like in git-diff(1):

With the default diff.colorMoved zebra mode you can get something like this git-diff(1):

And with the dimmed_zebra mode:

This feature is controlled by an option called --color-moved or diff.colorMoved in the gitconfig file. If you enable it, the default mode at the moment is the zebra mode that you saw in the screenshots before. git is sensible enough in zebra mode to only apply its algorithm for lines which are longer than 20 alphanumeric characters. That is also illustrated in those screenshots. Amended characters to moved lines are painted with a different color which is why it is called the zebra mode.

Also, two other modes exist – the plain and the dimmed_zebra mode. However, the plain mode is not so useful for our use-case because it does not differentiate between moved lines which were permutated a little bit. It only, as it says, checks if one, exact line was added somewhere else. dimmed_zebra is a bit more interesting – it only highlights when one block of moved text intersects with the text around it. You can try it out and see if that is something useful to you. In my opinion, this is the best mode.

At the end, let me introduce another project that is about making git-diff(1) even more beautiful. It can add some extra highlights or perform some other munges to the changed lines to make it even more clear what is happening. It is called diff-so-fancy. You can start using it simply by executing these commands:

git config --global core.pager "diff-so-fancy | less --tabs=4 -RFX"

Obviously, the diff-so-fancy binary needs to be available in $PATH. This is how the end result looks like if we applied this to our previous diff:

Happy hacking! Find more information here:

Why os.move() Sometimes Does Not Work And Why shutil.move() Is The Savior?

Recently I ran into an issue where this, for example, code fails:

import os

os.rename('/foo/a.txt', '/bar/b.txt')

Traceback (most recent call last):
File "", line 1, in 
OSError: [Errno 18] Invalid cross-device link

The documentation for the os module says that sometimes it might fail when the source and the destination are on different file-systems:

os.rename(srcdst*src_dir_fd=Nonedst_dir_fd=None)

Rename the file or directory src to dst. If dst is a directory, OSError will be raised. On Unix, if dst exists and is a file, it will be replaced silently if the user has permission. The operation may fail on some Unix flavors if src and dst are on different filesystems. If successful, the renaming will be an atomic operation (this is a POSIX requirement). On Windows, if dst already exists, OSError will be raised even if it is a file.

How could it fail? Renaming (moving) a file seems like such a rudimentary operation. Let’s try to investigate and find out the exact reasons…

Reason why it might fail

The fact that the move function is inside the os module implies that it uses the facilities provided by the operating system. As the Python documentation puts it:

This module provides a portable way of using operating system dependent functionality.

In this case, it (probably, depends on the implementation, obviously) uses the rename function in the C language because that what moving is and that in turn calls the rename system call. The system call is even kind of defined in the POSIX collection of standards. It is an extension of the function rename in the C standard but it sort of implies that an official system call exists as well. As the official text of POSIX says, rename can fail if:

[EXDEV] The links named by new and old are on different file systems and the implementation does not support links between file systems.

Even Linux does not support this so this error message is not that uncommon. As the rename(2) manual page says:

EXDEV oldpath and newpath are not on the same mounted filesystem. (Linux permits a filesystem to be mounted at multiple points, but rename() does not work across different mount points, even if the same filesystem is mounted on both.)

The curious case of Linux

The Linux kernel has, as we guessed, an official system call for renaming files. Actually, it even has a family of system calls related to this operation: renameat2, renameat, and rename. Internally, they all call the function sys_renameat2 with different actual parameters. And inside of it the code checks if src and dst are at the same mounted file-system. If not, –EXDEV is returned which is then propagated to the user-space program:

/* ... */
        error = -EXDEV;                                                                                                           
        if (old_path.mnt != new_path.mnt)                                                                                         
                goto exit2;  
/* ... */

Then the error is returned:

/* ... */
exit2:                                                                                                                            
        if (retry_estale(error, lookup_flags))                                                                                    
                should_retry = true;                                                                                              
        path_put(&new_path);                                                                                                      
        putname(to);                                                                                                              
exit1:                                                                                                                            
        path_put(&old_path);                                                                                                      
        putname(from);                                                                                                            
        if (should_retry) {                                                                                                       
                should_retry = false;                                                                                             
                lookup_flags |= LOOKUP_REVAL;                                                                                     
                goto retry;                                                                                                       
        }                                                                                                                         
exit:                                                                                                                             
        return error;                                                                                                             
}

This explicit check has been for forever in this function. Why is it there, though? I guess Linux just took an simpler (and more elegant, should I say) path here and added this constraint from the beginning just so that the code for the file-systems would not have to accompany for this case. The code of different file-systems is complex as it is right now. You could find the whole source code of this function in fs/namei.c.

Indeed, old_dir->i_op->rename() is later called in sys_renameat2old_dir is of type struct inode *, i_op is a pointer to a const struct inode_operations. That structure defines a bunch of pointers to functions that perform various operations with inodes. Then different file-systems define their own variable of type struct inode_operations and pass it to the kernel. It seems to me that it would be indeed a lot of work to make each file-systems rename() inode operation work with every other file-system. Plus, how would any file-system ensure future compatibility with other file-systems that the user could use by loading some custom made kernel module?

Fortunately, we could implement renaming files by other means, not just by directly calling the rename system call. This is where shutil.move() comes in…

Difference between os.move() and shutil.move()

shutil.move() side-steps this issue by being a bit more high-level. Before moving it checks whether src and dst reside on the same file-system. The different thing is here what it does in the case where they do not. os.move() would blindly fall on its face here but shutil.move() is “smarter” and it does not just call the system call with the expectation that it will succeed. Instead, shutil.move() copies the content of the src file and writes it to the dst file. Afterwards, the src file is removed. As the Python documentation puts it:

If the destination is on the current filesystem, then os.rename() is used. Otherwise, src is copied (using shutil.copy2()) to dst and then removed.

So not only it copies the content of the file but shutil.copy2() ensures that the meta-data is copied as well.This presents an issue because the operation might get interrupted between the actual copying of the content and before src is removed so potentially you might end up with two copies of the same file. Thus, shutil.move() is the preferred solution to the original problem presented at the start of the article but however be wary of the possible problems and make sure that your code handles that case if it might pose a problem to your program.