Why os.move() Sometimes Does Not Work And Why shutil.move() Is The Savior?

Recently I ran into an issue where this, for example, code fails:

import os

os.rename('/foo/a.txt', '/bar/b.txt')

Traceback (most recent call last):
File "", line 1, in 
OSError: [Errno 18] Invalid cross-device link

The documentation for the os module says that sometimes it might fail when the source and the destination are on different file-systems:

os.rename(srcdst*src_dir_fd=Nonedst_dir_fd=None)

Rename the file or directory src to dst. If dst is a directory, OSError will be raised. On Unix, if dst exists and is a file, it will be replaced silently if the user has permission. The operation may fail on some Unix flavors if src and dst are on different filesystems. If successful, the renaming will be an atomic operation (this is a POSIX requirement). On Windows, if dst already exists, OSError will be raised even if it is a file.

How could it fail? Renaming (moving) a file seems like such a rudimentary operation. Let’s try to investigate and find out the exact reasons…

Reason why it might fail

The fact that the move function is inside the os module implies that it uses the facilities provided by the operating system. As the Python documentation puts it:

This module provides a portable way of using operating system dependent functionality.

In this case, it (probably, depends on the implementation, obviously) uses the rename function in the C language because that what moving is and that in turn calls the rename system call. The system call is even kind of defined in the POSIX collection of standards. It is an extension of the function rename in the C standard but it sort of implies that an official system call exists as well. As the official text of POSIX says, rename can fail if:

[EXDEV] The links named by new and old are on different file systems and the implementation does not support links between file systems.

Even Linux does not support this so this error message is not that uncommon. As the rename(2) manual page says:

EXDEV oldpath and newpath are not on the same mounted filesystem. (Linux permits a filesystem to be mounted at multiple points, but rename() does not work across different mount points, even if the same filesystem is mounted on both.)

The curious case of Linux

The Linux kernel has, as we guessed, an official system call for renaming files. Actually, it even has a family of system calls related to this operation: renameat2, renameat, and rename. Internally, they all call the function sys_renameat2 with different actual parameters. And inside of it the code checks if src and dst are at the same mounted file-system. If not, –EXDEV is returned which is then propagated to the user-space program:

/* ... */
        error = -EXDEV;                                                                                                           
        if (old_path.mnt != new_path.mnt)                                                                                         
                goto exit2;  
/* ... */

Then the error is returned:

/* ... */
exit2:                                                                                                                            
        if (retry_estale(error, lookup_flags))                                                                                    
                should_retry = true;                                                                                              
        path_put(&new_path);                                                                                                      
        putname(to);                                                                                                              
exit1:                                                                                                                            
        path_put(&old_path);                                                                                                      
        putname(from);                                                                                                            
        if (should_retry) {                                                                                                       
                should_retry = false;                                                                                             
                lookup_flags |= LOOKUP_REVAL;                                                                                     
                goto retry;                                                                                                       
        }                                                                                                                         
exit:                                                                                                                             
        return error;                                                                                                             
}

This explicit check has been for forever in this function. Why is it there, though? I guess Linux just took an simpler (and more elegant, should I say) path here and added this constraint from the beginning just so that the code for the file-systems would not have to accompany for this case. The code of different file-systems is complex as it is right now. You could find the whole source code of this function in fs/namei.c.

Indeed, old_dir->i_op->rename() is later called in sys_renameat2old_dir is of type struct inode *, i_op is a pointer to a const struct inode_operations. That structure defines a bunch of pointers to functions that perform various operations with inodes. Then different file-systems define their own variable of type struct inode_operations and pass it to the kernel. It seems to me that it would be indeed a lot of work to make each file-systems rename() inode operation work with every other file-system. Plus, how would any file-system ensure future compatibility with other file-systems that the user could use by loading some custom made kernel module?

Fortunately, we could implement renaming files by other means, not just by directly calling the rename system call. This is where shutil.move() comes in…

Difference between os.move() and shutil.move()

shutil.move() side-steps this issue by being a bit more high-level. Before moving it checks whether src and dst reside on the same file-system. The different thing is here what it does in the case where they do not. os.move() would blindly fall on its face here but shutil.move() is “smarter” and it does not just call the system call with the expectation that it will succeed. Instead, shutil.move() copies the content of the src file and writes it to the dst file. Afterwards, the src file is removed. As the Python documentation puts it:

If the destination is on the current filesystem, then os.rename() is used. Otherwise, src is copied (using shutil.copy2()) to dst and then removed.

So not only it copies the content of the file but shutil.copy2() ensures that the meta-data is copied as well.This presents an issue because the operation might get interrupted between the actual copying of the content and before src is removed so potentially you might end up with two copies of the same file. Thus, shutil.move() is the preferred solution to the original problem presented at the start of the article but however be wary of the possible problems and make sure that your code handles that case if it might pose a problem to your program.

What is Actually True and False in Python?

Intro

Did you know that in Python 2.x you can do the following?

$ python2
Python 2.7.14 (default, Sep 20 2017, 01:25:59) 
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> True = False
>>> False
False
>>> True
False
>>> not True
True
>>> True == False
True
>>> True != False
False

How can it be that not True is True and True is equal to False? Why is it even possible to do this? Isn’t what is True and False in the language defined to be constant and unchangeable? What sense does it make to change the meaning of what is True and what is False? In any way, to fix this bug in the matrix, do this:

$ python2
Python 2.7.14 (default, Sep 20 2017, 01:25:59) 
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> True = False
>>> True = not False
>>> True
True
>>> False
False
>>> True == False
False

After this, everything will be back to normal. As you can see, you do not need to worry about anything because you can use various operators to assign a sane value to True again after you change it. Besides not you could use ==!= or any other operator which returns a boolean value.

This article will delve into the presented issue and explain why you are able to do this, first of all. Apart from that, there are some apt questions that are raised by this interesting behavior. They include:

  • What about Python 1.x or 3.x? Can you do the same?
  • How did the programming language developers miss this?
  • What could be the rationale behind these language design decisions?

I will try my best to look into and answer them. This is definitely an interesting piece of history of development of the Python programming language.

True and False in Python 1.x

The oldest major version of the Python programming language – 1.x – does not even have such a thing as False or True. You can see in this example:

try:
  print True
except NameError:
  print 'True not found'

This yields the text ‘True not found’ in the standard output:

$ docker run -it dahlia/python-1.5.2-docker
Python 1.5.2 (#1, Aug 11 2017, 14:21:33)  [GCC 4.8.4] on linux4
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> try:
...   print True
... except NameError:
...   print 'True not found'
... 
True not found

NameError is raised when you are using True in Python 1.x because Python tries to look up a variable called True and, obviously, that does not exist. Also, this shows that False does not exist as well:

try:
  print False
except NameError:
  print 'False not found'

And we get:

$ docker run -it dahlia/python-1.5.2-docker
Python 1.5.2 (#1, Aug 11 2017, 14:21:33)  [GCC 4.8.4] on linux4
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> try:
...   print False
... except NameError:
...   print 'False not found'
... 
False not found

In Python 1.x, like in some other languages, what is true or false is defined in terms of evaluation rules. Anything other than None, numeric zero of all types, empty sequences, and empty mappings are considered true (https://docs.python.org/release/1.6/ref/lambda.html):

In the context of Boolean operations, and also when expressions are used by control flow statements, the following values are interpreted as false: None, numeric zero of all types, empty sequences (strings, tuples and lists), and empty mappings (dictionaries). All other values are interpreted as true.

Thus, we could make our own True and False like this:

$ docker run -it dahlia/python-1.5.2-docker
Python 1.5.2 (#1, Aug 11 2017, 14:21:33)  [GCC 4.8.4] on linux4
Copyright 1991-1995 Stichting Mathematisch Centrum, Amsterdam
>>> False = 0
>>> True = not False
>>> print False, True
0 1

But, obviously, they are not protected from modification and they are not available in all Python programs so it is kind of pointless to have them unless you have large piece of software that was written in Python and you want to maintain an unified definition of what is True and False over all of it, and if you want to make it more future-proof because then only one small section will have to be changed to change the meaning of True and/or False.

A keen reader would notice that this also means that there is no dedicated boolean types. This section of the official language specification lists all of the types –  https://docs.python.org/release/1.6/ref/types.html. The official supported types in Python 1.x are these:

  • None
  • Ellipsis
  • Numbers
  • Sequences
  • Mappings
  • Callable types
  • Modules
  • Class and class instances
  • Files
  • Internal types

As you can see, there really is no dedicated type for boolean expressions. However, the situation was significantly improved in the next major version of Python – 2.x. Although not all negative aspects were fixed and they are still there in the language. Let’s talk about Python 2.x.

True and False in Python 2.x

Dedicated ‘bool’ type appeared in the 2.x version of the Python programming language as per this example:

$ python2
Python 2.7.14 (default, Sep 20 2017, 01:25:59) 
[GCC 7.2.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> type(True)
<type 'bool'>
>>> type(False)
<type 'bool'>

The type() function returns the type of the provided argument. As you can see, the type of True and False is bool. Boolean types were truly finally added in Python 2.x and they belong to the integer class of types (https://docs.python.org/2/reference/datamodel.html):

Booleans

These represent the truth values False and True. The two objects representing the values False and True are the only Boolean objects. The Boolean type is a subtype of plain integers, and Boolean values behave like the values 0 and 1, respectively, in almost all contexts, the exception being that when converted to a string, the strings “False” or “True” are returned, respectively.

However, even though a dedicated boolean was added but the new values True and False were not defined to be keywords thus their value could be changed. Instead, they were defined as “constants” that live in the built-in namespace (https://docs.python.org/2/library/constants.html):

A small number of constants live in the built-in namespace. They are:

False
The false value of the bool type.

New in version 2.3.

True
The true value of the bool type.

New in version 2.3.

Unfortunately but the fact that they are “constants” does not constitute that they are immutable in Python 2.x. All of this verbiage essentially just means that there are some variables of certain types and certain values pre-loaded into every Python program and the program itself is free to change their meaning.  Before 2.4, you even could assign a new value to None but later they changed it to raise an SyntaxError exception if you attempted to do that. Why they did not do that for True and False as well – I don’t know. I seriously wonder what insane use-cases or existing code they were accommodating for by not making the same change for True and False as well at the same time in 2.4.

Also, notice that the Python 2.x documentation makes a separation between “true” and “false” constants. “true” constants are those to which you cannot assign a new value because it raises an exception, and “false” constants are those to which you can. The official documentation even puts those words in quotes, I am not making it up. This could really make you say: “wat”.

Wat meme

If you ask me I see it as a huge inconsistency in language design and it makes no sense to me not to make same change from Python 2.4 on-wards to make it illegal to assign new values to True and False as well, and just remove the whole “false” constants notion in general. Perhaps they were afraid of making such a backwards incompatible change and so the developers waited until 3.x?

True and False in Python 3.x

This mess was finally permanently fixed in the next major version of Python, 3.x.  True and False, and other constants like None were turned into keywords. True was defined to be equal to the number 1 and False was defined to be equal the number 0. There are no more such thing as “false” or “true” constants. You can see that from this error message:

$ python
Python 3.6.3 (default, Oct 24 2017, 14:48:20) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> type(True)
<class 'bool'>
>>> type(False)
<class 'bool'>
>>> True = False
  File "<stdin>", line 1
SyntaxError: can't assign to keyword
>>> None = False
  File "<stdin>", line 1
SyntaxError: can't assign to keyword

As you can see, True and None (and others) became keywords and thus officially they became immutable. They are listed here https://docs.python.org/3/reference/lexical_analysis.html#keywords. Boolean type is still a sub-type of the Number types: https://docs.python.org/3/reference/datamodel.html#the-standard-type-hierarchy. The text that describes the boolean type is identical to the 2.x version.

Defining True and False to be constants that are immutable also brings some performance improvements because implementations of Python no longer have to look up what is the value of True and False every time they were used in an infinite loop, for example. Things were finally completely fixed by version 3.x and it only took almost 15 years to properly implement True and False.

What were they thinking?

From what I can tell is that Guido never cared at the beginning to add a separate boolean type just like, for example, the C programming language never had a boolean type until the C99 version of the standard. All of the boolean operations were simply expressed in terms of the evaluation rules.

The boolean type and the True/False constants were proposed in Python Enhancement Proposal (PEP) number 0285 (https://www.python.org/dev/peps/pep-0285/). However, it seems to me that at that time Python 1.x had a bunch of those constants that were mutable and these two new constants were added which kind of floated around and had an unknown status just like the others. After a bit of time, someone noticed that it does not make much sense to override the value of None/True/False and others. At that point they were converted into keywords thus rendering them immutable. The fix in the version 2.4 for the None value seemed like a bit of bandage aid applied by the developers to the language but it wasn’t fixed completely. I guess that they waited until the next major version bump because it’s a backwards-incompatible change.

It’s kind of humorous because some Python developers even (https://github.com/riptideio/pymodbus/issues/43) decided to include lines like these at the beginning of their programs:

True = 1 == 1
False = 1 == 0

It is crazy that they were afraid of people using their libraries who were messing with the values of True and False. Such is the fun story of True and False in the Python world.

EDIT:

2018 January 10 – changed the words to say that in Python 3.x True and False were defined to be equal to 1 and 0 respectively, they are not the actual numbers.