Why os.move() Sometimes Does Not Work And Why shutil.move() Is The Savior?

Recently I ran into an issue where this, for example, code fails:

import os

os.rename('/foo/a.txt', '/bar/b.txt')

Traceback (most recent call last):
File "", line 1, in 
OSError: [Errno 18] Invalid cross-device link

The documentation for the os module says that sometimes it might fail when the source and the destination are on different file-systems:

os.rename(srcdst*src_dir_fd=Nonedst_dir_fd=None)

Rename the file or directory src to dst. If dst is a directory, OSError will be raised. On Unix, if dst exists and is a file, it will be replaced silently if the user has permission. The operation may fail on some Unix flavors if src and dst are on different filesystems. If successful, the renaming will be an atomic operation (this is a POSIX requirement). On Windows, if dst already exists, OSError will be raised even if it is a file.

How could it fail? Renaming (moving) a file seems like such a rudimentary operation. Let’s try to investigate and find out the exact reasons…

Reason why it might fail

The fact that the move function is inside the os module implies that it uses the facilities provided by the operating system. As the Python documentation puts it:

This module provides a portable way of using operating system dependent functionality.

In this case, it (probably, depends on the implementation, obviously) uses the rename function in the C language because that what moving is and that in turn calls the rename system call. The system call is even kind of defined in the POSIX collection of standards. It is an extension of the function rename in the C standard but it sort of implies that an official system call exists as well. As the official text of POSIX says, rename can fail if:

[EXDEV] The links named by new and old are on different file systems and the implementation does not support links between file systems.

Even Linux does not support this so this error message is not that uncommon. As the rename(2) manual page says:

EXDEV oldpath and newpath are not on the same mounted filesystem. (Linux permits a filesystem to be mounted at multiple points, but rename() does not work across different mount points, even if the same filesystem is mounted on both.)

The curious case of Linux

The Linux kernel has, as we guessed, an official system call for renaming files. Actually, it even has a family of system calls related to this operation: renameat2, renameat, and rename. Internally, they all call the function sys_renameat2 with different actual parameters. And inside of it the code checks if src and dst are at the same mounted file-system. If not, –EXDEV is returned which is then propagated to the user-space program:

/* ... */
        error = -EXDEV;                                                                                                           
        if (old_path.mnt != new_path.mnt)                                                                                         
                goto exit2;  
/* ... */

Then the error is returned:

/* ... */
exit2:                                                                                                                            
        if (retry_estale(error, lookup_flags))                                                                                    
                should_retry = true;                                                                                              
        path_put(&new_path);                                                                                                      
        putname(to);                                                                                                              
exit1:                                                                                                                            
        path_put(&old_path);                                                                                                      
        putname(from);                                                                                                            
        if (should_retry) {                                                                                                       
                should_retry = false;                                                                                             
                lookup_flags |= LOOKUP_REVAL;                                                                                     
                goto retry;                                                                                                       
        }                                                                                                                         
exit:                                                                                                                             
        return error;                                                                                                             
}

This explicit check has been for forever in this function. Why is it there, though? I guess Linux just took an simpler (and more elegant, should I say) path here and added this constraint from the beginning just so that the code for the file-systems would not have to accompany for this case. The code of different file-systems is complex as it is right now. You could find the whole source code of this function in fs/namei.c.

Indeed, old_dir->i_op->rename() is later called in sys_renameat2old_dir is of type struct inode *, i_op is a pointer to a const struct inode_operations. That structure defines a bunch of pointers to functions that perform various operations with inodes. Then different file-systems define their own variable of type struct inode_operations and pass it to the kernel. It seems to me that it would be indeed a lot of work to make each file-systems rename() inode operation work with every other file-system. Plus, how would any file-system ensure future compatibility with other file-systems that the user could use by loading some custom made kernel module?

Fortunately, we could implement renaming files by other means, not just by directly calling the rename system call. This is where shutil.move() comes in…

Difference between os.move() and shutil.move()

shutil.move() side-steps this issue by being a bit more high-level. Before moving it checks whether src and dst reside on the same file-system. The different thing is here what it does in the case where they do not. os.move() would blindly fall on its face here but shutil.move() is “smarter” and it does not just call the system call with the expectation that it will succeed. Instead, shutil.move() copies the content of the src file and writes it to the dst file. Afterwards, the src file is removed. As the Python documentation puts it:

If the destination is on the current filesystem, then os.rename() is used. Otherwise, src is copied (using shutil.copy2()) to dst and then removed.

So not only it copies the content of the file but shutil.copy2() ensures that the meta-data is copied as well.This presents an issue because the operation might get interrupted between the actual copying of the content and before src is removed so potentially you might end up with two copies of the same file. Thus, shutil.move() is the preferred solution to the original problem presented at the start of the article but however be wary of the possible problems and make sure that your code handles that case if it might pose a problem to your program.

Making Unwinding Functions in C Simple: Do Not be Afraid of Using Gotos

Intro

Today I wanted to talk about unwinding and releasing resources in C functions. Let’s begin by stating that there are three main techniques for handling errors in the C programming language. Sometimes more than one technique may be used. Here is a list of them:

  • You must test the value functions return. Abnormal value indicates that some kind of error has happened and a normal value indicates that it was successful;
  • There is an external variable whose value you must check. For example, the POSIX variant of this is to have an variable called errno that changes to 0 when nothing bad happened and it has some kind of other value when an error occurs;
  • You pass a pointer to a function. The function changes the value of the variable it points to or even calls it with certain arguments if it is a function pointer depending on the result.

I have not mentioned one method but some people use atexit(3) to register functions that will be called at the end of a program which will release resources. However, this is unusual so I have not included it in the list.

This is very much related to our topic because when an error occurs, you will have to handle it. That process includes releasing the resources which were acquired before in the function. Especially if you are deep down in your function and then an error occurred, the choice that you make in how to release the resources will matter a lot so it is important to make the correct decision.

In C++ you have the destructors and so on but how are you going to do that in C?
Are you going to sprinkle all of your error paths with:

free(foo);
free(bar);

and so on? It might be your first choice to go down this route but I think a viable and preferred alternative to this is using gotos and labels. Obviously, they should be used very cautiously. It is a very powerful tool so there is a lot of peril involved and ways to abuse it so you have to be absolutely careful. For simple cases when you don’t have to release any resources a plain return works well but it is a different situation with multiple resource acquisitions. Compared with other methods, using gotos doesn’t force you to duplicate the error paths, the code distracts less from the normal path, and it is more readable. You can’t imagine how this could be true and you cannot believe me? Let me prove to you that you should use gotos in these more complex situations!

Tutorial: using gotos for cleaning up

First, you should begin by naming the goto labels according to the resources that it frees. You want to be able to discern which resource exactly is going to be freed. Also, because goto labels may be used for other purposes other than resource clean-up, it is a good idea to prefix the goto labels with “err_” to indicate that its purpose is for releasing resources when an error occurs. Due to the fact that you will have different labels for different resources that they release, they should only contain one statement after it before the next label or the final return, and only do what it actually says.

Some good examples of names: err_release_view, err_free_list, err_close_lsocket, and so on.

Order the labels in such order that resources which are acquired first are at the bottom. The order of labels which release the resources should be in the inverse order of which they were acquired.

Now whenever an error occurs, use goto to jump to that label which will release the resources that were already gotten. As a rule, you can remember this: always jump to that label which releases the most recently acquired resource. This rule makes it easy to remember.

It may remind you of the defer mechanism in Go and other programming languages where the programmer can specify a list of functions with certain arguments which will be called as soon as the function goes out of scope. We are essentially emulating the same thing with gotos. Just that the C version requires a bit more attention and carefulness.

Example code comparison

To show how readability could be improved by using this method I will present one function from the Linux kernel source code and how it was changed. This function was improved courtesy of Tobin C. Harding. Thanks!

Here is the first version which does not use gotos at all:

static int enqueue_txdev(struct ks_wlan_private *priv, unsigned char *p, unsigned long size, void (*complete_handler)(void *arg1, void *arg2), void *arg1, void *arg2)
{
  struct tx_device_buffer *sp;

  if (priv->dev_state < DEVICE_STATE_BOOT) {
    kfree(p);
    if (complete_handler)
      (*complete_handler) (arg1, arg2);
    return 1;
  }

  if ((TX_DEVICE_BUFF_SIZE - 1) <= cnt_txqbody(priv)) {
    /* in case of buffer overflow */
    DPRINTK(1, "tx buffer overflow\n");
    kfree(p);
    if (complete_handler)
      (*complete_handler) (arg1, arg2);
    return 1;
  }

  sp = &priv->tx_dev.tx_dev_buff[priv->tx_dev.qtail];
  sp->sendp = p;
  sp->size = size;
  sp->complete_handler = complete_handler;
  sp->arg1 = arg1;
  sp->arg2 = arg2;
  inc_txqtail(priv);

  return 0;
}

The version with goto:

static int enqueue_txdev(struct ks_wlan_private *priv, unsigned char *p, unsigned long size, void (*complete_handler)(void *arg1, void *arg2), void *arg1, void *arg2)
{
  struct tx_device_buffer *sp;
  int rc;

  if (priv->dev_state < DEVICE_STATE_BOOT) {
    rc = -EPERM;
    goto err_complete;
  }

  if ((TX_DEVICE_BUFF_SIZE - 1) <= cnt_txqbody(priv)) {
    /* in case of buffer overflow */
    DPRINTK(1, "tx buffer overflow\n");
    rc = -EOVERFLOW;
    goto err_complete;
  }

  sp = &priv->tx_dev.tx_dev_buff[priv->tx_dev.qtail];
  sp->sendp = p;
  sp->size = size;
  sp->complete_handler = complete_handler;
  sp->arg1 = arg1;
  sp->arg2 = arg2;
  inc_txqtail(priv);

  return 0;

err_complete:
  kfree(p);
  if (complete_handler)
    (*complete_handler) (arg1, arg2);
  return rc;
}

As we can see, the code is much more readable and the two error paths are not duplicated. The judicious use of gotos avoids the perils of producing spaghetti code. Also, don’t worry: this not the only case. The Linux kernel source has an uncountable number of such examples. It makes the code much more readable once you get used to this convention. Not to mention that the Linux kernel is one of the biggest, most complex C projects around. So you know that the developers wouldn’t make a decision to use such code constructs which would increase the complexity of the code even more.

One more thing – this cleanup code is simple and clean but imagine a situation where it is much more complex. What if something extra was done in the error path if, for example, closing a socket failed and some extra sub-system had to be informed or some other actions had to be performed? That would be quite some extra code in each path. In this case, the goto method would be so much more attractive.

Conclusion

Using gotos in your C code to clean up after errors have occurred is similar to the defer mechanism in Go. Having clean-up code in one place which may be called completely gets rid of code duplication in error paths. This in part makes the code more readable because the reader won’t be distracted by the error handling code which could possibly obscure the real path. Also, there is less possibility of errors because potentially much less code is duplicated. The gotos can be abused easily so you should be very cautious and follow the tips given in this article.

Bonus: your compiler might have an extension to help out with this

Some C compilers have extensions which help with resource cleanup. For example, the popular gcc supports the cleanup attribute which applies to variables which have automatic storage duration. If you apply this attribute, gcc will run a function with that variable as the argument. Any return value is ignored. Example usage:

void cleanup_free(void *p)

{
  free(*(void **)p);
}




void foo(void) 
{
  char __attribute__((cleanup(cleanup_free))) *bar;
  bar = malloc(128);
} 

This extra function is needed because if ordinary free(3) would be written then it  would receive a char** and, obviously, free(3) doesn’t know that it should be dereferenced one time first. If you compile this function and run it with valgrind then you will see that no memory was leaked. This is also useful with close(2) and other similar functions. However, there is one downside – if you want more granular control of what happens if, for example, close(2) fails then it is impossible with this because any return value is ignored silently.  Check out your compilers’ documentation if there is support for this kind of thing. Obviously, you should consider the alternative of writing portable code first.
Please comment if you find any errors or just want to discuss this.