Confusion With File Redirection And sudo

Introduction

I have noticed that newbies sometimes fail to understand why, for example, sudo echo “hi” > /tmp/test will create a file /tmp/test with whatever effective user ID and effective group ID the command is ran with. They fall into the trap thinking that root:root will be the owner of /tmp/test. However, that is certainly not true. This blog post will try to clarify why this happens and what the user can do to avoid this issue. This occurs due to peculiar parsing rules defined in POSIX. We will examine the standards which detail this behaviour, the source code of the popular shells bash/zsh, and some methods on how to avoid this problem with, for example, tee(1).

Why it happens

You, my dear reader, have to understand first of all that sudo is just a command, a binary just like any else. It is a special binary, though, because it runs whatever command you pass as arguments. However, the redirection part (> /tmp/test) is not part of the command. This special file redirection syntax is interpreted by the shell, not passed to sudo for it to be executed later. So the shell that you are running gets special instructions to run that command with file descriptor number 1 which will be opened to write to a newly created file /tmp/test.

Notice that at this point command has not been run yet. The shell, before starting the command, begins to prepare the first fd. The shell does this with whatever EUID/EGID that shell is running with so it creates that new file with not the root:root rights but with whatever EUID/EGID the shell is running with. Most shells probably prepare the file descriptor 1 with the dup*(2) family of functions or fcntl(2), and open(2) before doing a fork(2) and exec*(3) just after it.

This functionality is mandated by a section of POSIX: http://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_07. All POSIX-compatible shells follow that section. Obviously, some shells are not compatible with POSIX. This article targets POSIX-compatible shells, however. In either case, to find out all of the gory details of how file redirection works, either refer to that section, read the fine manual, and/or read the source code of the shell that you are using. The latter option gives you the most detail and is the most trustworthy whereas the former one is more accessible. I will not go on a tangent here and I will redirect you to this article that I wrote before if you want to find more about this.

Next we will review the code of two popular shells: GNU/Bash and the Z shell, to see how they realize this functionality in code.

How do popular shells implement this

GNU/Bash uses the GNU/Bison project for specifying the input command syntax. The syntax file is written in the yacc format. You can find the special syntax described in the parse.y file, in the root directory of GNU/Bash source. There are two special types defined: redirection and redirection_list. Here is an excerpt from parse.y file which defines how redirecting stdout to a file works or how passing a file to stdin works:

redirection:	'>' WORD
			{
			  source.dest = 1;
			  redir.filename = $2;
			  $$ = make_redirection (source, r_output_direction, redir, 0);
			}
	|	'<' WORD 
			{
			  source.dest = 0;
			  redir.filename = $2;
			  $$ = make_redirection (source, r_input_direction, redir, 0);
			}
        | ...

And here is the excerpt which shows that the redirection list goes after a command:

command:	simple_command
			{ $$ = clean_simple_command ($1); }
	|	shell_command
			{ $$ = $1; }
	|	shell_command redirection_list
			{
			  ...
			}
	|	...

My point is that you can see that GNU/Bash really has a special syntax for file redirection, it is not passed to sudo. The shell command is separate from the redirection list. With the Z shell it is a bit different because there is no one single place where the grammar is defined. Instead, it has a big file dedicated for the hand-rolled parsing engine that is written in C. Thus, it will not be possible to present the whole file but here are the excerpts from the file which prove to you once again that the file redirection feature is indeed based on a special syntax, it is not passed to the actual binary as arguments. In the Z shell source code, Src/parse.c we can find this comment:

/*
 * cmd : { redir } ( for | case | if | while | repeat |
 * subsh | funcdef | time | dinbrack | dinpar | simple ) { redir }
 *
 * zsh_construct is passed through to par_subsh(), q.v.
 */

Indeed, redirection is parsed later in the function that is responsible for parsing the command:

static int
par_cmd(int *cmplx, int zsh_construct)
{
    int r, nr = 0;

    r = ecused;
    if (IS_REDIROP(tok)) {
        *cmplx = 1;
        while (IS_REDIROP(tok)) {
            nr += par_redir(&r, NULL);
        }
    }

    switch (tok) {
    case FOR:
          ...
    case FOREACH:
          ...
    case SELECT:
          ...
    case CASE:
          ...
    case IF:
          ...
    case WHILE:
          ...
    case UNTIL:
          ...
    case REPEAT:
          ...
    ...
    }

    if (IS_REDIROP(tok)) {
        *cmplx = 1;
        while (IS_REDIROP(tok))
            (void)par_redir(&r, NULL);
    }

    ...
    return 1;
}

As you can see, the redirections may be at either the end or the beginning of the command. Let me repeat again: file redirection is special syntax and it is not passed to the actual thing being run. The GNU/Bash shell supports the file redirection syntax at the beginning of the line as well but I have just decided to not include it for brevity.

So how to actually redirect output to a file as root?

I guess the most popular solution to this is to simply use a pipe to redirect output to a file. I am not sure but probably the program, tee(1), was made for this purpose. Or it was made as an extension of the tee system call but still it is the perfect tool to solve our issue. So, the solution to this problem would be:

echo "hi" | sudo tee /tmp/test >&-

This will at first (probably, depending on your set up) ask for your password. Then, “hi” will be written to a file descriptor which will get passed on to tee(1). Then, tee(1) will dump everything into /tmp/test. The >&- (or >/dev/null) is needed so that tee(1) would not output anything. tee(1), unfortunately, copies the content from standard input to standard output as well.

Another way to solve this is to pass more commands to run via root either by using the -s sudo option. For example, this works:

sudo -s <<EOF
exec >/tmp/test
echo "hi"
EOF

Or, run another shell using sudo which will redirect everything to /tmp/test:

sudo /bin/bash -c 'exec >/tmp/test; echo "hi"'

Or, run a script with sudo which will internally redirect stdout to /tmp/test:

cat >./script.sh <<EOF
exec >/tmp/test
echo "hi"
EOF
chmod +x ./script.sh
sudo ./script.sh

Obviously, there are more (complex) ways how you could achieve this but these are the main methods. You are free to adopt these examples to your own case. Please comment if you find any errors or if you want to add anything about this topic.

Review of Dale Carnegie’s “How to Win Friends and Influence People” for programmers

Programmers and IT people are often shunned for being shy, timid, fearful, or just in general – we are being thought of someone who have no social (soft) skills. If you can in any way identify with this sentiment – you definitely would be rewarded deeply for reading this book and implementing the suggested tips in your own life. But let us start at the beginning – who is the author, who is he known for, and so on, and then I will tell you why you should deliciously digest this book.

First of all, Dale Carnegie is an innovator of a new genre of books that are called “self-help” books these days. Such books feature short chapters with a life lesson at the end of them. This makes it very approachable for programmers who might have a short attention span or not have much energy, or time after programming for the whole day. It is worth mentioning that Carnegie was so popular and known that even Warren Buffett, a well known businessman, used to hang (or still does) on his wall a certificate that he finished his courses when Carnegie was still around. So if it appeals to people like Warren Buffett very much, it would help you a lot as well. Let me introduce to you a few chapters and tell you how they would help you.

Foremost, being proficient in negotiation tactics is a very useful and critical skill for everyone, not just salesmen and managers. We, as programmers, are sometimes shy, undervalue ourselves when looking for a job and do not ask for a proper wage. This work of art will help you get rid of that imposer’s feeling. I think that understanding the critical people skills presented in this publication and implementing them in your real life will deeply change your understanding of human interaction mechanics. Negotiation is essential when you are trying to make another person think like you. This is why Carnegie aptly named a chapter “Twelve Ways to Win People to Your Way of Thinking” which is dedicated to teaching you methods how to achieve this goal. For example, you probably never thought about the technique to “let the other person feel like that idea is his or hers”. Presenting an idea to someone like it was thought by themselves makes them more likely to accept it because that person feels like it is theirs. There are even 11 more ways presented in “How to Win Friends and Influence People”. This kind of advice is what makes this book great and they are all useful to you as a programmer. Hopefully, learning them will give a rise to having your wage substantially increased.

Furthermore, one thing for sure is that programmers are notorious for being egoistic. I feel that this book teaches how to be a humble, understanding, and a thoughtful person. It strongly encourages to be honest and always consider all problems or thoughts from the perspective of the other person. For example, in part 2, “how to make people like you”, Dale Carnegie specifically wrote a chapter which specifically inspires you to become a good listener and encourage others to talk. Author argues, I agree and you probably too with the assertion that people like talking about themselves, about their hobbies, about their days and so on. You cannot win a person’s mind to your way of thinking without listening to them and making them feel good. Or in other words, “don’t kick over the beehive if you want the honey” which is another quote from other chapter. Thus, this book will help you kick out the bad habits that we usually have as programmers.

Not to mention the very simple, understandable English used in the book. It appeals to the layman so it is especially approachable to anyone. In general, this piece of work is not philosophic, don’t need to think much – you ought to just apply the principles explained in the book. However, it might not be for you if you are looking for deep discussion about why people behave in one or other way and about topics such as sociology. The courses organized by Dale Carnegie himself were attended by all kinds of people – from salesmen to ordinary plumbers so it had to appeal to the lowest common denominator which is that they were all normal, hard-working people who are looking to improve their social skills to furthermore reach a bigger point in their careers or to become leaders in their respective fields. Personally, I think that this perfectly OK for books like these and reduces the list of reasons why you should not read this book by one more thing.

On the other hand, obviously not everything is rainbow and roses. This book has some downsides too. I thought that some examples were seriously out-dated and not applicable to the current world. For instance, in part one, “fundamental techniques of handling people”, an example is introduced of a “famous” quarrel between the two USA presidents Theodore Roosevelt and William Howard Taft. It really isn’t known at all anymore, especially it is not known for people who do not live in the USA and this just makes the reader feel like they are missing some details. Moreover, some reviews on other sites say that the examples are so bad that it is only worth reading the “in a nut-shell” sections at the end of each chapter. But I don’t think they are that bad – some of the examples are really great and illustrate the point that the author is making very well. Also, some advice seemed repetitive and just presented from the other perspective. Exempli gratia, in part three Dale Carnegie says to “Be sympathetic with the other person’s ideas and desires”. It is similar or almost identical to the advice given in part two: “Be a good listener. Encourage others to talk about themselves” . I think that being a good listener already involves sympathizing with the person that you are having a conversation with. You could find more examples of these issues. However, they are not very noticeable and do not distract from the main ideas of the book.

All in all, it is a great book for programmers. But just because of the negatives that I have listed, I would rate it a shining 9 out of 10. You would not waste time by picking it up as your next read and it would greatly influence your person character development to the positive side. With programmers becoming more and more equal in skill, the people skills are what makes someone shine. This may sound like an advertisement but you should not hesitate and pick this book up whenever you can in the future. It is one of those must-reads. If there is one thing that you will definitely take away from this book is that you will learn how to emphasize with your conversation partner and generally improve your manners.