Concepts in Computing
CS4 - Winter 2007
Instructor: Fabio Pellacini

Lecture 19: Decideability

Introduction

Up to this point, we have examined how to design algorithms to solve computational problems, how to analyze those algorithms, and how to describe them using an English-like pseudo-code and the JavaScript language. It may therefore be somewhat surprising to learn that there are problems -- not even extremely esoteric ones -- for which there exist no algorithms to solve them.

Informally, we say that a problem is decidable if we can construct an algorithm that computes the answer to that problem, for any input of the correct form. If we cannot construct such an algorithm, we say that the problem is undecidable.

If we want to prove that a problem is decidable, all we have to do is to demonstrate an algorithm that solves the problem. For example, in an earlier lecture, we learned an algorithm for adding integers represented in binary; and we may therefore conclude that integer addition is a decidable problem.

Here are a couple of problems which cannot be solved by algorithms:

  • Suppose you have two computer programs -- perhaps one of them is the sample solution to a homework problem, and the other is a student's homework submission for that problem.

    These two programs are not identical, but you would like to determine if they do the same thing. Doing this by hand seems practically impossible. Ideally, you'd like to be able to write a third program which, given both of these programs, will tell you whether they compute the same results for any given input.

    Unfortunately, this is not possible!

  • Suppose you have written a computer program which performs a large and sophisticated computation. You feed it an enormous set of data, and set it running.

    After a while, you begin to wonder, "is this program ever going to stop and give me an answer? What if there is some kind of error in the program, which will cause it to run forever and ever without giving me a result?" You could stop the program, but what if it was only a few seconds away from being finished?

    What if you could design a computer program which could look at your code, and tell you right away "if you run this program, it will never halt." That would be a valuable tool! Sadly, this too is impossible. This is called the Halting Problem.

How do we prove that there is no algorithm to solve some problem? We might sit down, think really hard, and realize that we cannot think of any algorithm that will always work. We might, then, conclude that no such algorithm exists. But what if some really clever person comes along a few minutes later -- or a few weeks, months, or even years later -- and comes up with an algorithm that does work correctly? It's not sufficient that we can't find an algorithm by thinking really hard -- we need to prove that such an algorithm cannot exist, in order to show a problem to be undecidable.

Here's how we'll proceed:

  1. We'll talk about a mathematical model of computation called a Universal Register Machine.
  2. We'll look at how algorithms can be expressed using this mathematical model.
  3. We'll look at how you can use this model to prove the impossibility of a program that decides if other programs halt.

Intuition

  • Suppose we have a program M that takes as input an encoding of a program P and its argument, and returns whether or not P halts when given the argument.
    Halt 1
  • Build a program Q that takes as input an encoding of a program P and runs our program M to see whether P halts given its own encoding as an argument. If M indicates that P halts, then Q enters an infinite loop; if M indicates that P doesn't halt, then Q halts.
    Halt 2
  • What happens when we give Q as input an encoding of itself? If M decides that Q halts, then Q doesn't halt; if M decides that Q doesn't halt, then it does. The way out of this contradiction is to conclude that there is no such M.
    Halt 3

A Mathematical Model of Computation

A model is basically a simplified analogy for something complex. The point of constructing a model is to capture the essential attributes of an idea, without getting bogged down in a lot of details. What we'd like to construct is a model of a very simple type of computer that has the following properties:

  1. It should be very simple, so that it can be described clearly and without ambiguity, and manipulated mathematically without too much effort.
  2. It should be capable of expressing any computation we can come up with an algorithm for.

We have already learned about one model for a computer, that is the Von Neumann machine. That might seem simple enough, compared to what we're used to, but there are a lot of details to the Von Neumann machine that aren't really concerned with what the machine can do, but how it does it. The ALU, the bus, registers, memory, the MAR and MDR, the PC, the IR, etc., are tricky to keep track of. We would prefer to have something even simpler than that, provided we don't sacrifice any computing power (that is, provided we can compute all the same things on our new model, that we could on the Von Neumann machine).

A Simple Machine: The Universal Register Machine

Your textbook presents a model of computation called a Turing Machine, named after the luminary computer scientist Alan M. Turing. However, we will look at a different model, called a Universal Register Machine, or URM, which is a bit more familiar looking than the Turing Machine model.

A URM has three parts:

  • A set of registers, each of which has a unique natural number address. Each register can hold a single natural number value.
  • A program store, which is an array of instructions.
  • A program counter, which is a natural number indicating what instruction is to be executed next.

Here is a diagram that illustrates the components of a URM:

URM Example

This should look familiar to you -- it's very similar in structure to a von Neumann machine, except that instead of having separate memory and registers, we have only registers, and we do not worry about things like I/O or the bus -- the URM is an abstract model of computation, so we're not really concerned with how the pieces communicate in a real computer.

URM Instructions

A URM has a very simple instruction set. In fact, the whole instruction set consists entirely of the following four instructions:

Instruction Description
CLEAR x Set the value of register x to be zero.
INCREMENT x Add 1 to the value stored in register x.
COPY x, y Copy the value in register x to register y.
JUMPEQ x, y, z If the value in register x equals the value in register y, set the program counter to z; otherwise, the program counter is incremented.

Computing with a URM

What does it mean to compute with a URM? To specify a URM, you must supply the following information:

  1. The contents of the program store (instructions).
  2. The initial contents of the registers.

We will assume that the program counter (PC) always begins at 0. Furthermore, if the PC ever becomes greater than or equal to n, which is the size of the program store, we will assume that the URM halts (stops computing).

In other words, the URM's execution algorithm is as follows:

1.  PC = 0
2.  while (PC < n)
3.    if (program[PC] is "clear x")
4.      set R[x] to zero
5.    else if (program[PC] is "increment x")
6.      add 1 to R[x]
7.    else if (program[PC] is "copy x, y")
8.      set R[y] to the value of R[x]
9.    if (program[PC] is "jumpeq x, y, z" and R[x] == R[y])
10.     PC = z
11.   else
12.     PC = PC + 1
13. end of loop
14. halt.

With such a simple instruction set, it might seem like you could hardly implement any interesting algorithms at all! But, as it turns out, even this simple set of instructions is adequate to implement any algorithm you could write in JavaScript or the more complex instruction set of the von Neumann machine we discussed in lecture.

Models & Algorithms, The Church-Turing Thesis

Remember that at the outset, our goal was to come up with a model of a computer that was simple enough to reason about, but powerful enough to do anything a human can design an algorithm to do. Naturally, there is no way to prove that we can't come up with an algorithm that could not be computed by a URM; however, it is widely supposed to be true. This belief, proposed by Alonzo Church and Alan Turing, is usually called the Church-Turing Thesis.

Simply put (in terms of our URM model), the Church-Turing Thesis boils down to this:

Any algorithm a human can possibly invent can be translated into a URM program that computes that algorithm.

No one has ever found an algorithm that could not be translated into a URM, and in fact, no one has found another model of computation that has more power than a URM (i.e., all the currently known models of computation that have been developed can be shown to be equivalent). For this reason, although the Church-Turing Thesis is a "leap of faith" -- it must simply be believed, since it cannot be proven -- it is a very reasonable belief to hold, given our experience as scientists studying the properties of algorithms and computers.

If we can accept the Church-Turing Thesis, however, we now have a concrete way of proving assertions about algorithms. Why is that so? Because now, if we have an algorithm, we can construct a URM that computes that algorithm, and use logical methods to prove things about the behavior of that machine. If the URM is equivalent to the algorithm, then conclusions we make about the URM should also apply to the algorithm.

Thus, provided the Church-Turing Thesis is true, if we can show that no URM can be constructed to solve a problem, then we have done the equivalent of proving that no algorithm can be constructed to solve that problem.

The Halting Problem

One example of an undecidable problem is the Halting Problem:

Design an algorithm A which takes a program P and an input value x for that program. Algorithm A should halt and return 1 if running P(x) would eventually stop and return an answer; otherwise, A should halt and return 0, indicating that P(x) would run forever without stopping.

Encoding the Problem

What does it mean to "design a URM that solves the Halting Problem?" Recall that in order to compute with a URM, you need to know:

  1. A sequence of instructions (the program store).
  2. The starting values of all the registers.
  3. The initial value of the program counter.

By convention, let's assume that the URM's program counter always begins at location 0, and that we only need to give the initial values of the registers we "care" about (i.e., we'll assume any registers we don't specify initial values for have arbitrary unknown values).

So, to be precise, let us say that a URM M that solves the Halting Problem should behave as follows:

  1. We will put the encoding of a URM P into R[0]
  2. We will put the encoding of an input value x into R[1]
  3. When M halts, the value of R[0] is 1 if running the URM P on input x would eventually halt; and the value of R[0] is 0 if running P on x would not halt.

Since the registers of a URM contain natural numbers, we have to find a way to encode P as a natural number. Fortunately, that is not difficult -- all we really need to specify are the instructions. We can do this as follows:

  • For any natural number n, let B(n) denote the representation of n as an unsigned binary number (we know how to obtain this).
  • Execute the following algorithm, assuming n is the number of instructions in the URM's program store and that instruction is an array of those instructions:
    1.  Set i to 0
    2.  Set output to ""
    2.  While i < n, do
    3.    If instruction[i] == "CLEAR x" then
    4.      Append "1" + B(x) to output
    5.    Else if instruction[i] == "INCREMENT x" then
    6.      Append "2" + B(x) to output
    7.    Else if instruction[i] == "COPY x, y" then
    8.      Append "3" + B(x) + "5" + B(y) to output
    9.    Else if instruction[i] == "JUMPEQ x, y, z" then
    10.     Append "4" + B(x) + "5" + B(y) + "5" + B(z) to output
    11.   Add 1 to i
    12.   Append "6" to output
    13. End of loop
    14. Return output.
    15. Halt.
    

After doing this, we will have a number that "encodes" all the instructions in the program. For instance, given this URM program:

  0:  CLEAR 0
  1:  JUMPEQ 0, 1, 4
  2:  INCREMENT 0
  3:  JUMPEQ 0, 0, 1
  4:  COPY 1, 2

...here is the integer produced by the above algorithm:

106405151006206405051631510

It's a big value, but this is one of those cases where size doesn't matter: What we're after is a way of uniquely representing any possible URM program as a natural number, and this technique does that just fine. Here it is broken down into its pieces:

URM integer encoding

Notice that since all the argument values are encoded in binary, the digit "6" is free to be used as a separator between the instructions. Similarly, here is one of the encoded instructions broken out into its components:

URM instruction encoding

Here again, we can use the digit "5" to separate the binary encodings of the arguments, because it won't conflict with the 0's and 1's used by the binary representation. This is just a clever trick, but it guarantees that we get a unique natural number for any given URM program you might write.

To encode argument values, you can apply the same trick, using binary notation. For programs that require multiple input arguments, we'll adopt the convention that you encode each of the arguments in binary, then separate them with "7"'s. Again, this is just a trick that guarantees a unique representation. So, the arguments 5, 8, 14 might be encoded as:

  1017100071110

The Proof

Proving an existential statement (e.g., "There is at least one CS 4 student present in class today") -- find a single example. Similarly, disproving a universal statement (e.g., "All Dartmouth students are female") is also very easy. However, disproving universal statements (e.g., "All hikers carry water bottles") and disproving existential statements (our goal here -- "There is no algorithm to do ...") is very hard.

Since we can't really examine all possibilities, the proof often proceeds by logic: try to assume the opposite is true, and show that making that assumption leads to a logical inconsistency with other facts we know to be true. This technique is known as proof by contradiction.

Proof by Contradiction

Let's look at an example of proof by contradiction. Suppose we want to prove the statement "Chris is in Hanover". Let's first look at some things we know are true (or at least, that we knew to be true at the time this lecture was given!):

  1. If CS 4 is in session, then Chris is in Rockefeller.
  2. Anybody who is not in Hanover is not in Rockefeller.
  3. CS 4 is in session right now.

To prove by contradiction, we will pretend, for the sake of argument, that our statement is false. In other words, we are assuming that "Chris is not in Hanover." Then by rule (2), he must not be in Rockefeller. But, CS 4 is in session, by rule (3). Since that is true, rule (1) says that Chris is in Rockefeller. But these two statements are logically contradictory!

Since we got to this contradiction by assuming that our original statement was false, we have to conclude that it was a bad assumption. Therefore, we logically conclude that the original statement must have actually been true to begin with.

This is, of course, a very simple example, but it contains the essential characteristics of a proof by contradiction:

  1. You establish the statement you are trying to prove, call this Statement S.
  2. You establish the facts you know to be true about the world independent of whether S is true or false.
  3. You assume that S is false
  4. Using this assumption, prove that one or more of your facts (from step 2) is false.
  5. This is a logical contradiction, so you may assume that the assumption you made in step 3 is false -- therefore, your original assertion is true.

This general procedure is the outline of the method we will use to prove the undecidability of the Halting Problem.

Halting Problem, by Contradiction

Now we're ready to prove that the Halting Problem can't be solved. We'll do this by contradiction.

  1. Assume there exists some URM M which, when executed with R[0] containing the encoding of a URM P, and with R[1] containing the encoding of an input value x, determines whether running P on input x would eventually halt.
  2. Given that this M, construct a new URM program Q, in the following way:
    1. Q takes a single input, R[0]. Its first instruction, COPY 0, 1 copies it into R[1].
    2. After this copy, include the code for M. Thus Q will do exactly what M does. Increment the argument for each jump by 1, to account for the additional copy instruction that is Q's first step.
    3. Suppose n is the number of instructions in M's program. Anywhere in the code of M where there is a JUMPEQ x, y, z instruction where z > n, replace that instruction with JUMPEQ x, y, n+1.
      This means that, whenever M halts, it will do so by reaching location n+1 in the program store (recall that a URM halts whenever the PC runs off the end of the program).
    4. Beginning at location n+1, add the following instructions to Q's program:
        n+1: CLEAR 1
        n+2: INCREMENT 1
        n+3: JUMPEQ 0, 1, n+2
      
  3. Let us call Q's input x. Observe that, if running M on x, x leaves a 1 in R[0], then Q will loop forever without terminating; otherwise, Q will halt when its program counter reaches location n + 3 in the program store.
  4. Now, since Q is a URM, we can use the encoding procedure described above to obtain a natural number representing Q. Suppose we do this, and let E(Q) represent this number.
  5. Now, consider what happens if we run Q with R[0] set to E(Q)?
    Since we know M always halts with 0 or 1 stored in R[0], there are two possible outcomes:
    1. M returns 1. By the definition of M, this means that running Q on argument E(Q) will eventually halt.
    2. M returns 0. By the definition of M, this means that running Q on argument E(Q) will run forever without halting.
  6. But now consider case (5.a): As Q is defined, if M returns 1, then Q runs forever without halting when given x = E(Q) as input. But this contradicts the assumption that M returns 1 only if running Q on that input would eventually halt!
  7. So, now consider case (5.b): As Q is defined, if M returns 0, then Q halts when given x = E(Q) as input. But this contradicts the assumption that M returns 0 only if running Q on that input would run forever without halting!

In either case, we see that we have arrived at a logical impossibility. There were two assumptions that made this possible:

  1. The Church-Turing Thesis
  2. The assumption that M exists.

Thus, if we assume Assumption (1) is still true, the only possible explanation is that Assumption (2) must be false. And, we therefore conclude that there is no URM program which solves the Halting Problem.

Thus, assuming the Church-Turing Hypothesis is true, we conclude that no algorithm can solve the Halting Problem either.

Intuition, again

It helps to see the issue from multiple angles. At the start, we looked at a diagram of the intuition. We then showed how to formalize that intuition, in terms of URMs and proof by contradiction. Let's revisit the intuition (without the formality), in the JavaScript world.

Suppose there's a JavaScript library with a function in it called M. M takes two arguments, both strings. The first argument is a string containing a JavaScript function. For example, it could be the string:

  var P = "function hello(name) { alert('hello '+name); }";

The second argument is a string containing the argument to that function. For example, it could be the string:

  var arg = "Chris";

The people who gave us the library say that M can tell us whether or not a given function (described in a string) halts, when called with a given argument. Thus in our example, calling M(P,arg) should tell us whether or not hello("Chris") halts in a finite amount of time.

With such an M available in a library, we can write another function Q:

  function Q(A)
  {
    var result = M(A,A);
    if (result==true) {
      while (true) { 
        // Do nothing -- an infinite loop
      }
    }
    return 1;
  }

This is just writing some simple JavaScript, calling the library function with a couple of arguments, and doing something depending on what it returns. But now we can stick this JavaScript into a string:

  var B = "function Q(A) { var result ... }";

We can pass B to Q:

  var C = Q(B);

Now the question is: what is C?

  • If C is 1, then Q(B) must have halted. That means that M(B,B) returned false (since if it returned true, Q wouldn't have halted). By the definition of M, since M(B,B) is false, calling the function described in B on the argument B does not halt. But the function described in B is Q itself. So M says Q(B) doesn't halt, contradicting the first sentence in this paragraph.
  • Suppose Q(B) didn't halt (so C never got a value). That means M(B,B) returned true, and thus calling the function described in B on the argument B halts. But since B encodes Q, M says Q(B) halts, and again we have a contradiction.

In either case, there is a contradiction between what M says and what Q does. So M doesn't do what it is advertised to do -- accurately tell us whether or not a function halts given an argument. Since we didn't assume anything about M, this means that no such M can do that.

Philosophical Implications

The point of this exercise is to show that there are real limits to what kinds of problems can be solved on a computer. The Halting Problem is one example of a problem that is undecidable, but there are a great number of others, including:

Program equivalence
Given two programs P1 and P2, answer "yes" if for every possible input value x, P1(x) = P2(x); otherwise answer "no".
Program usefulness
Given a program P, answer "yes" if there is any input value x for which P(x) eventually halts, otherwise answer "no". (A program which doesn't halt for anything is certainly not useful!)
Minimal programs
Given a program P that computes some mathematical function f, answer "yes" if P is the shortest possible program that computes f, or answer "no" if there is a shorter program than P that computes the same thing.

As we've seen, the limitations of what is computable are not just esoteric -- even questions we care about, like the Halting Problem or the Program Equivalence Problem, are impossible in the general case.

On the other hand, that does not mean you can never solve these problems -- for instance, it may be possible for you to examine a particular program (either as a URM program, or in JavaScript) and determine that it will never halt:

  function P(x)
  {
    while(true) { }
  }
 0:  CLEAR 1
 1:  INCREMENT 1
 2:  JUMPEQ 1, 1, 1

The point is, you could never write a program that would give the correct answer for any JavaScript program you might write.