{"componentChunkName":"component---src-templates-blog-template-js","path":"/blog/lazy-dynamic-programming","result":{"data":{"markdownRemark":{"html":"

I recently stumbled upon the edit distance problem again while working on an ElasticSearch problem. It reminded me of the wonder and joy of studying algorithms. However well-trodden a path, another clever \"trick\" lies waiting to be found. Using the power of lazy evaluation to solve the edit distance problem is a beautiful rendition of an old favourite, yet current problem.

\n

Github repository with tests here. The following code samples are provided in Python where applicable and Haskell to demonstrate the constrast between the imperative and functional solutions. We will discuss 3 solutions including brute-force O(3N), standard dynamic programming O(|A| * |B|), and lazy dynamic programming O(|A| * (1 + D A B)).

\n

Based on the paper Lazy Dynamic-Programming Can be Eager by Dr. L. Allison (1992).

\n

The Problem

\n

Given 2 strings A and B, find the Levenshtein distance, the minimum number of point-mutations (delete, insert, substitutions) required to transform them into identical strings.

\n

For example, the word pair (\"cgggtatccaa\", \"ccctaggtccca\") has an edit distance of 6

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
EditiDiffResult
///\"cgggtatccaa\"
DEL1g\"cggtatccaa\"
SUB1g -> c\"ccgtatccaa\"
SUB2g -> c\"ccctatccaa\"
INS5g\"ccctagtccaa\"
INS6g\"ccctaggtccaa\"
SUB10a -> c\"ccctaggtccca\"
\n

Note that there may be multiple sets of point-mutations that arrive at the same solution.

\n

The edit distance problem is useful in a variety of applications including DNA sequencing, spell-check, and spam filtering.

\n

Subproblems

\n

The solution to this problem comes from the insight that the cost of mutating a string can be minimised by taking the fewest mutations required for a[:1], a[:2]..,a. Note that a similar principle applies if we check from the last character of the string a[-1], a[-2:].., a.

\n

To take the example above, (\"cgggtatccaa\", \"ccctaggtccca\"), if we start from the last character of either string, conversion of (a[-1], b[-1]), (\"a\", \"a\") requires a cost of 0. Then looking at (a[-2:], b[-2:]), (\"aa\", \"ca\"), the cost relies on the cost of the previous comparison, (\"a\", \"a\") and potential additions or deletions (\"\", \"a\"), (\"a\", \"\"). The last 2 are our base cases at cost 1.

\n

From this insight, we derive the following:

\n

The cost of converting substrings of a to b to each other is not incremented if the characters being examined match. In all other cases, a point-mutation is required so 1 is added to the minimum of 3 costs.

\n

Naive Solution: O(3N)

\n

where N = min(|A|, |B|).

\n

By checking all possible mutations recursively, we can incrementally build to the solution for the entire string. So far nothing special between either implementations.

\n

Python Implementation

\n
def edit_distance_rec(a, b):\n    if len(a) == 0:\n        return len(b)\n    if len(b) == 0:\n        return len(a)\n    if a[-1] == b[-1]:\n        return edit_distance_rec(a[:-1], b[:-1])\n    else:\n        return 1 + min(edit_distance_rec(a[:-1], b),\n                       edit_distance_rec(a, b[:-1]),\n                       edit_distance_rec(a[:-1], b[:-1]))
\n

Haskell Implementation

\n
recursiveEd :: String -> String -> Int\nrecursiveEd [] b = length b\nrecursiveEd a [] = length a\nrecursiveEd a@(ax: axs) b@(bx: bxs) = if ax == bx then recursiveEd axs bxs\n                        else 1 + minimum [recursiveEd axs bxs, recursiveEd axs b, recursiveEd a bxs]
\n

Tabular Dynamic Programming (O(|A| * |B|))

\n

The previous solution works but is slow as there are recurring sub-problems that are not exploited. Dynamic programming is typically used to cache these previous edit values to avoid re-computation.

\n

Python Implementation

\n
def edit_distance(a, b):\n    memo = [[0] * (len(a)+1) for _ in range(len(b)+1)]\n    memo[0] = [x for x in range(len(a)+1)]\n    for i, row in enumerate(memo):\n        row[0] = i\n    for i in range(1, len(b)+1):\n        for j in range(1, len(a)+1):\n            if b[i-1] == a[j-1]:\n                memo[i][j] = memo[i-1][j-1]\n            else:\n                memo[i][j] = min(memo[i-1][j]+1, memo[i][j-1]+1, memo[i-1][j-1]+1)\n    return memo[-1][-1], memo
\n

Haskell Implementation

\n

DP in a functional language with immutable variables utilises the caching property of unevaluated values, thunks in Haskell. Each row is built recursively by taking as input the previous row, the row count, and character being compared.

\n

An in-depth explanation of how DP problems are solved in Haskell can be found here.

\n

This is a copy of the implementation provided in the paper, translated to Haskell and commented.

\n
-- | Solve the edit distance problem\ndynProgEd :: \n    String      -- | first string a\n    -> String   -- | second string b\n    -> Int      -- | output to our edit distance problem\ndynProgEd [] b = length b\ndynProgEd a [] = length a\ndynProgEd a b = \n    let \n        -- | Calculate all subsequent rows from the previous of the memoisation table\n        nextRows :: \n            [Int]       -- | the previous row in the memoisation table\n            -> [Char]   -- | the remaining characters of string a to be compared\n            -> [[Int]]  -- | subsequent rows of the memoisation table\n        nextRows _ [] = []\n        nextRows prevRow (ax:axs) = \n            let \n                -- | Calculates the rest of the row from the previous character in the row\n                doRow :: \n                    [Char]      -- | the remaining characters of string b to be compared\n                    -> [Int]    -- | the previous row of the memoisation table starting from the column being examined\n                    -> Int      -- | the previous cost assignment west of the current square\n                    -> [Int]    -- | the rest of the row, cost of the remaining characters of string b\n                doRow [] _ _ = []\n                doRow (bx:bxs) (nw:n) w =\n                    let me = if ax == bx then nw \n                        else 1 + minimum [w, nw, (head n)]\n                    in me:doRow bxs n me\n                firstElement = 1 + (head prevRow)\n                thisRow = firstElement:(doRow b prevRow firstElement)\n            in thisRow:nextRows thisRow axs\n        table = [0..length b]:nextRows (head table) a\n    in last $ last table
\n

Lazy Dynamic Programming O(|A| * (1 + D A B))

\n

Haskell Implementation

\n

In the previous solutions, we built a memoisation table to calculate our edit distance values. However this is inefficient because the only value that matters is the one in the last position of some diagonal in table! This means that the number of intermediate calculations to get to that value can be minimised. How do we use laziness to do this? We will go through 3 general ideas then let the code speak about its specifics; they are:

\n
    \n
  1. Prefer diagonals over rows to enforce order of evaluation
    2. \n
  2. Force evaluation in a consistent direction
    3. \n
  3. Be lazy - declare calculation of new values based on un-evaluated dependencies
    4. \n
\n

Let's take the simple case where a = "aaa" and b = "aaaaa". Imagine for a moment that we were using the earlier method of expanding the entire memoisation table - it would look like this.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
''aaaaa
''012345
a101234
a210123
a321012
\n

Because we are lazily evaluating the entire table not all of these values will be filled in, but for now this gives us a visual representation of what we are trying to do. For this simplified example, in theory we should ever only need to calculate the values marked by X and Y here. The Y values need to be known as we look at the NW, W and N values of a particular entry to determine X.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
''aaaaa
''XY2345
aYXY234
a2YXY23
a32YX12
\n

It suggests that we need to rethink the way we represent the table. Instead of rows and columns, we structure our data as 3 diagonal groups instead. This allows us to better enforce our intended order of evaluation.

\n
mainDiag :: [Int]\nmainDiag = [0, 0, 0, 0]\n\nlowers :: [[Int]]\nlower = [\n    [1, 1, 1],\n    [2, 2],\n    [3]\n]\n\nuppers :: [[Int]]\nuppers = [\n    [1, 1, 1, 1],\n    [2, 2, 2, 2],\n    [3, 3, 3],\n    [4, 4],\n    [5]\n]
\n

One thing we notice is that if len(a) == len(b), the last value in mainDiag is our answer. Otherwise if len(a) != len(b), we look in either lowers or uppers for the corresponding corner value.

\n

Depending on the input strings of course, this path starting in the top-left can snake to the left or the right of the table and cause further evaluation of uppers or lowers. This change in direction is governed by our simple minimum [W, NW, N] when a point mutation is required. Where possible we prefer expanding in one direction rather than in both. The wonderful solution to this problem was to simply introduce:

\n
specialMin3 a b c = if a < b then a else min b c
\n

Now we worry about how to fill in our values. lowers is represented by L, mainDiag by M, and uppers by U. We initialise our 3 diagonal stores to depend on their NW, W and N values. To construct the next value of mainDiag here for example, it needs to take as input head uppers and head lowers. This is allowed even if uppers and lowers have not been evaluated. Coming from an imperative programming background, this was something quite difficult to wrap my head around, but reading more about thunks and how functional programming works was incredibly useful in bridging this gap.

\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n
''aaaaa
''MU----
aL?----
a------
a------
\n

And that's it! This sums up the main points I had trouble with understanding the algorithm. This is a copy of the implementation provided in the paper, translated to Haskell and commented.

\n
-- | Solve the edit distance problem\nlazyDynProgEd :: \n    String      -- | first string a\n    -> String   -- | second string b\n    -> Int      -- | output to our edit distance problem\nlazyDynProgEd [] b = length b\nlazyDynProgEd a [] = length a\nlazyDynProgEd a b = \n    let mainDiag = oneDiag a b (head uppers) (-1:(head lowers))     -- | -1 initialises the top-left square in the mainDiag to 0\n        uppers = eachDiag a b (mainDiag:uppers)                     -- | uppers passed to eachDiag expands to form the remaining diags\n        lowers = eachDiag b a (mainDiag:lowers)                     -- | lowers passed to eachDiag expands to form the remaining diags\n        -- | Constructor for a diagonal between diagAbove and diagBelow which lazily expands as values become known\n        oneDiag :: \n            String      -- | first string a\n            -> String   -- | second string b\n            -> [Int]    -- | diagAbove\n            -> [Int]    -- | diagBelow\n            -> [Int]    -- | edit distance values for the middle diagonal\n        oneDiag a b diagAbove diagBelow =\n            let \n                doDiag :: \n                    String      -- | the remaining characters of string a to be compared\n                    -> String   -- | the remaining characters of string b to be compared\n                    -> Int      -- | nw element to the current square in the diagonal (the previous element in this diagonal)\n                    -> [Int]    -- | diagAbove\n                    -> [Int]    -- | diagBelow\n                    -> [Int]    -- | edit distance values for the middle diagonal from the 2 input strings\n                doDiag [] _ _ _ _ = []\n                doDiag _ [] _ _ _ = []\n                doDiag (ax:axs) (bx:bxs) nw n w =\n                    let me = if ax == bx then nw\n                        else 1 + specialMin3 (head w) nw (head n)\n                    in me:(doDiag axs bxs me (tail n) (tail w))\n                firstElement = 1 + head diagBelow\n                thisDiag = firstElement:(doDiag a b firstElement diagAbove (tail diagBelow))\n            in thisDiag\n        -- | Constructor for either upper or lower diagonals. The functions a', bx', and bx' are apostrophed to differentiate them from strings a and b in the outer scope.\n        eachDiag :: \n            String      -- | first string a'\n            -> String   -- | the remaining characters of string b' to be compared\n            -> [[Int]]  -- | (lastDiag which started from bx'):(list of diagonals from comparing the remaining characters in bxs')\n            -> [[Int]]  -- | a list containing all diagonals from comparing the remaining characters in b'\n        eachDiag a' [] diags = []\n        eachDiag a' (bx':bxs') (lastDiag:diags) =\n            let nextDiag = head $ tail diags                    -- | grab the successor of thisDiag in uppers/lowers to form thisDiag\n                thisDiag = oneDiag a' bxs' nextDiag lastDiag\n            in thisDiag:(eachDiag a' bxs' diags)\n        lengthDiff = (length a) - (length b)\n    -- | only evaluate what is necessary\n    in last (if lengthDiff == 0 then mainDiag \n    else if lengthDiff > 0 then (lowers !! (lengthDiff-1)) \n    else (uppers !! (abs (lengthDiff)-1)))
\n
-- | always expand lowers if possible\nspecialMin3 :: Ord a => a -> a -> a -> a\nspecialMin3 a b c = if a < b then a else min b c
\n

Final thoughts

\n

The paper explains the algorithm more concisely and it is worth reading. My thoughts here are about the points that I didn't quite appreciate until I had a closer look at this, at face value, complex algorithm, so I hope it was at least somewhat useful for the reader. Do visit the repo here and if you like, make changes and run the tests provided.

\n

References

\n

Allison, L. (1992). Lazy dynamic-programming can be eager. Information Processing Letters, 43(4), 207-212. doi:10.1016/0020-0190(92)90202-7

","timeToRead":11,"frontmatter":{"date":"11 April 2020","title":"Lazy Dynamic-Programming for Edit Distance","summary":"Using the power of lazy evaluation to solve an old favourite, yet current problem."}}},"pageContext":{"slug":"/blog/lazy-dynamic-programming"}},"staticQueryHashes":["671693147","671693147"]}