Upper/lower case issues with non-english texts

[humanzz]

Hello,
I was using the ahocorasick library to perform some text processing on English Wikipedia dump. As it contains several non-english words, it lead me to discovering an issue in how the library handles cases.

I wrote a test case to explain it. Here it is

@Test
public void caseInsensitiveTrieWithSomeUnicodeCharactersCreatesEmitsWithWrongStart(){
    // when this is lower cased, it becomes a string of length 2!
    String upperLengthOne = "İ";
    String normalI = "I";
    Trie trie = Trie.builder()
                    .caseInsensitive()
                    //.onlyWholeWords()
                    .addKeyword(upperLengthOne)
                    .build();
    // because when lower cased it becomes 2 characters, the emit gets confused and creates a string that starts at -1
    // this can cause further problems if we index into the original string with this emit start and end.
    // This happens if we make the trie builder.onlyWholeWords() in which case we get exceptions
    assertEquals(-1, trie.parseText(upperLengthOne).stream().findAny().get().getStart());
}

The whole problem happens when lower-casing the keyword has a different length than the original keyword. I can't think of a quick easy fix for this issue, but I guess the least that can be done about it is that the library should throw an error when the lowercased keyword's length is != the original keywords length warning the library's user to this issue.

[humanzz]

Building on the fix to unicode issues in #8, I fixed the problem in the pull request above.

[matanox]

Was a solution finally merged?

[ghost]

I don't believe the merge can happen until some of the conflicts have been resolved.

[matanox]

Sounds like a mildly horrific bug to keep in circulation :)
Thanks for spotting it