Java Code Examples for java.text.Normalizer#Form

The following examples show how to use java.text.Normalizer#Form . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: NormalizerBase.java    From dragonwell8_jdk with GNU General Public License v2.0 6 votes vote down vote up
/**
 * Test if a string is in a given normalization form.
 * This is semantically equivalent to source.equals(normalize(source, mode)).
 *
 * Unlike quickCheck(), this function returns a definitive result,
 * never a "maybe".
 * For NFD, NFKD, and FCD, both functions work exactly the same.
 * For NFC and NFKC where quickCheck may return "maybe", this function will
 * perform further tests to arrive at a true/false result.
 * @param str       the input string to be checked to see if it is normalized
 * @param form      the normalization form
 * @param options   the optional features to be enabled.
 */
public static boolean isNormalized(String str, Normalizer.Form form, int options) {
    switch (form) {
    case NFC:
        return (NFC.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    case NFD:
        return (NFD.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    case NFKC:
        return (NFKC.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    case NFKD:
        return (NFKD.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    }

    throw new IllegalArgumentException("Unexpected normalization form: " +
                                       form);
}
 
Example 2
Source File: NormalizerBase.java    From openjdk-jdk8u with GNU General Public License v2.0 6 votes vote down vote up
/**
 * Test if a string is in a given normalization form.
 * This is semantically equivalent to source.equals(normalize(source, mode)).
 *
 * Unlike quickCheck(), this function returns a definitive result,
 * never a "maybe".
 * For NFD, NFKD, and FCD, both functions work exactly the same.
 * For NFC and NFKC where quickCheck may return "maybe", this function will
 * perform further tests to arrive at a true/false result.
 * @param str       the input string to be checked to see if it is normalized
 * @param form      the normalization form
 * @param options   the optional features to be enabled.
 */
public static boolean isNormalized(String str, Normalizer.Form form, int options) {
    switch (form) {
    case NFC:
        return (NFC.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    case NFD:
        return (NFD.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    case NFKC:
        return (NFKC.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    case NFKD:
        return (NFKD.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    }

    throw new IllegalArgumentException("Unexpected normalization form: " +
                                       form);
}
 
Example 3
Source File: NormalizerBase.java    From TencentKona-8 with GNU General Public License v2.0 6 votes vote down vote up
/**
 * Test if a string is in a given normalization form.
 * This is semantically equivalent to source.equals(normalize(source, mode)).
 *
 * Unlike quickCheck(), this function returns a definitive result,
 * never a "maybe".
 * For NFD, NFKD, and FCD, both functions work exactly the same.
 * For NFC and NFKC where quickCheck may return "maybe", this function will
 * perform further tests to arrive at a true/false result.
 * @param str       the input string to be checked to see if it is normalized
 * @param form      the normalization form
 * @param options   the optional features to be enabled.
 */
public static boolean isNormalized(String str, Normalizer.Form form, int options) {
    switch (form) {
    case NFC:
        return (NFC.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    case NFD:
        return (NFD.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    case NFKC:
        return (NFKC.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    case NFKD:
        return (NFKD.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    }

    throw new IllegalArgumentException("Unexpected normalization form: " +
                                       form);
}
 
Example 4
Source File: NormalizerBase.java    From openjdk-jdk8u-backup with GNU General Public License v2.0 6 votes vote down vote up
/**
 * Test if a string is in a given normalization form.
 * This is semantically equivalent to source.equals(normalize(source, mode)).
 *
 * Unlike quickCheck(), this function returns a definitive result,
 * never a "maybe".
 * For NFD, NFKD, and FCD, both functions work exactly the same.
 * For NFC and NFKC where quickCheck may return "maybe", this function will
 * perform further tests to arrive at a true/false result.
 * @param str       the input string to be checked to see if it is normalized
 * @param form      the normalization form
 * @param options   the optional features to be enabled.
 */
public static boolean isNormalized(String str, Normalizer.Form form, int options) {
    switch (form) {
    case NFC:
        return (NFC.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    case NFD:
        return (NFD.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    case NFKC:
        return (NFKC.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    case NFKD:
        return (NFKD.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    }

    throw new IllegalArgumentException("Unexpected normalization form: " +
                                       form);
}
 
Example 5
Source File: NormalizerBase.java    From openjdk-8-source with GNU General Public License v2.0 6 votes vote down vote up
/**
 * Test if a string is in a given normalization form.
 * This is semantically equivalent to source.equals(normalize(source, mode)).
 *
 * Unlike quickCheck(), this function returns a definitive result,
 * never a "maybe".
 * For NFD, NFKD, and FCD, both functions work exactly the same.
 * For NFC and NFKC where quickCheck may return "maybe", this function will
 * perform further tests to arrive at a true/false result.
 * @param str       the input string to be checked to see if it is normalized
 * @param form      the normalization form
 * @param options   the optional features to be enabled.
 */
public static boolean isNormalized(String str, Normalizer.Form form, int options) {
    switch (form) {
    case NFC:
        return (NFC.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    case NFD:
        return (NFD.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    case NFKC:
        return (NFKC.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    case NFKD:
        return (NFKD.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    }

    throw new IllegalArgumentException("Unexpected normalization form: " +
                                       form);
}
 
Example 6
Source File: NormalizerBase.java    From jdk8u60 with GNU General Public License v2.0 6 votes vote down vote up
/**
 * Test if a string is in a given normalization form.
 * This is semantically equivalent to source.equals(normalize(source, mode)).
 *
 * Unlike quickCheck(), this function returns a definitive result,
 * never a "maybe".
 * For NFD, NFKD, and FCD, both functions work exactly the same.
 * For NFC and NFKC where quickCheck may return "maybe", this function will
 * perform further tests to arrive at a true/false result.
 * @param str       the input string to be checked to see if it is normalized
 * @param form      the normalization form
 * @param options   the optional features to be enabled.
 */
public static boolean isNormalized(String str, Normalizer.Form form, int options) {
    switch (form) {
    case NFC:
        return (NFC.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    case NFD:
        return (NFD.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    case NFKC:
        return (NFKC.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    case NFKD:
        return (NFKD.quickCheck(str.toCharArray(),0,str.length(),false,NormalizerImpl.getNX(options))==YES);
    }

    throw new IllegalArgumentException("Unexpected normalization form: " +
                                       form);
}
 
Example 7
Source File: StringFunctions.java    From presto with Apache License 2.0 5 votes vote down vote up
@Description("Transforms the string to normalized form")
@ScalarFunction
@LiteralParameters({"x", "y"})
@SqlType(StandardTypes.VARCHAR)
public static Slice normalize(@SqlType("varchar(x)") Slice slice, @SqlType("varchar(y)") Slice form)
{
    Normalizer.Form targetForm;
    try {
        targetForm = Normalizer.Form.valueOf(form.toStringUtf8());
    }
    catch (IllegalArgumentException e) {
        throw new PrestoException(INVALID_FUNCTION_ARGUMENT, "Normalization form must be one of [NFD, NFC, NFKD, NFKC]");
    }
    return utf8Slice(Normalizer.normalize(slice.toStringUtf8(), targetForm));
}
 
Example 8
Source File: NormalizerBase.java    From TencentKona-8 with GNU General Public License v2.0 5 votes vote down vote up
/**
 * Normalizes a <code>String</code> using the given normalization form.
 *
 * @param str      the input string to be normalized.
 * @param form     the normalization form
 * @param options   the optional features to be enabled.
 */
public static String normalize(String str, Normalizer.Form form, int options) {
    int len = str.length();
    boolean asciiOnly = true;
    if (len < 80) {
        for (int i = 0; i < len; i++) {
            if (str.charAt(i) > 127) {
                asciiOnly = false;
                break;
            }
        }
    } else {
        char[] a = str.toCharArray();
        for (int i = 0; i < len; i++) {
            if (a[i] > 127) {
                asciiOnly = false;
                break;
            }
        }
    }

    switch (form) {
    case NFC :
        return asciiOnly ? str : NFC.normalize(str, options);
    case NFD :
        return asciiOnly ? str : NFD.normalize(str, options);
    case NFKC :
        return asciiOnly ? str : NFKC.normalize(str, options);
    case NFKD :
        return asciiOnly ? str : NFKD.normalize(str, options);
    }

    throw new IllegalArgumentException("Unexpected normalization form: " +
                                       form);
}
 
Example 9
Source File: NormalizerBase.java    From openjdk-jdk9 with GNU General Public License v2.0 5 votes vote down vote up
private static Mode toMode(Normalizer.Form form) {
    switch (form) {
    case NFC :
        return NFC;
    case NFD :
        return NFD;
    case NFKC :
        return NFKC;
    case NFKD :
        return NFKD;
    }

    throw new IllegalArgumentException("Unexpected normalization form: " +
                                       form);
}
 
Example 10
Source File: NormalizerBase.java    From jdk8u-jdk with GNU General Public License v2.0 5 votes vote down vote up
/**
 * Normalizes a <code>String</code> using the given normalization form.
 *
 * @param str      the input string to be normalized.
 * @param form     the normalization form
 * @param options   the optional features to be enabled.
 */
public static String normalize(String str, Normalizer.Form form, int options) {
    int len = str.length();
    boolean asciiOnly = true;
    if (len < 80) {
        for (int i = 0; i < len; i++) {
            if (str.charAt(i) > 127) {
                asciiOnly = false;
                break;
            }
        }
    } else {
        char[] a = str.toCharArray();
        for (int i = 0; i < len; i++) {
            if (a[i] > 127) {
                asciiOnly = false;
                break;
            }
        }
    }

    switch (form) {
    case NFC :
        return asciiOnly ? str : NFC.normalize(str, options);
    case NFD :
        return asciiOnly ? str : NFD.normalize(str, options);
    case NFKC :
        return asciiOnly ? str : NFKC.normalize(str, options);
    case NFKD :
        return asciiOnly ? str : NFKD.normalize(str, options);
    }

    throw new IllegalArgumentException("Unexpected normalization form: " +
                                       form);
}
 
Example 11
Source File: NormalizerBase.java    From hottub with GNU General Public License v2.0 5 votes vote down vote up
/**
 * Normalizes a <code>String</code> using the given normalization form.
 *
 * @param str      the input string to be normalized.
 * @param form     the normalization form
 * @param options   the optional features to be enabled.
 */
public static String normalize(String str, Normalizer.Form form, int options) {
    int len = str.length();
    boolean asciiOnly = true;
    if (len < 80) {
        for (int i = 0; i < len; i++) {
            if (str.charAt(i) > 127) {
                asciiOnly = false;
                break;
            }
        }
    } else {
        char[] a = str.toCharArray();
        for (int i = 0; i < len; i++) {
            if (a[i] > 127) {
                asciiOnly = false;
                break;
            }
        }
    }

    switch (form) {
    case NFC :
        return asciiOnly ? str : NFC.normalize(str, options);
    case NFD :
        return asciiOnly ? str : NFD.normalize(str, options);
    case NFKC :
        return asciiOnly ? str : NFKC.normalize(str, options);
    case NFKD :
        return asciiOnly ? str : NFKD.normalize(str, options);
    }

    throw new IllegalArgumentException("Unexpected normalization form: " +
                                       form);
}
 
Example 12
Source File: AbstractUnicodeNormalizer.java    From cyberduck with GNU General Public License v3.0 4 votes vote down vote up
public AbstractUnicodeNormalizer(final Normalizer.Form form) {
    this.form = form;
}
 
Example 13
Source File: NormalizerBase.java    From openjdk-jdk9 with GNU General Public License v2.0 4 votes vote down vote up
public static boolean isNormalized(String str, Normalizer.Form form) {
    return NormalizerBase.isNormalized(str, toMode(form), UNICODE_LATEST);
}
 
Example 14
Source File: NormalizerBase.java    From Bytecoder with Apache License 2.0 4 votes vote down vote up
public static boolean isNormalized(String str, Normalizer.Form form) {
    return NormalizerBase.isNormalized(str, toMode(form), UNICODE_LATEST);
}
 
Example 15
Source File: NormalizerBase.java    From openjdk-jdk8u-backup with GNU General Public License v2.0 2 votes vote down vote up
/**
 * Test if a string is in a given normalization form.
 * This is semantically equivalent to source.equals(normalize(source, mode)).
 *
 * Unlike quickCheck(), this function returns a definitive result,
 * never a "maybe".
 * For NFD, NFKD, and FCD, both functions work exactly the same.
 * For NFC and NFKC where quickCheck may return "maybe", this function will
 * perform further tests to arrive at a true/false result.
 * @param str       the input string to be checked to see if it is normalized
 * @param form      the normalization form
 * @param options   the optional features to be enabled.
 */
public static boolean isNormalized(String str, Normalizer.Form form) {
    return isNormalized(str, form, UNICODE_LATEST);
}
 
Example 16
Source File: NormalizerBase.java    From jdk8u-jdk with GNU General Public License v2.0 2 votes vote down vote up
/**
 * Test if a string is in a given normalization form.
 * This is semantically equivalent to source.equals(normalize(source, mode)).
 *
 * Unlike quickCheck(), this function returns a definitive result,
 * never a "maybe".
 * For NFD, NFKD, and FCD, both functions work exactly the same.
 * For NFC and NFKC where quickCheck may return "maybe", this function will
 * perform further tests to arrive at a true/false result.
 * @param str       the input string to be checked to see if it is normalized
 * @param form      the normalization form
 * @param options   the optional features to be enabled.
 */
public static boolean isNormalized(String str, Normalizer.Form form) {
    return isNormalized(str, form, UNICODE_LATEST);
}
 
Example 17
Source File: NormalizerBase.java    From openjdk-8 with GNU General Public License v2.0 2 votes vote down vote up
/**
 * Test if a string is in a given normalization form.
 * This is semantically equivalent to source.equals(normalize(source, mode)).
 *
 * Unlike quickCheck(), this function returns a definitive result,
 * never a "maybe".
 * For NFD, NFKD, and FCD, both functions work exactly the same.
 * For NFC and NFKC where quickCheck may return "maybe", this function will
 * perform further tests to arrive at a true/false result.
 * @param str       the input string to be checked to see if it is normalized
 * @param form      the normalization form
 * @param options   the optional features to be enabled.
 */
public static boolean isNormalized(String str, Normalizer.Form form) {
    return isNormalized(str, form, UNICODE_LATEST);
}
 
Example 18
Source File: NormalizerBase.java    From hottub with GNU General Public License v2.0 2 votes vote down vote up
/**
 * Test if a string is in a given normalization form.
 * This is semantically equivalent to source.equals(normalize(source, mode)).
 *
 * Unlike quickCheck(), this function returns a definitive result,
 * never a "maybe".
 * For NFD, NFKD, and FCD, both functions work exactly the same.
 * For NFC and NFKC where quickCheck may return "maybe", this function will
 * perform further tests to arrive at a true/false result.
 * @param str       the input string to be checked to see if it is normalized
 * @param form      the normalization form
 * @param options   the optional features to be enabled.
 */
public static boolean isNormalized(String str, Normalizer.Form form) {
    return isNormalized(str, form, UNICODE_LATEST);
}
 
Example 19
Source File: Bytes.java    From bytes-java with Apache License 2.0 2 votes vote down vote up
/**
 * Creates a new instance from normalized form of given utf-8 encoded string
 *
 * @param utf8String to get the internal byte array from
 * @param form       to normalize, usually you want {@link java.text.Normalizer.Form#NFKD} for compatibility
 * @return new instance
 */
public static Bytes from(CharSequence utf8String, Normalizer.Form form) {
    return from(Normalizer.normalize(utf8String, form), StandardCharsets.UTF_8);
}
 
Example 20
Source File: NormalizerBase.java    From jdk8u-jdk with GNU General Public License v2.0 2 votes vote down vote up
/**
 * Normalizes a <code>String</code> using the given normalization form.
 *
 * @param str      the input string to be normalized.
 * @param form     the normalization form
 */
public static String normalize(String str, Normalizer.Form form) {
    return normalize(str, form, UNICODE_LATEST);
}