How to get a custom style value in the interaction manager's getCommonCharacterFormat() function ?

There's no general code for that.  You will have to write something on your own.  If for example you want to check for a specific style having a common value across the range a new function modeled on getCommonCharacterFormat would work:

    public static function getCommonCharacterStyle(range:ElementRange,styleName:String):*
    {       
        var leaf:FlowLeafElement = range.firstLeaf;
        var value:* = leaf.getStyle(styleName);
        if (value === undefined)
            return undefined;    // no common value or common value is undefined
       
        for (;;)
        {
            if (leaf == range.lastLeaf)
                break;
            leaf = leaf.getNextLeaf();
            if (value !== leaf.getStyle(styleName))
                return undefined;    // no common value
        }
       
        return value;     // common value
    }

Depending on your use case some variation of this will supply the information your application needs.  More complicated, but doable, would be extending this to work on all userStyles in leaf returning an object of key-value pairs.

Hope that helps,

Richard