Problem with parsing JSON

Nov 17, 2015 at 5:31 PM
Edited Nov 17, 2015 at 8:05 PM
I have a REST API that returns a string with files metadata. Among those files I have one that has "￿" (0xEF0xBF0xBF, link ) character in name. When I try to parse such a response I get exception that says:
e = {_message="* Line 1, Column 13 Syntax error: Malformed string literal" } 
This is the string I'm trying to parse:
{"files":[["Thunderbolt ￿Gigabit￿ Ethernet ￿Adapter.txt",["bc288518-c426-4dbd-9600-a213a35d1c04",1447772221866,9,"Bartosz","Siewior"]],["System.Windows.Interactivity.dll",["56ce759b-019b-4723-8fca-7af877908971",1440507238241,55904,"Bartosz","Siewior"]]],"folderPermission":[["MODIFY"]],"directories":[]}


The code I'm using to receive and parse the string:
// ...
    client.request(request).then([&](web::http::http_response response) {
            status_code = response.status_code();
            //response.extract_json(true).then([&](pplx::task<web::json::value> previousTask) {
            response.extract_string().then([&](pplx::task<utility::string_t> previousTask) {
                try {

                    utility::string_t str_response = previousTask.get();
                    web::json::value root;
                    root = web::json::value::parse(str_response);
                }
                catch (const web::http::http_exception& e) {
                    std::wstringstream ss;
                    ss << e.what();
                    str_response = ss.str();
                }
                catch (const web::json::json_exception& e) {
                    std::wstringstream ss;
                    ss << e.what();
                    str_response = ss.str();
                }

                TaskExecutionData data = { task.id, status_code, str_response.c_str() };
                callback(data);
            }).wait();
        }).wait();
VS2013 JSON Visualizer can parse and show the result properly: VS_json_visualizer

I've tried Casablanca version 2.5.0 and 2.6.0, neither of them is able to parse that string... Do you have any ides?
Nov 19, 2015 at 12:23 AM
The issue is that on windows we parse using UTF16 and the particular character you're having issues with maps to 0xFFFF. This is also the character returned by std::char_traits<wchar_t>::eof(), so we detect the code point as an "end of stream" signal and terminate the parse appropriately.

The UTF8 parser should not have these issues, so as a workaround you can probably do something similar to how the current web::json::value::parse() function works in json_parsing.cpp:1245, except using char as the template parameter instead of utility::char_t.
Nov 19, 2015 at 4:26 PM
Thanks, what you proposed seems to be working. Do you have plan to fix the UTF16 in near feature? Should I create and issue for that?