How to pattern match structures returned by mochijson2?

Question

I've just started tinkering with Erlang and am building a very simple test web application which is just intended to show my twitter timeline.

I'm using webmachine to write the app and erlyDTL to render the templates.

My question is related to the structures returned by mochiweb's mochijson2:decode/1 function.

I can successfully fetch and deocde my timeline as in the following example:

1> Url = "http://api.twitter.com/1/statuses/user_timeline.json?screen_name=<TWITTER_SCREEN_NAME_HERE>".
2> inets:start().
3> {ok, {_, _, Response}} = httpc:request(Url).
4> DecodedJson = mochijson2:decode(Response).

The mochijson2:decode/1 function returns a list of tuples of the format:

[{struct, proplist()}, {struct, proplist()}, ...]

However, to pass the timeline into erlyDTL I need to get rid of the struct atom tag and simply pass a list of proplists to the webmachine resource (rendered by erlyDTL). Being quite new to pattern matching, I figured that the following list comprehension would achieve this:

Timeline = [Tweet || {struct, Tweet} <- DecodedJson].

Indeed, this works perfectly for all of the items in each Tweet proplist except for one, <<"user">>, the value of which is itself a {struct, proplist()} tuple. I can't for the life of me figure out how to nuke the struct atom from this nested tuple and was wondering if anyone could provide an example of Erlang code which would pattern match both the outer Tweet in {struct, Tweet} and the User {struct, User} contained within each Tweet.

The end goal is to be able to access each tweet in Django template language as in the following example:

{{ tweet.text }}  <- works
{{ tweet.created_at }}  <- works
{{ tweet.user.profile_image_url }}  <- ???

Any help would be greatly appreciated!

Answer 1

In a recent project ive worked on, we were dealing with big JSON data structures coming from EXT JS front end apps. One example of the JSON object is here below (this is just the skeleton of the JSON ):

{
        "presence_token":"734737328233HDHSBSHSYEYEYWYWGWE",
        "presence_time":"HH:Mins:Secs",
        "friend_requests":
        [
            {
                "from":"Username",
                "type":"buddy",
                "date":"DD/MM/YY",
                "time":"HH:Mins:Secs",
                "name":"Your Full name",
                "email":"user@example.com"
            }
        ],
        "group_status":
        [
            {
                "group_name":"ecampus",
                "status":"running",
                "members":["phil","josh","shazz"],
                "start_date":"DD/MM/YY",
                "start_time":"HH:Mins:Secs"             
            },
            {
                "group_name":"buganda",
                "status":"off"
            }
        ],
        "friend_status":
        [
            {
                "friend":"Friend_username",
                "status":"online",
                "log_on_time":"HH:Mins:Secs",
                "state":"available",
                "name":"Friend_Fullname",
                "email":"user@example.com"              
            },
            {
                "friend":"Friend_username",
                "status":"offline",                 
                "name":"Friend_Fullname",
                "email":"user@example.com"              
            }           
        ]           
    }

After mochijson2:decode/1, the struct object i had appears like this:

        {struct,[{<<"presence_token">>,
                  <<"734737328233HDHSBSHSYEYEYWYWGWE">>},
                 {<<"presence_time">>,<<"HH:Mins:Secs">>},
                 {<<"friend_requests">>,
                  [{struct,[{<<"from">>,<<"Username">>},
                            {<<"type">>,<<"buddy">>},
                            {<<"date">>,<<"DD/MM/YY">>},
                            {<<"time">>,<<"HH:Mins:Secs">>},
                            {<<"name">>,<<"Your Full name">>},
                            {<<"email">>,<<"user@example.com">>}]}]},
                 {<<"group_status">>,
                  [{struct,[{<<"group_name">>,<<"ecampus">>},
                            {<<"status">>,<<"running">>},
                            {<<"members">>,[<<"phil">>,<<"josh">>,<<"shazz">>]},
                            {<<"start_date">>,<<"DD/MM/YY">>},
                            {<<"start_time">>,<<"HH:Mins:Secs">>}]},
                   {struct,[{<<"group_name">>,<<"buganda">>},
                            {<<"status">>,<<"off">>}]}]},
                 {<<"friend_status">>,
                  [{struct,[{<<"friend">>,<<"Friend_username">>},
                            {<<"status">>,<<"online">>},
                            {<<"log_on_time">>,<<"HH:Mins:Secs">>},
                            {<<"state">>,<<"available">>},
                            {<<"name">>,<<"Friend_Fullname">>},
                            {<<"email">>,<<"user@example.com">>}]},
                   {struct,[{<<"friend">>,<<"Friend_username">>},
                            {<<"status">>,<<"offline">>},
                            {<<"name">>,<<"Friend_Fullname">>},
                            {<<"email">>,<<"user@example.com">>}]}]}]}

Now i decided to create a module which will convert this struct into a "deep" proplist, this module would contain a function struct:all_keys/1 which if i feed it with the struct object it generates lists and tuples in an organised way. Here is the code:

-module(struct).
-export([all_keys/1]).

is_struct({struct,_}) -> true;
is_struct(_) -> false.

to_binary(S) when is_list(S)-> list_to_binary(S);
to_binary(S) when is_integer(S)-> S;
to_binary(S) when is_atom(S)-> to_binary(atom_to_list(S));
to_binary(S) -> S.

to_value(V) when is_binary(V)-> binary_to_list(V);
to_value(V) when is_integer(V)-> V;
to_value(V) when is_list(V)-> 
    try list_to_integer(V) of
        PP -> PP
    catch
        _:_ -> 
            try list_to_float(V) of
                PP2 -> PP2
            catch
                _:_ -> V
            end
    end;
to_value(A)-> A.

to_value2({struct,L})->
    all_keys({struct,L});
to_value2([{struct,_L}|_Rest] = LL)->
    [all_keys(XX) || XX <- LL];
to_value2(D) when is_binary(D)-> to_value(D);
to_value2(D) when is_list(D)-> 
    [to_value2(Any) || Any <- D].    

all_keys({struct,L})-> 
    [{to_value(Key),to_value2(Val)} || {Key,Val} <- L];
all_keys(List)-> [all_keys(X) || X <- List].

Now, calling struct:all_keys(Struct_object) will give this output:

[{"presence_token",P_token},
 {"presence_time",P_time},
 {"friend_requests",
  [[{"from","Username"},
    {"type","buddy"},
    {"date","DD/MM/YY"},
    {"time","HH:Mins:Secs"},
    {"name","Your Full name"},
    {"email","user@example.com"}]]},
 {"group_status",
  [[{"group_name","ecampus"},
    {"status","running"},
    {"members",["phil","josh","shazz"]},
    {"start_date","DD/MM/YY"},
    {"start_time","HH:Mins:Secs"}],
   [{"group_name","buganda"},{"status","off"}]]},
 {"friend_status",
  [[{"friend","Friend_username"},
    {"status","online"},
    {"log_on_time","HH:Mins:Secs"},
    {"state","available"},
    {"name","Friend_Fullname"},
    {"email","user@example.com"}],
   [{"friend","Friend_username"},
    {"status","offline"},
    {"name","Friend_Fullname"},
    {"email","user@example.com"}]]}]

The above proplist is easier to work with than the struct object. However, you may find another version of the struct module especially in a famous mochiweb example called Sticky Notes whose link i do not have right now.The struct module i have pasted above should be able to help you with using mochijson2. success

Answer 2

Here's what we use internally for a similar purpose:

%% @doc Flatten {struct, [term()]} to [term()] recursively.
destruct({struct, L}) ->
    destruct(L);
destruct([H | T]) ->
    [destruct(H) | destruct(T)];
destruct({K, V}) ->
    {K, destruct(V)};
destruct(Term) ->
    Term.

For other uses of mochijson2 terms, KVC might be useful to you: https://github.com/etrepum/kvc

Answer 3

Based on the structure you described, you could try:

timeline(List) -> timeline(List, []).

timeline([], Result) -> lists:reverse(Result);
timeline([{struct, S}|T], Result) -> timeline(T, [S|Result]);
timeline([{<<"user">>, {struct, S}}|T], Result) -> timeline(T, [S|Result]);
timeline([_|T], Result) -> timeline(T, Result).

I put that code in a module called twitter:

> twitter:timeline([{struct, 1}, {struct, 2}, {<<"user">>, {struct, 3}}, 5]).
[1,2,3]

You may want to replace <<"user">> with _ depending on your specific needs. You probably also want to introduce some kind of exception handling since you are dealing with input from the outside world.