CODEWITHSUNDEEP
pythondictionaryyamlpyyaml
mwcz
mwcz

Reputation: 9311

Can PyYAML dump dict items in non-alphabetical order?

I'm using yaml.dump to output a dict. It prints out each item in alphabetical order based on the key.

>>> d = {"z":0,"y":0,"x":0}
>>> yaml.dump( d, default_flow_style=False )
'x: 0\ny: 0\nz: 0\n'

Is there a way to control the order of the key/value pairs?

In my particular use case, printing in reverse would (coincidentally) be good enough. For completeness though, I'm looking for an answer that shows how to control the order more precisely.

I've looked at using collections.OrderedDict but PyYAML doesn't (seem to) support it. I've also looked at subclassing yaml.Dumper, but I haven't been able to figure out if it has the ability to change item order.

Upvotes: 117

Views: 107979

Answers (9)

Anthon
Anthon

Reputation: 76792

There are two things you need to do to get this as you want:

  • you need to use something else than a dict, because it doesn't keep the items ordered
  • you need to dump that alternative in the appropriate way.¹
import sys
import ruamel.yaml
from ruamel.yaml.comments import CommentedMap

yaml = ruamel.yaml.YAML()    

d = CommentedMap()
d['z'] = 0
d['y'] = 0
d['x'] = 0

yaml.dump(d, sys.stdout)

output:

z: 0
y: 0
x: 0

¹ This was done using ruamel.yaml a YAML 1.2 parser, of which I am the author.

Upvotes: 2

wim
wim

Reputation: 363253

For Python 3.7+, dicts preserve insertion order. Since PyYAML 5.1.x, you can disable the sorting of keys (#254). Unfortunately, the sorting keys behaviour does still default to True.

>>> import yaml
>>> yaml.dump({"b":1, "a": 2})
'a: 2\nb: 1\n'
>>> yaml.dump({"b":1, "a": 2}, sort_keys=False)
'b: 1\na: 2\n'

My project oyaml is a monkeypatch/drop-in replacement for PyYAML. It will preserve dict order by default in all Python versions and PyYAML versions.

>>> import oyaml as yaml  # pip install oyaml
>>> yaml.dump({"b":1, "a": 2})
'b: 1\na: 2\n'

Additionally, it will dump the collections.OrderedDict subclass as normal mappings, rather than Python objects.

>>> from collections import OrderedDict
>>> d = OrderedDict([("b", 1), ("a", 2)])
>>> import yaml
>>> yaml.dump(d)
'!!python/object/apply:collections.OrderedDict\n- - - b\n    - 1\n  - - a\n    - 2\n'
>>> yaml.safe_dump(d)
RepresenterError: ('cannot represent an object', OrderedDict([('b', 1), ('a', 2)]))
>>> import oyaml as yaml
>>> yaml.dump(d)
'b: 1\na: 2\n'
>>> yaml.safe_dump(d)
'b: 1\na: 2\n'

Upvotes: 36

Cooper.Wu
Cooper.Wu

Reputation: 4575

If you upgrade PyYAML to 5.1 version, now, it supports dump without sorting the keys like this:

yaml.dump(data, sort_keys=False)

As shown in help(yaml.Dumper), sort_keys defaults to True:

Dumper(stream, default_style=None, default_flow_style=False,
  canonical=None, indent=None, width=None, allow_unicode=None,
  line_break=None, encoding=None, explicit_start=None, explicit_end=None,
  version=None, tags=None, sort_keys=True)

(These are passed as kwargs to yaml.dump)

Upvotes: 254

George Swani Antony
George Swani Antony

Reputation: 1

The following setting makes sure the content is not sorted in the output:

yaml.sort_base_mapping_type_on_output = False

Upvotes: -1

Peter Bašista
Peter Bašista

Reputation: 907

If safe_dump (i.e. dump with Dumper=SafeDumper) is used, then calling yaml.add_representer has no effect. In such case it is necessary to call add_representer method explicitly on SafeRepresenter class:

yaml.representer.SafeRepresenter.add_representer(
    OrderedDict, ordered_dict_representer
)

Upvotes: 0

Ark-kun
Ark-kun

Reputation: 6812

One-liner to rule them all:

yaml.add_representer(dict, lambda self, data: yaml.representer.SafeRepresenter.represent_dict(self, data.items()))

That's it. Finally. After all those years and hours, the mighty represent_dict has been defeated by giving it the dict.items() instead of just dict

Here is how it works:

This is the relevant PyYaml source code:

    if hasattr(mapping, 'items'):
        mapping = list(mapping.items())
        try:
            mapping = sorted(mapping)
        except TypeError:
            pass
    for item_key, item_value in mapping:

To prevent the sorting we just need some Iterable[Pair] object that does not have .items().

dict_items is a perfect candidate for this.

Here is how to do this without affecting the global state of the yaml module:

#Using a custom Dumper class to prevent changing the global state
class CustomDumper(yaml.Dumper):
    #Super neat hack to preserve the mapping key order. See https://stackoverflow.com/a/52621703/1497385
    def represent_dict_preserve_order(self, data):
        return self.represent_dict(data.items())    

CustomDumper.add_representer(dict, CustomDumper.represent_dict_preserve_order)

return yaml.dump(component_dict, Dumper=CustomDumper)

Upvotes: 15

orodbhen
orodbhen

Reputation: 2694

This is really just an addendum to @Blender's answer. If you look in the PyYAML source, at the representer.py module, You find this method:

def represent_mapping(self, tag, mapping, flow_style=None):
    value = []
    node = MappingNode(tag, value, flow_style=flow_style)
    if self.alias_key is not None:
        self.represented_objects[self.alias_key] = node
    best_style = True
    if hasattr(mapping, 'items'):
        mapping = mapping.items()
        mapping.sort()
    for item_key, item_value in mapping:
        node_key = self.represent_data(item_key)
        node_value = self.represent_data(item_value)
        if not (isinstance(node_key, ScalarNode) and not node_key.style):
            best_style = False
        if not (isinstance(node_value, ScalarNode) and not node_value.style):
            best_style = False
        value.append((node_key, node_value))
    if flow_style is None:
        if self.default_flow_style is not None:
            node.flow_style = self.default_flow_style
        else:
            node.flow_style = best_style
    return node

If you simply remove the mapping.sort() line, then it maintains the order of items in the OrderedDict.

Another solution is given in this post. It's similar to @Blender's, but works for safe_dump. The common element is the converting of the dict to a list of tuples, so the if hasattr(mapping, 'items') check evaluates to false.

Update:

I just noticed that The Fedora Project's EPEL repo has a package called python2-yamlordereddictloader, and there's one for Python 3 as well. The upstream project for that package is likely cross-platform.

Upvotes: 3

Blender
Blender

Reputation: 298432

There's probably a better workaround, but I couldn't find anything in the documentation or the source.

Python 2 (see comments)

I subclassed OrderedDict and made it return a list of unsortable items:

from collections import OrderedDict

class UnsortableList(list):
    def sort(self, *args, **kwargs):
        pass

class UnsortableOrderedDict(OrderedDict):
    def items(self, *args, **kwargs):
        return UnsortableList(OrderedDict.items(self, *args, **kwargs))

yaml.add_representer(UnsortableOrderedDict, yaml.representer.SafeRepresenter.represent_dict)

And it seems to work:

>>> d = UnsortableOrderedDict([
...     ('z', 0),
...     ('y', 0),
...     ('x', 0)
... ])
>>> yaml.dump(d, default_flow_style=False)
'z: 0\ny: 0\nx: 0\n'

Python 3 or 2 (see comments)

You can also write a custom representer, but I don't know if you'll run into problems later on, as I stripped out some style checking code from it:

import yaml

from collections import OrderedDict

def represent_ordereddict(dumper, data):
    value = []

    for item_key, item_value in data.items():
        node_key = dumper.represent_data(item_key)
        node_value = dumper.represent_data(item_value)

        value.append((node_key, node_value))

    return yaml.nodes.MappingNode(u'tag:yaml.org,2002:map', value)

yaml.add_representer(OrderedDict, represent_ordereddict)

But with that, you can use the native OrderedDict class.

Upvotes: 49

krishnakishorek
krishnakishorek

Reputation: 70

I was also looking for an answer to the question "how to dump mappings with the order preserved?" I couldn't follow the solution given above as i am new to pyyaml and python. After spending some time on the pyyaml documentation and other forums i found this.

You can use the tag

!!omap

to dump the mappings by preserving the order. If you want to play with the order i think you have to go for keys:values

The links below can help for better understanding.

https://bitbucket.org/xi/pyyaml/issue/13/loading-and-then-dumping-an-omap-is-broken

http://yaml.org/type/omap.html

Upvotes: -1

Related Questions