CODEWITHSUNDEEP
hadoopsystem-administrationcloudera
James Adams
James Adams

Reputation: 8727

Cloudera CDH3 installation failure, how to get around this?

I am attempting to install CHD3 onto a 3 node cluster. I launch the installations via the Cloudera Manager. All three installations fail.

I see this error after the Cloudera installation fails in /var/log/cloudera-scm-agent/cloudera-scm-agent.out:

  File "/usr/lib64/cmf/agent/src/cmf/agent.py", line 19, in <module>
    import psutil
  File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/psutil-0.3.0-py2.6-linux-x86_64.egg/psutil/__init__.py", line 84, in <module>
    TOTAL_PHYMEM = _psplatform.phymem_usage()[0]
  File "/usr/lib64/cmf/agent/build/env/lib/python2.6/site-packages/psutil-0.3.0-py2.6-linux-x86_64.egg/psutil/_pslinux.py", line 122, in phymem_usage
    percent = usage_percent(total - (free + buffers + cached), total,
TypeError: unsupported operand type(s) for +: 'int' and 'NoneType'

Apparently the Python interpreter running on start up sees "free", "buffers", or "cached" as having a NoneType and this error causes the entire installation to roll back.

Can anyone advise as to why this occurs and/or a way around the problem?

Thanks in advance.

Upvotes: 0

Views: 966

Answers (1)

AdamIerymenko
AdamIerymenko

Reputation: 1329

The problem is here, in phymem_usage() in _pslinux.py:

def phymem_usage():
    # total, used and free values are matched against free cmdline utility
    # the percentage matches top/htop and gnome-system-monitor
    f = open('/proc/meminfo', 'r')
    try:
        total = free = buffers = cached = None
        for line in f:
            if line.startswith('MemTotal:'):
                total = int(line.split()[1]) * 1024
            elif line.startswith('MemFree:'):
                free = int(line.split()[1]) * 1024
            elif line.startswith('Buffers:'):
                buffers = int(line.split()[1]) * 1024
            elif line.startswith('Cached:'):
                cached = int(line.split()[1]) * 1024
                break
        used = total - free
        percent = usage_percent(total - (free + buffers + cached), total,
                                _round=1)
        return ntuple_sysmeminfo(total, used, free, percent)
    finally:
        f.close()

Note that it is examining /proc/meminfo and is converting fields to integers without checking if those fields exist. On some systems, including some virtualization technologies, Buffers or Cache may be missing. (The LSB spec states that most of these fields are optional.)

A quick fix would be to change /proc/meminfo to /tmp/meminfo, "cat /proc/meminfo >/tmp/meminfo", and add a line like:

Buffers: 0 kB

Upvotes: 1

Related Questions