CODEWITHSUNDEEP
performancehttperlangelixir
helios35
helios35

Reputation: 1637

High CPU utilization when downloading files

When downloading files via HTTP using Erlang's httpc, CPU utilization is much higher than with curl or wget, for instance. The code I'm using to measure the download speed can be seen at the bottom of this post.

The high CPU utilization is problematic especially on low-end devices. I'm running Erlang on an ARM-SoC which is only slightly more powerful than the first Raspberry PI, and this piece of code results in 100% CPU utilization and a download speed of only 6.1 MiB/s. With curl and wget, CPU utilization stays slightly below 100% and it manages to almost fully utilize the network interface (10.7 MiB/s or 85.6 MBit/s on a 100 MBit/s network interface).

I tried using other HTTP libraries including ibrowse and hackney, but the same issue persists. My guess is that it has to do with Erlang's socket performance, but I could be wrong. So my question is, what exactly is responsible for those slow download speeds, and are there any workarounds for it? I know of libraries like https://github.com/puzza007/katipo which use libcurl and therefore probably won't have the same issue, but I would prefer not to use any libraries that make use of NIFs.

defmodule DownloadPerformanceTest do
  @testfile 'http://speed.hetzner.de/100MB.bin'
  @filesize 104857600
  @save_to '/dev/null'

  def test() do
    Application.start(:inets)
    then = :erlang.system_time(:micro_seconds)
    {:ok, :saved_to_file} = :httpc.request(:get, {@testfile, []}, [], [{:stream, @save_to}])
    now = :erlang.system_time(:micro_seconds)
    diff = now - then
    bw = bandwidth_to_human_readable(@filesize, diff)
    IO.puts "Download took #{:erlang.trunc(diff / 1_000_000)} seconds, average speed: #{bw}"
  end

  defp bandwidth_to_human_readable(bytes, microseconds) do
    bytes_per_second = bytes / (microseconds / 1000000)
    exponent = :erlang.trunc(:math.log2(bytes_per_second) / :math.log2(1024))
    prefix = case exponent do
               0 -> {:ok, ""}
               1 -> {:ok, "Ki"}
               2 -> {:ok, "Mi"}
               3 -> {:ok, "Gi"}
               4 -> {:ok, "Ti"}
               5 -> {:ok, "Pi"}
               6 -> {:ok, "Ei"}
               7 -> {:ok, "Zi"}
               8 -> {:ok, "Yi"}
               _ -> {:error, :too_large}
             end
    case prefix do
      {:ok, prefix} ->
        quantity = Float.round(bytes_per_second / :math.pow(1024, exponent), 2)
        unit = "#{prefix}B/s"
        "#{quantity} #{unit}"
      {:error, :too_large} ->
        "#{bytes_per_second} B/s"
    end
  end
end

Upvotes: 2

Views: 1296

Answers (1)

Supreet Sethi
Supreet Sethi

Reputation: 1806

Going back to the benchmark, three clear issues, I am able ascertain

  • You are using a remote resource which is effected by externalities making benchmark numbers. Hence for testing, I changed to local resource
  • Secondly, except for hackney, none of the other libraries stream the payload to file. Although saving to /dev/null, the file saving has a cost.
  • Test needs to be run more than once (may be three times)

Once, I removed save action in download_loop_hackney(), hackney is the fastest

defp download_loop_hackney(client, file) do
    case :hackney.stream_body(client) do
      {:ok, _result} ->

        #IO.binwrite(file, result)
        download_loop_hackney(client, file)
      :done ->
        :ok = File.close(file)
    end
  end

The benchmark numbers are thus

download_http: download took 0 seconds, average speed: 211.05 MiB/s
download_ibrowse: download took 0 seconds, average speed: 223.15 MiB/s
download_hackney: download took 0 seconds, average speed: 295.83 MiB/s
download_tcp: download took 0 seconds, average speed: 595.84 MiB/s

Upvotes: 1

Related Questions