B Furtado
Reputation: 1500
How to interpret the results of profiling?
I am not a computer scientist and I have been having a hard time trying to make my agent-based model reasonably fast (Implementing Pool on a for loop with a lot of inputs, Multiprocessing pool example (parallel) is slower than sequential. Trying to understand pool in python, Returning lists of instances makes no difference? And what about performance?)
Everybody keeps telling me to profile. I gave it another shot, but I have no idea how to interpret the results (using timeit I know which function is slower, but I could not use Pool or @jit).
So given the results below, how should I proceed? Thanks a bunch.
98008290 function calls (96393650 primitive calls) in 568.193 seconds
Ordered by: standard name
ncalls tottime percall cumtime percall filename:lineno(function)
1 0.000 0.000 0.000 0.000 <decorator-gen-0>:1(<module>)
1 0.000 0.000 0.000 0.000 <decorator-gen-10>:1(<module>)
1 0.000 0.000 0.000 0.000 <decorator-gen-11>:1(<module>)
1 0.000 0.000 0.000 0.000 <decorator-gen-1>:1(<module>)
6 0.000 0.000 0.006 0.001 <decorator-gen-1>:1(non_reentrant)
1 0.000 0.000 0.000 0.000 <decorator-gen-2>:1(<module>)
1 0.000 0.000 0.000 0.000 <decorator-gen-3>:1(<module>)
1 0.000 0.000 0.000 0.000 <decorator-gen-4>:1(<module>)
1 0.000 0.000 0.000 0.000 <decorator-gen-5>:1(<module>)
1 0.000 0.000 0.000 0.000 <decorator-gen-6>:1(<module>)
1 0.000 0.000 0.000 0.000 <decorator-gen-7>:1(<module>)
1 0.000 0.000 0.000 0.000 <decorator-gen-8>:1(<module>)
3 0.000 0.000 0.002 0.001 <decorator-gen-8>:1(_held_figure)
1 0.000 0.000 0.000 0.000 <decorator-gen-9>:1(<module>)
1416 0.003 0.000 0.003 0.000 <frozen importlib._bootstrap>:1000(__init__)
2 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:1003(module_repr)
1245 0.034 0.000 0.166 0.000 <frozen importlib._bootstrap>:1019(init_module_attrs)
1245 0.011 0.000 0.209 0.000 <frozen importlib._bootstrap>:1099(create)
1245/32 0.008 0.000 23.613 0.738 <frozen importlib._bootstrap>:1122(_exec)
169/105 0.003 0.000 4.104 0.039 <frozen importlib._bootstrap>:1156(_load_backward_compatible)
1414/32 0.034 0.000 23.620 0.738 <frozen importlib._bootstrap>:1186(_load_unlocked)
15 0.000 0.000 0.002 0.000 <frozen importlib._bootstrap>:1223(_fix_up_module)
1650 0.002 0.000 0.006 0.000 <frozen importlib._bootstrap>:1266(find_spec)
19 0.000 0.000 0.006 0.000 <frozen importlib._bootstrap>:1287(load_module)
1245 0.019 0.000 0.019 0.000 <frozen importlib._bootstrap>:129(_new_module)
19 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:1311(is_package)
1631 0.004 0.000 0.009 0.000 <frozen importlib._bootstrap>:1336(find_spec)
1631 0.022 0.000 0.254 0.000 <frozen importlib._bootstrap>:1401(_open_registry)
1631 0.034 0.000 0.307 0.000 <frozen importlib._bootstrap>:1408(_search_registry)
139 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:141(__init__)
1631 0.006 0.000 0.314 0.000 <frozen importlib._bootstrap>:1423(find_spec)
139 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:144(__enter__)
1245/32 0.022 0.000 23.612 0.738 <frozen importlib._bootstrap>:1465(exec_module)
139 0.001 0.000 0.003 0.000 <frozen importlib._bootstrap>:147(__exit__)
556 0.001 0.000 0.001 0.000 <frozen importlib._bootstrap>:148(<genexpr>)
1245 0.055 0.000 2.437 0.002 <frozen importlib._bootstrap>:1534(get_code)
1245 0.008 0.000 0.008 0.000 <frozen importlib._bootstrap>:1591(__init__)
1245 0.001 0.000 0.001 0.000 <frozen importlib._bootstrap>:1616(get_filename)
1245 0.941 0.001 1.037 0.001 <frozen importlib._bootstrap>:1621(get_data)
1245 0.011 0.000 0.384 0.000 <frozen importlib._bootstrap>:1631(path_stats)
120 0.001 0.000 0.001 0.000 <frozen importlib._bootstrap>:1697(__init__)
120/92 0.004 0.000 4.087 0.044 <frozen importlib._bootstrap>:1708(load_module)
1888 0.017 0.000 0.066 0.000 <frozen importlib._bootstrap>:172(__init__)
120 0.001 0.000 0.006 0.000 <frozen importlib._bootstrap>:1726(is_package)
240 0.001 0.000 0.001 0.000 <frozen importlib._bootstrap>:1729(<genexpr>)
181 0.058 0.000 0.128 0.001 <frozen importlib._bootstrap>:1853(_path_hooks)
2614 0.012 0.000 0.142 0.000 <frozen importlib._bootstrap>:1870(_path_importer_cache)
135 0.002 0.000 0.461 0.003 <frozen importlib._bootstrap>:1887(_legacy_get_spec)
1631 0.035 0.000 2.252 0.001 <frozen importlib._bootstrap>:1902(_get_spec)
2857 0.033 0.000 0.035 0.000 <frozen importlib._bootstrap>:192(acquire)
1631 0.009 0.000 2.261 0.001 <frozen importlib._bootstrap>:1934(find_spec)
179 0.006 0.000 0.012 0.000 <frozen importlib._bootstrap>:1979(__init__)
1253 0.001 0.000 0.001 0.000 <frozen importlib._bootstrap>:1985(<genexpr>)
1365 0.016 0.000 0.055 0.000 <frozen importlib._bootstrap>:2011(_get_spec)
2335 0.118 0.000 1.603 0.001 <frozen importlib._bootstrap>:2016(find_spec)
178 0.016 0.000 0.100 0.001 <frozen importlib._bootstrap>:2061(_fill_cache)
178 0.003 0.000 0.004 0.000 <frozen importlib._bootstrap>:2090(<setcomp>)
179 0.002 0.000 0.071 0.000 <frozen importlib._bootstrap>:2102(path_hook_for_FileFinder)
7719 0.012 0.000 0.017 0.000 <frozen importlib._bootstrap>:2120(__enter__)
7719 0.013 0.000 0.020 0.000 <frozen importlib._bootstrap>:2124(__exit__)
1176 0.007 0.000 0.019 0.000 <frozen importlib._bootstrap>:2138(_find_spec_legacy)
1650/1648 0.097 0.000 2.739 0.002 <frozen importlib._bootstrap>:2147(_find_spec)
2857 0.031 0.000 0.033 0.000 <frozen importlib._bootstrap>:217(release)
216 0.001 0.000 0.001 0.000 <frozen importlib._bootstrap>:2187(_sanity_check)
1730/92 0.045 0.000 23.736 0.258 <frozen importlib._bootstrap>:2207(_find_and_load_unlocked)
1730/92 0.057 0.000 23.753 0.258 <frozen importlib._bootstrap>:2234(_find_and_load)
216/214 0.004 0.000 0.051 0.000 <frozen importlib._bootstrap>:2240(_gcd_import)
338220/334774 1.206 0.000 10.385 0.000 <frozen importlib._bootstrap>:2264(_handle_fromlist)
108/107 0.001 0.000 0.052 0.000 <frozen importlib._bootstrap>:2322(__import__)
1730 0.006 0.000 0.006 0.000 <frozen importlib._bootstrap>:257(__init__)
1730 0.017 0.000 0.134 0.000 <frozen importlib._bootstrap>:261(__enter__)
1730 0.007 0.000 0.033 0.000 <frozen importlib._bootstrap>:268(__exit__)
2857 0.040 0.000 0.106 0.000 <frozen importlib._bootstrap>:274(_get_module_lock)
1888 0.003 0.000 0.003 0.000 <frozen importlib._bootstrap>:288(cb)
1127 0.011 0.000 0.044 0.000 <frozen importlib._bootstrap>:293(_lock_unlock_module)
2335 0.007 0.000 0.007 0.000 <frozen importlib._bootstrap>:30(_relax_case)
1804/32 0.007 0.000 23.649 0.739 <frozen importlib._bootstrap>:313(_call_with_frames_removed)
2490 0.035 0.000 0.140 0.000 <frozen importlib._bootstrap>:437(cache_from_source)
2490 0.008 0.000 0.014 0.000 <frozen importlib._bootstrap>:45(_r_long)
9163 0.047 0.000 0.132 0.000 <frozen importlib._bootstrap>:50(_path_join)
9990 0.024 0.000 0.024 0.000 <frozen importlib._bootstrap>:518(_verbose_message)
9163 0.050 0.000 0.071 0.000 <frozen importlib._bootstrap>:52(<listcomp>)
1365/1155 0.007 0.000 4.094 0.004 <frozen importlib._bootstrap>:534(_check_name_wrapper)
38 0.000 0.000 0.006 0.000 <frozen importlib._bootstrap>:546(_requires_builtin_wrapper)
2618 0.042 0.000 0.063 0.000 <frozen importlib._bootstrap>:56(_path_split)
1245 0.055 0.000 0.070 0.000 <frozen importlib._bootstrap>:599(_validate_bytecode_header)
1245 0.017 0.000 0.823 0.001 <frozen importlib._bootstrap>:654(_compile_bytecode)
5307 0.020 0.000 1.598 0.000 <frozen importlib._bootstrap>:68(_path_stat)
2 0.000 0.000 0.000 0.000 <frozen importlib._bootstrap>:690(_module_repr)
1245 0.003 0.000 0.003 0.000 <frozen importlib._bootstrap>:728(__init__)
1245 0.004 0.000 0.004 0.000 <frozen importlib._bootstrap>:732(__enter__)
1245 0.017 0.000 0.032 0.000 <frozen importlib._bootstrap>:739(__exit__)
4980 0.005 0.000 0.005 0.000 <frozen importlib._bootstrap>:742(<genexpr>)
1727 0.014 0.000 0.597 0.000 <frozen importlib._bootstrap>:78(_path_is_mode_type)
1801 0.016 0.000 0.016 0.000 <frozen importlib._bootstrap>:790(__init__)
2490 0.017 0.000 0.104 0.000 <frozen importlib._bootstrap>:824(cached)
1245 0.008 0.000 0.010 0.000 <frozen importlib._bootstrap>:842(parent)
1247 0.001 0.000 0.001 0.000 <frozen importlib._bootstrap>:850(has_location)
49/48 0.001 0.000 0.464 0.010 <frozen importlib._bootstrap>:859(spec_from_loader)
1547 0.006 0.000 0.547 0.000 <frozen importlib._bootstrap>:87(_path_isfile)
1395 0.020 0.000 0.487 0.000 <frozen importlib._bootstrap>:884(spec_from_file_location)
180 0.001 0.000 0.057 0.000 <frozen importlib._bootstrap>:92(_path_isdir)
1 0.000 0.000 0.000 0.000 <string>:1(<module>)
1 0.000 0.000 0.000 0.000 <string>:1(error)
1 0.000 0.000 0.000 0.000 <string>:11(com_error)
313 0.001 0.000 0.001 0.000 <string>:12(__new__)
1 0.000 0.000 0.000 0.000 <string>:2(<module>)
1 0.000 0.000 0.000 0.000 <string>:2(__init__)
4 0.000 0.000 0.000 0.000 <string>:2(_parse_args)
1 0.000 0.000 0.000 0.000 <string>:5(AndersonResult)
1 0.000 0.000 0.000 0.000 <string>:5(Anderson_ksampResult)
1 0.000 0.000 0.000 0.000 <string>:5(AnsariResult)
1 0.000 0.000 0.000 0.000 <string>:5(ArgInfo)
1 0.000 0.000 0.000 0.000 <string>:5(ArgSpec)
1 0.000 0.000 0.000 0.000 <string>:5(Arguments)
1 0.000 0.000 0.000 0.000 <string>:5(ArrayFormula)
1 0.000 0.000 0.000 0.000 <string>:5(Attribute)
1 0.000 0.000 0.000 0.000 <string>:5(BartlettResult)
1 0.000 0.000 0.000 0.000 <string>:5(BinnedStatistic2dResult)
1 0.000 0.000 0.000 0.000 <string>:5(FlignerResult)
1 0.000 0.000 0.000 0.000 <string>:5(Formula)
1 0.002 0.002 0.002 0.002 <string>:5(FriedmanchisquareResult)
1 0.000 0.000 0.000 0.000 <string>:5(FullArgSpec)
1 0.000 0.000 0.000 0.000 <string>:5(FunctionAttributes)
1 0.000 0.000 0.000 0.000 <string>:5(HistogramResult)
19424 2.346 0.000 2.346 0.000 {pandas.algos.take_2d_axis1_object_object}
82904 0.318 0.000 0.318 0.000 {pandas.index.get_value_at}
199 0.051 0.000 0.051 0.000 {pandas.lib.clean_index_list}
57 0.002 0.000 0.002 0.000 {pandas.lib.get_blkno_indexers}
2918 0.095 0.000 0.095 0.000 {pandas.lib.infer_dtype}
38 0.001 0.000 0.001 0.000 {pandas.lib.is_bool_array}
76 0.000 0.000 0.000 0.000 {pandas.lib.is_bool}
166001 0.169 0.000 0.169 0.000 {pandas.lib.is_float}
249112 0.215 0.000 0.215 0.000 {pandas.lib.is_integer}
20 0.000 0.000 0.000 0.000 {pandas.lib.is_timedelta_array}
38 0.002 0.000 0.002 0.000 {pandas.lib.isnullobj}
414822 1.023 0.000 1.023 0.000 {pandas.lib.isscalar}
11 0.054 0.005 0.054 0.005 {pandas.lib.maybe_convert_numeric}
56 0.006 0.000 0.006 0.000 {pandas.lib.maybe_convert_objects}
3 0.000 0.000 0.018 0.006 {pandas.lib.reduce}
65 0.009 0.000 0.105 0.002 {pandas.lib.scalar_compare}
28 0.001 0.000 0.001 0.000 {pandas.lib.to_object_array_tuples}
415734 2.533 0.000 4.116 0.000 {pandas.lib.values_from_object}
20 0.000 0.000 0.000 0.000 {pandas.tslib.is_timestamp_array}
48 0.001 0.000 0.001 0.000 {reduce.nansum}
Upvotes: 3
Views: 6800
Answers (2)
Ciprian Tomoiagă
Reputation: 3990
I would also recommend flamegraphs for interpreting profiling results. It works especially well with PyFlame, which is a PTracing profiler (it has minimal overhead).
A flamegraph looks like this:
Upvotes: 6
Alexander Davydov
Reputation: 395
You might want to use some of the tools available to analyze/visualize the cProfile output like pstats or RunShakeRun
Upvotes: 0
Related Questions
- Python profiling
- Better breakdown of Python profiling
- Interpreting Memory Profiling results in Python
- Interpreting Python profiler
- Performance analyzer
- Understanding Python profile output
- Interpreting the output of python memory_profiler
- Clarifying this output from the pstats module
- Python profiler not giving enough information
- Question regarding python profiling