Ruby meta programming performance
In 2015, Pragtob wrote a blog post about Ruby meta programming. Well, you may wonder why I’m talking about a blog that wrote five years ago while the computer science is progressing at ever-increasing speed. I searched for it, because in work we were hit by this problem.
In work we use Cocoapods to manage dependency. In a project with few pods you won’t notice the speed of Cocoapods. But what if, let’s say, 600+ pods? It can take half a minute just for Cocoapods to generate XCode Project files! We managed to narrow down the performance issue to this very Ruby file. On the surface, it’s pretty harmless. There is a helper method that can generate other methods using meta programming, saving the hassle to manually write things like sorting results, memorizing… However, there are also some methods that merely return a constant string but also use this helper. Methods defined using define_method have small overhead, but when the method get called tens of thousands of times, the overhead is big enough to use some monkey patching to optimize it. I monkey patched those methods that use define_method but simply return a constant string. Re-define those methods with plain def resulted in a 5s optimization. 5s may not sound like much, but in programming world, it’s like 3 years!
Normally, stories end here. However, I decided to dig a little deeper. You see, I can squeeze 5s from just re-define methods with def. Can’t Ruby interpreter do something similar in the first place? I peeked into Ruby interpreter source code and found out that methods defined with plain def and methods that defined with define_method indeed treated differently. For def, source code is here. And the define_method source code is here. The major difference is the type of def functions are VM_METHOD_TYPE_CFUNC, while the type of define_method is VM_METHOD_TYPE_BMETHOD. Unfortunately, I failed to discover why VM_METHOD_TYPE_CFUNC is faster than VM_METHOD_TYPE_BMETHOD, nor did I know why they’re of different type in the first place. A quick search on the net doesn’t help anything as well. So I just gave up on this matter. However, I did know more about Ruby through the process. I now know the default class is kernel, everything you defined outside any class goes into kernel. And the class type is actually… a class too! The reason why symbols are Ruby is unique is that rb_intern converts string to internal ID type, same string converted to same ID. And Ruby maintains a central function table to determine which function you’re calling, function visibility and the class it belonged to are stored with the function itself.
Tests
Well, if you’re interested the overhead in Ruby 3.0, 5 years after Pragtob’s post. Here is the code to test it. You’ll need benchmark-ips gem to run it.
require 'benchmark/ips'
require 'set'
class BuildSettings
def self.define_method_cocoapods(method_name, build_setting: false, &implementation)
(@build_settings_names ||= Set.new) << method_name.to_s.upcase if build_setting
raw_method_name = :"_raw_#{method_name}"
define_method(raw_method_name, &implementation)
private(raw_method_name)
define_method(method_name) do
retval = send(raw_method_name)
retval
end
end
define_method_cocoapods :meta_fn, build_setting: true do
'A simple string'
end
def direct_fn
'A simple string'
end
define_method_cocoapods :meta_complex_fn, build_setting: true do
10.times do
foo = 'String'
end
end
def direct_complex_fn
10.times do
foo = 'String'
end
end
end
Benchmark.ips do |x|
build_settings = BuildSettings.new
x.report('direct') { build_settings.direct_fn }
x.report('meta') { build_settings.meta_fn }
x.report('complex meta') { build_settings.meta_complex_fn }
x.report('complex direct') { build_settings.direct_complex_fn }
x.compare!
end
And on my computer, the result is:
Warming up --------------------------------------
direct 1.020M i/100ms
meta 436.276k i/100ms
complex meta 118.988k i/100ms
complex direct 143.041k i/100ms
Calculating -------------------------------------
direct 10.511M (± 3.3%) i/s - 53.022M in 5.050267s
meta 4.372M (± 3.0%) i/s - 22.250M in 5.093786s
complex meta 1.196M (± 2.6%) i/s - 6.068M in 5.079384s
complex direct 1.458M (± 2.1%) i/s - 7.295M in 5.005728s
Comparison:
direct: 10511119.1 i/s
meta: 4372098.4 i/s - 2.40x (± 0.00) slower
complex direct: 1457994.7 i/s - 7.21x (± 0.00) slower
complex meta: 1195505.1 i/s - 8.79x (± 0.00) slower
Not much has progressed since 2015?
Not quite so, we didn’t have TruffleRuby back in 2015! And on TruffleRuby define_method and def have same performance! Run this test through TruffleRuby produced this output:
Warming up --------------------------------------
direct 317.071M i/100ms
meta 334.676M i/100ms
complex meta 328.865M i/100ms
complex direct 331.058M i/100ms
Calculating -------------------------------------
direct 3.306B (± 4.9%) i/s - 16.488B in 5.001178s
meta 3.305B (± 3.2%) i/s - 16.734B in 5.068132s
complex meta 3.267B (± 3.6%) i/s - 16.443B in 5.040465s
complex direct 3.314B (± 3.3%) i/s - 16.553B in 5.000735s
Comparison:
complex direct: 3313971061.1 i/s
direct: 3306325701.2 i/s - same-ish: difference falls within error
meta: 3305189666.5 i/s - same-ish: difference falls within error
complex meta: 3266924543.9 i/s - same-ish: difference falls within error
And it’s multitude faster than CRuby too! However to my dismay, Cocoapods doesn’t run well enough with TruffleRuby. I guess we’ll never have good things.
For those of you want to find exactly why define_method is slower on CRuby, here are the lines I found relevant during my research:
- https://github.com/ruby/ruby/blob/b5e94916bfb6aca65211047dcc4c55481c5b30a2/proc.c#L2241 define_method -> rb_add_func
- https://github.com/ruby/ruby/blob/53a094ea45567cdfb7b8aab2f3dde96a15b89565/class.c#L1799 rb_define_method -> rb_add_method_cfunc
- https://github.com/ruby/ruby/blob/a0a8f2abf533702b2cd96e79f700ce5b9cd94f50/vm_method.c#L322 rb_add_method_cfunc -> rb_add_func
- https://github.com/ruby/ruby/blob/a0a8f2abf533702b2cd96e79f700ce5b9cd94f50/vm_method.c#L907 rb_add_method -> rb_method_entry_make
- https://github.com/ruby/ruby/blob/a0a8f2abf533702b2cd96e79f700ce5b9cd94f50/vm_method.c#L762 rb_method_entry_make -> rb_method_definition_create
- https://github.com/ruby/ruby/blob/a0a8f2abf533702b2cd96e79f700ce5b9cd94f50/vm_method.c#L555 rb_method_definition_create
If you know the reason, please share it over the internet.
