These are chat archives for opal/opal

16th
Oct 2017
Colin Gunn
@balmoral
Oct 16 2017 00:19

I frequently/randomly get this error when source maps are being fetched from the server:

RuntimeError: can't add a new key into hash during iteration
        /Users/col/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/set.rb:171:in `replace'
        /Users/col/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/set.rb:171:in `replace'
        /Users/col/.rvm/rubies/ruby-2.4.0/lib/ruby/2.4.0/set.rb:381:in `collect!'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/loader.rb:85:in `asset_from_cache'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/loader.rb:57:in `block in load'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/loader.rb:311:in `block in fetch_asset_from_dependency_cache'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/loader.rb:307:in `each'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/loader.rb:307:in `each_with_index'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/loader.rb:307:in `fetch_asset_from_dependency_cache'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/loader.rb:44:in `load'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/cached_environment.rb:20:in `block in initialize'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/cached_environment.rb:47:in `load'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/bundle.rb:23:in `block in call'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/utils.rb:200:in `dfs'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/bundle.rb:24:in `call'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/processor_utils.rb:75:in `call_processor'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/processor_utils.rb:57:in `block in call_processors'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/processor_utils.rb:56:in `reverse_each'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/processor_utils.rb:56:in `call_processors'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/loader.rb:134:in `load_from_unloaded'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/loader.rb:60:in `block in load'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/loader.rb:317:in `fetch_asset_from_dependency_cache'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/loader.rb:44:in `load'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/cached_environment.rb:20:in `block in initialize'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/cached_environment.rb:47:in `load'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/base.rb:66:in `find_asset'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/environment.rb:30:in `find_asset'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/sprockets-3.7.1/lib/sprockets/base.rb:92:in `[]'
        /Users/col/.rvm/gems/ruby-2.4.0/bundler/gems/opal-sprockets-c5ce8dc6c5c5/lib/opal/sprockets/source_map_server.rb:87:in `call'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/rack-2.0.3/lib/rack/etag.rb:25:in `call'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/rack-2.0.3/lib/rack/conditional_get.rb:25:in `call'
        /Users/col/.rvm/gems/ruby-2.4.0/gems/rack-2.0.3/lib/rack/builder.rb:153:in `call

I seem to remember it occuring a while back with Volt too. Haven’t noticed any downside, maybe just will miss a source map.

Jamie Gaskins
@jgaskins
Oct 16 2017 00:20
@janbiedermann Added 350 load paths and files and, sure enough, it's slow as shit on Sprockets 3.7.1. Your patch doesn't make it any faster for me, though, so I'm not sure what we're doing differently.
@balmoral Are you using Puma?
Colin Gunn
@balmoral
Oct 16 2017 00:21
@dancinglightning @elia ^Was only a semantic nicety, so NO great loss. :smile:
@jgaskins mostly using puma, but also testing with thin when experimenting with websockets (seem to get different debug output)
Jamie Gaskins
@jgaskins
Oct 16 2017 00:22
That's usually a multi-threading issue with Sprockets — one thread is trying to add something to the cache while another is iterating.
I primarily see it while using Puma because threads :-)
Colin Gunn
@balmoral
Oct 16 2017 00:24
Thought it might be something like that - is there any way of making it thread safe?
Jamie Gaskins
@jgaskins
Oct 16 2017 00:26
I assume something like this might work:
class App < Roda
  assets = Roda::OpalAssets.new
  mutex = Mutex.new

  route do |r|
    mutex.synchronize { assets.route r }
  end
end
It would only serve one asset at a time but the rest of the app would run multithreaded
I've been ignoring it mainly because it doesn't seem to harm anything.
I haven't done the mutex thing myself because that's what Rails does and Roda::OpalAssets serves my apps 3x as fast. :-)
Jamie Gaskins
@jgaskins
Oct 16 2017 00:37
@janbiedermann I'd be surprised if any app really needs that many load paths and files, though. That sounds like an excessive number of files to load and an especially massive number of load paths for a single front-end app.
Colin Gunn
@balmoral
Oct 16 2017 00:37
@jgaskins I’ve placed the mutex so that it only syncs when it’s a serving source map request - works great! Thanks for the tip. :smile:
Jamie Gaskins
@jgaskins
Oct 16 2017 00:37
@balmoral Nice!
Elia Schito
@elia
Oct 16 2017 00:38

@jgaskins this is what seems to have resolved the super slow compiling times we were noticing lately:

assets_cache_path = "#{::Rails.root}/tmp/cache/assets/#{::Rails.env}"
Rails.application.assets.cache = Sprockets::Cache::FileStore.new(assets_cache_path, 200.megabytes)

maybe you're hitting the 25mb limit too…

Jamie Gaskins
@jgaskins
Oct 16 2017 00:41
Oh, good point, I didn't think about cache limits. I'm using the in-memory cache, which IIRC is capped at 1000 entries by default. With > 400 assets, plus source maps, my sample app doesn't have much breathing room.
Colin Gunn
@balmoral
Oct 16 2017 01:11
@jgaskins code snippet for source maps thread safety, no noticeable slow down in response…
    def source_map_prefix
      '/__OPAL_SOURCE_MAPS__'
    end

    def init_source_maps
      if Opal::Config.source_map_enabled = config.source_maps? && development?
        ::Opal::Sprockets::SourceMapHeaderPatch.inject!(source_map_prefix)
        opal_source_map_server = Opal::SourceMapServer.new(sprockets, source_map_prefix)
        builder = Rack::Builder.new do
           use Rack::ConditionalGet
           use Rack::ETag
           run opal_source_map_server
        end
        mutex = Mutex.new
        @source_map_server = lamba do |env|
          mutex.synchronize do
            builder.call(env)
          end
        end
      end
    end

    def route(r)
      …
      r.on source_map_prefix[1..-1] do
        r.run @source_map_server
      end 
      ...
    end
Colin Gunn
@balmoral
Oct 16 2017 01:22
@elia @janbiedermann @jgaskins load time for my app (nearly 1000 files) halved by doubling sprockets cache memory store size
Jan Biedermann
@janbiedermann
Oct 16 2017 08:41
@balmoral if you got the cache size right, you can determine by measuring the time taken for a javascript_include_tag with Benchmark.measure do, for me that was double digit seconds, when the cache size is right it schould be 0.00xxx s on a page load without compile.