Wednesday, November 29, 2017

Collecting Unique Destination Hosts with Tcpdump

Simple solution to collecting all destination hosts for IP traffic. Just start it and when you ctrl-C, you'll have a nice sorted list of unique hosts and the port or assumed protocol based on the port:

sudo tcpdump 'ip' | cut -d " " -f 5 > /tmp/ipout.tmp || cat /tmp/ipout.tmp | sort | uniq && rm /tmp/ipout.tmp

Note: this assumes you're just doing it in a single macOS terminal window at a time.

Monday, November 20, 2017

Docker compose healthcheck

Running too many docker containers on limited hardware might not work well with Docker's native healthcheck defaults.

If using healthcheck, it's probably best to supply additional configuration. Specifically, v2.3 and v3.4 now have start_period in milliseconds. Also, consider changing the startup order. Use 'docker ps' to check the healthcheck status, which is listed by the STATUS value in parentheses.

Sunday, June 18, 2017

Manually Removing GoogleSoftwareUpdate in macOS

Wrapped it in a basic check for it's existence to avoid having to sudo if not necessary:

if [ -d "/Library/Google/GoogleSoftwareUpdate/" ]; then
  echo "sudo'ing to remove GoogleSoftwareUpdate. Might need to enter your password."
  sudo rm -rf /Library/Google/GoogleSoftwareUpdate/
  sudo rm -rf /Users/**/Library/Google/GoogleSoftwareUpdate
  sudo rm -f /Users/**/Library/Logs/GoogleSoftwareUpdateAgent.log
  sudo rm -f /Users/**/Library/LaunchAgents/com.google.keystone.agent.plist
fi

Reboot to avoid a bunch of errors in your system log. Have to keep the software up to date on your own after removing it. Do it at your own risk.

Monday, May 22, 2017

Crystal GC and "Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS Program received and didn't handle signal IOT (6)"

Using Crystal 0.22.0, sometimes the GC seems like it is slowing things down a bit or getting hung up, but it turns out that collecting more frequently doesn't seem to help much, and slows it down more overall.

When you eventually run out of memory, you'll hit (or at least I did in macOS):

Too many heap sections: Increase MAXHINCR or MAX_HEAP_SECTS
Program received and didn't handle signal IOT (6)

The impact of those hangs is not remedied by doing forced cleanup, e.g.:

if GC.stats.free_bytes < 90000
  GC.collect
end

I tried that in various ways, but you can't squeeze more memory out or trick GC into running more to run faster, at least not in my testing.

I noticed in this StackOverflow answer, Mono: Too many heap sections Increase MAXHINCR or MAX_HEAP_SECTS when app takes more than 4 GB in memory, that Mono also has a problem with the Boehm GC. Perhaps the two are similar and there's a fix for Crystal or it's compilation that would help with this.

Friday, May 19, 2017

CRLF will be replaced by LF in db/structure.sql.

$ git add db/structure.sql
warning: CRLF will be replaced by LF in db/structure.sql.
The file will have its original line endings in your working directory.

Thought we had all our line-ending nonsense straightened out, but could see that structure.sql before being updated by rake db:migrate just had \n and after had some \r\n:

$ file structure.sql.before_db_migrate
before.txt: ASCII text, with very long lines
$ file structure.sql.after_db_migrate 
after.txt: ASCII text, with very long lines, with CRLF, LF line terminators
$ cat -v structure.sql.after_db_migrate
...
CREATE FUNCTION some_function_name() RETURNS void
    LANGUAGE ...
    AS $$^M
BEGIN^M
  REFRESH ...;^M
  RETURN;^M
END; $$;

So, it was in a single function of the schema. Not Rails's/ActiveRecord's fault and not a git config problem.

Friday, May 12, 2017

List Terminal Commands and Utilities and Listing All Text Editors in Posix and Linux

This is a neat command I've not used before in bash:

compgen -c | sort | uniq

That lists all of the commands available.

Getting an (incomplete) list of available text editors is a bit tougher, but we can build on the command above and guess:

compgen -c | sort | uniq | grep '^dit$\|^emacs$\|^uemacs$\|^vi$\|^vim$\|^mu$\|^nano$\|^rnano$\|^pico$\|^jed$\|^joe$'

Thursday, May 4, 2017

Working Around ActiveRecord PostgreSQL Savepoint Problems (RELEASE SAVEPOINT active_record_1)

Ended up not using it, so putting it here if someone runs into problems with AR 4.x savepoints.

The error:

ActiveRecord::StatementInvalid:         ActiveRecord::StatementInvalid: PG::InFailedSqlTransaction: ERROR:  current transaction is aborted, commands ignored until end of transaction block
        : RELEASE SAVEPOINT active_record_1
            .../gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:155:in `async_exec'
            .../gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:155:in `block in execute'
            .../gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract_adapter.rb:484:in `block in log'
            .../gems/activesupport-4.2.8/lib/active_support/notifications/instrumenter.rb:20:in `instrument'
            .../gems/activerecord-4.2.8/lib/active_record/connection_adapters/abstract_adapter.rb:478:in `log'
            .../gems/activerecord-4.2.8/lib/active_record/connection_adapters/postgresql/database_statements.rb:154:in `execute'
            ...

The fix:

module SavepointsFix
  class << self
    attr_accessor :enabled, :debug, :active_savepoints, :rollback_on_fail

    def get_backtrace
      begin
        raise
      rescue => e
        e.backtrace.join("\n")
      end
    end

  end

  def create_savepoint(name = current_savepoint_name)
    if SavepointsFix.enabled
      begin
        puts "SAVEPOINT #{name}. call stack:\n#{SavepointsFix.get_backtrace}\n\n" if SavepointsFix.debug
        result = super(name)
        SavepointsFix.active_savepoints.push name
        result
      rescue Exception => e
        puts "#{e.message}\n#{e.backtrace.join("\n  ")}" if SavepointsFix.debug
        if SavepointsFix.rollback_on_fail
          begin
            ActiveRecord::Base.connection.execute 'ROLLBACK'
            active_savepoints = []
          rescue => e
            puts "#{e.message}\n#{e.backtrace.join("\n  ")}" if SavepointsFix.debug
          end
        end
      else
      end
    else
      super(name)
    end
  end

  def exec_rollback_to_savepoint(name = current_savepoint_name)
    if SavepointsFix.enabled
      if SavepointsFix.active_savepoints.length > 0 && SavepointsFix.active_savepoints.last == name
        puts "ROLLBACK TO SAVEPOINT #{name}. call stack:\n#{SavepointsFix.get_backtrace}\n\n" if SavepointsFix.debug
        begin
          result = super(name)
          SavepointsFix.active_savepoints.pop
          result
        rescue Exception => e
          puts "#{e.message}\n#{e.backtrace.join("\n  ")}" if SavepointsFix.debug
          if SavepointsFix.rollback_on_fail
            begin
              ActiveRecord::Base.connection.execute 'ROLLBACK'
              active_savepoints = []
            rescue => e
              puts "#{e.message}\n#{e.backtrace.join("\n  ")}" if SavepointsFix.debug
            end
          end
        end
      else
        puts "IGNORED EXTRA: ROLLBACK TO SAVEPOINT #{name}. call stack:\n#{SavepointsFix.get_backtrace}\n\n" if SavepointsFix.debug
      end
    else
      super(name)
    end
  end

  def release_savepoint(name = current_savepoint_name)
    if SavepointsFix.enabled
      if SavepointsFix.active_savepoints.length > 0 && SavepointsFix.active_savepoints.last == name
        puts "RELEASE SAVEPOINT #{name}. call stack:\n#{SavepointsFix.get_backtrace}\n\n" if SavepointsFix.debug
        begin
          result = super(name)
          SavepointsFix.active_savepoints.pop
          result
        rescue Exception => e
          puts "#{e.message}\n#{e.backtrace.join("\n  ")}" if SavepointsFix.debug
          if SavepointsFix.rollback_on_fail
            begin
              ActiveRecord::Base.connection.execute 'ROLLBACK'
              active_savepoints = []
            rescue => e
              puts "#{e.message}\n#{e.backtrace.join("\n  ")}" if SavepointsFix.debug
            end
          end
        end
      else
        puts "IGNORED EXTRA: RELEASE SAVEPOINT #{name}. call stack:\n#{SavepointsFix.get_backtrace}\n\n" if SavepointsFix.debug
      end
    else
      super(name)
    end
  end
end

SavepointsFix.active_savepoints = []
SavepointsFix.debug = false
SavepointsFix.rollback_on_fail = true

::ActiveRecord::ConnectionAdapters::PostgreSQLAdapter.prepend SavepointsFix

class Minitest::Test
  alias_method :run_with_savepoint_behavior, :run

  def run
    SavepointsFix.debug = false
    result = run_with_savepoint_behavior    
    result
  ensure
    SavepointsFix.debug = false
  end

  def debug_savepoints(&blk)
    SavepointsFix.debug = true
    yield
  ensure
    SavepointsFix.debug = false
  end

  def with_savepoints_fix(&blk)
    SavepointsFix.enabled = true
    yield
  ensure
    SavepointsFix.enabled = false
  end
end

Usage in a minitest test:

with_savepoints_fix do
  some_model.destroy!
end