module Mongo::Retryable

Defines basic behavior around retrying operations.

@since 2.1.0

Public Instance Methods

legacy_write_with_retry(server = nil, session = nil) { |server| ... } click to toggle source

Implements legacy write retrying functionality by yielding to the passed block one or more times.

This method is used for operations which are not supported by modern retryable writes, such as delete_many and update_many.

@param [ Server ] server The server which should be used for the

operation. If not provided, the current primary will be retrieved from
the cluster.

@param [ nil | Session ] session Optional session to use with the operation.

@yieldparam [ Server ] server The server to which the write should be sent.

@api private

# File lib/mongo/retryable.rb, line 285
def legacy_write_with_retry(server = nil, session = nil)
  # This is the pre-session retry logic, and is not subject to
  # current retryable write specifications.
  # In particular it does not retry on SocketError and SocketTimeoutError.
  attempt = 0
  begin
    attempt += 1
    server ||= select_server(cluster, ServerSelector.primary, session)
    yield server
  rescue Error::OperationFailure => e
    e.add_note('legacy retry')
    e.add_note("attempt #{attempt}")
    server = nil
    if attempt > client.max_write_retries
      raise e
    end
    if e.write_retryable? && !(session && session.in_transaction?)
      log_retry(e, message: 'Legacy write retry')
      cluster.scan!(false)
      retry
    else
      raise e
    end
  end
end
nro_write_with_retry(session, write_concern) { |server| ... } click to toggle source

Retryable writes wrapper for operations not supporting modern retryable writes.

If the driver is configured to use modern retryable writes, this method yields to the passed block exactly once, thus not retrying any writes.

If the driver is configured to use legacy retryable writes, this method delegates to legacy_write_with_retry which performs write retries using legacy logic.

@param [ nil | Session ] session Optional session to use with the operation. @param [ nil | Hash | WriteConcern::Base ] write_concern The write concern.

@yieldparam [ Server ] server The server to which the write should be sent.

@api private

# File lib/mongo/retryable.rb, line 257
def nro_write_with_retry(session, write_concern, &block)
  if session && session.client.options[:retry_writes]
    server = select_server(cluster, ServerSelector.primary, session)
    begin
      yield server
    rescue Error::SocketError, Error::SocketTimeoutError, Error::OperationFailure => e
      e.add_note('retries disabled')
      raise e
    end
  else
    legacy_write_with_retry(nil, session, &block)
  end
end
read_with_one_retry(options = nil) { || ... } click to toggle source

Execute a read operation with a single retry on network errors.

This method is used by the driver for some of the internal housekeeping operations. Application-requested reads should use read_with_retry rather than this method.

@api private

@example Execute the read.

read_with_one_retry do
  ...
end

@note This only retries read operations on socket errors.

@param [ Hash ] options Options. @param [ Proc ] block The block to execute.

@option options [ String ] :retry_message Message to log when retrying.

@return [ Result ] The result of the operation.

@since 2.2.6

# File lib/mongo/retryable.rb, line 154
def read_with_one_retry(options = nil)
  yield
rescue Error::SocketError, Error::SocketTimeoutError => e
  retry_message = options && options[:retry_message]
  log_retry(e, message: retry_message)
  yield
end
read_with_retry(session = nil, server_selector = nil) { |server| ... } click to toggle source

Execute a read operation with retrying.

This method performs server selection for the specified server selector and yields to the provided block, which should execute the initial query operation and return its result. The block will be passed the server selected for the operation. If the block raises an exception, and this exception corresponds to a read retryable error, and read retries are enabled for the client, this method will perform server selection again and yield to the block again (with potentially a different server). If the block returns successfully, the result of the block is returned.

If modern retry reads are on (which is the default), the initial read operation will be retried once. If legacy retry reads are on, the initial read operation will be retried zero or more times depending on the :max_read_retries client setting, the default for which is 1. To disable read retries, turn off modern read retries by setting retry_reads: false and set :max_read_retries to 0 on the client.

@api private

@example Execute the read.

read_with_retry(session, server_selector) do |server|
  ...
end

@param [ Mongo::Session ] session The session that the operation is being

run on.

@param [ Mongo::ServerSelector::Selectable ] server_selector Server

selector for the operation.

@param [ Proc ] block The block to execute.

@return [ Result ] The result of the operation.

# File lib/mongo/retryable.rb, line 99
def read_with_retry(session = nil, server_selector = nil, &block)
  if session.nil? && server_selector.nil?
    # Older versions of Mongoid call read_with_retry without arguments.
    # This is already not correct in a MongoDB 3.6+ environment with
    # sessions. For compatibility we emulate the legacy driver behavior
    # here but upgrading Mongoid is strongly recommended.
    unless $_mongo_read_with_retry_warned
      $_mongo_read_with_retry_warned = true
      Logger.logger.warn("Legacy read_with_retry invocation - please update the application and/or its dependencies")
    end
    # Since we don't have a session, we cannot use the modern read retries.
    # And we need to select a server but we don't have a server selector.
    # Use PrimaryPreferred which will work as long as there is a data
    # bearing node in the cluster; the block may select a different server
    # which is fine.
    server_selector = ServerSelector.get(mode: :primary_preferred)
    legacy_read_with_retry(nil, server_selector, &block)
  elsif session && session.retry_reads?
    modern_read_with_retry(session, server_selector, &block)
  elsif client.max_read_retries > 0
    legacy_read_with_retry(session, server_selector, &block)
  else
    server = select_server(cluster, server_selector, session)
    begin
      yield server
    rescue Error::SocketError, Error::SocketTimeoutError, Error::OperationFailure => e
      e.add_note('retries disabled')
      raise e
    end
  end
end
read_with_retry_cursor(session, server_selector, view) { |server| ... } click to toggle source

Execute a read operation returning a cursor with retrying.

This method performs server selection for the specified server selector and yields to the provided block, which should execute the initial query operation and return its result. The block will be passed the server selected for the operation. If the block raises an exception, and this exception corresponds to a read retryable error, and read retries are enabled for the client, this method will perform server selection again and yield to the block again (with potentially a different server). If the block returns successfully, the result of the block (which should be a Mongo::Operation::Result) is used to construct a Mongo::Cursor object for the result set. The cursor is then returned.

If modern retry reads are on (which is the default), the initial read operation will be retried once. If legacy retry reads are on, the initial read operation will be retried zero or more times depending on the :max_read_retries client setting, the default for which is 1. To disable read retries, turn off modern read retries by setting retry_reads: false and set :max_read_retries to 0 on the client.

@api private

@example Execute a read returning a cursor.

cursor = read_with_retry_cursor(session, server_selector, view) do |server|
  # return a Mongo::Operation::Result
  ...
end

@param [ Mongo::Session ] session The session that the operation is being

run on.

@param [ Mongo::ServerSelector::Selectable ] server_selector Server

selector for the operation.

@param [ CollectionView ] view The CollectionView defining the query. @param [ Proc ] block The block to execute.

@return [ Cursor ] The cursor for the result set.

# File lib/mongo/retryable.rb, line 59
def read_with_retry_cursor(session, server_selector, view, &block)
  read_with_retry(session, server_selector) do |server|
    result = yield server
    Cursor.new(view, result, server, session: session)
  end
end
write_with_retry(session, write_concern, ending_transaction = false) { |server, txn_num, false| ... } click to toggle source

Implements write retrying functionality by yielding to the passed block one or more times.

If the session is provided (hence, the deployment supports sessions), and modern retry writes are enabled on the client, the modern retry logic is invoked. Otherwise the legacy retry logic is invoked.

If ending_transaction parameter is true, indicating that a transaction is being committed or aborted, the operation is executed exactly once. Note that, since transactions require sessions, this method will raise ArgumentError if ending_transaction is true and session is nil.

@api private

@example Execute the write.

write_with_retry do
  ...
end

@note This only retries operations on not master failures, since it is

the only case we can be sure a partial write did not already occur.

@param [ nil | Session ] session Optional session to use with the operation. @param [ nil | Hash | WriteConcern::Base ] write_concern The write concern. @param [ true | false ] ending_transaction True if the write operation is abortTransaction or

commitTransaction, false otherwise.

@param [ Proc ] block The block to execute.

@yieldparam [ Server ] server The server to which the write should be sent. @yieldparam [ Integer ] txn_num Transaction number (NOT the ACID kind).

@return [ Result ] The result of the operation.

@since 2.1.0

# File lib/mongo/retryable.rb, line 196
def write_with_retry(session, write_concern, ending_transaction = false, &block)
  if ending_transaction && !session
    raise ArgumentError, 'Cannot end a transaction without a session'
  end

  unless ending_transaction || retry_write_allowed?(session, write_concern)
    return legacy_write_with_retry(nil, session, &block)
  end

  # If we are here, session is not nil. A session being nil would have
  # failed retry_write_allowed? check.

  server = select_server(cluster, ServerSelector.primary, session)

  unless ending_transaction || server.retry_writes?
    return legacy_write_with_retry(server, session, &block)
  end

  txn_num = if session.in_transaction?
    session.txn_num
  else
    session.next_txn_num
  end
  begin
    yield(server, txn_num, false)
  rescue Error::SocketError, Error::SocketTimeoutError => e
    e.add_note('modern retry')
    e.add_note("attempt 1")
    if session.in_transaction? && !ending_transaction
      raise e
    end
    retry_write(e, session, txn_num, &block)
  rescue Error::OperationFailure => e
    e.add_note('modern retry')
    e.add_note("attempt 1")
    if e.unsupported_retryable_write?
      raise_unsupported_error(e)
    elsif (session.in_transaction? && !ending_transaction) || !e.write_retryable?
      raise e
    end

    retry_write(e, session, txn_num, &block)
  end
end

Private Instance Methods

legacy_read_with_retry(session, server_selector) { |server| ... } click to toggle source
# File lib/mongo/retryable.rb, line 334
def legacy_read_with_retry(session, server_selector)
  attempt = 0
  server = select_server(cluster, server_selector, session)
  begin
    attempt += 1
    yield server
  rescue Error::SocketError, Error::SocketTimeoutError => e
    e.add_note('legacy retry')
    e.add_note("attempt #{attempt}")
    if attempt > client.max_read_retries || (session && session.in_transaction?)
      raise e
    end
    log_retry(e, message: 'Legacy read retry')
    server = select_server(cluster, server_selector, session)
    retry
  rescue Error::OperationFailure => e
    e.add_note('legacy retry')
    e.add_note("attempt #{attempt}")
    if e.retryable? && !(session && session.in_transaction?)
      if attempt > client.max_read_retries
        raise e
      end
      log_retry(e, message: 'Legacy read retry')
      sleep(client.read_retry_interval)
      server = select_server(cluster, server_selector, session)
      retry
    else
      raise e
    end
  end
end
log_retry(e, options = nil) click to toggle source

Log a warning so that any application slow down is immediately obvious.

# File lib/mongo/retryable.rb, line 454
def log_retry(e, options = nil)
  message = if options && options[:message]
    options[:message]
  else
    "Retry"
  end
  Logger.logger.warn "#{message} due to: #{e.class.name} #{e.message}"
end
modern_read_with_retry(session, server_selector) { |server| ... } click to toggle source
# File lib/mongo/retryable.rb, line 313
def modern_read_with_retry(session, server_selector, &block)
  server = select_server(cluster, server_selector, session)
  begin
    yield server
  rescue Error::SocketError, Error::SocketTimeoutError => e
    e.add_note('modern retry')
    e.add_note("attempt 1")
    if session.in_transaction?
      raise e
    end
    retry_read(e, server_selector, session, &block)
  rescue Error::OperationFailure => e
    e.add_note('modern retry')
    e.add_note("attempt 1")
    if session.in_transaction? || !e.write_retryable?
      raise e
    end
    retry_read(e, server_selector, session, &block)
  end
end
raise_unsupported_error(e) click to toggle source

Retry writes on MMAPv1 should raise an actionable error; append actionable information to the error message and preserve the backtrace.

# File lib/mongo/retryable.rb, line 465
def raise_unsupported_error(e)
  new_error = Error::OperationFailure.new("#{e.class}: #{e} "\
    "This MongoDB deployment does not support retryable writes. Please add "\
    "retryWrites=false to your connection string or use the retry_writes: false Ruby client option")
  new_error.set_backtrace(e.backtrace)
  raise new_error
end
retry_read(original_error, server_selector, session) { |server, true| ... } click to toggle source
# File lib/mongo/retryable.rb, line 381
def retry_read(original_error, server_selector, session, &block)
  begin
    server = select_server(cluster, server_selector, session)
  rescue => e
    original_error.add_note("later retry failed: #{e.class}: #{e}")
    raise original_error
  end

  log_retry(original_error, message: 'Read retry')

  begin
    yield server, true
  rescue Error::SocketError, Error::SocketTimeoutError => e
    e.add_note('modern retry')
    e.add_note("attempt 2")
    raise e
  rescue Error::OperationFailure => e
    e.add_note('modern retry')
    unless e.write_retryable?
      original_error.add_note("later retry failed: #{e.class}: #{e}")
      raise original_error
    end
    e.add_note("attempt 2")
    raise e
  rescue => e
    e.add_note('modern retry')
    original_error.add_note("later retry failed: #{e.class}: #{e}")
    raise original_error
  end
end
retry_write(original_error, session, txn_num) { |server, txn_num, true| ... } click to toggle source
# File lib/mongo/retryable.rb, line 412
def retry_write(original_error, session, txn_num, &block)
  # We do not request a scan of the cluster here, because error handling
  # for the error which triggered the retry should have updated the
  # server description and/or topology as necessary (specifically,
  # a socket error or a not master error should have marked the respective
  # server unknown). Here we just need to wait for server selection.
  server = select_server(cluster, ServerSelector.primary, session)
  unless server.retry_writes?
    # Do not need to add "modern retry" here, it should already be on
    # the first exception.
    original_error.add_note('did not retry because server selected for retry does not supoprt retryable writes')
    raise original_error
  end
  log_retry(original_error, message: 'Write retry')
  yield(server, txn_num, true)
rescue Error::SocketError, Error::SocketTimeoutError => e
  e.add_note('modern retry')
  e.add_note('attempt 2')
  raise e
rescue Error::OperationFailure => e
  e.add_note('modern retry')
  if e.write_retryable?
    e.add_note('attempt 2')
    raise e
  else
    original_error.add_note("later retry failed: #{e.class}: #{e}")
    raise original_error
  end
rescue => e
  # Do not need to add "modern retry" here, it should already be on
  # the first exception.
  original_error.add_note("later retry failed: #{e.class}: #{e}")
  raise original_error
end
retry_write_allowed?(session, write_concern) click to toggle source
# File lib/mongo/retryable.rb, line 366
def retry_write_allowed?(session, write_concern)
  unless session && session.retry_writes?
    return false
  end

  if write_concern.nil?
    true
  else
    unless write_concern.is_a?(WriteConcern::Base)
      write_concern = WriteConcern.get(write_concern)
    end
    write_concern.acknowledged?
  end
end
select_server(cluster, server_selector, session) click to toggle source

This is a separate method to make it possible for the test suite to assert that server selection is performed during retry attempts.

# File lib/mongo/retryable.rb, line 449
def select_server(cluster, server_selector, session)
  server_selector.select_server(cluster, nil, session)
end