com.sleepycat.je.log
Class FileManager

java.lang.Object
  extended by com.sleepycat.je.log.FileManager

public class FileManager
extends Object

The FileManager presents the abstraction of one contiguous file. It doles out LSNs.


Nested Class Summary
static class FileManager.FileMode
           
(package private)  class FileManager.LogEndFileDescriptor
          The LogEndFileDescriptor is used to write and fsync the end of the log.
 
Field Summary
static String BAD_SUFFIX
           
static String BUP_SUFFIX
           
static String DEL_SUFFIX
           
(package private) static String[] DEL_SUFFIXES
           
(package private) static boolean IO_EXCEPTION_TESTING_ON_READ
           
(package private) static boolean IO_EXCEPTION_TESTING_ON_WRITE
           
static String JE_SUFFIX
           
(package private) static String[] JE_SUFFIXES
           
(package private)  long lastFileNumberTouched
           
(package private)  long lastFileTouchedOffset
           
static boolean LOGWRITE_EXCEPTION_TESTING
           
static long N_BAD_WRITES
           
(package private)  LongStat nBytesReadFromWriteQueue
           
(package private)  LongStat nBytesWrittenFromWriteQueue
           
(package private)  IntStat nFileOpens
           
(package private)  LongStat nFSyncTime
           
(package private)  LongStat nLogFSyncs
           
(package private)  IntStat nOpenFiles
           
(package private)  LongStat nRandomReadBytes
           
(package private)  LongStat nRandomReads
           
(package private)  LongStat nRandomWriteBytes
           
(package private)  LongStat nRandomWrites
           
(package private)  LongStat nReadsFromWriteQueue
           
(package private)  LongStat nSequentialReadBytes
           
(package private)  LongStat nSequentialReads
           
(package private)  LongStat nSequentialWriteBytes
           
(package private)  LongStat nSequentialWrites
           
(package private)  LongStat nWriteQueueOverflow
           
(package private)  LongStat nWriteQueueOverflowFailures
           
(package private)  LongStat nWritesFromWriteQueue
           
(package private)  StatGroup stats
           
static long STOP_ON_WRITE_COUNT
           
static boolean THROW_ON_WRITE
           
(package private) static boolean THROW_RRE_FOR_UNIT_TESTS
           
static String TMP_SUFFIX
           
 boolean VERIFY_CHECKSUMS
           
static long WRITE_COUNT
           
 
Constructor Summary
FileManager(EnvironmentImpl envImpl, File dbEnvHome, boolean readOnly)
          Set up the file cache and initialize the file manager to point to the beginning of the log.
 
Method Summary
(package private)  boolean bumpLsn(long size)
          Increase the current log position by "size" bytes.
 boolean checkEnvHomePermissions(boolean rdOnly)
          Ensure that if the environment home dir is on readonly media or in a readonly directory that the environment has been opened for readonly access.
 void clear()
          Close all file handles and empty the cache.
 void close()
          Clear the file lock.
static boolean continueAfterWriteException()
          Returns whether we are in a test mode where we attempt to continue after a write IOException.
 boolean deleteFile(long fileNum)
          Delete log file NNNNNNNN.
 boolean filesExist()
           
static int firstLogEntryOffset()
           
 void force(Set<Long> fileNums)
          Fsync this set of log files.
(package private)  void forceNewLogFile()
          Set the flag that causes a new file to be written before the next write.
 File[] gatherDataDirs()
           
 Long[] getAllFileNumbers()
          Get all JE file numbers.
(package private)  Set<Long> getCacheKeys()
           
 long getCurrentFileNum()
          Returns the highest (current) file number.
 FileHandle getFileHandle(long fileNum)
          Return a read only file handle that corresponds to this file number.
(package private)  long getFileHeaderPrevOffset(long fileNum)
           
 int getFileLogVersion(long fileNum)
          Returns the log version for the given file.
static String getFileName(long fileNum, String suffix)
           
 Long getFirstFileNum()
          public for cleaner.
 Long getFollowingFileNum(long currentFileNum1, boolean forward)
          Get the next file number before/after currentFileNum.
 String getFullFileName(long fileNum, String suffix)
           
 String getFullFileName(String fileName)
           
(package private)  String[] getFullFileNames(long fileNum)
           
 Long getLastFileNum()
           
 long getLastUsedLsn()
          Return the last allocated LSN in the log.
 long getNextLsn()
          Return the next available LSN in the log.
 long getNFSyncRequests()
           
 long getNFSyncs()
           
 long getNFSyncTimeouts()
           
 Long getNumFromName(String fileName)
          Get the file number from a file name.
(package private)  long getPrevEntryOffset()
           
 boolean getReadOnly()
           
(package private)  boolean getUseWriteQueue()
          For unit tests.
(package private)  void groupSync()
          Flush a file using the group sync mechanism, trying to amortize off other syncs.
 boolean hasQueuedWrites()
          Returns whether anything is in the write queue.
 boolean isFileValid(long fileNum)
          For assertions that check whether a file is valid or has been deleted via log cleaning.
 String[] listFileNames(long minFileNumber, long maxFileNumber)
          Find .jdb files which are >= the minimimum file number and <= the maximum file number.
(package private)  String[] listFileNames(String[] suffixes)
          Find JE files.
static String[] listFiles(File envDirFile, String[] suffixes, boolean envMultiSubDir)
          Find JE files, flavor for unit test support.
 File[] listJDBFiles()
           
 File[] listJDBFilesInternalMultiDir(JEFileFilter fileFilter)
           
 File[] listJDBFilesInternalSingleDir(JEFileFilter fileFilter)
           
(package private)  StatGroup loadStats(StatsConfig config)
           
 boolean lockEnvironment(boolean rdOnly, boolean exclusive)
          Lock the environment.
 void makeInvisible(long fileNum, List<Long> lsns)
          Mark the specified log entries as invisible and obsolete.
(package private)  void readFromFile(RandomAccessFile file, ByteBuffer readBuffer, long offset, long fileNo)
          Read a buffer from a file at a given offset.
(package private)  boolean readFromFile(RandomAccessFile file, ByteBuffer readBuffer, long offset, long fileNo, boolean dataKnownToBeInFile)
          Read a buffer from a file at a given offset.
 void releaseExclusiveLock()
           
 boolean renameFile(long fileNum, String newSuffix)
          Rename this file to NNNNNNNN.suffix.
(package private)  void restoreLastPosition()
           
(package private)  void saveLastPosition()
           
 void setIncludeDeletedFiles(boolean includeDeletedFiles)
           
 void setLastPosition(long nextAvailableLsn, long lastUsedLsn, long prevOffset)
          Set the file manager's "end of log".
 void setSyncAtFileEnd(boolean sync)
          May be used to disable sync at file end to speed unit tests.
(package private)  void syncLogEnd()
          FSync the end of the log.
(package private)  void syncLogEndAndFinishFile()
          Sync the end of the log, close off this log file.
 void testWriteQueueLock()
          For unit testing only.
 void testWriteQueueUnlock()
          For unit testing only.
 void truncateLog(long fileNum, long offset)
           
 void truncateSingleFile(long fileNum, long offset)
          Truncate a log at this position.
(package private)  void writeLogBuffer(LogBuffer fullBuffer, boolean flushRequired)
          Write out a log buffer to the file.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

IO_EXCEPTION_TESTING_ON_WRITE

static boolean IO_EXCEPTION_TESTING_ON_WRITE

IO_EXCEPTION_TESTING_ON_READ

static boolean IO_EXCEPTION_TESTING_ON_READ

THROW_RRE_FOR_UNIT_TESTS

static boolean THROW_RRE_FOR_UNIT_TESTS

WRITE_COUNT

public static long WRITE_COUNT

STOP_ON_WRITE_COUNT

public static long STOP_ON_WRITE_COUNT

N_BAD_WRITES

public static long N_BAD_WRITES

THROW_ON_WRITE

public static boolean THROW_ON_WRITE

JE_SUFFIX

public static final String JE_SUFFIX
See Also:
Constant Field Values

DEL_SUFFIX

public static final String DEL_SUFFIX
See Also:
Constant Field Values

BAD_SUFFIX

public static final String BAD_SUFFIX
See Also:
Constant Field Values

DEL_SUFFIXES

static final String[] DEL_SUFFIXES

JE_SUFFIXES

static final String[] JE_SUFFIXES

TMP_SUFFIX

public static final String TMP_SUFFIX
See Also:
Constant Field Values

BUP_SUFFIX

public static final String BUP_SUFFIX
See Also:
Constant Field Values

VERIFY_CHECKSUMS

public boolean VERIFY_CHECKSUMS

lastFileNumberTouched

long lastFileNumberTouched

lastFileTouchedOffset

long lastFileTouchedOffset

stats

final StatGroup stats

nRandomReads

final LongStat nRandomReads

nRandomWrites

final LongStat nRandomWrites

nSequentialReads

final LongStat nSequentialReads

nSequentialWrites

final LongStat nSequentialWrites

nRandomReadBytes

final LongStat nRandomReadBytes

nRandomWriteBytes

final LongStat nRandomWriteBytes

nSequentialReadBytes

final LongStat nSequentialReadBytes

nSequentialWriteBytes

final LongStat nSequentialWriteBytes

nFileOpens

final IntStat nFileOpens

nOpenFiles

final IntStat nOpenFiles

nBytesReadFromWriteQueue

final LongStat nBytesReadFromWriteQueue

nBytesWrittenFromWriteQueue

final LongStat nBytesWrittenFromWriteQueue

nReadsFromWriteQueue

final LongStat nReadsFromWriteQueue

nWritesFromWriteQueue

final LongStat nWritesFromWriteQueue

nWriteQueueOverflow

final LongStat nWriteQueueOverflow

nWriteQueueOverflowFailures

final LongStat nWriteQueueOverflowFailures

nLogFSyncs

final LongStat nLogFSyncs

nFSyncTime

final LongStat nFSyncTime

LOGWRITE_EXCEPTION_TESTING

public static final boolean LOGWRITE_EXCEPTION_TESTING
Constructor Detail

FileManager

public FileManager(EnvironmentImpl envImpl,
                   File dbEnvHome,
                   boolean readOnly)
            throws EnvironmentLockedException
Set up the file cache and initialize the file manager to point to the beginning of the log.

Parameters:
configManager -
dbEnvHome - environment home directory
Throws:
IllegalArgumentException - via Environment ctor
EnvironmentLockedException - via Environment ctor
Method Detail

continueAfterWriteException

public static boolean continueAfterWriteException()
Returns whether we are in a test mode where we attempt to continue after a write IOException. This method will never return true in production use.


setLastPosition

public void setLastPosition(long nextAvailableLsn,
                            long lastUsedLsn,
                            long prevOffset)
Set the file manager's "end of log".

Parameters:
nextAvailableLsn - LSN to be used for the next log entry
lastUsedLsn - last LSN to have a valid entry, may be null
prevOffset - value to use for the prevOffset of the next entry. If the beginning of the file, this is 0.

saveLastPosition

void saveLastPosition()

restoreLastPosition

void restoreLastPosition()

setSyncAtFileEnd

public void setSyncAtFileEnd(boolean sync)
May be used to disable sync at file end to speed unit tests. Must only be used for unit testing, since log corruption may result.


getFirstFileNum

public Long getFirstFileNum()
public for cleaner.

Returns:
the number of the first file in this environment.

getReadOnly

public boolean getReadOnly()

getLastFileNum

public Long getLastFileNum()
Returns:
the number of the last file in this environment.

getCurrentFileNum

public long getCurrentFileNum()
Returns the highest (current) file number. Because a long value cannot be read atomically without synchronization, this method should be called while holding the log write latch.


getUseWriteQueue

boolean getUseWriteQueue()
For unit tests.


isFileValid

public boolean isFileValid(long fileNum)
For assertions that check whether a file is valid or has been deleted via log cleaning.


setIncludeDeletedFiles

public void setIncludeDeletedFiles(boolean includeDeletedFiles)

getAllFileNumbers

public Long[] getAllFileNumbers()
Get all JE file numbers.

Returns:
an array of all JE file numbers.

getFollowingFileNum

public Long getFollowingFileNum(long currentFileNum1,
                                boolean forward)
Get the next file number before/after currentFileNum.

Parameters:
currentFileNum1 - the file we're at right now. Note that it may not exist, if it's been cleaned and renamed.
forward - if true, we want the next larger file, if false we want the previous file
Returns:
null if there is no following file, or if filenum doesn't exist

filesExist

public boolean filesExist()
Returns:
true if there are any files at all.

getNumFromName

public Long getNumFromName(String fileName)
Get the file number from a file name.

Parameters:
the - file name
Returns:
the file number

listFileNames

String[] listFileNames(String[] suffixes)
Find JE files. Return names sorted in ascending fashion.

Parameters:
suffix - which type of file we're looking for
Returns:
array of file names Used by unit tests so package protection.

listFileNames

public String[] listFileNames(long minFileNumber,
                              long maxFileNumber)
Find .jdb files which are >= the minimimum file number and <= the maximum file number. Return names sorted in ascending fashion.

Returns:
array of file names

listFiles

public static String[] listFiles(File envDirFile,
                                 String[] suffixes,
                                 boolean envMultiSubDir)
Find JE files, flavor for unit test support.

Parameters:
suffix - which type of file we're looking for
Returns:
array of file names

listJDBFiles

public File[] listJDBFiles()

listJDBFilesInternalSingleDir

public File[] listJDBFilesInternalSingleDir(JEFileFilter fileFilter)

listJDBFilesInternalMultiDir

public File[] listJDBFilesInternalMultiDir(JEFileFilter fileFilter)

gatherDataDirs

public File[] gatherDataDirs()

getFullFileNames

String[] getFullFileNames(long fileNum)
Returns:
the full file name and path for the nth JE file.

getFullFileName

public String getFullFileName(long fileNum,
                              String suffix)
Returns:
the full file name and path for this file name.

getFullFileName

public String getFullFileName(String fileName)

getFileName

public static String getFileName(long fileNum,
                                 String suffix)
Returns:
the file name for the nth file.

renameFile

public boolean renameFile(long fileNum,
                          String newSuffix)
                   throws IOException,
                          DatabaseException
Rename this file to NNNNNNNN.suffix. If that file already exists, try NNNNNNNN.suffix.1, etc. Used for deleting files or moving corrupt files aside.

Parameters:
fileNum - the file we want to move
newSuffix - the new file suffix
Returns:
true if successful, false if File.renameTo returns false, which can occur on Windows if the file was recently closed.
Throws:
IOException
DatabaseException

deleteFile

public boolean deleteFile(long fileNum)
                   throws IOException,
                          DatabaseException
Delete log file NNNNNNNN.

Parameters:
fileNum - the file we want to move
Returns:
true if successful, false if File.delete returns false, which can occur on Windows if the file was recently closed.
Throws:
IOException
DatabaseException

getFileLogVersion

public int getFileLogVersion(long fileNum)
                      throws DatabaseException
Returns the log version for the given file.

Throws:
DatabaseException

getFileHandle

public FileHandle getFileHandle(long fileNum)
                         throws FileNotFoundException,
                                ChecksumException,
                                DatabaseException
Return a read only file handle that corresponds to this file number. Retrieve it from the cache or open it anew and validate the file header. This method takes a latch on this file, so that the file descriptor will be held in the cache as long as it's in use. When the user is done with the file, the latch must be released.

Parameters:
fileNum - which file
Returns:
the file handle for the existing or newly created file
Throws:
FileNotFoundException
ChecksumException
DatabaseException

getFileHeaderPrevOffset

long getFileHeaderPrevOffset(long fileNum)
                       throws ChecksumException,
                              DatabaseException
Returns:
the prevOffset field stored in the file header.
Throws:
ChecksumException
DatabaseException

getPrevEntryOffset

long getPrevEntryOffset()
Returns:
the file offset of the last LSN that was used. For constructing the headers of log entries. If the last LSN that was used was in a previous file, or this is the very first LSN of the whole system, return 0.

bumpLsn

boolean bumpLsn(long size)
Increase the current log position by "size" bytes. Move the prevOffset pointer along.

Parameters:
size - is an unsigned int
Returns:
true if we flipped to the next log file.

writeLogBuffer

void writeLogBuffer(LogBuffer fullBuffer,
                    boolean flushRequired)
              throws DatabaseException
Write out a log buffer to the file.

Parameters:
fullBuffer - buffer to write
flushRequired - true if this write can not be queued on the Write Queue.
Throws:
DatabaseException

readFromFile

void readFromFile(RandomAccessFile file,
                  ByteBuffer readBuffer,
                  long offset,
                  long fileNo)
            throws DatabaseException
Read a buffer from a file at a given offset. We know that the desired data exists in this file. There's no need to incur extra costs such as checks of the file length, nor to return status as to whether this file contains the data.

Throws:
DatabaseException

readFromFile

boolean readFromFile(RandomAccessFile file,
                     ByteBuffer readBuffer,
                     long offset,
                     long fileNo,
                     boolean dataKnownToBeInFile)
               throws DatabaseException
Read a buffer from a file at a given offset.

Returns:
true if the read buffer is filled, false, if there is nothing left in the file to read
Throws:
DatabaseException

syncLogEnd

void syncLogEnd()
          throws DatabaseException
FSync the end of the log.

Throws:
DatabaseException

syncLogEndAndFinishFile

void syncLogEndAndFinishFile()
                       throws DatabaseException,
                              IOException
Sync the end of the log, close off this log file. Should only be called under the log write latch.

Throws:
DatabaseException
IOException

hasQueuedWrites

public boolean hasQueuedWrites()
Returns whether anything is in the write queue.


testWriteQueueLock

public void testWriteQueueLock()
For unit testing only.


testWriteQueueUnlock

public void testWriteQueueUnlock()
For unit testing only.


groupSync

void groupSync()
         throws DatabaseException
Flush a file using the group sync mechanism, trying to amortize off other syncs.

Throws:
DatabaseException

clear

public void clear()
           throws IOException,
                  DatabaseException
Close all file handles and empty the cache.

Throws:
IOException
DatabaseException

close

public void close()
           throws IOException
Clear the file lock.

Throws:
IOException

lockEnvironment

public boolean lockEnvironment(boolean rdOnly,
                               boolean exclusive)
Lock the environment. Return true if the lock was acquired. If exclusive is false, then this implements a single writer, multiple reader lock. If exclusive is true, then implement an exclusive lock. There is a lock file and there are two regions of the lock file: byte 0, and byte 1. Byte 0 is the exclusive writer process area of the lock file. If an environment is opened for write, then it attempts to take an exclusive write lock on byte 0. Byte 1 is the shared reader process area of the lock file. If an environment is opened for read-only, then it attempts to take a shared lock on byte 1. This is how we implement single writer, multi reader semantics. The cleaner, each time it is invoked, attempts to take an exclusive lock on byte 1. The owning process already either has an exclusive lock on byte 0, or a shared lock on byte 1. This will necessarily conflict with any shared locks on byte 1, even if it's in the same process and there are no other holders of that shared lock. So if there is only one read-only process, it will have byte 1 for shared access, and the cleaner can not run in it because it will attempt to get an exclusive lock on byte 1 (which is already locked for shared access by itself). If a write process comes along and tries to run the cleaner, it will attempt to get an exclusive lock on byte 1. If there are no other reader processes (with shared locks on byte 1), and no other writers (which are running cleaners on with exclusive locks on byte 1), then the cleaner will run.


releaseExclusiveLock

public void releaseExclusiveLock()
                          throws DatabaseException
Throws:
DatabaseException

checkEnvHomePermissions

public boolean checkEnvHomePermissions(boolean rdOnly)
                                throws DatabaseException
Ensure that if the environment home dir is on readonly media or in a readonly directory that the environment has been opened for readonly access.

Returns:
true if the environment home dir is readonly.
Throws:
IllegalArgumentException - via Environment ctor
DatabaseException

truncateSingleFile

public void truncateSingleFile(long fileNum,
                               long offset)
                        throws IOException,
                               DatabaseException
Truncate a log at this position. Used by recovery to a timestamp utilities and by recovery to set the end-of-log position, see LastFileReader.setEndOfFile().

This method forces a new log file to be written next, if the last file (the file truncated to) has an old version in its header. This ensures that when the log is opened by an old version of JE, a version incompatibility will be detected. [#11243]

Throws:
IOException
DatabaseException

truncateLog

public void truncateLog(long fileNum,
                        long offset)
                 throws IOException,
                        DatabaseException
Throws:
IOException
DatabaseException

makeInvisible

public void makeInvisible(long fileNum,
                          List<Long> lsns)
Mark the specified log entries as invisible and obsolete. The entries are written here, but are fsync'ed later. If there is any problem or exception during the setting, the method will throw an EnvironmentFailureException. These changes are made directly to the file, but recently logged log entries may also be resident in the log buffers. The caller must take care to call LogManager.flush() before this method, to ensure that all entries are on disk. In addition, we must ensure that after this step, the affected log entries will only be read via a FileReader, and will not be faulted in by the LogManager. Entries may be present in the log and in the log buffers, but only the on disk version is modified by this method. The LogManager can read directly from the log buffers and may read the incorrect, non-invisible version of the log entry, rather than the invisible version from the file. This should not be an issue, because invisible log entries should be detached from the in-memory tree before they are made invisible.

Parameters:
fileNum - target file.
lsns - The list of LSNs to make invisible, must be sorted in ascending order.

force

public void force(Set<Long> fileNums)
Fsync this set of log files. Used for replication syncup rollback.


forceNewLogFile

void forceNewLogFile()
Set the flag that causes a new file to be written before the next write.


firstLogEntryOffset

public static int firstLogEntryOffset()
Returns:
the size in bytes of the file header log entry.

getNextLsn

public long getNextLsn()
Return the next available LSN in the log. Note that this is unsynchronized, so is only valid as an approximation of log size.


getLastUsedLsn

public long getLastUsedLsn()
Return the last allocated LSN in the log. Note that this is unsynchronized, so if it is called outside the log write latch it is only valid as an approximation of log size.


getNFSyncs

public long getNFSyncs()

getNFSyncRequests

public long getNFSyncRequests()

getNFSyncTimeouts

public long getNFSyncTimeouts()

loadStats

StatGroup loadStats(StatsConfig config)

getCacheKeys

Set<Long> getCacheKeys()


Copyright (c) 2004-2012 Oracle. All rights reserved.