Java Code Examples for org.apache.hadoop.hbase.KeyValue#ROW_LENGTH_SIZE

The following examples show how to use org.apache.hadoop.hbase.KeyValue#ROW_LENGTH_SIZE . You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example 1
Source File: TableReporter.java    From hbase-operator-tools with Apache License 2.0 5 votes vote down vote up
/**
 * Estimate based on keyvalue's serialization format in the RPC layer. Note that there is an extra
 * SIZEOF_INT added to the size here that indicates the actual length of the cell for cases where
 * cell's are serialized in a contiguous format (For eg in RPCs).
 * @return Estimate of the <code>cell</code> size in bytes plus an extra SIZEOF_INT indicating the
 *         actual cell length.
 */
public static int estimatedSerializedSizeOf(final Cell cell) {
  if (cell instanceof ExtendedCell) {
    return ((ExtendedCell) cell).getSerializedSize(true) + Bytes.SIZEOF_INT;
  }

  return getSumOfCellElementLengths(cell) +
      // Use the KeyValue's infrastructure size presuming that another implementation would have
      // same basic cost.
      KeyValue.ROW_LENGTH_SIZE + KeyValue.FAMILY_LENGTH_SIZE +
      // Serialization is probably preceded by a length (it is in the KeyValueCodec at least).
      Bytes.SIZEOF_INT;
}
 
Example 2
Source File: RowColBloomHashKey.java    From hbase with Apache License 2.0 5 votes vote down vote up
@Override
public int length() {
  // For ROW_COL blooms we use bytes
  // <RK length> (2 bytes) , <RK>, 0 (one byte CF length), <CQ>, <TS> (8 btes), <TYPE> ( 1 byte)
  return KeyValue.ROW_LENGTH_SIZE + this.t.getRowLength() + KeyValue.FAMILY_LENGTH_SIZE
      + this.t.getQualifierLength() + KeyValue.TIMESTAMP_TYPE_SIZE;
}
 
Example 3
Source File: CompressionState.java    From hbase with Apache License 2.0 5 votes vote down vote up
/** 
 * Analyze the key and fill the state assuming we know previous state.
 * Uses mark() and reset() in ByteBuffer to avoid moving the position.
 * <p>
 * This method overrides all the fields of this instance, except
 * {@link #prevOffset}, which is usually manipulated directly by encoders
 * and decoders.
 * @param in Buffer at the position where key starts
 * @param keyLength Length of key in bytes
 * @param valueLength Length of values in bytes
 * @param commonPrefix how many first bytes are common with previous KeyValue
 * @param previousState State from previous KeyValue
 */
void readKey(ByteBuffer in, int keyLength, int valueLength,
    int commonPrefix, CompressionState previousState) {
  this.keyLength = keyLength;
  this.valueLength = valueLength;

  // fill the state
  in.mark(); // mark beginning of key

  if (commonPrefix < KeyValue.ROW_LENGTH_SIZE) {
    rowLength = in.getShort();
    ByteBufferUtils.skip(in, rowLength);

    familyLength = in.get();

    qualifierLength = keyLength - rowLength - familyLength -
        KeyValue.KEY_INFRASTRUCTURE_SIZE;
    ByteBufferUtils.skip(in, familyLength + qualifierLength);
  } else {
    rowLength = previousState.rowLength;
    familyLength = previousState.familyLength;
    qualifierLength = previousState.qualifierLength +
        keyLength - previousState.keyLength;
    ByteBufferUtils.skip(in, (KeyValue.ROW_LENGTH_SIZE +
        KeyValue.FAMILY_LENGTH_SIZE) +
        rowLength + familyLength + qualifierLength);
  }

  readTimestamp(in);

  type = in.get();

  in.reset();
}
 
Example 4
Source File: DiffKeyDeltaEncoder.java    From hbase with Apache License 2.0 4 votes vote down vote up
private int compressSingleKeyValue(DataOutputStream out, Cell cell, Cell prevCell)
    throws IOException {
  int flag = 0; // Do not use more bits that can fit into a byte
  int kLength = KeyValueUtil.keyLength(cell);
  int vLength = cell.getValueLength();

  long timestamp;
  long diffTimestamp = 0;
  int diffTimestampFitsInBytes = 0;
  int timestampFitsInBytes;
  int commonPrefix = 0;

  if (prevCell == null) {
    timestamp = cell.getTimestamp();
    if (timestamp < 0) {
      flag |= FLAG_TIMESTAMP_SIGN;
      timestamp = -timestamp;
    }
    timestampFitsInBytes = ByteBufferUtils.longFitsIn(timestamp);
    flag |= (timestampFitsInBytes - 1) << SHIFT_TIMESTAMP_LENGTH;
    // put column family
    byte familyLength = cell.getFamilyLength();
    out.write(familyLength);
    PrivateCellUtil.writeFamily(out, cell, familyLength);
  } else {
    // Finding common prefix
    int preKeyLength = KeyValueUtil.keyLength(prevCell);
    commonPrefix = PrivateCellUtil.findCommonPrefixInFlatKey(cell, prevCell, true, false);
    if (kLength == preKeyLength) {
      flag |= FLAG_SAME_KEY_LENGTH;
    }
    if (vLength == prevCell.getValueLength()) {
      flag |= FLAG_SAME_VALUE_LENGTH;
    }
    if (cell.getTypeByte() == prevCell.getTypeByte()) {
      flag |= FLAG_SAME_TYPE;
    }
    // don't compress timestamp and type using prefix encode timestamp
    timestamp = cell.getTimestamp();
    diffTimestamp = prevCell.getTimestamp() - timestamp;
    boolean negativeTimestamp = timestamp < 0;
    if (negativeTimestamp) {
      timestamp = -timestamp;
    }
    timestampFitsInBytes = ByteBufferUtils.longFitsIn(timestamp);
    boolean minusDiffTimestamp = diffTimestamp < 0;
    if (minusDiffTimestamp) {
      diffTimestamp = -diffTimestamp;
    }
    diffTimestampFitsInBytes = ByteBufferUtils.longFitsIn(diffTimestamp);
    if (diffTimestampFitsInBytes < timestampFitsInBytes) {
      flag |= (diffTimestampFitsInBytes - 1) << SHIFT_TIMESTAMP_LENGTH;
      flag |= FLAG_TIMESTAMP_IS_DIFF;
      if (minusDiffTimestamp) {
        flag |= FLAG_TIMESTAMP_SIGN;
      }
    } else {
      flag |= (timestampFitsInBytes - 1) << SHIFT_TIMESTAMP_LENGTH;
      if (negativeTimestamp) {
        flag |= FLAG_TIMESTAMP_SIGN;
      }
    }
  }
  out.write(flag);
  if ((flag & FLAG_SAME_KEY_LENGTH) == 0) {
    ByteBufferUtils.putCompressedInt(out, kLength);
  }
  if ((flag & FLAG_SAME_VALUE_LENGTH) == 0) {
    ByteBufferUtils.putCompressedInt(out, vLength);
  }
  ByteBufferUtils.putCompressedInt(out, commonPrefix);
  short rLen = cell.getRowLength();
  if (commonPrefix < rLen + KeyValue.ROW_LENGTH_SIZE) {
    // Previous and current rows are different. Copy the differing part of
    // the row, skip the column family, and copy the qualifier.
    PrivateCellUtil.writeRowKeyExcludingCommon(cell, rLen, commonPrefix, out);
    PrivateCellUtil.writeQualifier(out, cell, cell.getQualifierLength());
  } else {
    // The common part includes the whole row. As the column family is the
    // same across the whole file, it will automatically be included in the
    // common prefix, so we need not special-case it here.
    // What we write here is the non common part of the qualifier
    int commonQualPrefix = commonPrefix - (rLen + KeyValue.ROW_LENGTH_SIZE)
        - (cell.getFamilyLength() + KeyValue.FAMILY_LENGTH_SIZE);
    PrivateCellUtil.writeQualifierSkippingBytes(out, cell, cell.getQualifierLength(),
      commonQualPrefix);
  }
  if ((flag & FLAG_TIMESTAMP_IS_DIFF) == 0) {
    ByteBufferUtils.putLong(out, timestamp, timestampFitsInBytes);
  } else {
    ByteBufferUtils.putLong(out, diffTimestamp, diffTimestampFitsInBytes);
  }

  if ((flag & FLAG_SAME_TYPE) == 0) {
    out.write(cell.getTypeByte());
  }
  PrivateCellUtil.writeValue(out, cell, vLength);
  return kLength + vLength + KeyValue.KEYVALUE_INFRASTRUCTURE_SIZE;
}
 
Example 5
Source File: FastDiffDeltaEncoder.java    From hbase with Apache License 2.0 4 votes vote down vote up
private int compressSingleKeyValue(DataOutputStream out, Cell cell, Cell prevCell)
    throws IOException {
  int flag = 0; // Do not use more bits than will fit into a byte
  int kLength = KeyValueUtil.keyLength(cell);
  int vLength = cell.getValueLength();

  if (prevCell == null) {
    // copy the key, there is no common prefix with none
    out.write(flag);
    ByteBufferUtils.putCompressedInt(out, kLength);
    ByteBufferUtils.putCompressedInt(out, vLength);
    ByteBufferUtils.putCompressedInt(out, 0);
    PrivateCellUtil.writeFlatKey(cell, (DataOutput)out);
    // Write the value part
    PrivateCellUtil.writeValue(out, cell, cell.getValueLength());
  } else {
    int preKeyLength = KeyValueUtil.keyLength(prevCell);
    int preValLength = prevCell.getValueLength();
    // find a common prefix and skip it
    int commonPrefix = PrivateCellUtil.findCommonPrefixInFlatKey(cell, prevCell, true, false);

    if (kLength == preKeyLength) {
      flag |= FLAG_SAME_KEY_LENGTH;
    }
    if (vLength == prevCell.getValueLength()) {
      flag |= FLAG_SAME_VALUE_LENGTH;
    }
    if (cell.getTypeByte() == prevCell.getTypeByte()) {
      flag |= FLAG_SAME_TYPE;
    }

    byte[] curTsBuf = Bytes.toBytes(cell.getTimestamp());
    int commonTimestampPrefix = findCommonTimestampPrefix(curTsBuf,
        Bytes.toBytes(prevCell.getTimestamp()));

    flag |= commonTimestampPrefix << SHIFT_TIMESTAMP_LENGTH;

    // Check if current and previous values are the same. Compare value
    // length first as an optimization.
    if (vLength == preValLength
        && PrivateCellUtil.matchingValue(cell, prevCell, vLength, preValLength)) {
      flag |= FLAG_SAME_VALUE;
    }

    out.write(flag);
    if ((flag & FLAG_SAME_KEY_LENGTH) == 0) {
      ByteBufferUtils.putCompressedInt(out, kLength);
    }
    if ((flag & FLAG_SAME_VALUE_LENGTH) == 0) {
      ByteBufferUtils.putCompressedInt(out, vLength);
    }
    ByteBufferUtils.putCompressedInt(out, commonPrefix);
    short rLen = cell.getRowLength();
    if (commonPrefix < rLen + KeyValue.ROW_LENGTH_SIZE) {
      // Previous and current rows are different. Copy the differing part of
      // the row, skip the column family, and copy the qualifier.
      PrivateCellUtil.writeRowKeyExcludingCommon(cell, rLen, commonPrefix, out);
      PrivateCellUtil.writeQualifier(out, cell, cell.getQualifierLength());
    } else {
      // The common part includes the whole row. As the column family is the
      // same across the whole file, it will automatically be included in the
      // common prefix, so we need not special-case it here.
      // What we write here is the non common part of the qualifier
      int commonQualPrefix = commonPrefix - (rLen + KeyValue.ROW_LENGTH_SIZE)
          - (cell.getFamilyLength() + KeyValue.FAMILY_LENGTH_SIZE);
      PrivateCellUtil.writeQualifierSkippingBytes(out, cell, cell.getQualifierLength(),
        commonQualPrefix);
    }
    // Write non common ts part
    out.write(curTsBuf, commonTimestampPrefix, KeyValue.TIMESTAMP_SIZE - commonTimestampPrefix);

    // Write the type if it is not the same as before.
    if ((flag & FLAG_SAME_TYPE) == 0) {
      out.write(cell.getTypeByte());
    }

    // Write the value if it is not the same as before.
    if ((flag & FLAG_SAME_VALUE) == 0) {
      PrivateCellUtil.writeValue(out, cell, vLength);
    }
  }
  return kLength + vLength + KeyValue.KEYVALUE_INFRASTRUCTURE_SIZE;
}
 
Example 6
Source File: PrefixKeyDeltaEncoder.java    From hbase with Apache License 2.0 4 votes vote down vote up
private void writeKeyExcludingCommon(Cell cell, int commonPrefix, DataOutputStream out)
    throws IOException {
  short rLen = cell.getRowLength();
  if (commonPrefix < rLen + KeyValue.ROW_LENGTH_SIZE) {
    // Previous and current rows are different. Need to write the differing part followed by
    // cf,q,ts and type
    PrivateCellUtil.writeRowKeyExcludingCommon(cell, rLen, commonPrefix, out);
    byte fLen = cell.getFamilyLength();
    out.writeByte(fLen);
    PrivateCellUtil.writeFamily(out, cell, fLen);
    PrivateCellUtil.writeQualifier(out, cell, cell.getQualifierLength());
    out.writeLong(cell.getTimestamp());
    out.writeByte(cell.getTypeByte());
  } else {
    // The full row key part is common. CF part will be common for sure as we deal with Cells in
    // same family. Just need write the differing part in q, ts and type
    commonPrefix = commonPrefix - (rLen + KeyValue.ROW_LENGTH_SIZE)
        - (cell.getFamilyLength() + KeyValue.FAMILY_LENGTH_SIZE);
    int qLen = cell.getQualifierLength();
    int commonQualPrefix = Math.min(commonPrefix, qLen);
    int qualPartLenToWrite = qLen - commonQualPrefix;
    if (qualPartLenToWrite > 0) {
      PrivateCellUtil.writeQualifierSkippingBytes(out, cell, qLen, commonQualPrefix);
    }
    commonPrefix -= commonQualPrefix;
    // Common part in TS also?
    if (commonPrefix > 0) {
      int commonTimestampPrefix = Math.min(commonPrefix, KeyValue.TIMESTAMP_SIZE);
      if (commonTimestampPrefix < KeyValue.TIMESTAMP_SIZE) {
        byte[] curTsBuf = Bytes.toBytes(cell.getTimestamp());
        out.write(curTsBuf, commonTimestampPrefix, KeyValue.TIMESTAMP_SIZE
            - commonTimestampPrefix);
      }
      commonPrefix -= commonTimestampPrefix;
      if (commonPrefix == 0) {
        out.writeByte(cell.getTypeByte());
      }
    } else {
      out.writeLong(cell.getTimestamp());
      out.writeByte(cell.getTypeByte());
    }
  }
}
 
Example 7
Source File: ClientKeyValue.java    From phoenix with BSD 3-Clause "New" or "Revised" License 4 votes vote down vote up
@Override
public int getLength() {
  return KEYVALUE_INFRASTRUCTURE_SIZE + KeyValue.ROW_LENGTH_SIZE + row.getLength()
      + KeyValue.FAMILY_LENGTH_SIZE + family.getLength() + qualifier.getLength()
      + KeyValue.TIMESTAMP_SIZE + KeyValue.TYPE_SIZE + value.getLength();
}