com.mongodb.hadoop.util.MongoConfigUtil Java Examples

The following examples show how to use com.mongodb.hadoop.util.MongoConfigUtil. You can vote up the ones you like or vote down the ones you don't like, and go to the original project or source file by following the links above each example. You may check out the related API usage on the sidebar.
Example #1
Source File: SchoolProficiencyMapper.java    From secure-data-service with Apache License 2.0 6 votes vote down vote up
@SuppressWarnings({ "rawtypes", "unchecked" })
@Override
protected void setup(Mapper.Context context) throws IOException, InterruptedException {
    super.setup(context);

    ConfigSections cfg = JobConfiguration.fromHadoopConfiguration(context.getConfiguration());
    MetadataConfig meta = cfg.getMetadata();
    bands = meta.getCutPoints();

    BSONObject obj = MongoConfigUtil.getFields(context.getConfiguration());
    if (obj != null) {
        fields = obj.keySet().toArray(new String[0]);
    } else {
        throw new IllegalArgumentException("Invalid configuration found. Aggregates must "
            + "specify a the hadoop.map.fields property.");
    }
}
 
Example #2
Source File: MongoAggFormatterTest.java    From secure-data-service with Apache License 2.0 6 votes vote down vote up
/**
 * Test method for
 * {@link org.slc.sli.aggregation.mapreduce.io.MongoAggFormatter#getRecordWriter(org.apache.hadoop.mapreduce.TaskAttemptContext)}
 * .
 *
 * @throws Exception
 */
@Test
public void testGetRecordWriter() throws Exception {

    DBCollection mockCollection = Mockito.mock(DBCollection.class);
    UserGroupInformation ugi = Mockito.mock(UserGroupInformation.class);

    PowerMockito.mockStatic(UserGroupInformation.class);

    Mockito.when(UserGroupInformation.getCurrentUser()).thenReturn(ugi);

    TaskAttemptContext c = new MockTaskAttemptContext();
    Configuration config = c.getConfiguration();

    PowerMockito.mockStatic(MongoConfigUtil.class);
    Mockito.when(MongoConfigUtil.getOutputCollection(config)).thenReturn(mockCollection);

    MongoAggFormatter f = new MongoAggFormatter();
    assertTrue(f.getRecordWriter(new MockTaskAttemptContext()) instanceof MongoAggWriter);
}
 
Example #3
Source File: MongoAggWriterTest.java    From secure-data-service with Apache License 2.0 6 votes vote down vote up
public void expect(T expected) throws IOException {
    this.expected = expected;

    mockCollection = Mockito.mock(DBCollection.class);

    ugi = Mockito.mock(UserGroupInformation.class);

    PowerMockito.mockStatic(UserGroupInformation.class);
    Mockito.when(UserGroupInformation.getCurrentUser()).thenReturn(ugi);

    ctx = new MockTaskAttemptContext();
    config = ctx.getConfiguration();

    PowerMockito.mockStatic(MongoConfigUtil.class);
    Mockito.when(MongoConfigUtil.getOutputCollection(config)).thenReturn(mockCollection);

    key = new TenantAndIdEmittableKey("testTenant", "testId");
    key.setTenantId(new Text("Midgar"));
    key.setId(new Text("abcdefg01234567890"));
    writer = new MongoAggWriter(mockCollection, ctx);
}
 
Example #4
Source File: BulkImportJob.java    From zerowing with MIT License 5 votes vote down vote up
private void setupJob() {
  _job.setInputFormatClass(MongoInputFormat.class);
  _job.setMapperClass(BulkImportMapper.class);
  _job.setMapOutputKeyClass(ImmutableBytesWritable.class);
  _job.setMapOutputValueClass(Put.class);

  MongoConfigUtil.setInputURI(getConfiguration(), _mongoURI);
  MongoConfigUtil.setReadSplitsFromSecondary(getConfiguration(), true);
}
 
Example #5
Source File: MongoAggFormatter.java    From secure-data-service with Apache License 2.0 4 votes vote down vote up
@Override
public RecordWriter<EmittableKey, BSONWritable> getRecordWriter(
    org.apache.hadoop.mapreduce.TaskAttemptContext context) {
    Configuration config = context.getConfiguration();
    return new MongoAggWriter(MongoConfigUtil.getOutputCollection(config), context);
}