@InterfaceAudience.Public @InterfaceStability.Stable public abstract class MultipleOutputFormat<K,V> extends FileOutputFormat<K,V>
| Constructor and Description |
|---|
MultipleOutputFormat() |
| Modifier and Type | Method and Description |
|---|---|
protected K |
generateActualKey(K key,
V value)
Generate the actual key from the given key/value.
|
protected V |
generateActualValue(K key,
V value)
Generate the actual value from the given key and value.
|
protected String |
generateFileNameForKeyValue(K key,
V value,
String name)
Generate the file output file name based on the given key and the leaf file
name.
|
protected String |
generateLeafFileName(String name)
Generate the leaf name for the output file name.
|
protected abstract RecordWriter<K,V> |
getBaseRecordWriter(FileSystem fs,
JobConf job,
String name,
Progressable arg3) |
protected String |
getInputFileBasedOutputFileName(JobConf job,
String name)
Generate the outfile name based on a given name and the input file name.
|
RecordWriter<K,V> |
getRecordWriter(FileSystem fs,
JobConf job,
String name,
Progressable arg3)
Create a composite record writer that can write key/value data to different
output files
|
checkOutputSpecs, getCompressOutput, getOutputCompressorClass, getOutputPath, getPathForCustomFile, getTaskOutputPath, getUniqueName, getWorkOutputPath, setCompressOutput, setOutputCompressorClass, setOutputPathpublic RecordWriter<K,V> getRecordWriter(FileSystem fs, JobConf job, String name, Progressable arg3) throws IOException
getRecordWriter in interface OutputFormat<K,V>getRecordWriter in class FileOutputFormat<K,V>fs - the file system to usejob - the job conf for the jobname - the leaf file name for the output file (such as part-00000")arg3 - a progressable for reporting progress.IOExceptionprotected String generateLeafFileName(String name)
name - the leaf file name for the output fileprotected String generateFileNameForKeyValue(K key, V value, String name)
key - the key of the output dataname - the leaf file nameprotected K generateActualKey(K key, V value)
key - the key of the output datavalue - the value of the output dataprotected V generateActualValue(K key, V value)
key - the key of the output datavalue - the value of the output dataprotected String getInputFileBasedOutputFileName(JobConf job, String name)
MRJobConfig.MAP_INPUT_FILE does not exists (i.e. this is not for a map only job),
the given name is returned unchanged. If the config value for
"num.of.trailing.legs.to.use" is not set, or set 0 or negative, the given
name is returned unchanged. Otherwise, return a file name consisting of the
N trailing legs of the input file name where N is the config value for
"num.of.trailing.legs.to.use".job - the job configname - the output file nameprotected abstract RecordWriter<K,V> getBaseRecordWriter(FileSystem fs, JobConf job, String name, Progressable arg3) throws IOException
fs - the file system to usejob - a job conf objectname - the name of the file over which a record writer object will be
constructedarg3 - a progressable objectIOExceptionCopyright © 2024 Apache Software Foundation. All rights reserved.