CODEWITHSUNDEEP
hadoopapache-pighdfs
FuriousGeorge
FuriousGeorge

Reputation: 171

How do I store data in multiple, partitioned files on HDFS using Pig

I've got a pig job that analyzes a large number of log files and generates a relationship between a group of attributes and a bag of IDs that have those attributes. I'd like to store that relationship on HDFS, but I'd like to do so in a way that is friendly for other Hive/Pig/MapReduce jobs to operate on the data, or subsets of the data without having to ingest the full output of my pig job, as that is a significant amount of data.

For example, if the schema of my relationship is something like:

relation: {group: (attr1: long,attr2: chararray,attr3: chararray),ids: {(id: chararray)}}

I'd really like to be able to partition this data, storing it in a file structure that looks like:

/results/attr1/attr2/attr3/file(s)

where the attrX values in the path are the values from the group, and the file(s) contain only ids. This would allow me to easily subset my data for subsequent analysis without duplicating data.

Is such a thing possible, even with a custom StoreFunc? Is there a different approach that I should be taking to accomplish this goal?

I'm pretty new to Pig, so any help or general suggestions about my approach would be greatly appreciated.

Thanks in advance.

Upvotes: 3

Views: 1057

Answers (1)

FuriousGeorge
FuriousGeorge

Reputation: 171

Multistore wasn't a perfect fit for what I was trying to do, but it proved a good example of how to write a custom StoreFunc that writes multiple, partitioned output files. I downloaded the Pig source code and created my own storage function that parsed the group tuple, using each of the items to build up the HDFS path, and then parsed the bag of ids, writing one ID per line into the result file.

Upvotes: 1

Related Questions