Customizing Yarn container

Question

I'm testing spring-yarn integration API and I'm little confused about what is the best practice of Yarn container customization in terms of:

1) If I want to use spring-boot-yarn combo, what is the correct way of telling the spring boot to pick up my implementation of yarn container instead of DefaultYarnContainer...The only way I figured out was via ImportResource annotation at container project class containing main method, which was pointing to spring application xml with declaration:

<yarn:container container class="myhadoop.yarn.container.custom.MyContainerImplementation"/>

Component scan doesn't work at all...Spring boot was still using DefaultYarnContainer...

2) If I understand Yarn architecture correctly then application master is responsible for launching the container. But If I change DefaultYarnContainer for my implementation then I need to start container manually via run method, nothing was starting it, please what is the correct way?

Thanks a lot in advance for help

Answer 1

You don't need to implement all beans for container, just one is enough.

You're right. I didn't notice of the following method in SpringYarnConfig:

@Override
    public void configure(YarnContainerConfigurer container) throws Exception {
        if (StringUtils.hasText(sycp.getContainerClass())) {
            container
                .containerClass(sycp.getContainerClass());
        } else if (yarnContainerClass != null){
            container
                .containerClass(yarnContainerClass);
        } else if (yarnContainerRef != null) {
            if (yarnContainerRef instanceof YarnContainer) {
                container
                    .containerRef((YarnContainer) yarnContainerRef);
            }
        } else if (StringUtils.hasText(containerClass)) {
            container.containerClass(containerClass);
        }
    }

where it's clear that one is really enough...:)

Use of event publisher and state listener are really meant as building blocks what you'd need to call yourself.

Yes, I see that DefaultYarnContainer is handling notifyXXXX methods(fires up container state changes) calling by himself...Alright, I will definitelly play with it more.

Janne, thanks a lot for help. You provided perfect closer insight into Spring-Yarn.

Answer 2

Janne, thanks a lot! This way is far more elegant and it works...Here is what I did:

@EnableAutoConfiguration
@Configuration
@ComponentScan
public class ContainerApplication {

@Autowired
private MyContainerImplementation myContainerImplementation;

@Bean(name="yarnContainerClass")
public Class<? extends YarnContainer> getYarnContainerClass() {
    return MyContainerImplementation.class;
}

@Bean(name="yarnContainerRef")
public MyContainerImplementation getYarnContainerRef() {
    return myContainerImplementation;
}

@Bean(name="customContainerClass")
public String getCustomContainerClass() {
    return "myhadoop.yarn.container.custom.MyContainerImplementation";
}

public static void main(String[] args) {
    SpringApplication.run(ContainerApplication.class, args);
}

}

I added MyContainerImplementation into yml as you pointed and my container implementation was started by application master without me running the run method manually, because I see the following lines in the hadoop logs:

LifecycleObjectSupport:started
myhadoop.yarn.container.custom.MyContainerImplementation@5e2cd950
.
.
LifecycleObjectSupport: stopped
myhadoop.yarn.container.custom.MyContainerImplementation@5e2cd950

Anyway, I have additional question. I wanted to test low-level yarn things like ContainerStateListener and YarnPublisher, but they're not called at all..:-( Here is my test customized container:

@Component
public class MyContainerImplementation extends AbstractYarnContainer {
private static final Log log =    LogFactory.getLog(MyContainerImplementation.class);

public MyContainerImplementation() {
    super();

    log.info("...Initializing yarn MyContainerImplementation....");

    this.setYarnEventPublisher(new DefaultYarnEventPublisher() {
        @Override
        public void publishContainerAllocated(Object source, Container container) {
            super.publishContainerAllocated(source, container);
            log.info("Yarn container allocated: "+container.getResource().getMemory());
        }
    });

    this.addContainerStateListener(new ContainerStateListener() {

        @Override
        public void state(ContainerState state, Object exit) {
            switch(state) {
                case COMPLETED: {
                    log.info("...Container started successfully!...");
                    break;
                }
                case FAILED: {
                    log.info("...Starting of container failed!...");
                    break;
                }
                default: {
                    log.info("Unexpected container state...exiting!...");
                }
            }
        }
    });
}

public void runInternal() {
    log.info("...Running internal method...");
}

}

Do I need to add additional configuration to make ContainerStateListener and YarnPublisher to work?

Answer 3

If boot is doing auto-configuration for yarn container, there are few ways to define the actual container which defaults to DefaultYarnContainer.

Logic of this can be found from here https://github.com/spring-projects/spring-hadoop/blob/master/spring-yarn/spring-yarn-boot/src/main/java/org/springframework/yarn/boot/YarnContainerAutoConfiguration.java#L107

1. Use spring.yarn.container.containerClass=foo.jee.MyContainer in yml
2. Create class as bean with name yarnContainerClass
3. Create your container impl as bean with name yarnContainerRef
4. Create bean as name customContainerClass which would be a class as string