Invalid template resource property 'Ref'

I am new to writing yaml scripts and I keep getting this error 'Invalid template resource property 'Ref'' when I try to create a stack on Cloudformation. Is there something I am missing in my code. Please do let me know. Thanks! I have written this template to bring up an emr cluster using yaml.

AWSTemplateFormatVersion: 2010-09-09
Description: EMR Cluster for Spark
Metadata:
  AWS::CloudFormation::Interface:
    ParameterGroups:
      - Label:
          default: Common Parameters
        Parameters:
          - SystemValue
          - SubSystemValue
          - RevenueStreamValue
          - EnvironmentValue
          - KMSArn
      - Label:
          default: EMR Parameters
          default: Lambda Parameters
        Parameters:
          - EMRClusterName
          - KeyName
          - MasterInstanceType
          - CoreInstanceType
          - NumberOfCoreInstances
          - VpcId
          - VPCSubnetIds
          - LogUriFolder
          - S3DataUri 
          - ReleaseLabel
          - ApplicationsList
          - PackageBucket
          - EbsRootVolumeSize
          - FunctionName1Value
      - Label:
          default: S3 Parameters
        Parameters: 
          - EDWBucketName  
    ParameterLabels:
      SystemValue:
        default: System
      SubSystemValue:
        default: SubSystem
      RevenueStreamValue:
        default: Revenue Stream
      EnvironmentValue:
        default: Environment Value  
      FunctionName1Value:
        default: Lambda-1 Name
      EMRClusterName:
        default: EMR Cluster Name       
      KeyName:
        default: Key Name
      MasterInstanceType:
        default: Master Instance Type
      CoreInstanceType:
        default: Core Instance Type
      NumberOfCoreInstances:
        default: Number Of Core Instances
      VpcId:
        default: VPC ID 
      VPCSubnetIds:
        default: VPC Subnet ID
      LogUriFolder:
        default: Log Uri Folder
      S3DataUri:
        default: S3 Data Uri
      ReleaseLabel:
        default: Release Label
      ApplicationsList:
        default: Applications List
      KMSArn:
        default: KMS Arn
      EDWBucketName:
        default: EDW Bucket Name
      PackageBucket:
        default: Package Bucket Name
      EbsRootVolumeSize:
        default: Ebs Root Volume Size
#########################################################################
Parameters:
  KMSArn:
    Type: String
    Description: Enter KMS ARN based on your subsystem.
    Default: 'a36ef9be-97e1-4949-9b04-c1979eda5955'  
  SystemValue:
    Type: String
    Description: Enter System Name
    Default: 'Messaging'
  SubSystemValue:
    Type: String
    Description: Enter SubSystem Name
    Default: 'EDW'
  RevenueStreamValue:
    Type: String
    Description: Enter Revenue Stream Name
    Default: 'edw'
  FunctionName1Value:
    Type: String
    Default: 'EMRCluster'
    Description: Enter 1st Lambda Function Name(Do not append System & Sub-System Name).
  EDWBucketName:
    Type: String
    Default: 'crx-dev-messaging-edw'
    Description: Enter crx-[env]-messaging-edw
  PackageBucket:
    Type: String
    Default: 'crx-dev-deployments'
    Description: Enter crx-[env]-deployments  
  EnvironmentValue:
    Type: String
    Default: 'dev'
  LambdaRuntime:
    Type: String
    Default: 'Python-2.7' 
  TagEnvironmentValue:
    Type: String
    Default: 'dev'
  VpcId:
    Type: 'AWS::EC2::VPC::Id'
    Default: vpc-7c368507
  VpcCIDR:
    Type: String
    Default: '10.10.16.0/20'
  VPCSubnetIds:
    Type: AWS::EC2::Subnet::Id
    Default: subnet-4d527c62    
  Env:
    Type: String
    Default: 'dev'    
  KeyName:
    Type: AWS::EC2::KeyPair::KeyName
    Description: Ane<Env>-Messaging-EDW
    Default: development
  MasterInstanceType:
    Type: String
    Description: Instance type to be used for the master instance.
    Default: 'm5.xlarge'
    AllowedValues:
      - m5.xlarge
      - m1.medium
  CoreInstanceType:
    Type: String
    Description: Instance type to be used for core instances.
    Default: 'm5.xlarge'
    AllowedValues:
      - m5.xlarge
      - m1.medium
  NumberOfCoreInstances:
    Description: Must be a valid number - 2
    Type: Number
    Default: '2'
  LogUriFolder:
    Type: String
    Description: S3 Folder name for spark logs (spark-logs)
    Default: 's3n://crx-dev-messaging-edw/spark-logs/'
  S3DataUri: 
    Type: String
    Description: Must be a valid S3 bucket URL
    Default: 's3n://crx-messaging-edw'
  EbsRootVolumeSize:
    Type: String
    Description: Specify the volume size
    Default: '200'
  ReleaseLabel:
    Type: String
    Description: Must be a valid EMR release version emr-6.0.0
    Default: 'emr-6.0.0'
  ApplicationsList:
    Type: String 
    Description: Please select which application will be installed on the cluster
      this would be either Ganglia,spark,Hadoop etc.,
    Default: 'Spark'
    AllowedValues: 
      - Hadoop
      - Ganglia
      - Spark
#########################################################################
Resources:
  Bucket1:
    Type: AWS::S3::Bucket
    Properties:
      BucketName: !Sub 'crx-${EnvironmentValue}-${SubSystemValue}' 
      BucketEncryption: 
        ServerSideEncryptionConfiguration: 
        - ServerSideEncryptionByDefault:
            SSEAlgorithm: aws:kms
            KMSMasterKeyID: !Ref KMSArn 
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true

  EbsRootVolumeSize: !Ref EbsRootVolumeSize
  SvcAccessSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub SG-${Env}-${SystemValue}-${SubSystemValue}-ServiceAccess
      VpcId: !Ref VpcId
      GroupDescription: !Sub 'SG-${SystemValue}-${SubSystemValue}-Service Access Security Group'
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 443
          ToPort: 443
          CidrIp: !Ref VpcCIDR
      SecurityGroupEgress:
        - IpProtocol: -1
          CidrIp: 0.0.0.0/0
      Tags:
        - Key: Environment
          Value: !Ref TagEnvironmentValue
        - Key: RevenueStream
          Value: !Ref RevenueStreamValue
        - Key: SubSystem
          Value: !Ref SubSystemValue
        - Key: System
          Value: !Ref SystemValue
        - Key: Name
          Value: !Sub 'SG-${Env}-${SystemValue}-${SubSystemValue}-ServiceAccess'

  EmrMasterSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub 'SG-${Env}-${SystemValue}-${SubSystemValue}-EMRMaster'
      VpcId: !Ref VpcId
      GroupDescription: !Sub 'SG-${SystemValue}-${SubSystemValue}-EMR Managed Master Security Group'
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 443
          ToPort: 443
          CidrIp: !Ref VpcCIDR
      SecurityGroupEgress:
        - IpProtocol: -1
          CidrIp: 0.0.0.0/0
      Tags:
        - Key: Environment
          Value: !Ref TagEnvironmentValue
        - Key: RevenueStream
          Value: !Ref RevenueStreamValue
        - Key: SubSystem
          Value: !Ref SubSystemValue
        - Key: System
          Value: !Ref SystemValue
        - Key: Name
          Value: !Sub 'SG-${Env}-${SystemValue}-${SubSystemValue}-EMRMaster'
  EmrSlaveSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Properties:
      GroupName: !Sub 'SG-${Env}-${SystemValue}-${SubSystemValue}-EMRSlave'
      VpcId: !Ref VpcId
      GroupDescription: !Sub 'SG-${SystemValue}-${SubSystemValue}-EMR Managed Slave Security Group'
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 443
          ToPort: 443
          CidrIp: !Ref VpcCIDR
      SecurityGroupEgress:
        - IpProtocol: -1
          CidrIp: 0.0.0.0/0
      Tags:
        - Key: Environment
          Value: !Ref TagEnvironmentValue
        - Key: RevenueStream
          Value: !Ref RevenueStreamValue
        - Key: SubSystem
          Value: !Ref SubSystemValue
        - Key: System
          Value: !Ref SystemValue
        - Key: Name
          Value: !Sub 'SG-${Env}-${SystemValue}-${SubSystemValue}-EMRSlave'
  LambdaSecurityGroup:
    Type: AWS::EC2::SecurityGroup
    Condition : HasFunctionName1
    Properties:
      GroupName: !Sub 'SG-${SGEnvironmentValue}-${SystemValue}-${SubSystemValue}'
      VpcId: !Ref VpcId
      GroupDescription: !Sub 'SG-${SystemValue}-${SubSystemValue}-Lambda Security Group'
      SecurityGroupIngress:
        - IpProtocol: tcp
          FromPort: 443
          ToPort: 443
          CidrIp: !Ref VpcCIDR
      SecurityGroupEgress:
        - IpProtocol: -1
          CidrIp: 0.0.0.0/0
      Tags:
        - Key: Environment
          Value: !Ref TagEnvironmentValue
        - Key: RevenueStream
          Value: !Ref RevenueStreamValue
        - Key: SubSystem
          Value: !Ref SubSystemValue
        - Key: System
          Value: !Ref SystemValue
        - Key: Name
          Value: !Sub 'SG-${SGEnvironmentValue}-${SystemValue}-${SubSystemValue}'

  Lambda1:
    Type: AWS::Lambda::Function
    DependsOn:
    - LambdaExecutionRole
    - ManagerApiPolicy
    Properties:
      Description: Script to launch EMR
      FunctionName: !Sub ' ${SystemValue}-${SubSystemValue}-${FunctionName1Value}'
      Handler: lambda_function.lambda_handler
      Code:
        S3Bucket: !Ref PackageBucket
        S3Key: emr-launch.zip
      Role: !GetAtt 
        - LambdaExecutionRole
        - Arn
      Runtime: !Ref LambdaRuntime
      Timeout: '900'
      MemorySize: 512
      Environment:
        Variables:
          parameterPath: !Sub '/crx/${SystemValue}/${SubSystemValue}/'
          region: !Ref 'AWS::Region'
      VpcConfig:
        SubnetIds: !Ref VPCSubnetIds
        SecurityGroupIds:
          - !GetAtt 
            - LambdaSecurityGroup
            - GroupId
      Tags:
        - Value: !Sub '${SystemValue}-${SubSystemValue}-${FunctionName1Value}'
          Key: Name
        - Key: SubSystem
          Value: !Ref SubSystemValue
        - Key: System
          Value: !Ref SystemValue
        - Value: !Ref RevenueStreamValue
          Key: RevenueStream
        - Value: !Ref TagEnvironmentValue
          Key: Environment

  LambdaExecutionRole:
    Type: AWS::IAM::Role

    DependsOn:
      - LambdaSecurityGroup
    Properties:
      RoleName: !Sub '${SystemValue}-${SubSystemValue}'
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
        - Effect: Allow
          Principal:
            Service:
              - lambda.amazonaws.com
              - states.amazonaws.com
              - events.amazonaws.com
            Action:
              - 'sts:AssumeRole'  

  ManagerApiPolicy:
    Type: 'AWS::IAM::ManagedPolicy'

    DependsOn:
      - LambdaExecutionRole
    Properties:
      ManagedPolicyName: !Sub '${SystemValue}-${SubSystemValue}'
      PolicyDocument:
        Version: 2012-10-17
        Statement:
          - Sid: common
            Effect: Allow
            Action:
              - 'ec2:CreateNetworkInterface'
              - 'ec2:Describe*'
              - 'ec2:Get*'
              - 'ec2:DeleteNetworkInterface'
              - 'kms:GenerateDataKey'
              - 'kms:ListAliases'
              - 'kms:ListKeys'
              - 'kms:ReEncryptTo'
              - 'kms:CreateKey'
              - 'kms:GenerateRandom'
              - 'kms:ReEncryptFrom'
              - 'logs:CreateLogGroup'
              - 'logs:PutLogEvents'
              - 'logs:CreateLogStream'
              - 'ssm:DescribeParameters'
              - 'lambda:InvokeFunction'

            Resource: '*'
          - Sid: ssm
            Effect: Allow
            Action:
              - 'ssm:GetParametersByPath'
              - 'ssm:GetParameters'
              - 'ssm:GetParameter'
            Resource:
              - !Join 
                - ''
                - - 'arn:aws:ssm:'
                  - !Ref 'AWS::Region'
                  - ':'
                  - !Ref 'AWS::AccountId'
                  - ':parameter/crx/'
                  - !Sub '${SystemValue}/${SubSystemValue}*'


          - Sid: kms
            Effect: Allow
            Action: 'kms:*'
            Resource:
              - !Ref KMSArn
              - !Join 
                - ''
                - - 'arn:aws:kms:'
                  - !Ref 'AWS::Region'
                  - ':'
                  - !Ref 'AWS::AccountId'
                  - ':alias/'
                  - !Sub '${SystemValue}-${SubSystemValue}'

          - Sid: s3
            Effect: Allow
            Action: 
              - 's3:PutObject'
              - 's3:GetObject'  
              - 's3:List*'             
            Resource: 
              - !Join 
                - ''
                - - 'arn:aws:s3:'                
                  - '::'                
                  - !Sub 'crx-${EnvironmentValue}-${PackageBucketName}'
              - !Join 
                - ''
                - - 'arn:aws:s3:'                
                  - '::'                
                  - !Sub 'crx-${EnvironmentValue}-${PackageBucketName}/*'
              - !Join 
                - ''
                - - 'arn:aws:s3:'                
                  - '::'                
                  - !Sub 'crx-${EnvironmentValue}-public-assets'
              - !Join 
                - ''
                - - 'arn:aws:s3:'                
                  - '::'                
                  - !Sub 'crx-${EnvironmentValue}-public-assets/*'                  


      Roles:
        - !Ref LambdaExecutionRole

  PermissionForEventsToInvokeLambda: 
    Type: 'AWS::Lambda::Permission'
    Properties: 
      FunctionName: 
        !Ref EmrCloudWatchEventLambda
      Action: lambda:InvokeFunction
      Principal: events.amazonaws.com
      SourceArn: 
        Fn::GetAtt: 
        - EventRuleEMRtest
        - Arn

  EMRClusterServiceRole:
    Type: 'AWS::IAM::Role'
    Properties:
      AssumeRolePolicyDocument:
        Version: 2012-10-17
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - ec2.amazonaws.com
          Action:
          - 'sts:AssumeRole'
    ManagedPolicyArns:
            - arn:aws:iam::aws:policy/service-role/AmazonElasticMapReduceforEC2Role
    Path: /

  EMRClusterinstanceProfile:
    Type: AWS::IAM::InstanceProfile
    Properties:
      Path: /
      Roles:
      - !Ref EMRClusterinstanceProfileRole

Upvotes: 0

Answers (3)

Pat Myron

Reputation: 4628

The CloudFormation Linter and Visual Studio Code extension catch these template errors:

W2030 You must specify a valid Default value for LambdaRuntime (Python-2.7).
Valid values are ['dotnetcore1.0', 'dotnetcore2.0', 'dotnetcore2.1', 'dotnetcore3.1', 'go1.x', 'java8', 'java11', 'nodejs', 'nodejs4.3-edge', 'nodejs4.3', 'nodejs6.10', 'nodejs8.10', 'nodejs10.x', 'nodejs12.x', 'provided', 'python2.7', 'python3.6', 'python3.7', 'python3.8', 'ruby2.5', 'ruby2.7']
template.yaml:111:5

E3001 Invalid resource attribute Ref for resource EbsRootVolumeSize
template.yaml:191:3

E3001 Type not defined for resource EbsRootVolumeSize
template.yaml:191:3

E3004 Circular Dependencies for resource EbsRootVolumeSize. Circular dependency with [EbsRootVolumeSize]
template.yaml:191:3

E8002 Condition HasFunctionName1 is not defined.
template.yaml:270:5

E1019 Parameter SGEnvironmentValue for Fn::Sub not found at Resources/LambdaSecurityGroup/Properties/GroupName/Fn::Sub
template.yaml:272:7

E1019 Parameter SGEnvironmentValue for Fn::Sub not found at Resources/LambdaSecurityGroup/Properties/Tags/4/Value/Fn::Sub
template.yaml:293:11

E3002 Property SubnetIds should be of type List or Parameter should be a list for resource Lambda1
template.yaml:318:9

E1019 Parameter PackageBucketName for Fn::Sub not found at Resources/ManagerApiPolicy/Properties/PolicyDocument/Statement/3/Resource/0/Fn::Join/1/2/Fn::Sub
template.yaml:422:13

E1019 Parameter PackageBucketName for Fn::Sub not found at Resources/ManagerApiPolicy/Properties/PolicyDocument/Statement/3/Resource/1/Fn::Join/1/2/Fn::Sub
template.yaml:422:13

E1012 Ref EmrCloudWatchEventLambda not found as a resource or parameter
template.yaml:451:7

E1010 Invalid GetAtt EventRuleEMRtest.Arn for resource PermissionForEventsToInvokeLambda
template.yaml:456:9

E3001 Invalid resource attribute ManagedPolicyArns for resource EMRClusterServiceRole
template.yaml:472:5

E3001 Invalid resource attribute Path for resource EMRClusterServiceRole
template.yaml:474:5

E1012 Ref EMRClusterinstanceProfileRole not found as a resource or parameter
template.yaml:481:9