How to send HTTP POST data in multipart/form-data to REST API in Perl

Question

I am sending POST HTTP request to a website REST API from my Perl script using multipart/form-data which requires basic authentication. I am passing some params as key value pairs in Content along with image file to be uploaded. I passed credentials in header with encoding. The response from the REST API is error code 422 data validation. My code snippet is below:

    use warnings;
    use Data::Dumper;
    use LWP::UserAgent;
    use HTTP::Headers;
    use HTTP::Request;
    use MIME::Base64;
    use JSON;
    use JSON::Parse 'parse_json';
    
    my $url = 'https://api.xyxabc.org/objects';
    my $imgmd5 = '3740239d74858504f5345365a1e3eb33';
    my $file= "images/1839.png";
    
    my %options = (         
            "username" =>  '___APIKEY___',
            "password" => '' );             # PASSWORD FIELD IS TO BE SENT BLANK
            
    my $ua = LWP::UserAgent->new(keep_alive=>1);
    $ua->agent("MyApp/0.1 ");
    my $h = HTTP::Headers->new(
        Authorization       => 'Basic ' .  encode_base64($options{username} . ':' . $options{password}),
        Content                 =>  [
                                                'name'=> 'Ethereum',
                                                'lang' => 'en',
                                                'description' => 'Ethereum is a decentralized open-source',
                                                'main_image' => $imgmd5,                                            
                                                'parents[0][Objects][id]' => '42100',
                                                'Objects[imageFiles][0]' => $file,
                                                'objectsPropertiesValues[142][ObjectsPropertiesValues][property_id]' => 142,                                        
        ],
        Content_Type    => 'multipart/form-data',
    );
    
    my $r = HTTP::Request->new('POST', $url, $h);
    my $response = $ua->request($r);
    my $jp = JSON::Parse->new ();
    print Dumper $response->status_line;
    my $jsonobj = $response->decoded_content;    
        eval {
                        $jsonobj = $jp->parse ($jsonobj);
            };
                if ($@) {
        print $@;    
    }
    print Dumper $jsonobj;

The error is:

$VAR1 = '422 Data Validation Failed.';
$VAR1 = [
          {
            'field' => 'name',
            'message' => 'Name cannot be blank.'
          },
          {
            'message' => 'Language cannot be blank.',
            'field' => 'lang'
          }
        ];

What I am doing wrong? basically server is not getting well formed query string and headers as I understand. I am passing some 32 key value pairs along with image file to be uploaded in actual script and I have produced here a minimal script. I know that all the params variables are fine as when I post this HTTP request via postman, it complains with different error.

I executed similar query yesterday night via postman with different param values and it got executed along with image uploaded. But now both postman and Perl script are complaining. I need two things: 1.First why the POST request via postman is complaining? 2. Second I am building Perl LWP script to upload data to this website REST API and I need a functioning script as produced above.

I will obliged if someone helps.

simbabque · Accepted Answer

Steffen's answer shows you how to do it the most simple way.

If you want a bit more control, and especially if you want to do more than one request, give my solution a go.

The authorization you're doing is correct. I would suggest you move that into a default header on the $ua object. That makes sense if you're making multiple requests.

use strict;
use warnings;

use LWP::UserAgent;
use JSON 'from_json';
use MIME::Base64 'encode_base64';
use Data::Dumper;

my $url    = 'http://localhost:3000';
my $imgmd5 = '3740239d74858504f5345365a1e3eb33';
my $file   = "images/1839.png";

my %options = (
    "username" => '___APIKEY___',
    "password" => ''
);

my $ua = LWP::UserAgent->new( keep_alive => 1 );
$ua->agent("MyApp/0.1 ");
$ua->default_header( Authorization => 'Basic '
      . encode_base64( $options{username} . ':' . $options{password} ) );

Note I changed the URL to a local address. We'll see why and how I'm testing this code further down.

For your request, you can use HTTP::Request::Common as Steffen suggested, or you can pass it all to the post method on your $ua. It takes a multitude of different argument combinations and is very flexible. We want to send a form with key/value pairs, and a header for the content type.

my $res = $ua->post(
    $url,                          # where to send it
    Content_Type => 'form-data',   # key/value pairs of headers
    Content =>                     # the form VVV
    {
        'name'                    => 'Ethereum',
        'lang'                    => 'en',
        'description'             => 'Ethereum is a decentralized open-source',
        'main_image'              => $imgmd5,
        'parents[0][Objects][id]' => '42100',
        'Objects[imageFiles][0]'  => $file,
        'objectsPropertiesValues[142][ObjectsPropertiesValues][property_id]' =>
          142,
    }
);

I've changed some of the modules you're using. You don't need JSON::Parser. Just the JSON module is sufficient. If you let LWP decode your content you can use from_json, because the body has already been turned into Perl's string representation from whatever character encoding (probably utf-8) it came in.

It's now as simple as this.

if ( $res->is_success ) {
    my $json = eval { from_json( $res->decoded_content ) };
    print Dumper $json;
}

To debug this, I'm using the ojo module that comes with Mojolicious. It allows you to create a web application in a one-liner.

In a terminal, I am running this command. It spawns an app that listens on port 3000, route / with any method, and returns a fixed JSON object for your code to receive.

$ perl -Mojo -E 'a("/" => { json => { foo => 123 } })->start' daemon
Web application available at http://127.0.0.1:3000

Next, I am making a request.

$ perl 66829616.pl
$VAR1 = {
          'foo' => 123
        };

This works. But we don't know yet if we're sending the right headers. Let's look at that. Install the LWP::ConsoleLogger module and load LWP::ConsoleLogger::Everywhere. It will dump requests and responses from all your LWP objects.

$ perl -MLWP::ConsoleLogger::Everywhere 66829616.pl
POST http://localhost:3000

POST Params:

.------------------------------------+-----------------------------------------.
| Key                                | Value                                   |
+------------------------------------+-----------------------------------------+
| Objects[imageFiles][0]             | images/1839.png                         |
| description                        | Ethereum is a decentralized open-source |
| lang                               | en                                      |
| main_image                         | 3740239d74858504f5345365a1e3eb33        |
| name                               | Ethereum                                |
| objectsPropertiesValues[142][Obje- | 142                                     |
| ctsPropertiesValues][property_id]  |                                         |
| parents[0][Objects][id]            | 42100                                   |
'------------------------------------+-----------------------------------------'

.---------------------------------+-------------------------------------.
| Request (before sending) Header | Value                               |
+---------------------------------+-------------------------------------+
| Authorization                   | Basic X19fQVBJS0VZX19fOg==          |
| Content-Length                  | 633                                 |
| Content-Type                    | multipart/form-data; boundary=xYzZY |
| User-Agent                      | MyApp/0.1 libwww-perl/6.52          |
'---------------------------------+-------------------------------------'

.------------------------------------------------------------------------------.
| Content                                                                      |
+------------------------------------------------------------------------------+
| [ REDACTED by LWP::ConsoleLogger.  Do not know how to display multipart/for- |
| m-data; boundary=xYzZY. ]                                                    |
'------------------------------------------------------------------------------'

.------------------------------------------------------------------------------.
| Text                                                                         |
+------------------------------------------------------------------------------+
| [ REDACTED by LWP::ConsoleLogger.  Do not know how to display multipart/for- |
| m-data; boundary=xYzZY. ]                                                    |
'------------------------------------------------------------------------------'

.--------------------------------+-------------------------------------.
| Request (after sending) Header | Value                               |
+--------------------------------+-------------------------------------+
| Authorization                  | Basic X19fQVBJS0VZX19fOg==          |
| Content-Length                 | 633                                 |
| Content-Type                   | multipart/form-data; boundary=xYzZY |
| User-Agent                     | MyApp/0.1 libwww-perl/6.52          |
'--------------------------------+-------------------------------------'

==> 200 OK

.---------------------+--------------------------------.
| Response Header     | Value                          |
+---------------------+--------------------------------+
| Client-Date         | Sat, 27 Mar 2021 11:01:31 GMT  |
| Client-Peer         | 127.0.0.1:3000                 |
| Client-Response-Num | 1                              |
| Content-Length      | 11                             |
| Content-Type        | application/json;charset=UTF-8 |
| Date                | Sat, 27 Mar 2021 11:01:31 GMT  |
| Server              | Mojolicious (Perl)             |
'---------------------+--------------------------------'

.-------------.
| Content     |
+-------------+
| {"foo":123} |
'-------------'

.---------------------.
| Text                |
+---------------------+
| {                   |
|     foo => 123,     |
| }                   |
'---------------------'

$VAR1 = {
          'foo' => 123
        };

As you can see, your auth header is there, and we're using the right content type.

Notice that the User-Agent header includes libww-perl. That's because you have a space at the end of the string you pass to agent(). Remove the whitespace to stop it doing that.

$ua->agent("MyApp/0.1 "); # append libwww/perl
$ua->agent("MyApp/0.1");  # don't append

If you wanted to turn this into a more extendable API client, you could use Moo (or Moose) to write a module like this. Put this into a file API/Factopedia.pm in your lib directory.


package API::Factopedia;

use HTTP::Request::Common qw(POST PUT);
use LWP::UserAgent;
use JSON 'from_json';
use MIME::Base64 'encode_base64';

use Moo;

has ua => (
    is      => 'ro',
    lazy    => 1,
    builder => sub {
        my ($self) = @_;
        my $ua = LWP::UserAgent->new( keep_alive => 1, agent => 'MyApp/0.1' );
        $ua->default_header( Authorization => $self->_create_auth );
        return $ua;
    },
);

has [qw/ username password /] => (
    is       => 'ro',
    required => 1
);

has base_uri => (
    is      => 'ro',
    default => 'https://api.factopedia.org'
);

=head2 _create_auth

Returns the basic authentication credentials to use based on username and password.

=cut

sub _create_auth {
    my ($self) = @_;
    return 'Basic ' . encode_base64( $self->username . ':' . $self->password );
}

=head2 create_object

Creates an object in the API. Takes a hashref of formdata and returns a JSON response.

    my $json = $api->create_object({ ... });

=cut

sub create_object {
    my ( $self, $data ) = @_;

    return $self->_request(
        POST(
            $self->_url('/objects'),
            Content_Type => 'form-data',
            Content      => $data
        )
    );
}

=head2 update_object

Updates a given 

=cut

sub update_object {
    my ( $self, $id, $data ) = @_;

    # parameter validation (probably optional)
    die unless $id;
    die if $id =~ m/\D/;

    return $self->_request(
        PUT(
            $self->_url("/object/$id"),
            Content_Type => 'form-data',
            Content      => $data
        )
    );
}

=head2 _request

Queries the API, decodes the response, handles errors and returns JSON. 
Takes an L object.

=cut

sub _request {
    my ( $self, $req ) = @_;

    my $res = $self->ua->request($req);

    if ( $res->is_success ) {
        return from_json( $res->decoded_content );
    }

    # error handling here
}

=head2 _url

Returns the full API URL for a given endpoint.

=cut

sub _url {
    my ( $self, $endpoint ) = @_;

    return $self->base_uri . $endpoint;
}

no Moo;

1;

The client allows injecting a custom user agent object for testing and lets you easily override the _create_auth method in a subclass or by replacing it in a unit test. You can also pass in a different base URI for testing, as shown below.

Now all you need to do in your script is create an instance, and call the create_object method.

use strict;
use warnings;
use Data::Dumper;

use API::Factopedia;

my $url    = 'http://localhost:3000';
my $imgmd5 = '3740239d74858504f5345365a1e3eb33';
my $file   = "images/1839.png";

my $client = API::Factopedia->new(
    username => '__APIKEY__',
    password => '',
    base_uri => 'http://localhost:3000', # overwritted for test
);

print Dumper $client->create_object(
    {
        'name'                    => 'Ethereum',
        'lang'                    => 'en',
        'description'             => 'Ethereum is a decentralized open-source',
        'main_image'              => $imgmd5,
        'parents[0][Objects][id]' => '42100',
        'Objects[imageFiles][0]'  => $file,
        'objectsPropertiesValues[142][ObjectsPropertiesValues][property_id]' =>
          142,
    }
);

To test with our one-liner, we need to change the endpoint from / to /objects and rerun it.

The output will be the same.

If you want to extend this client to do additional endpoints, you just need to add simple methods to your module. I've done that with PUT for updating an object.

How to send HTTP POST data in multipart/form-data to REST API in Perl

Answers (2)

Related Questions