CODEWITHSUNDEEP
httpcurlcookiesheader
Matt Clark
Matt Clark

Reputation: 28629

Save cookies between two curl requests

I know that using cURL I can see my received cookies / headers by using

curl --head www.google.com

And I know that I can add headers to my request using

curl --cookie "Key=Value" www.google.com

I am currently working on testing an issue which requires persistent cookies, and there can be a lot of them.

How can I efficiently preserve cookies between two cURL requests?

If possible using a temporary file for storage.

Upvotes: 138

Views: 173057

Answers (5)

StackzOfZtuff
StackzOfZtuff

Reputation: 3110

Use --cookie and --cookie-jar parameters

Here is an example with Google.

If you run curl with -v (--verbose) then it will prefix lines with *, < or > as follows:

  • * internal action taken by curl. E.g. curl reading from or writing to cookie-jar file:
    • * WARNING: failed to open cookie file
    • * Added cookie, if we get a cookie from server and write to cookie-jar
    • * Replaced cookie if we get a cookie from server and OVERwrite to cookie-jar. This will happen if we send a cookie to server and the server REJECTS and replaces it.
  • < received from server. E.g. cookies we receive:
    • < Set-Cookie:
  • > sent to server. E.g. cookies we send:
    • > Cookie:

At the beginning there is no cookie-jar file. Curl will complain and then create that file.

$ cat cookie-jar.txt
cat: cookie-jar.txt: No such file or directory


$ curl -v https://www.google.com/ --cookie cookie-jar.txt --cookie-jar cookie-jar.txt 2>&1 | grep -i cookie | sed 's_^[<>] .*_&\n_'
* WARNING: failed to open cookie file "cookie-jar.txt"
* Added cookie AEC="AVYB7cqL-OQvxmmtwVLNJw9AokjyxcVxGMb6G_st_46QOcVyszGthpwRegs" for domain google.com, path /, expire 1745678783
< Set-Cookie: AEC=AVYB7cqL-OQvxmmtwVLNJw9AokjyxcVxGMb6G_st_46QOcVyszGthpwRegs; expires=Sat, 26-Apr-2025 14:46:23 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax

* Added cookie __Secure-ENID="23.SE=OmfFv0dP1S9U8BdJ9URyZ88ULi1FpEBJq3G3d_5BZhEEHUkiXoy27Sx4lFBt6gDZcnAabVKJt50nidGBSP8hnHUR2seYTBd-pQpG8x71f3XoJlPAbv62d9A8oz60x-D9_BslxQNdsOnwotM-rEt7qJ8352dSH1j7QVIGPnOr0hQlb1UaTU3BdO8VmSJIsi8pk_Em-gKw5uU" for domain google.com, path /, expire 1764313481
< Set-Cookie: __Secure-ENID=23.SE=OmfFv0dP1S9U8BdJ9URyZ88ULi1FpEBJq3G3d_5BZhEEHUkiXoy27Sx4lFBt6gDZcnAabVKJt50nidGBSP8hnHUR2seYTBd-pQpG8x71f3XoJlPAbv62d9A8oz60x-D9_BslxQNdsOnwotM-rEt7qJ8352dSH1j7QVIGPnOr0hQlb1UaTU3BdO8VmSJIsi8pk_Em-gKw5uU; expires=Fri, 28-Nov-2025 07:04:41 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax

So the cookie-jar file was created and now two cookies are in there:

$ cat cookie-jar.txt
# Netscape HTTP Cookie File
# https://curl.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.

#HttpOnly_.google.com   TRUE    /       TRUE    1764313481      __Secure-ENID   23.SE=OmfFv0dP1S9U8BdJ9URyZ88ULi1FpEBJq3G3d_5BZhEEHUkiXoy27Sx4lFBt6gDZcnAabVKJt50nidGBSP8hnHUR2seYTBd-pQpG8x71f3XoJlPAbv62d9A8oz60x-D9_BslxQNdsOnwotM-rEt7qJ8352dSH1j7QVIGPnOr0hQlb1UaTU3BdO8VmSJIsi8pk_Em-gKw5uU
#HttpOnly_.google.com   TRUE    /       TRUE    1745678783      AEC     AVYB7cqL-OQvxmmtwVLNJw9AokjyxcVxGMb6G_st_46QOcVyszGthpwRegs

Then when we run the same command again curl will use the file (which now actually exists) to read and send the cookies. Curl will also update the cookie-jar file.

$ curl -v https://www.google.com/ --cookie cookie-jar.txt --cookie-jar cookie-jar.txt 2>&1 | grep -i cookie | sed 's_^[<>] .*_&\n_'
> Cookie: __Secure-ENID=23.SE=OmfFv0dP1S9U8BdJ9URyZ88ULi1FpEBJq3G3d_5BZhEEHUkiXoy27Sx4lFBt6gDZcnAabVKJt50nidGBSP8hnHUR2seYTBd-pQpG8x71f3XoJlPAbv62d9A8oz60x-D9_BslxQNdsOnwotM-rEt7qJ8352dSH1j7QVIGPnOr0hQlb1UaTU3BdO8VmSJIsi8pk_Em-gKw5uU; AEC=AVYB7cqL-OQvxmmtwVLNJw9AokjyxcVxGMb6G_st_46QOcVyszGthpwRegs

* Added cookie SOCS="CAAaBgiAqPu4Bg" for domain google.com, path /, expire 1764254856
< Set-Cookie: SOCS=CAAaBgiAqPu4Bg; expires=Thu, 27-Nov-2025 14:47:36 GMT; path=/; domain=.google.com; Secure; SameSite=lax

* Replaced cookie __Secure-ENID="23.SE=A4bC0q4soiHrkO1kkvQ-Vfn9e7CIwL9gbldPbKESKWCtKWGLynyUDx5cYx2n9cx9MVbug87WK2kKhOcZNDkTr3u_vWDrjW2MRs1mTG5m596PRfLsLkjKvoLY5lqUXK9kq7XV1cVH5lo3WriaYkcGzoX5RbXMmQDvh04aovhgXxuikkZL6A5qXbnA58pVnA9_eEznKNtEjZBEccH_7j4" for domain google.com, path /, expire 1764313481
< Set-Cookie: __Secure-ENID=23.SE=A4bC0q4soiHrkO1kkvQ-Vfn9e7CIwL9gbldPbKESKWCtKWGLynyUDx5cYx2n9cx9MVbug87WK2kKhOcZNDkTr3u_vWDrjW2MRs1mTG5m596PRfLsLkjKvoLY5lqUXK9kq7XV1cVH5lo3WriaYkcGzoX5RbXMmQDvh04aovhgXxuikkZL6A5qXbnA58pVnA9_eEznKNtEjZBEccH_7j4; expires=Fri, 28-Nov-2025 07:04:41 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax

$ cat cookie-jar.txt
# Netscape HTTP Cookie File
# https://curl.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.

.google.com     TRUE    /       TRUE    1764254856      SOCS    CAAaBgiAqPu4Bg
#HttpOnly_.google.com   TRUE    /       TRUE    1745678783      AEC     AVYB7cqL-OQvxmmtwVLNJw9AokjyxcVxGMb6G_st_46QOcVyszGthpwRegs
#HttpOnly_.google.com   TRUE    /       TRUE    1764313481      __Secure-ENID   23.SE=A4bC0q4soiHrkO1kkvQ-Vfn9e7CIwL9gbldPbKESKWCtKWGLynyUDx5cYx2n9cx9MVbug87WK2kKhOcZNDkTr3u_vWDrjW2MRs1mTG5m596PRfLsLkjKvoLY5lqUXK9kq7XV1cVH5lo3WriaYkcGzoX5RbXMmQDvh04aovhgXxuikkZL6A5qXbnA58pVnA9_eEznKNtEjZBEccH_7j4

Kinda like TSV.

Detailed info on file format here: https://curl.se/docs/http-cookies.html

TLDR: at a rough glance the cookie-jar file format seems to be a variant on TSV (tab separated values)...

$ cat cookie-jar.txt -vETn
     1  # Netscape HTTP Cookie File$
     2  # https://curl.se/docs/http-cookies.html$
     3  # This file was generated by libcurl! Edit at your own risk.$
     4  $
     5  .google.com^ITRUE^I/^ITRUE^I1764254856^ISOCS^ICAAaBgiAqPu4Bg$
     6  #HttpOnly_.google.com^ITRUE^I/^ITRUE^I1745678783^IAEC^IAVYB7cqL-OQvxmmtwVLNJw9AokjyxcVxGMb6G_st_46QOcVyszGthpwRegs$
     7  #HttpOnly_.google.com^ITRUE^I/^ITRUE^I1764313481^I__Secure-ENID^I23.SE=A4bC0q4soiHrkO1kkvQ-Vfn9e7CIwL9gbldPbKESKWCtKWGLynyUDx5cYx2n9cx9MVbug87WK2kKhOcZNDkTr3u_vWDrjW2MRs1mTG5m596PRfLsLkjKvoLY5lqUXK9kq7XV1cVH5lo3WriaYkcGzoX5RbXMmQDvh04aovhgXxuikkZL6A5qXbnA58pVnA9_eEznKNtEjZBEccH_7j4$

...so you can csvlook to (sort of) look at it:

$ cat cookie-jar.txt | grep -P '\t' | csvlook --tabs --no-header --no-inference | cut -c-150
| a                     | b    | c | d    | e          | f             | g
| --------------------- | ---- | - | ---- | ---------- | ------------- | -----------------------------------------------------------------------------
| .google.com           | TRUE | / | TRUE | 1764254856 | SOCS          | CAAaBgiAqPu4Bg
| #HttpOnly_.google.com | TRUE | / | TRUE | 1745678783 | AEC           | AVYB7cqL-OQvxmmtwVLNJw9AokjyxcVxGMb6G_st_46QOcVyszGthpwRegs
| #HttpOnly_.google.com | TRUE | / | TRUE | 1764313481 | __Secure-ENID | 23.SE=A4bC0q4soiHrkO1kkvQ-Vfn9e7CIwL9gbldPbKESKWCtKWGLynyUDx5cYx2n9cx9MVbug87

Upvotes: 1

dw1
dw1

Reputation: 1845

curl -b cookie.txt -c cookie.txt <url> or curl --cookie cookie.txt --cookie-jar cookie.txt <url> will both store and send saved cookies.

Upvotes: 117

gihook
gihook

Reputation: 329

just to extend on the accepted answer. Using --cookie and --cookie-jar flags can be used without file writes, using process substitution:

  1. write to STDOUT and save into $cookie variable
cookie=$(curl -c - <url>)
  1. read cookie from $cookie variable
curl --cookie <(echo "$cookie") <url>

Upvotes: 15

Bruce Tong
Bruce Tong

Reputation: 1431

I would like to thank everyone here for helping me find a solution to login to my wordpress site using CURL. This is my little contribution, hope that it will also help some one who stumble across this in the future

For windows 10 build newer than 1706 you can use curl from the command line. 

curl -c c:\Users\<your-name>\Desktop\cookie.txt -F "log=<your username>" -F "pwd=<your-pwd>" https://acme.net/wp-login.php

you won't see any response in your terminal. it just refreshes itself but your cookie is already stored.

then type in the below to read and send the cookies you just stored.

curl -b c:\Users\<your-name>\Desktop\cookie.txt -L https://acme.net/my-profile/

the entire webpage that should only be accessible for logged in users will be rendered inside your terminal including any data that is loaded dynamically via javascript.

Even if the end point of your WP login url is '/register/log-in' or '/login' i still recommend you use '/wp-login.php' as the endpoint. That's because some WP themes have a login nonce in a hidden input on their login page. The native WP login page does not have this.

other sources you can refer to:

https://wpmayor.com/login-to-wordpress-dashboard-via-curl/

https://gist.github.com/subfuzion/08c5d85437d5d4f00e58

https://www.youtube.com/watch?v=B4ilccLUQVs

https://makandracards.com/makandra/48262-how-to-use-cookies-with-curl

Upvotes: 5

Remy Lebeau
Remy Lebeau

Reputation: 597215

Use the --cookie-jar or --dump-header parameter to save received cookies to a file. The --cookie parameter can read back the cookies from that file later.

-b, --cookie <name=data>

(HTTP) Pass the data to the HTTP server as a cookie. It is supposedly the data previously received from the server in a "Set-Cookie:" line. The data should be in the format "NAME1=VALUE1; NAME2=VALUE2".

If no '=' symbol is used in the line, it is treated as a filename to use to read previously stored cookie lines from, which should be used in this session if they match. Using this method also activates the cookie engine which will make curl record incoming cookies too, which may be handy if you're using this in combination with the -L, --location option. The file format of the file to read cookies from should be plain HTTP headers (Set-Cookie style) or the Netscape/Mozilla cookie file format.

The file specified with -b, --cookie is only used as input. No cookies will be written to the file. To store cookies, use the -c, --cookie-jar option.

Exercise caution if you are using this option and multiple transfers may occur. If you use the NAME1=VALUE1; format, or in a file use the Set-Cookie format and don't specify a domain, then the cookie is sent for any domain (even after redirects are followed) and cannot be modified by a server-set cookie. If the cookie engine is enabled and a server sets a cookie of the same name then both will be sent on a future transfer to that server, likely not what you intended. To address these issues set a domain in Set-Cookie (doing that will include sub-domains) or use the Netscape format.

If this option is used several times, the last one will be used.

-c, --cookie-jar <file name>

(HTTP) Specify to which file you want curl to write all cookies after a completed operation. Curl writes all cookies previously read from a specified file as well as all cookies received from remote server(s). If no cookies are known, no data will be written. The file will be written using the Netscape cookie file format. If you set the file name to a single dash, "-", the cookies will be written to stdout.

This command line option will activate the cookie engine that makes curl record and use cookies. Another way to activate it is to use the -b, --cookie option.

If the cookie jar can't be created or written to, the whole curl operation won't fail or even report an error clearly. Using -v will get a warning displayed, but that is the only visible feedback you get about this possibly lethal situation.

Since 7.43.0 cookies that were imported in the Set-Cookie format without a domain name are not exported by this option.

If this option is used several times, the last specified file name will be used.

-D, --dump-header <file>

Write the protocol headers to the specified file.

This option is handy to use when you want to store the headers that an HTTP site sends to you. Cookies from the headers could then be read in a second curl invocation by using the -b, --cookie option! The -c, --cookie-jar option is a better way to store cookies.

When used in FTP, the FTP server response lines are considered being "headers" and thus are saved there.

If this option is used several times, the last one will be used

Alternatively, instead of using the command-line cURL app, write some code that uses the libCurl library. That will give you more direct control over cookie handling. libCurl has several features related to HTTP cookies:

Options for curl_easy_getinfo():

Options for curl_easy_setopt():

Then you can store the cookies however you want, and assign them as needed to later HTTP sessions.

Upvotes: 146

Related Questions