Dumping very bad idea of updating this post. New place for this script: https://gist.github.com/Wogol/66e9936b6d49cc5fecca59eaeca1ca2e Im trying to create a .command macOS script (Should also work under GNU/Linux) that use Youtube-dl that is simple to use. I have fixed so it downloads description, thumbnail, subtitles, json, creates folder structure and also saves video ID, uploader and upload date. ISSUES WITH THE MAIN SCRIPT: FIXED (13th august) Problem I struggle with is the option of Audio & Video or Audio only. For some reason only audio works in the script. The download script for the video&audio dont work but if I paste that same command line ("The DEBUG output line") in a terminal window it works. Scratching my head. Youtube-dl gives me this message: ERROR: requested format not available FIXED (31th august) Get max resolution of video working. Have found information to force mp4 or max resolution but not combined them. ISSUES WITH INFORMATION FILE: Also creating a info file with title, channel name, release date, description. Im now struggling with getting video information from .json and youtube-dl to be exported into the info.txt file. FIXED (5th september) textfile=""$folder"info.txt" not working. Gives this error: (There I want to add youtube-dl folder. ytdl.command: line 104: ~/Downloads/ytdl/dog_vids/info.txt: No such file or directory FIXED (5th september) Find youtube-dl folder and get it to work with grep. Something like this: youtube-dl --simulate --SHOW_THE_OUTPUT_PATH -o $folder'/%(title)s/%(title)s - (%(id)s) - %(uploader)s - %(upload_date)s.%(ext)s' https://www.youtube.com/watch?v=dQw4w9WgXcQ FIXED (5th september) With grep command i named the json file "*.json" because there will only be one per directory but I dont like that solution. (Could be answered with point above) FIXED (5th september) How to make so grep dont grab "? It now adds them before and after everything. FIXED (5th september) How to get the tags information from json file? Tags look like this: "tags": ["music", "video", "classic"] FIXED (5th september) Run the creation of info file part of the script in the background of downloading the video? CURRENT VERSION TRYING TO GET IT WORKING (12 august) textfile=""$folder"info.txt" echo TITLE >> ~/Downloads/ytdl/dog_vids/info.txt youtube-dl -e $url >> ~/Downloads/ytdl/dog_vids/info.txt echo \ >> ~/Downloads/ytdl/dog_vids/info.txt echo CHANNEL >> $textfile echo \ >> $textfile echo CHANNEL URL >> $textfile echo \ >> $textfile echo UPLOAD DATE >> $textfile echo \ >> $textfile echo URL >> $textfile echo $url >> $textfile echo \ >> $textfile echo TAGS >> $textfile echo \ >> $textfile echo DESCRIPTION >> $textfile youtube-dl --get-description $url >> $textfile EXPERIMENT FUTURE VERSION - EXTRACTING INFORMATION FROM JSON FILE This isnt a working script. Showing how I want it with $textfile, $ytdlfolder and $jsonfile. url=https://www.youtube.com/watch?v=dQw4w9WgXcQ textfile=""$folder""$YOUTUBE-DL_PATH"info.txt" ytdlfolder="$folder""$YOUTUBE-DL_PATH" jsonfile="$folder""$YOUTUBE-DL_JSON-FILE" Echo TITLE >> $textfile grep -o '"title": *"[^"]*"' $jsonfile | grep -o '"[^"]*"$' >> $textfile Echo \ >> $textfile Echo CHANNEL >> $textfile grep -o '"uploader": *"[^"]*"' $jsonfile | grep -o '"[^"]*"$' >> $textfile Echo \ >> $textfile Echo CHANNEL URL >> $textfile grep -o '"uploader_url": *"[^"]*"' *.json | grep -o '"[^"]*"$' >> $textfile Echo \ >> $textfile Echo UPLOAD DATE >> $textfile grep -o '"upload_date": *"[^"]*"' *.json | grep -o '"[^"]*"$' >> $textfile Echo \ >> $textfile Echo TAGS >> $textfile grep -o '"tags": *"[^"]*"' *.json | grep -o '"[^"]*"$' >> $textfile Echo \ >> $textfile echo URL >> $textfile echo $url >> $textfile echo \ >> $textfile Echo DESCRIPTION >> $textfile youtube-dl --get-description $url >> $textfile THE SCRIPT: 12 august. Moved url to the top so when user paste the url they get the videos title back. This so the user know they got the right video. Added max resolution 1920x1080. (Do not work) 13 august. Downloading Audio & Video works. 31 august. Fixed force mp4 and max heigh of 1080. 5 september. Finally working script. Read more about it here (Or scroll down): Youtube-dl download script debug 2020-09-17 Folders can now have spaces in them. 2020-09-22 Select menus is now one column. Minor fixes. Now all the bugs is fixed. Issues left is only optimizations. #! /bin/bash ################################################################################ # Script Name: Youtube-dl Easy Download Script # Description: Easy to use script to download YouTube videos with a couple of # options. # # What this script do: # - Downloads video in MP4 with highest quality and max resolution 1920x1080. # - Downloads thumbnail and subtitles. # - Gives user option where to download the video and video or only audio. # - Creates a folder with same name as video title and puts all files there. # - Creates a .txt file with information about the video. # # # Author: Wogol - Stackoverflow.com, Github.com # License: The GNU General Public License v3.0 - GNU GPL-3 # # # Big thanks to the people at youtube-dl GitHub and Stack Overflow. Without # their help this would never ever been possible for me. # # Special thanks to: # Reino @ Stack Overflow # # ##### # # Software required: youtube-dl, xidel, printf # # macOS: 1. Install Homebrew: https://brew.sh # 2. Terminal command: brew install youtube-dl xidel # # Linux: Depends on package manager your distribution use. # # ##### # # Version history: # 2020-09-22 # - Select menus is now one column. # - Minor fixes. # - Now all the bugs is fixed. Issues left is only optimizations. # # 2020-09-17 # - Folders can now have spaces in them. # # 2020-09-05 # - First working version. # # ##### # # Issues left: # - In the beginning there is a confirmation that show the title of the # video so user know they got the correct video. It takes youtube-dl a # couple of seconds. To speed up the script it is DISABLED by default. # # - Have found out that the script dont need xidel to get json information # but youtube-dl can get it. Dont know how to use youtube-dl --dump-json # to get the same result. # # - To get the path to the .txt file script use youtube-dl. This gives the # script a pause for a few seconds. Best would get to get the path some how # without connecting to YouTube again but use the output from youtube-dl # some how. ... or run it in the background when video is downloading. # ################################################################################ clear # - WELCOME MESSAGE - echo COLUMNS=$(tput cols) title="-= Youtube-dl Easy Download Script =-" printf "%*s\n" $(((${#title}+$COLUMNS)/2)) "$title" # - PASTE URL - echo -e "\n*** - Paste URL address and hit RETURN. Example:\nhttps://www.youtube.com/watch?v=dQw4w9WgXcQ --OR-- https://youtu.be/dQw4w9WgXcQ\n" read url # - VIDEO TITLE - # So users know they have the correct URL. #echo -e "\nThe video is: (This takes 3-4 seconds, or more ...)" #youtube-dl -e $url #echo # - DOWNLOAD LOCATION - # DIRECTORY MUST END WITH SLASH: / echo -e "\n\n*** - Choose download folder:\n" COLUMNS=0 PS3='Choose: ' select directory in "~/Downloads/ytdl/Rick Astley/" "~/Downloads/ytdl/Never Gonna Give You Up/" "~/Downloads/ytdl/Other Rick Videos/" ; do echo -e "\nOption $REPLY selected. Download directory is:\n $directory" # - AUDIO/VIDEO SETTINGS - echo -e "\n\n*** - Choose download settings:\n" COLUMNS=0 PS3='Choose: ' options=("Audio & Video" "Audio only") select settingsopt in "${options[@]}" do case $settingsopt in "Audio & Video") av="-f bestvideo[ext=mp4][height<=1080]+bestaudio[ext=m4a]/best[ext=mp4]/best --merge-output-format mp4" ;; "Audio only") av="-f bestaudio[ext=m4a]/bestaudio" ;; esac echo -e "\nOption $REPLY selected:\n $settingsopt" # - THE DOWNLOAD SCRIPT - echo -e "\n\n*** - Starting download:\n" youtube-dl $av --write-thumbnail --all-subs --restrict-filenames -o "$directory%(title)s/%(title)s.%(ext)s" $url # - INFORMATION FILE - textfile=$(youtube-dl --get-filename --restrict-filenames -o "$directory%(title)s/%(title)s.txt" $url) xidel -s "$url" -e ' let $json:=json( //script/extract(.,"ytplayer.config = (.+?\});",1)[.] )/args, $a:=json($json/player_response)/videoDetails, $b:=json($json/player_response)/microformat return ( "- TITLE -", $a/title,"", "- CHANNEL -", $a/author,"", "- CHANNEL URL -", $b//ownerProfileUrl,"", "- UPLOAD DATE -", $b//publishDate,"", "- URL -", $json/loaderUrl,"", "- TAGS -", $a/keywords,"", "- DESCRIPTION -", $a/shortDescription ) ' --printed-json-format=compact >> "$textfile" # - THE END - echo COLUMNS=$(tput cols) ending="Download Complete!" printf "%*s\n\n" $(((${#ending}+$COLUMNS)/2)) "$ending" exit done done

Reputation: 41

Youtube-dl download script debug

Dumping very bad idea of updating this post. New place for this script: https://gist.github.com/Wogol/66e9936b6d49cc5fecca59eaeca1ca2e

Im trying to create a .command macOS script (Should also work under GNU/Linux) that use Youtube-dl that is simple to use. I have fixed so it downloads description, thumbnail, subtitles, json, creates folder structure and also saves video ID, uploader and upload date.

ISSUES WITH THE MAIN SCRIPT:

FIXED (13th august) Problem I struggle with is the option of Audio & Video or Audio only. For some reason only audio works in the script. The download script for the video&audio dont work but if I paste that same command line ("The DEBUG output line") in a terminal window it works. Scratching my head. Youtube-dl gives me this message: ERROR: requested format not available
FIXED (31th august) Get max resolution of video working. Have found information to force mp4 or max resolution but not combined them.

ISSUES WITH INFORMATION FILE:

Also creating a info file with title, channel name, release date, description. Im now struggling with getting video information from .json and youtube-dl to be exported into the info.txt file.

FIXED (5th september) textfile=""$folder"info.txt" not working. Gives this error: (There I want to add youtube-dl folder.

    ytdl.command: line 104: ~/Downloads/ytdl/dog_vids/info.txt: No such file or directory

FIXED (5th september) Find youtube-dl folder and get it to work with grep. Something like this:

   youtube-dl --simulate --SHOW_THE_OUTPUT_PATH -o $folder'/%(title)s/%(title)s - (%(id)s) - %(uploader)s - %(upload_date)s.%(ext)s' https://www.youtube.com/watch?v=dQw4w9WgXcQ

FIXED (5th september) With grep command i named the json file "*.json" because there will only be one per directory but I dont like that solution. (Could be answered with point above)
FIXED (5th september) How to make so grep dont grab "? It now adds them before and after everything.
FIXED (5th september) How to get the tags information from json file? Tags look like this:

    "tags": ["music", "video", "classic"]

FIXED (5th september) Run the creation of info file part of the script in the background of downloading the video?

CURRENT VERSION TRYING TO GET IT WORKING

(12 august)

    textfile=""$folder"info.txt"
    
    echo TITLE >> ~/Downloads/ytdl/dog_vids/info.txt
    youtube-dl -e $url >> ~/Downloads/ytdl/dog_vids/info.txt
    echo \ >> ~/Downloads/ytdl/dog_vids/info.txt
    
    echo CHANNEL >> $textfile
    echo \ >> $textfile
    
    echo CHANNEL URL >> $textfile
    echo \ >> $textfile
    
    echo UPLOAD DATE >> $textfile
    echo \ >> $textfile
    
    echo URL >> $textfile
    echo $url >> $textfile
    echo \ >> $textfile
    
    echo TAGS >> $textfile
    echo \ >> $textfile
    
    echo DESCRIPTION >> $textfile
    youtube-dl --get-description $url >> $textfile

EXPERIMENT FUTURE VERSION - EXTRACTING INFORMATION FROM JSON FILE

This isnt a working script. Showing how I want it with $textfile, $ytdlfolder and $jsonfile.

    url=https://www.youtube.com/watch?v=dQw4w9WgXcQ
    
    textfile=""$folder""$YOUTUBE-DL_PATH"info.txt"
    ytdlfolder="$folder""$YOUTUBE-DL_PATH"
    jsonfile="$folder""$YOUTUBE-DL_JSON-FILE"

    Echo TITLE >> $textfile
    grep -o '"title": *"[^"]*"' $jsonfile | grep -o '"[^"]*"$' >> $textfile
    Echo \ >> $textfile
    
    Echo CHANNEL >> $textfile
    grep -o '"uploader": *"[^"]*"' $jsonfile | grep -o '"[^"]*"$' >> $textfile
    Echo \ >> $textfile
    
    Echo CHANNEL URL >> $textfile
    grep -o '"uploader_url": *"[^"]*"' *.json | grep -o '"[^"]*"$' >> $textfile
    Echo \ >> $textfile
    
    Echo UPLOAD DATE >> $textfile
    grep -o '"upload_date": *"[^"]*"' *.json | grep -o '"[^"]*"$' >> $textfile
    Echo \ >> $textfile
    
    Echo TAGS >> $textfile
    grep -o '"tags": *"[^"]*"' *.json | grep -o '"[^"]*"$' >> $textfile
    Echo \ >> $textfile
    
    echo URL >> $textfile
    echo $url >> $textfile
    echo \ >> $textfile
    
    Echo DESCRIPTION >> $textfile
    youtube-dl --get-description $url >> $textfile

THE SCRIPT:

12 august.

Moved url to the top so when user paste the url they get the videos title back. This so the user know they got the right video.
Added max resolution 1920x1080. (Do not work)

13 august.

Downloading Audio & Video works.

31 august.

Fixed force mp4 and max heigh of 1080.

5 september.

Finally working script. Read more about it here (Or scroll down): Youtube-dl download script debug

2020-09-17

Folders can now have spaces in them.

2020-09-22

Select menus is now one column.
Minor fixes.
Now all the bugs is fixed. Issues left is only optimizations.

    #! /bin/bash
    ################################################################################
    # Script Name: Youtube-dl Easy Download Script
    # Description: Easy to use script to download YouTube videos with a couple of
    #              options.
    #
    # What this script do:
    #   - Downloads video in MP4 with highest quality and max resolution 1920x1080.
    #   - Downloads thumbnail and subtitles.
    #   - Gives user option where to download the video and video or only audio.
    #   - Creates a folder with same name as video title and puts all files there.
    #   - Creates a .txt file with information about the video.
    #
    #
    # Author:      Wogol - Stackoverflow.com, Github.com
    # License:     The GNU General Public License v3.0 - GNU GPL-3
    #
    # 
    # Big thanks to the people at youtube-dl GitHub and Stack Overflow. Without
    # their help this would never ever been possible for me.
    #
    # Special thanks to:
    #              Reino @ Stack Overflow
    #
    # #####
    #
    # Software required:                 youtube-dl, xidel, printf
    #
    # macOS:       1. Install Homebrew:  https://brew.sh
    #              2. Terminal command:  brew install youtube-dl xidel
    #
    # Linux:       Depends on package manager your distribution use.
    #
    # #####
    #
    # Version history:
    # 2020-09-22
    #   - Select menus is now one column.
    #   - Minor fixes.
    #   - Now all the bugs is fixed. Issues left is only optimizations.
    #
    # 2020-09-17
    #   - Folders can now have spaces in them.
    #
    # 2020-09-05
    #   - First working version.
    #
    # #####
    #
    # Issues left:
    #   - In the beginning there is a confirmation that show the title of the
    #     video so user know they got the correct video. It takes youtube-dl a
    #     couple of seconds. To speed up the script it is DISABLED by default.
    #
    #   - Have found out that the script dont need xidel to get json information
    #     but youtube-dl can get it. Dont know how to use youtube-dl --dump-json
    #     to get the same result.
    #
    #   - To get the path to the .txt file script use youtube-dl. This gives the
    #     script a pause for a few seconds. Best would get to get the path some how
    #     without connecting to YouTube again but use the output from youtube-dl
    #     some how. ... or run it in the background when video is downloading.
    #
    ################################################################################



    clear


    
    # - WELCOME MESSAGE -

    echo

    COLUMNS=$(tput cols)
    title="-= Youtube-dl Easy Download Script =-" 
    printf "%*s\n" $(((${#title}+$COLUMNS)/2)) "$title"



    # - PASTE URL -

    echo -e "\n*** - Paste URL address and hit RETURN. Example:\nhttps://www.youtube.com/watch?v=dQw4w9WgXcQ --OR-- https://youtu.be/dQw4w9WgXcQ\n"

    read url



    # - VIDEO TITLE -

    # So users know they have the correct URL.

    #echo -e "\nThe video is: (This takes 3-4 seconds, or more ...)"
    #youtube-dl -e $url
    #echo



    # - DOWNLOAD LOCATION -

    
    # DIRECTORY MUST END WITH SLASH: /


    echo -e "\n\n*** - Choose download folder:\n"
    
    COLUMNS=0
    PS3='Choose: '
    select directory in "~/Downloads/ytdl/Rick Astley/" "~/Downloads/ytdl/Never Gonna Give You Up/" "~/Downloads/ytdl/Other Rick Videos/" ; do
    echo -e "\nOption $REPLY selected. Download directory is:\n $directory"



    # - AUDIO/VIDEO SETTINGS -

    echo -e "\n\n*** - Choose download settings:\n"

    COLUMNS=0
    PS3='Choose: '
    options=("Audio & Video" "Audio only")
    select settingsopt in "${options[@]}"
        do
    case $settingsopt in

    "Audio & Video")
    av="-f bestvideo[ext=mp4][height<=1080]+bestaudio[ext=m4a]/best[ext=mp4]/best --merge-output-format mp4"

    ;;

    "Audio only")
    av="-f bestaudio[ext=m4a]/bestaudio"

    ;;

    esac

    echo -e "\nOption $REPLY selected:\n $settingsopt"



    # - THE DOWNLOAD SCRIPT -

    echo -e "\n\n*** - Starting download:\n"
    youtube-dl $av --write-thumbnail --all-subs --restrict-filenames -o "$directory%(title)s/%(title)s.%(ext)s" $url



    # - INFORMATION FILE -

    textfile=$(youtube-dl --get-filename --restrict-filenames -o "$directory%(title)s/%(title)s.txt" $url)

    xidel -s "$url" -e '
    let $json:=json(
        //script/extract(.,"ytplayer.config = (.+?\});",1)[.]
      )/args,
      $a:=json($json/player_response)/videoDetails,
      $b:=json($json/player_response)/microformat
    return (
    "- TITLE -",
    $a/title,"",
    "- CHANNEL -",
    $a/author,"",
    "- CHANNEL URL -",
    $b//ownerProfileUrl,"",
    "- UPLOAD DATE -",
    $b//publishDate,"",
    "- URL -",
    $json/loaderUrl,"",
    "- TAGS -",
    $a/keywords,"",
    "- DESCRIPTION -",
    $a/shortDescription
    )
    ' --printed-json-format=compact >> "$textfile"



    # - THE END -
    
    echo
    COLUMNS=$(tput cols) 
    ending="Download Complete!"
    printf "%*s\n\n" $(((${#ending}+$COLUMNS)/2)) "$ending"
    
    exit
    
    done
    done

Upvotes: 1

Answers (3)

Wogol

Reputation: 41

Finally got the script working.

Have got a lot of help from many people but big thanks to Reino with his help in this thread: Grep command questions - Grep text from program output?

The script has issues and it can be optimized but I dont know how to fix them. This is the first bash script I have created.

The goals with this was to create a script that:

Simple and easy to use.
No terminal commands.
Initial sorting in different directories.
Video or only audio.
MP4 with max resolution 1920x1080 because everything supports it out of the box.
Text file with additional information about the video.

These are features I miss in programs like Downie (macOS) and Clipgrab.

For other people to use this script and future fixing I tried to create a Github page... not my cup of tea so to say.

Script is in the first post on this page.

Youtube-dl download script debug

Upvotes: 1

Wogol

Reputation: 41

Have now fixed so the script force mp4 and max height of 1080.

    -f bestvideo[ext=mp4][height<=1080]+bestaudio[ext=m4a]/best[ext=mp4]/best --merge-output-format mp4

Now the rest of the issues left.

Upvotes: 0

Wogol

Reputation: 41

I fixed so Audio & Video download now works. The problem was ' in the av line. Removed them and now it works fine. Also updated the av line from the man/manual for youtube-dl.

Not working:

    av="-f 'bestvideo[ext=mp4]+bestaudio[ext=m4a]/bestvideo+bestaudio' --merge-output-format mp4"

Working:

    av="-f bestvideo[ext=mp4]+bestaudio[ext=m4a]/best[ext=mp4]/best --merge-output-format mp4"

Upvotes: 0

Youtube-dl download script debug

Answers (3)

Related Questions