updated documentation #85

add option to apply metadata in existing file.

Apologies for the late merge, you sent this pull request right as school was beginning to pick up in earnest and I forgot about it in that rush. Thanks for the great work!

.gitignore

@@ -11,4 +11,5 @@
 .ripper.log
 ffmpeg
 ffprobe
-youtube-dl
+youtube-dl
+*.temp

README.md

@@ -54,6 +54,8 @@ Arguments:
     -s, --song <song>           Specify song name to download
     -A, --album <album>         Specify the album name to download
     -p, --playlist <playlist>   Specify the playlist name to download
+    -u, --url <url>             Specify the youtube url to download from (for single songs only)
+    -g, --give-url              Specify the youtube url sources while downloading (for albums or playlists only only)
 
 Examples:
     $ irs --song "Bohemian Rhapsody" --artist "Queen"
@@ -75,6 +77,8 @@ Examples:
 Just download the latest release for your platform 
 [here](https://github.com/cooperhammond/irs/releases).
 
+Note that the binaries right now have only been tested on WSL. They *should* run on most linux distros, and OS X, but if they don't please make an issue above.
+
 ### From Source
 
 If you're one of those cool people who compiles from source
@@ -92,6 +96,8 @@ If you're one of those cool people who compiles from source
     ```yaml
     binary_directory: ~/.irs/bin
     music_directory: ~/Music
+    filename_pattern: "{track_number} - {title}"
+    directory_pattern: "{artist}/{album}"
     client_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
     client_secret: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
     single_folder_playlist:
@@ -118,6 +124,9 @@ Here's what they do:
 ```yaml
 binary_directory: ~/.irs/bin
 music_directory: ~/Music
+search_terms: "lyrics"
+filename_pattern: "{track_number} - {title}"
+directory_pattern: "{artist}/{album}"
 client_key: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
 client_secret: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
 single_folder_playlist:
@@ -128,8 +137,10 @@ single_folder_playlist:
  - `binary_directory`: a path specifying where the downloaded binaries should
     be placed
  - `music_directory`: a path specifying where downloaded mp3s should be placed.
-    Note that there will be more structure created inside that folder, usually
-    in the format of `music-dir>artist-name>album-name>track`
+ - `search_terms`: additional search terms to plug into youtube, which can be
+    potentially useful for not grabbing erroneous audio.
+ - `filename_pattern`: a pattern for the output filename of the mp3
+ - `directory_pattern`: a pattern for the folder structure your mp3s are saved in
  - `client_key`: a client key from your spotify API application
  - `client_secret`: a client secret key from your spotify API application
  - `single_folder_playlist/enabled`: if set to true, all mp3s from a downloaded
@@ -141,6 +152,55 @@ single_folder_playlist:
     the album name and album image of the mp3 with the title of your playlist
     and the image for your playlist respectively
 
+
+In a pattern following keywords will be replaced:
+
+| Keyword | Replacement | Example |
+| :----: | :----: | :----: |
+| `{artist}` | Artist Name | Queen |
+| `{title}` | Track Title | Bohemian Rhapsody |
+| `{album}` | Album Name | Stone Cold Classics |
+| `{track_number}` | Track Number | 9 |
+| `{total_tracks}` | Total Tracks in Album | 14 |
+| `{disc_number}` | Disc Number | 1 |
+| `{day}` | Release Day | 01 |
+| `{month}` | Release Month | 01 |
+| `{year}` | Release Year | 2006 |
+| `{id}` | Spotify ID | 6l8GvAyoUZwWDgF1e4822w |
+
+Beware OS-restrictions when naming your mp3s.
+
+Pattern Examples:
+```yaml
+music_directory: ~/Music
+filename_pattern: "{track_number} - {title}"
+directory_pattern: "{artist}/{album}"
+```
+Outputs: `~/Music/Queen/Stone Cold Classics/9 - Bohemian Rhapsody.mp3`
+<br><br>
+```yaml
+music_directory: ~/Music
+filename_pattern: "{artist} - {title}"
+directory_pattern: ""
+```
+Outputs: `~/Music/Queen - Bohemian Rhapsody.mp3`
+<br><br>
+```yaml
+music_directory: ~/Music
+filename_pattern: "{track_number} of {total_tracks} - {title}"
+directory_pattern: "{year}/{artist}/{album}"
+```
+Outputs: `~/Music/2006/Queen/Stone Cold Classics/9 of 14 - Bohemian Rhapsody.mp3`
+<br><br>
+```yaml
+music_directory: ~/Music
+filename_pattern: "{track_number}. {title}"
+directory_pattern: "irs/{artist} - {album}"
+```
+Outputs: `~/Music/irs/Queen - Stone Cold Classics/9. Bohemian Rhapsody.mp3`
+<br>
+
+
 ## How it works
 
 **At it's core** `irs` downloads individual songs. It does this by interfacing
@@ -174,4 +234,4 @@ doing here, _pretty please_ shoot me an email.
 2. Create your feature branch (`git checkout -b my-new-feature`)
 3. Commit your changes (`git commit -am 'Add some feature'`)
 4. Push to the branch (`git push origin my-new-feature`)
-5. Create a new Pull Request
+5. Create a new Pull Request

shard.lock

@@ -1,6 +1,10 @@
-version: 1.0
+version: 2.0
 shards:
-  ydl_binaries:
-    github: cooperhammond/ydl-binaries
-    commit: c82e3937fee20fd076b1c73e24b2d0205e2cf0da
+  json_mapping:
+    git: https://github.com/crystal-lang/json_mapping.cr.git
+    version: 0.1.0
+
+  ydl_binaries:
+    git: https://github.com/cooperhammond/ydl-binaries.git
+    version: 1.1.1+git.commit.c82e3937fee20fd076b1c73e24b2d0205e2cf0da
 

shard.yml

@@ -1,5 +1,5 @@
 name: irs
-version: 1.0.0
+version: 1.4.0
 
 authors:
   - Cooper Hammond <kepoorh@gmail.com>
@@ -12,4 +12,6 @@ license: MIT
 
 dependencies:
   ydl_binaries:
-    github: cooperhammond/ydl-binaries
+    github: cooperhammond/ydl-binaries
+  json_mapping:
+    github: crystal-lang/json_mapping.cr

spec/irs_spec.cr

@@ -1,9 +1,35 @@
 require "./spec_helper"
 
-describe Irs do
+describe CLI do
   # TODO: Write tests
 
-  it "works" do
-    false.should eq(true)
+  it "can show help" do
+    run_CLI_with_args(["--help"])
+  end
+
+  it "can show version" do
+    run_CLI_with_args(["--version"])
+  end
+
+  # !!TODO: make a long and short version of the test suite
+  # TODO: makes so this doesn't need user input
+  it "can install ytdl and ffmpeg binaries" do
+    # run_CLI_with_args(["--install"])
+  end
+
+  it "can show config file loc" do
+    run_CLI_with_args(["--config"])
+  end
+
+  it "can download a single song" do
+    run_CLI_with_args(["--song", "Bohemian Rhapsody", "--artist", "Queen"])
+  end
+
+  it "can download an album" do
+    run_CLI_with_args(["--artist", "Arctic Monkeys", "--album", "Da Frame 2R / Matador"])
+  end
+
+  it "can download a playlist" do
+    run_CLI_with_args(["--artist", "prakkillian", "--playlist", "IRS Testing"])
   end
 end

spec/spec_helper.cr

@@ -1,2 +1,10 @@
 require "spec"
-require "../src/irs"
+
+# https://github.com/mosop/stdio
+
+require "../src/bottle/cli"
+
+def run_CLI_with_args(argv : Array(String))
+    cli = CLI.new(argv)
+    cli.act_on_args
+end

src/bottle/cli.cr

@@ -20,6 +20,10 @@ class CLI
     [["-s", "--song"], "song", "string"],
     [["-A", "--album"], "album", "string"],
     [["-p", "--playlist"], "playlist", "string"],
+    [["-u", "--url"], "url", "string"],
+    [["-S", "--select"], "select", "bool"],
+    [["--ask-skip"], "ask_skip", "bool"],
+    [["--apply"], "apply_file", "string"]
   ]
 
   @args : Hash(String, String)
@@ -48,6 +52,12 @@ class CLI
         #{Style.blue "-s, --song <song>"}           Specify song name to download
         #{Style.blue "-A, --album <album>"}         Specify the album name to download
         #{Style.blue "-p, --playlist <playlist>"}   Specify the playlist name to download
+        #{Style.blue "-u, --url <url>"}             Specify the youtube url to download from
+        #{Style.blue "                 "}           (for albums and playlists, the command-line
+        #{Style.blue "                 "}           argument is ignored, and it should be '')
+        #{Style.blue "-S, --select"}                Use a menu to choose each song's video source
+        #{Style.blue "--ask-skip"}                  Before every playlist/album song, ask to skip
+        #{Style.blue "--apply <file>"}              Apply metadata to a existing file
 
     #{Style.bold "Examples:"}
         $ #{Style.green %(irs --song "Bohemian Rhapsody" --artist "Queen")}
@@ -69,34 +79,35 @@ class CLI
 
     if @args["help"]? || @args.keys.size == 0
       help
-      exit
+
     elsif @args["version"]?
       version
-      exit
+
     elsif @args["install"]?
       YdlBinaries.get_both(Config.binary_location)
-      exit
+
     elsif @args["config"]?
       puts ENV["IRS_CONFIG_LOCATION"]?
-      exit
+
     elsif @args["song"]? && @args["artist"]?
       s = Song.new(@args["song"], @args["artist"])
       s.provide_client_keys(Config.client_key, Config.client_secret)
-      s.grab_it
-      s.organize_it(Config.music_directory)
-      exit
+      s.grab_it(flags: @args)
+      s.organize_it()
+
     elsif @args["album"]? && @args["artist"]?
       a = Album.new(@args["album"], @args["artist"])
       a.provide_client_keys(Config.client_key, Config.client_secret)
-      a.grab_it
+      a.grab_it(flags: @args)
+
     elsif @args["playlist"]? && @args["artist"]?
       p = Playlist.new(@args["playlist"], @args["artist"])
       p.provide_client_keys(Config.client_key, Config.client_secret)
-      p.grab_it
+      p.grab_it(flags: @args)
+
     else
       puts Style.red("Those arguments don't do anything when used that way.")
       puts "Type `irs -h` to see usage."
-      exit 1
     end
   end
 

src/bottle/config.cr

@@ -7,8 +7,11 @@ require "../search/spotify"
 EXAMPLE_CONFIG = <<-EOP
 #{Style.dim "exampleconfig.yml"}
 #{Style.dim "===="}
+#{Style.blue "search_terms"}: #{Style.green "\"lyrics\""}
 #{Style.blue "binary_directory"}: #{Style.green "~/.irs/bin"}
 #{Style.blue "music_directory"}: #{Style.green "~/Music"}
+#{Style.blue "filename_pattern"}: #{Style.green "\"{track_number} - {title}\""}
+#{Style.blue "directory_pattern"}: #{Style.green "\"{artist}/{album}\""}
 #{Style.blue "client_key"}: #{Style.green "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"}
 #{Style.blue "client_secret"}: #{Style.green "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"}
 #{Style.blue "single_folder_playlist"}: 
@@ -22,8 +25,11 @@ module Config
   extend self
 
   @@arguments = [
+    "search_terms",
     "binary_directory",
     "music_directory",
+    "filename_pattern",
+    "directory_pattern",
     "client_key",
     "client_secret",
     "single_folder_playlist: enabled",
@@ -41,6 +47,10 @@ module Config
     exit 1
   end
 
+  def search_terms : String
+    return @@conf["search_terms"].to_s
+  end
+
   def binary_location : String
     path = @@conf["binary_directory"].to_s
     return Path[path].expand(home: true).to_s
@@ -50,6 +60,14 @@ module Config
     path = @@conf["music_directory"].to_s
     return Path[path].expand(home: true).to_s
   end
+  
+  def filename_pattern : String
+    return @@conf["filename_pattern"].to_s
+  end
+  
+  def directory_pattern : String
+    return @@conf["directory_pattern"].to_s
+  end
 
   def client_key : String
     return @@conf["client_key"].to_s

src/bottle/pattern.cr

@@ -0,0 +1,28 @@
+module Pattern
+  extend self
+
+  def parse(formatString : String, metadata : JSON::Any)
+    formatted : String = formatString
+
+    date : Array(String) = (metadata["album"]? || JSON.parse("{}"))["release_date"]?.to_s.split('-')
+
+    keys : Hash(String, String) = {
+      "artist" => ((metadata.dig?("artists") || JSON.parse("{}"))[0]? || JSON.parse("{}"))["name"]?.to_s,
+      "title" => metadata["name"]?.to_s,
+      "album" => (metadata["album"]? || JSON.parse("{}"))["name"]?.to_s,
+      "track_number" => metadata["track_number"]?.to_s,
+      "disc_number" => metadata["disc_number"]?.to_s,
+      "total_tracks" => (metadata["album"]? || JSON.parse("{}"))["total_tracks"]?.to_s,
+      "year" => date[0]?.to_s,
+      "month" => date[1]?.to_s,
+      "day" => date[2]?.to_s,
+      "id" => metadata["id"]?.to_s
+    }
+
+    keys.each do |pair|
+      formatted = formatted.gsub("{#{pair[0]}}", pair[1] || "")
+    end
+
+    return formatted
+  end
+end

src/bottle/version.cr

@@ -1,3 +1,3 @@
 module IRS
-  VERSION = "0.1.0"
+  VERSION = "1.4.0"
 end

src/glue/album.cr

@@ -9,7 +9,7 @@ class Album < SpotifyList
 
   # Uses the `spotify_searcher` defined in parent `SpotifyList` to find the
   # correct metadata of the list
-  def find_it
+  def find_it : JSON::Any
     album = @spotify_searcher.find_item("album", {
       "name"   => @list_name.as(String),
       "artist" => @list_author.as(String),
@@ -42,6 +42,6 @@ class Album < SpotifyList
   end
 
   private def organize(song : Song)
-    song.organize_it(@home_music_directory)
+    song.organize_it()
   end
 end

src/glue/list.cr

@@ -17,6 +17,9 @@ abstract class SpotifyList
     "searching" => [
       Style.bold("Searching for %l by %a ... \r"),
       Style.green("+ ") + Style.bold("%l by %a                                 \n")
+    ],
+    "url" => [
+      Style.bold("When prompted for a URL, provide a youtube URL or press enter to scrape for one\n")
     ]
   }
 
@@ -24,11 +27,19 @@ abstract class SpotifyList
   end
 
   # Finds the list, and downloads all of the songs using the `Song` class
-  def grab_it
+  def grab_it(flags = {} of String => String)
+    ask_url = flags["url"]?
+    ask_skip = flags["ask_skip"]?
+    is_playlist = flags["playlist"]?
+  
     if !@spotify_searcher.authorized?
       raise("Need to call provide_client_keys on Album or Playlist class.")
     end
 
+    if ask_url
+      outputter("url", 0)
+    end
+
     outputter("searching", 0)
     list = find_it()
     outputter("searching", 1)
@@ -36,22 +47,28 @@ abstract class SpotifyList
 
     i = 0
     contents.each do |datum|
+      i += 1
       if datum["track"]?
         datum = datum["track"]
       end
 
       data = organize_song_metadata(list, datum)
 
-      song = Song.new(data["name"].to_s, data["artists"][0]["name"].to_s)
+      s_name = data["name"].to_s
+      s_artist = data["artists"][0]["name"].to_s
+
+      song = Song.new(s_name, s_artist)
       song.provide_spotify(@spotify_searcher)
       song.provide_metadata(data)
 
-      puts Style.bold("[#{data["track_number"]}/#{contents.size}]")
-      song.grab_it
+      puts Style.bold("[#{i}/#{contents.size}]")
 
-      organize(song)
-
-      i += 1
+      unless ask_skip && skip?(s_name, s_artist, is_playlist)
+        song.grab_it(flags: flags)
+        organize(song)
+      else
+        puts "Skipping..."
+      end
     end
   end
 
@@ -60,6 +77,13 @@ abstract class SpotifyList
     @spotify_searcher.authorize(client_key, client_secret)
   end
 
+  private def skip?(name, artist, is_playlist)
+    print "Skip #{Style.blue name}" +
+      (is_playlist ? " (by #{Style.green artist})": "") + "? "
+    response = gets
+    return response && response.lstrip.downcase.starts_with? "y"
+  end
+
   private def outputter(key : String, index : Int32)
     text = @outputs[key][index]
       .gsub("%l", @list_name)

src/glue/mapper.cr

@@ -1,4 +1,5 @@
 require "json"
+require "json_mapping"
 
 class PlaylistExtensionMapper
   JSON.mapping(
@@ -45,6 +46,7 @@ class TrackMapper
       type: Int32,
       setter: true
     },
+    duration_ms: Int32,
     type: String,
     uri: String
   )

src/glue/playlist.cr

@@ -13,7 +13,7 @@ class Playlist < SpotifyList
 
   # Uses the `spotify_searcher` defined in parent `SpotifyList` to find the
   # correct metadata of the list
-  def find_it
+  def find_it : JSON::Any
     @playlist = @spotify_searcher.find_item("playlist", {
       "name"     => @list_name.as(String),
       "username" => @list_author.as(String),
@@ -67,9 +67,10 @@ class Playlist < SpotifyList
         FileUtils.mkdir_p(strpath)
       end
       safe_filename = song.filename.gsub(/[\/]/, "").gsub("  ", " ")
-      File.rename("./" + song.filename, (path / safe_filename).to_s)
+      FileUtils.cp("./" + song.filename, (path / safe_filename).to_s)
+      FileUtils.rm("./" + song.filename)
     else
-      song.organize_it(@home_music_directory)
+      song.organize_it()
     end
   end
 end

src/glue/song.cr

@@ -4,6 +4,8 @@ require "../search/youtube"
 require "../interact/ripper"
 require "../interact/tagger"
 
+require "../bottle/config"
+require "../bottle/pattern"
 require "../bottle/styles"
 
 class Song
@@ -24,7 +26,10 @@ class Song
     ],
     "url" => [
       "  Searching for URL ...\r",
-      Style.green("  + ") + Style.dim("URL found                       \n")
+      Style.green("  + ") + Style.dim("URL found                       \n"),
+      "  Validating URL ...\r",
+      Style.green("  + ") + Style.dim("URL validated                   \n"),
+      "  URL?: "
     ],
     "download" => [
       "  Downloading video:\n",
@@ -47,11 +52,16 @@ class Song
   end
 
   # Find, downloads, and tags the mp3 song that this class represents.
+  # Optionally takes a youtube URL to download from
   #
   # ```
   # Song.new("Bohemian Rhapsody", "Queen").grab_it
   # ```
-  def grab_it
+  def grab_it(url : (String | Nil) = nil, flags = {} of String => String)
+    passed_url : (String | Nil) = flags["url"]?
+    passed_file : (String | Nil) = flags["apply_file"]?
+    select_link = flags["select"]?
+
     outputter("intro", 0)
 
     if !@spotify_searcher.authorized? && !@metadata
@@ -79,43 +89,78 @@ class Song
     end
 
     data = @metadata.as(JSON::Any)
-    @filename = data["track_number"].to_s + " - #{data["name"].to_s}.mp3"
+    @song_name = data["name"].as_s
+    @artist_name = data["artists"][0]["name"].as_s
+    @filename = "#{Pattern.parse(Config.filename_pattern, data)}.mp3"
 
-    outputter("url", 0)
-    url = Youtube.find_url(@song_name, @artist_name, search_terms: "lyrics")
-    if !url
-      raise("There was no url found on youtube for " +
-            %("#{@song_name}" by "#{@artist_name}. ) +
-            "Check your input and try again.")
+    if passed_file
+      puts Style.green("  +") + Style.dim(" Moving file: ") + passed_file
+      File.rename(passed_file, @filename)
+    else
+      if passed_url
+        if passed_url.strip != ""
+          url = passed_url
+        else
+          outputter("url", 4)
+          url = gets
+          if !url.nil? && url.strip == ""
+            url = nil
+          end
+        end
+      end
+
+      if !url
+        outputter("url", 0)
+        url = Youtube.find_url(data, flags: flags)
+        if !url
+          raise("There was no url found on youtube for " +
+                %("#{@song_name}" by "#{@artist_name}. ) +
+                "Check your input and try again.")
+        end
+        outputter("url", 1)
+      else
+        outputter("url", 2)
+        url = Youtube.validate_url(url)
+        if !url
+          raise("The url is an invalid youtube URL " +
+                "Check the URL and try again")
+        end
+        outputter("url", 3)
+      end
+
+      outputter("download", 0)
+      Ripper.download_mp3(url.as(String), @filename)
+      outputter("download", 1)
     end
-    outputter("url", 1)
-
-    outputter("download", 0)
-    Ripper.download_mp3(url.as(String), @filename)
-    outputter("download", 1)
 
     outputter("albumart", 0)
     temp_albumart_filename = ".tempalbumart.jpg"
-    HTTP::Client.get(data["album"]["images"][0]["url"].to_s) do |response|
+    HTTP::Client.get(data["album"]["images"][0]["url"].as_s) do |response|
       File.write(temp_albumart_filename, response.body_io)
     end
     outputter("albumart", 0)
 
     # check if song's metadata has been modded in playlist, update artist accordingly
-    if data["artists"][-1]["owner"]? 
-      @artist = data["artists"][-1]["name"].to_s
+    if data["artists"][-1]["owner"]?
+      @artist = data["artists"][-1]["name"].as_s
     else
-      @artist = data["artists"][0]["name"].to_s
+      @artist = data["artists"][0]["name"].as_s
     end
-    @album = data["album"]["name"].to_s
+    @album = data["album"]["name"].as_s
 
     tagger = Tags.new(@filename)
     tagger.add_album_art(temp_albumart_filename)
-    tagger.add_text_tag("title", data["name"].to_s)
+    tagger.add_text_tag("title", data["name"].as_s)
     tagger.add_text_tag("artist", @artist)
-    tagger.add_text_tag("album", @album)
-    tagger.add_text_tag("genre", 
-      @spotify_searcher.find_genre(data["artists"][0]["id"].to_s))
+
+    if !@album.empty?
+      tagger.add_text_tag("album", @album)
+    end
+
+    if genre = @spotify_searcher.find_genre(data["artists"][0]["id"].as_s)
+      tagger.add_text_tag("genre", genre)
+    end
+
     tagger.add_text_tag("track", data["track_number"].to_s)
     tagger.add_text_tag("disc", data["disc_number"].to_s)
 
@@ -127,26 +172,31 @@ class Song
     outputter("finished", 0)
   end
 
-  # Will organize the song into the user's provided music directory as
-  # music_directory > artist_name > album_name > song
+  # Will organize the song into the user's provided music directory
+  # in the user's provided structure
   # Must be called AFTER the song has been downloaded.
   #
   # ```
   # s = Song.new("Bohemian Rhapsody", "Queen").grab_it
-  # s.organize_it("/home/cooper/Music")
-  # # Will move the mp3 file to
+  # s.organize_it()
+  # # With
+  # # directory_pattern = "{artist}/{album}"
+  # # filename_pattern = "{track_number} - {title}"
+  # # Mp3 will be moved to
   # # /home/cooper/Music/Queen/A Night At The Opera/1 - Bohemian Rhapsody.mp3
   # ```
-  def organize_it(music_directory : String)
-    path = Path[music_directory].expand(home: true)
-    path = path / @artist_name.gsub(/[\/]/, "").gsub("  ", " ")
-    path = path / @album.gsub(/[\/]/, "").gsub("  ", " ")
+  def organize_it()
+    path = Path[Config.music_directory].expand(home: true)
+    Pattern.parse(Config.directory_pattern, @metadata.as(JSON::Any)).split('/').each do |dir|
+      path = path / dir.gsub(/[\/]/, "").gsub("  ", " ")
+    end
     strpath = path.to_s
     if !File.directory?(strpath)
       FileUtils.mkdir_p(strpath)
     end
     safe_filename = @filename.gsub(/[\/]/, "").gsub("  ", " ")
-    File.rename("./" + @filename, (path / safe_filename).to_s)
+    FileUtils.cp("./" + @filename, (path / safe_filename).to_s)
+    FileUtils.rm("./" + @filename)
   end
 
   # Provide metadata so that it doesn't have to find it. Useful for overwriting

src/interact/future.cr

@@ -0,0 +1,157 @@
+# copy and pasted from crystal 0.33.1
+# https://github.com/crystal-lang/crystal/blob/18e76172444c7bd07f58bf360bc21981b667668d/src/concurrent/future.cr#L138
+
+
+# :nodoc:
+class Concurrent::Future(R)
+  enum State
+    Idle
+    Delayed
+    Running
+    Completed
+    Canceled
+  end
+
+  @value : R?
+  @error : Exception?
+  @delay : Float64
+
+  def initialize(run_immediately = true, delay = 0.0, &@block : -> R)
+    @state = State::Idle
+    @value = nil
+    @error = nil
+    @channel = Channel(Nil).new
+    @delay = delay.to_f
+    @cancel_msg = nil
+
+    spawn_compute if run_immediately
+  end
+
+  def get
+    wait
+    value_or_raise
+  end
+
+  def success?
+    completed? && !@error
+  end
+
+  def failure?
+    completed? && @error
+  end
+
+  def canceled?
+    @state == State::Canceled
+  end
+
+  def completed?
+    @state == State::Completed
+  end
+
+  def running?
+    @state == State::Running
+  end
+
+  def delayed?
+    @state == State::Delayed
+  end
+
+  def idle?
+    @state == State::Idle
+  end
+
+  def cancel(msg = "Future canceled, you reached the [End of Time]")
+    return if @state >= State::Completed
+    @state = State::Canceled
+    @cancel_msg = msg
+    @channel.close
+    nil
+  end
+
+  private def compute
+    return if @state >= State::Delayed
+    run_compute
+  end
+
+  private def spawn_compute
+    return if @state >= State::Delayed
+
+    @state = @delay > 0 ? State::Delayed : State::Running
+
+    spawn { run_compute }
+  end
+
+  private def run_compute
+    delay = @delay
+
+    if delay > 0
+      sleep delay
+      return if @state >= State::Canceled
+      @state = State::Running
+    end
+
+    begin
+      @value = @block.call
+    rescue ex
+      @error = ex
+    ensure
+      @channel.close
+      @state = State::Completed
+    end
+  end
+
+  private def wait
+    return if @state >= State::Completed
+    compute
+    @channel.receive?
+  end
+
+  private def value_or_raise
+    raise Exception.new(@cancel_msg) if @state == State::Canceled
+
+    value = @value
+    if value.is_a?(R)
+      value
+    elsif error = @error
+      raise error
+    else
+      raise "compiler bug"
+    end
+  end
+end
+
+# Spawns a `Fiber` to compute *&block* in the background after *delay* has elapsed.
+# Access to get is synchronized between fibers.  *&block* is only called once.
+# May be canceled before *&block* is called by calling `cancel`.
+# ```
+# d = delay(1) { Process.kill(Process.pid) }
+# long_operation
+# d.cancel
+# ```
+def delay(delay, &block : -> _)
+  Concurrent::Future.new delay: delay, &block
+end
+
+# Spawns a `Fiber` to compute *&block* in the background.
+# Access to get is synchronized between fibers.  *&block* is only called once.
+# ```
+# f = future { http_request }
+# ... other actions ...
+# f.get #=> String
+# ```
+def future(&exp : -> _)
+  Concurrent::Future.new &exp
+end
+
+# Conditionally spawns a `Fiber` to run *&block* in the background.
+# Access to get is synchronized between fibers.  *&block* is only called once.
+# *&block* doesn't run by default, only when `get` is called.
+# ```
+# l = lazy { expensive_computation }
+# spawn { maybe_use_computation(l) }
+# spawn { maybe_use_computation(l) }
+# ```
+def lazy(&block : -> _)
+  Concurrent::Future.new run_immediately: false, &block
+end
+

src/interact/logger.cr

@@ -1,3 +1,5 @@
+require "./future"
+
 class Logger
   @done_signal = "---DONE---"
 

src/search/ranking.cr

@@ -0,0 +1,144 @@
+alias VID_VALUE_CLASS = String
+alias VID_METADATA_CLASS = Hash(String, VID_VALUE_CLASS)
+alias YT_METADATA_CLASS = Array(VID_METADATA_CLASS)
+
+module Ranker
+  extend self
+
+  GARBAGE_PHRASES = [
+    "cover", "album", "live", "clean", "version", "full", "full album", "row",
+    "at", "@", "session", "how to", "npr music", "reimagined", "version",
+    "trailer"
+  ]
+
+  GOLDEN_PHRASES = [
+    "official video", "official music video",
+  ]
+
+  # Will rank videos according to their title and the user input, returns a sorted array of hashes
+  # of the points a song was assigned and its original index
+  # *spotify_metadata* is the metadate (from spotify) of the song that you want
+  # *yt_metadata* is an array of hashes with metadata scraped from the youtube search result page
+  # *query* is the query that you submitted to youtube for the results you now have
+  # ```
+  # Ranker.rank_videos(spotify_metadata, yt_metadata, query)
+  # => [
+  #      {"points" => x, "index" => x},
+  #      ...
+  #    ]
+  # ```
+  # "index" corresponds to the original index of the song in yt_metadata
+  def rank_videos(spotify_metadata : JSON::Any, yt_metadata : YT_METADATA_CLASS,
+                  query : String) : Array(Hash(String, Int32))
+    points = [] of Hash(String, Int32)
+    index = 0
+
+    actual_song_name = spotify_metadata["name"].as_s
+    actual_artist_name = spotify_metadata["artists"][0]["name"].as_s
+
+    yt_metadata.each do |vid|
+      pts = 0
+
+      pts += points_string_compare(actual_song_name, vid["title"])
+      pts += points_string_compare(actual_artist_name, vid["title"])
+      pts += count_buzzphrases(query, vid["title"])
+      pts += compare_timestamps(spotify_metadata, vid)
+
+      points.push({
+        "points" => pts,
+        "index"  => index,
+      })
+      index += 1
+    end
+
+    # Sort first by points and then by original index of the song
+    points.sort! { |a, b|
+      if b["points"] == a["points"]
+        a["index"] <=> b["index"]
+      else
+        b["points"] <=> a["points"]
+      end
+    }
+
+    return points
+  end
+
+  # SINGULAR COMPONENT OF RANKING ALGORITHM
+  private def compare_timestamps(spotify_metadata : JSON::Any, node : VID_METADATA_CLASS) : Int32
+    # puts spotify_metadata.to_pretty_json()
+    actual_time = spotify_metadata["duration_ms"].as_i
+    vid_time = node["duration_ms"].to_i
+
+    difference = (actual_time - vid_time).abs 
+
+    # puts "actual: #{actual_time}, vid: #{vid_time}"
+    # puts "\tdiff: #{difference}"
+    # puts "\ttitle: #{node["title"]}"
+
+    if difference <= 1000
+      return 3
+    elsif difference <= 2000
+      return 2
+    elsif difference <= 5000
+      return 1
+    else 
+      return 0
+    end
+  end
+
+  # SINGULAR COMPONENT OF RANKING ALGORITHM
+  # Returns an `Int` based off the number of points worth assigning to the
+  # matchiness of the string. First the strings are downcased and then all
+  # nonalphanumeric characters are stripped.
+  # If *item1* includes *item2*, return 3 pts.
+  # If after the items have been blanked, *item1* includes *item2*,
+  #   return 1 pts.
+  # Else, return 0 pts.
+  private def points_string_compare(item1 : String, item2 : String) : Int32
+    if item2.includes?(item1)
+      return 3
+    end
+
+    item1 = item1.downcase.gsub(/[^a-z0-9]/, "")
+    item2 = item2.downcase.gsub(/[^a-z0-9]/, "")
+
+    if item2.includes?(item1)
+      return 1
+    else
+      return 0
+    end
+  end
+
+  # SINGULAR COMPONENT OF RANKING ALGORITHM
+  # Checks if there are any phrases in the title of the video that would
+  # indicate audio having what we want.
+  # *video_name* is the title of the video, and *query* is what the user the
+  # program searched for. *query* is needed in order to make sure we're not
+  # subtracting points from something that's naturally in the title
+  private def count_buzzphrases(query : String, video_name : String) : Int32
+    good_phrases = 0
+    bad_phrases = 0
+
+    GOLDEN_PHRASES.each do |gold_phrase|
+      gold_phrase = gold_phrase.downcase.gsub(/[^a-z0-9]/, "")
+
+      if query.downcase.gsub(/[^a-z0-9]/, "").includes?(gold_phrase)
+        next
+      elsif video_name.downcase.gsub(/[^a-z0-9]/, "").includes?(gold_phrase)
+        good_phrases += 1
+      end
+    end
+
+    GARBAGE_PHRASES.each do |garbage_phrase|
+      garbage_phrase = garbage_phrase.downcase.gsub(/[^a-z0-9]/, "")
+
+      if query.downcase.gsub(/[^a-z0-9]/, "").includes?(garbage_phrase)
+        next
+      elsif video_name.downcase.gsub(/[^a-z0-9]/, "").includes?(garbage_phrase)
+        bad_phrases += 1
+      end
+    end
+
+    return good_phrases - bad_phrases
+  end
+end

src/search/spotify.cr

@@ -60,9 +60,10 @@ class SpotifySearcher
   # ```
   def find_item(item_type : String, item_parameters : Hash, offset = 0,
                 limit = 20) : JSON::Any?
-    query = generate_query(item_type, item_parameters, offset, limit)
+    query = generate_query(item_type, item_parameters)
 
-    url = @root_url.join("search?q=#{query}").to_s
+    url = "search?q=#{query}&type=#{item_type}&limit=#{limit}&offset=#{offset}"
+    url = @root_url.join(url).to_s
 
     response = HTTP::Client.get(url, headers: @access_header)
     error_check(response)
@@ -204,8 +205,14 @@ class SpotifySearcher
   # ```
   # SpotifySearcher.new.authorize(...).find_genre("1dfeR4HaWDbWqFHLkxsg1d")
   # ```
-  def find_genre(id : String) : String
-    genre = get_item("artist", id)["genres"][0].to_s
+  def find_genre(id : String) : String | Nil
+    genre = get_item("artist", id)["genres"]
+
+    if genre.as_a.empty?
+      return nil
+    end
+
+    genre = genre[0].to_s
     genre = genre.split(" ").map { |x| x.capitalize }.join(" ")
 
     return genre
@@ -222,8 +229,7 @@ class SpotifySearcher
 
   # Generates url to run a GET request against to the Spotify open API
   # Returns a `String.`
-  private def generate_query(item_type : String, item_parameters : Hash,
-                             offset : Int32, limit : Int32) : String
+  private def generate_query(item_type : String, item_parameters : Hash) : String
     query = ""
 
     # parameter keys to exclude in the api request. These values will be put
@@ -235,9 +241,9 @@ class SpotifySearcher
       if k == "name"
         # will remove the "name:<title>" param from the query
         if item_type == "playlist"
-          query += item_parameters[k].gsub(" ", "+") + "+"
+          query += item_parameters[k] + "+"
         else
-          query += param_encode(item_type, item_parameters[k])
+          query += as_field(item_type, item_parameters[k])
         end
 
         # check if the key is to be excluded
@@ -248,14 +254,21 @@ class SpotifySearcher
         # NOTE: playlist names will be inserted into the query normally, without
         # a parameter.
       else
-        query += param_encode(k, item_parameters[k])
+        query += as_field(k, item_parameters[k])
       end
     end
 
-    # extra api info
-    query += "&type=#{item_type}&limit=#{limit}&offset=#{offset}"
+    return URI.encode(query.rchop("+"))
+  end
 
-    return query
+  # Returns a `String` encoded for the spotify api
+  #
+  # ```
+  # query_encode("album", "A Night At The Opera")
+  # => "album:A Night At The Opera+"
+  # ```
+  private def as_field(key, value) : String
+    return "#{key}:#{value}+"
   end
 
   # Ranks the given items based off of the info from parameters.
@@ -321,15 +334,6 @@ class SpotifySearcher
     end
   end
 
-  # Returns a `String` encoded for the spotify api
-  #
-  # ```
-  # query_encode("album", "A Night At The Opera")
-  # => "album:A+Night+At+The+Opera"
-  # ```
-  private def param_encode(key : String, value : String) : String
-    return key.gsub(" ", "+") + ":" + value.gsub(" ", "+") + "+"
-  end
 end
 
 # puts SpotifySearcher.new()

src/search/youtube.cr

@@ -1,5 +1,13 @@
 require "http"
 require "xml"
+require "json"
+require "uri"
+
+require "./ranking"
+
+require "../bottle/config"
+require "../bottle/styles"
+
 
 module Youtube
   extend self
@@ -9,172 +17,184 @@ module Youtube
     "yt-uix-tile-link yt-ui-ellipsis yt-ui-ellipsis-2 yt-uix-sessionlink      spf-link ",
   ]
 
-  GARBAGE_PHRASES = [
-    "cover", "album", "live", "clean", "version", "full", "full album", "row",
-    "at", "@", "session", "how to", "npr music", "reimagined", "hr version",
-    "trailer",
-  ]
-
-  GOLDEN_PHRASES = [
-    "official video", "official music video",
-  ]
+  # Note that VID_VALUE_CLASS, VID_METADATA_CLASS, and YT_METADATA_CLASS are found in ranking.cr
 
   # Finds a youtube url based off of the given information.
   # The query to youtube is constructed like this:
   #   "<song_name> <artist_name> <search terms>"
   # If *download_first* is provided, the first link found will be downloaded.
+  # If *select_link* is provided, a menu of options will be shown for the user to choose their poison
   #
   # ```
   # Youtube.find_url("Bohemian Rhapsody", "Queen")
   # => "https://www.youtube.com/watch?v=dQw4w9WgXcQ"
   # ```
-  def find_url(song_name : String, artist_name : String, search_terms = "",
-               download_first = false) : String?
-    query = (song_name + " " + artist_name + " " + search_terms).strip.gsub(" ", "+")
+  def find_url(spotify_metadata : JSON::Any,
+               flags = {} of String => String) : String?
 
-    url = "https://www.youtube.com/results?search_query=" + query
+    search_terms = Config.search_terms
 
-    response = HTTP::Client.get(url)
+    select_link = flags["select"]?
 
-    valid_nodes = get_video_link_nodes(response.body)
+    song_name = spotify_metadata["name"].as_s
+    artist_name = spotify_metadata["artists"][0]["name"].as_s
 
-    if valid_nodes.size == 0
-      puts "There were no results for that query."
+    human_query = "#{song_name} #{artist_name} #{search_terms.strip}"
+    params = HTTP::Params.encode({"search_query" => human_query})
+
+    response = HTTP::Client.get("https://www.youtube.com/results?#{params}")
+
+    yt_metadata = get_yt_search_metadata(response.body)
+
+    if yt_metadata.size == 0
+      puts "There were no results for this query on youtube: \"#{human_query}\""
       return nil
     end
 
     root = "https://youtube.com"
+    ranked = Ranker.rank_videos(spotify_metadata, yt_metadata, human_query)
 
-    return root + valid_nodes[0]["href"] if download_first
-
-    ranked = rank_videos(song_name, artist_name, query, valid_nodes)
+    if select_link
+      return root + select_link_menu(spotify_metadata, yt_metadata)
+    end
 
     begin
-      return root + valid_nodes[ranked[0]["index"]]["href"]
+      puts Style.dim("  Video: ") + yt_metadata[ranked[0]["index"]]["title"]
+      return root + yt_metadata[ranked[0]["index"]]["href"]
     rescue IndexError
       return nil
     end
+
+    exit 1
   end
 
-  # Will rank videos according to their title and the user input
-  # Return:
-  # [
-  #   {"points" => x, "index" => x},
-  #   ...
-  # ]
-  private def rank_videos(song_name : String, artist_name : String,
-                          query : String, nodes : Array(XML::Node)) : Array(Hash(String, Int32))
-    points = [] of Hash(String, Int32)
-    index = 0
-
-    nodes.each do |node|
-      pts = 0
-
-      pts += points_compare(song_name, node["title"])
-      pts += points_compare(artist_name, node["title"])
-      pts += count_buzzphrases(query, node["title"])
-
-      points.push({
-        "points" => pts,
-        "index"  => index,
-      })
+  # Presents a menu with song info for the user to choose which url they want to download
+  private def select_link_menu(spotify_metadata : JSON::Any,
+                               yt_metadata : YT_METADATA_CLASS) : String
+    puts Style.dim("  Spotify info: ") +
+         Style.bold("\"" + spotify_metadata["name"].to_s) + "\" by \"" +
+         Style.bold(spotify_metadata["artists"][0]["name"].to_s + "\"") +
+         " @ " + Style.blue((spotify_metadata["duration_ms"].as_i / 1000).to_i.to_s) + "s"
+    puts "  Choose video to download:"
+    index = 1
+    yt_metadata.each do |vid|
+      print "    " + Style.bold(index.to_s + " ")
+      puts "\"" + vid["title"] + "\" @ " + Style.blue((vid["duration_ms"].to_i / 1000).to_i.to_s) + "s"
       index += 1
-    end
-
-    # Sort first by points and then by original index of the song
-    points.sort! { |a, b|
-      if b["points"] == a["points"]
-        a["index"] <=> b["index"]
-      else
-        b["points"] <=> a["points"]
-      end
-    }
-
-    return points
-  end
-
-  # Returns an `Int` based off the number of points worth assigning to the
-  # matchiness of the string. First the strings are downcased and then all
-  # nonalphanumeric characters are stripped.
-  # If *item1* includes *item2*, return 3 pts.
-  # If after the items have been blanked, *item1* includes *item2*,
-  #   return 1 pts.
-  # Else, return 0 pts.
-  private def points_compare(item1 : String, item2 : String) : Int32
-    if item2.includes?(item1)
-      return 3
-    end
-
-    item1 = item1.downcase.gsub(/[^a-z0-9]/, "")
-    item2 = item2.downcase.gsub(/[^a-z0-9]/, "")
-
-    if item2.includes?(item1)
-      return 1
-    else
-      return 0
-    end
-  end
-
-  # Checks if there are any phrases in the title of the video that would
-  # indicate audio having what we want.
-  # *video_name* is the title of the video, and *query* is what the user the
-  # program searched for. *query* is needed in order to make sure we're not
-  # subtracting points from something that's naturally in the title
-  private def count_buzzphrases(query : String, video_name : String) : Int32
-    good_phrases = 0
-    bad_phrases = 0
-
-    GOLDEN_PHRASES.each do |gold_phrase|
-      gold_phrase = gold_phrase.downcase.gsub(/[^a-z0-9]/, "")
-
-      if query.downcase.gsub(/[^a-z0-9]/, "").includes?(gold_phrase)
-        next
-      elsif video_name.downcase.gsub(/[^a-z0-9]/, "").includes?(gold_phrase)
-        good_phrases += 1
+      if index > 5
+        break
       end
     end
 
-    GARBAGE_PHRASES.each do |garbage_phrase|
-      garbage_phrase = garbage_phrase.downcase.gsub(/[^a-z0-9]/, "")
-
-      if query.downcase.gsub(/[^a-z0-9]/, "").includes?(garbage_phrase)
-        next
-      elsif video_name.downcase.gsub(/[^a-z0-9]/, "").includes?(garbage_phrase)
-        bad_phrases += 1
+    input = 0
+    while true # not between 1 and 5
+      begin
+        print Style.bold("  > ")
+        input = gets.not_nil!.chomp.to_i
+        if input < 6 && input > 0
+          break
+        end
+      rescue
+        puts Style.red("  Invalid input, try again.")
       end
     end
 
-    return good_phrases - bad_phrases
+    return yt_metadata[input-1]["href"]
+
   end
 
   # Finds valid video links from a `HTTP::Client.get` request
-  # Returns an `Array` of `XML::Node`
-  private def get_video_link_nodes(doc : String) : Array(XML::Node)
-    nodes = XML.parse(doc).xpath_nodes("//a")
-    valid_nodes = [] of XML::Node
+  # Returns an `Array` of `NODES_CLASS` containing additional metadata from Youtube
+  private def get_yt_search_metadata(response_body : String) : YT_METADATA_CLASS
+    yt_initial_data : JSON::Any = JSON.parse("{}")
 
-    nodes.each do |node|
-      if video_link_node?(node)
-        valid_nodes.push(node)
+    response_body.each_line do |line|
+      # timestamp 11/8/2020:
+      # youtube's html page has a line previous to this literally with 'scraper_data_begin' as a comment
+      if line.includes?("var ytInitialData")
+        # Extract JSON data from line
+        data = line.split(" = ")[2].delete(';')
+        dataEnd = (data.index("</script>") || 0) - 1
+
+        begin
+          yt_initial_data = JSON.parse(data[0..dataEnd])
+        rescue
+          break
+        end
       end
     end
 
-    return valid_nodes
+    if yt_initial_data == JSON.parse("{}")
+      puts "Youtube has changed the way it organizes its webpage, submit a bug"
+      puts "saying it has done so on https://github.com/cooperhammond/irs"
+      exit(1)
+    end
+
+    # where the vid metadata lives
+    yt_initial_data = yt_initial_data["contents"]["twoColumnSearchResultsRenderer"]["primaryContents"]["sectionListRenderer"]["contents"]
+
+    video_metadata = [] of VID_METADATA_CLASS
+
+    i = 0
+    while true
+      begin
+        # video title
+        raw_metadata = yt_initial_data[0]["itemSectionRenderer"]["contents"][i]["videoRenderer"]
+
+        metadata = {} of String => VID_VALUE_CLASS
+
+        metadata["title"] = raw_metadata["title"]["runs"][0]["text"].as_s
+        metadata["href"] = raw_metadata["navigationEndpoint"]["commandMetadata"]["webCommandMetadata"]["url"].as_s
+        timestamp = raw_metadata["lengthText"]["simpleText"].as_s
+        metadata["timestamp"] = timestamp
+        metadata["duration_ms"] = ((timestamp.split(":")[0].to_i * 60 +
+                               timestamp.split(":")[1].to_i) * 1000).to_s
+
+
+        video_metadata.push(metadata)
+      rescue IndexError
+        break
+      rescue Exception
+      end
+      i += 1
+    end
+
+    return video_metadata
   end
 
-  # Tests if the provided `XML::Node` has a valid link to a video
-  # Returns a `Bool`
-  private def video_link_node?(node : XML::Node) : Bool
-    # If this passes, then the node links to a playlist, not a video
-    if node["href"]?
-      return false if node["href"].includes?("&list=")
-    end
+  # Returns as a valid URL if possible
+  #
+  # ```
+  # Youtube.validate_url("https://www.youtube.com/watch?v=NOTANACTUALVIDEOID")
+  # => nil
+  # ```
+  def validate_url(url : String) : String | Nil
+    uri = URI.parse url
+    return nil if !uri
 
-    VALID_LINK_CLASSES.each do |valid_class|
-      if node["class"]?
-        return true if node["class"].includes?(valid_class)
+    query = uri.query
+    return nil if !query
+
+    # find the video ID
+    vID = nil
+    query.split('&').each do |q|
+      if q.starts_with?("v=")
+        vID = q[2..-1]
       end
     end
-    return false
+    return nil if !vID
+
+    url = "https://www.youtube.com/watch?v=#{vID}"
+
+    # this is an internal endpoint to validate the video ID
+    params = HTTP::Params.encode({"format" => "json", "url" => url})
+    response = HTTP::Client.get "https://www.youtube.com/oembed?#{params}"
+    return nil unless response.success?
+
+    res_json = JSON.parse(response.body)
+    title = res_json["title"].as_s
+    puts Style.dim("  Video: ") + title
+
+    return url
   end
 end