FormAPI Blog

Testing One-Line Bash Scripts With Dry Runs

While working on my Hugo timestamp blog post, I wrote a little script to add a timestamp to my existing blog posts and set the original slug in the YAML:

for POST in *.md; do
  TIMESTAMP=$(ruby -r yaml -e \
    "puts YAML.load_file('$POST')['date'].strftime('%Y%m%d%H%M%S')")
  ! [[ $POST =~ ^[0-9]{14} ]] \
    && perl -i -p0e "s/(---\n\n)/slug: ${POST/.md/}\n\1/s" $POST \
    && mv $POST $TIMESTAMP-$POST
done

I thought it might be interesting to talk about the process of writing this script. The trick is to always put echo before your main command, so you can experiment with the script and see what it’s going to do. When everything looks good, you can remove echo to run the command instead of printing it.

When you use sed or perl for search-and-replace, you can do the same thing by omitting the -i flag. By default, they’ll just print out the result to the console. When the output looks correct, you can add the -i flag to modify the file.

If you’d like to see the whole process, I’ve included some of my bash history to show how I wrote it. I’ve added some comments to explain what I was thinking.

(Full disclosure: I edited this a bit and removed a lot of the trial-and-error with regexes.)

# Let's start by printing out all the filenames:

~/code/form_api/blog $ for POST in content/posts/*.md; do echo $POST; done
content/posts/20180924155149-add-a-timestamp-to-hugo-post-filenames.md
content/posts/bitcoin-treasure-hunt-solution.md
content/posts/bitcoin-treasure-hunt.md
content/posts/fast-docker-builds-for-rails-and-webpack.md
content/posts/how-to-include-a-screen-recording-in-a-blog-post-on-mac.md
content/posts/json-patch-with-rails-5-and-react.md
content/posts/using-applescript-to-set-up-iterm2-for-rails-development.md

# --------------------------------------------------------------------------------
# I know YAML can parse the "front matter" from Markdown posts,
# so I'll just use Ruby to parse the dates and print out a formatted timestamp:

~/code/form_api/blog $ for POST in content/posts/*.md; do TIMESTAMP=$(ruby -e "YAML.load_file('$POST')"); echo $TIMESTAMP; done
'-e:1:in `<main>': uninitialized constant YAML (NameError)

-e:1:in `<main>': uninitialized constant YAML (NameError)

-e:1:in `<main>': uninitialized constant YAML (NameError)

# ...


# --------------------------------------------------------------------------------
# Right, I need to require yaml. Can do that with the '-r' flag:

~/code/form_api/blog $ for POST in content/posts/*.md; do TIMESTAMP=$(ruby -r yaml -e "YAML.load_file('$POST')"); echo $TIMESTAMP; done





# --------------------------------------------------------------------------------
# Ok at least it didn't crash. I'll try to get the date:

~/code/form_api/blog $ for POST in content/posts/*.md; do TIMESTAMP=$(ruby -r yaml -e "YAML.load_file('$POST').date"); echo $TIMESTAMP; done
-e:1:in `<main>': undefined method `date' for #<Hash:0x00007ff4f2822a28> (NoMethodError)
Did you mean?  update

-e:1:in `<main>': undefined method `date' for #<Hash:0x00007f8ca48d6728> (NoMethodError)
Did you mean?  update

-e:1:in `<main>': undefined method `date' for #<Hash:0x00007fce1689a788> (NoMethodError)
Did you mean?  update

# ...


# --------------------------------------------------------------------------------
# Not sure why I called .date instead of ['date']. I've been doing too much JavaScript...

~/code/form_api/blog $ for POST in content/posts/*.md; do TIMESTAMP=$(ruby -r yaml -e "YAML.load_file('$POST')['date']"); echo $TIMESTAMP; done







# --------------------------------------------------------------------------------
# Great, it didn't crash, but it didn't print anything. I need to use `puts`:
# (Side note: It would be nice if Ruby could print out the last value from a script.)

~/code/form_api/blog $ for POST in content/posts/*.md; do TIMESTAMP=$(ruby -r yaml -e "puts YAML.load_file('$POST')['date']"); echo $TIMESTAMP; done
2018-09-24 15:51:49 +0700
2017-11-25 05:57:29 +0700
2017-11-24 15:00:00 +0700
# ...


# --------------------------------------------------------------------------------
# Ok, now I'll parse the date from the string:

~/code/form_api/blog $ for POST in content/posts/*.md; do TIMESTAMP=$(ruby -r yaml -e "puts Date.parse(YAML.load_file('$POST')['date'])"); echo $TIMESTAMP; done
-e:1:in `parse': no implicit conversion of Time into String (TypeError)
	from -e:1:in `<main>'

-e:1:in `parse': no implicit conversion of Time into String (TypeError)
	from -e:1:in `<main>'

-e:1:in `parse': no implicit conversion of Time into String (TypeError)
	from -e:1:in `<main>'

# ...

# --------------------------------------------------------------------------------
# Oh right, YAML already parses the date. Now I just need to format it:

~/code/form_api/blog $ for POST in content/posts/*.md; do TIMESTAMP=$(ruby -r yaml -e "puts YAML.load_file('$POST')['date'].strtime('%Y%m%d')"); echo $TIMESTAMP; done
-e:1:in `<main>': undefined method `strtime' for 2018-09-24 15:51:49 +0700:Time (NoMethodError)
Did you mean?  strftime

-e:1:in `<main>': undefined method `strtime' for 2017-11-25 05:57:29 +0700:Time (NoMethodError)
Did you mean?  strftime

-e:1:in `<main>': undefined method `strtime' for 2017-11-24 15:00:00 +0700:Time (NoMethodError)
Did you mean?  strftime

# ...


# --------------------------------------------------------------------------------
# Thanks, I did mean `strftime`:

~/code/form_api/blog $ for POST in content/posts/*.md; do TIMESTAMP=$(ruby -r yaml -e "puts YAML.load_file('$POST')['date'].strftime('%Y%m%d')"); echo $TIMESTAMP; done
20180924
20171125
20171124
20180802
20180923
20171119
20180923

# --------------------------------------------------------------------------------
# Ok cool, now I'll just add hours, minutes, and seconds:

~/code/form_api/blog $ for POST in content/posts/*.md; do TIMESTAMP=$(ruby -r yaml -e "puts YAML.load_file('$POST')['date'].strftime('%Y%m%d%H%M%s')"); echo $TIMESTAMP; done
2018092415511537779109
2017112505571511564249
2017112415001511510400
2018080218231533209014
2018092322041537715076
2017111906401511048422
2018092312141537679643

# --------------------------------------------------------------------------------
# Hmm, that timestamp is too long. Why did I think it was %s?
# %s is the number of seconds since 1970-01-01. I need %S:

~/code/form_api/blog $ for POST in content/posts/*.md; do TIMESTAMP=$(ruby -r yaml -e "puts YAML.load_file('$POST')['date'].strftime('%Y%m%d%H%M%S')"); echo $TIMESTAMP; done
20171119064022
20171124150000
20171125055729
20180802182334
20180923121403
20180923220436
20180924155149
20180924164500

# --------------------------------------------------------------------------------
# OK great, I've got my timestamps. Also this would be a lot easier
# if I just `cd` into the directory:

~/code/form_api/blog $ cd content/posts


# --------------------------------------------------------------------------------
# OK now I'll write the `mv` command and make sure everything looks good:

~/code/form_api/blog/content/posts $ for POST in *.md; do TIMESTAMP=$(ruby -r yaml -e "puts YAML.load_file('$POST')['date'].strftime('%Y%m%d%H%M%S')"); echo mv $POST $TIMESTAMP-$POST; done
mv 20180924155149-add-a-timestamp-to-hugo-post-filenames.md 20180924155149-20180924155149-add-a-timestamp-to-hugo-post-filenames.md
mv bitcoin-treasure-hunt-solution.md 20171125055729-bitcoin-treasure-hunt-solution.md
mv bitcoin-treasure-hunt.md 20171124150000-bitcoin-treasure-hunt.md
mv fast-docker-builds-for-rails-and-webpack.md 20180802182334-fast-docker-builds-for-rails-and-webpack.md
mv how-to-include-a-screen-recording-in-a-blog-post-on-mac.md 20180923220436-how-to-include-a-screen-recording-in-a-blog-post-on-mac.md
mv json-patch-with-rails-5-and-react.md 20171119064022-json-patch-with-rails-5-and-react.md
mv using-applescript-to-set-up-iterm2-for-rails-development.md 20180923121403-using-applescript-to-set-up-iterm2-for-rails-development.md

# --------------------------------------------------------------------------------
# That's pretty good, but I want to ignore that one post that already has a timestamp:

~/code/form_api/blog/content/posts $ for POST in *.md; do TIMESTAMP=$(ruby -r yaml -e "puts YAML.load_file('$POST')['date'].strftime('%Y%m%d%H%M%S')"); ! [[ $POST =~ ^[0-9]{14} ]] && echo mv $POST $TIMESTAMP-$POST; done
mv bitcoin-treasure-hunt-solution.md 20171125055729-bitcoin-treasure-hunt-solution.md
mv bitcoin-treasure-hunt.md 20171124150000-bitcoin-treasure-hunt.md
mv fast-docker-builds-for-rails-and-webpack.md 20180802182334-fast-docker-builds-for-rails-and-webpack.md
mv how-to-include-a-screen-recording-in-a-blog-post-on-mac.md 20180923220436-how-to-include-a-screen-recording-in-a-blog-post-on-mac.md
mv json-patch-with-rails-5-and-react.md 20171119064022-json-patch-with-rails-5-and-react.md
mv using-applescript-to-set-up-iterm2-for-rails-development.md 20180923121403-using-applescript-to-set-up-iterm2-for-rails-development.md

# --------------------------------------------------------------------------------
# OK I've got all the mv commands working. Now I just need to set the original
# "slug" for each post. I'm going to use a regex with Perl:
# (Perl is great for multi-line search and replace.)

# ... Ignore all the regex attempts ...

~/code/form_api/blog/content/posts $ perl -p0e "s/(---\n\n)/slug: \n\1/s" json-patch-with-rails-5-and-react.md  | head -n 7
---
author: "Nathan Broadbent"
title: "Better Rails Performance with JSON Patch"
date: 2017-11-19T06:40:22+07:00
slug:
---

# --------------------------------------------------------------------------------
# Ok I've figured out how to inject `slug: ` before the
# last "---" in the YAML. Now I need to remove ".md" from the filename
# to get the slug, and put it all together:

~/code/form_api/blog/content/posts $ for POST in *.md; do TIMESTAMP=$(ruby -r yaml -e "puts YAML.load_file('$POST')['date'].strftime('%Y%m%d%H%M%S')"); ! [[ $POST =~ ^[0-9]{14} ]] && perl -p0e "s/(---\n\n)/slug: ${POST/.md/}\n\1/s" $POST | head -n7 && echo mv $POST $TIMESTAMP-$POST; done
---
title: "Bitcoin Treasure Hunt Answers"
date: 2017-11-25T05:57:29+07:00
draft: false
slug: bitcoin-treasure-hunt-solution
mv bitcoin-treasure-hunt-solution.md 20171125055729-bitcoin-treasure-hunt-solution.md
---
title: "Bitcoin Treasure Hunt"
date: 2017-11-24T08:00:00+00:00
draft: false
slug: bitcoin-treasure-hunt
mv bitcoin-treasure-hunt.md 20171124150000-bitcoin-treasure-hunt.md

# ...

# --------------------------------------------------------------------------------
# Everything looks good! Now I'm ready to run it by removing the `echo` in front
# of `mv`, and adding an `-i` to the perl command.
~/code/form_api/blog/content/posts $ for POST in *.md; do TIMESTAMP=$(ruby -r yaml -e "puts YAML.load_file('$POST')['date'].strftime('%Y%m%d%H%M%S')"); ! [[ $POST =~ ^[0-9]{14} ]] && perl -i -p0e "s/(---\n\n)/slug: ${POST/.md/}\n\1/s" $POST && mv $POST $TIMESTAMP-$POST; done


# Double-check the renamed files:

~/code/form_api/blog/content/posts $ ll
-rw-r--r--  1 🍀  🖥   5617 Sep 24 20:28 [1]  20171119064022-json-patch-with-rails-5-and-react.md
-rw-r--r--  1 🍀  🖥   7047 Sep 24 20:28 [2]  20171124150000-bitcoin-treasure-hunt.md
-rw-r--r--  1 🍀  🖥  14057 Sep 24 20:28 [3]  20171125055729-bitcoin-treasure-hunt-solution.md
-rw-r--r--  1 🍀  🖥  13808 Sep 24 20:28 [4]  20180802182334-fast-docker-builds-for-rails-and-webpack.md
-rw-r--r--  1 🍀  🖥   7499 Sep 24 20:28 [5]  20180923121403-using-applescript-to-set-up-iterm2-for-rails-development.md
-rw-r--r--  1 🍀  🖥   1938 Sep 24 20:28 [6]  20180923220436-how-to-include-a-screen-recording-in-a-blog-post-on-mac.md
-rw-r--r--  1 🍀  🖥   4064 Sep 24 20:20 [7]  20180924155149-adding-a-timestamp-to-hugo-post-filenames.md

# Make sure a blog post looks ok:

~/code/form_api/blog/content/posts $ head -n8 20171119064022-json-patch-with-rails-5-and-react.md
---
author: "Nathan Broadbent"
title: "Better Rails Performance with JSON Patch"
date: 2017-11-19T06:40:22+07:00
slug: json-patch-with-rails-5-and-react
---

[FormAPI](https://formapi.io/) is a service that generates PDFs. The backend is written with Ruby on Rails,


# All done!

The nice thing about this technique is that you have the freedom to make lots of mistakes and try different things. The script was just a “dry run” that printed out the result, until I removed echo and added -i to the perl command.

Thanks for reading, and I hope you find this technique useful!