This is the second of two interrelated posts talking about the IDGAF development process, using my setlist management software project as a prime example of what not to do when developing software. This part examines the workflow process itself and the resulting codebase of technical debt.

What is this workflow you speak of?

IDGAF: Imagine Defining Greatness After Failing

Okay, so what does that really mean on a practical level? It's simple really, take any project in mind and just get it done. Cut every corner, waste loads of resources if necessary, ensure you're having fun with it and get it to a point where "this actually worked". By the time you've gotten there, version two of the project is already imagined, fully and clearly. Now, repeat the loop. Tackle version two in the same fashion, except this time address all the pain-points from version one that you can and just get it done. Moar repeats. Done. More. Done. More. Until there's no reason to touch the project anymore because it just does what it needs to do and you are free to move on to other things that can get done now that this thing is done.

That's hopefully-obvious for simple side-projects, because sure, any workflow will do with those "not important" things. The things that don't keep the lights on and food on the table. So what about those projects or jobs we need to do that "are important" things?

Well, there's a million and one options for that and in truth they are all probably better than what I'm describing here - and let's be clear... I am not advocating this workflow, I am simply talking about it and most likely there's many other names for it, that I'm simply not aware of, or they're just not as fun of a narrative to use as this one.

In technical terms, one could say this is an iterative workflow, that assumes no release is "final" and that there is a sufficient reason or driving necessity for getting it done now and without formality. Organic evolution perhaps?

Why are we talking about this?

Fairly simple: nobody intentionally chooses this pattern, it's simply the result of doing one's best to get things done, now, in order to get more things done, later... and of course, it's the approach I inadvertently (carelessly) took with my OSDJS project. Fine, I needed something to blog about and trash talking the code I wrote is an easy thing to write about... because there's just so much garbage in the midst of something that could be generally useful.

Side-side note...

If anyone from Rekordbox or other major DJ software company is interested in this work, whether to integrate some form of it into your commercial products or just want to know more - I'm totally open to talking. I built this software because your commercial solutions to the problem are lacking, uh, finesse? flexibility? I dunno, what to call it but would happily use something more refined and polished than my clunker.

Okay, you promised talk about bad code!

I like your enthusiasm! Let's jump right to it then.

To understand bad code, let's take an example of good code.

{{< highlight go >}} // this code is solid gold bug free // because it's just a bunch of comments // pointing out that what's good now /* may not be good in the future */ {{< /highlight >}}

Alright alight, let's talk about this fable of "good code". Most programmers will agree with the following list of some really great things good code does:

never trusts user input, always validate
cleans up after itself (resources, memory, temp files, etc)
performs it's function in testable ways
gaurds against errors (if a func used returns an error object, check it!)
no magic numbers / values hard-coded, or have detailed docs / comments answering the following to some sufficient degree:
- why the value was chosen
- what other possible values are there
- were there any "DO NOT PICK THIS VALUE" circumstances?
naming conventions are descriptive, or if that would make them too long, use acronyms that make sense with an inline comment spelling out the name once
small, focused, purpose-driven, "I do one thing and I do it well" functions
no code is repeated more than twice (investigate if having another helper func is a bad idea and if not bad, do it)
was not blindly copied from stackoverflow verbatim

That's a cursory list for any codebase to aspire to and I'm certain there are oodles of points I'm forgetting and even more that I'm somehow assuming every developer just knows. The above are basically all the sins I've committed in the OSDJS project and some are featured in this posting.

Starting out easy with poor UX

Maybe you noticed in the previous post that there's the File menu with only two items: New Playlist and Quit. Having noticed that, then perhaps you've asked your self "how does he get new tracks into the library?" but then noticed the command-line usage documentation mentions an import command... which yes it means that to import stuff I have to close the main application window and then do some funk on the CLI like this:

$ ./bin/osdjs import -r -c /path/to/new/music/tracks/

where -r is for recursive and -c is for copying the MP3 into the library. Okay, so that's not so painful. Fine, we don't need a popup dialog in the GUI to prompt for the importing, however, you did code that import in a way that could be used in the GUI though right? right?

It may not seem like it on first glance, but yes, I did in fact have some foresight to build the actual import process into the library package and simply use that. However, the actual import is only for singular files and so there's some directory walking depending on the CLI arguments given.

{{< highlight go >}} func PerformCommand() error { file_path := flags.GetTextValue("import", "file") cp := flags.GetBoolValue("import", "copy") ln := flags.GetBoolValue("import", "link") recurse := flags.GetBoolValue("import", "recursive") if cp && ln { return text.E("cannot copy and link at the same time") } force := flags.GetBoolValue("import", "force") m := library.MoveImport if cp { m = library.CopyImport } else if ln { m = library.LinkImport } if fs.IsDir(file_path) { if recurse { count := 0 filepath.Walk(file_path, func(p string, f os.FileInfo, err error) error { if err != nil { log.Error(err.Error()) return nil } if filepath.Ext(p) == ".mp3" { err := library.ImportMusic(p, m, force) if err != nil { if err.Error() == "in library" { log.Notice("already exists: %v", p) } else { log.Error("import error: %v - %v", err.Error(), p) } } else { count++ } } return nil }) log.Notice("recursively imported %v tracks from: %v", count, file_path) } else { files, err := ioutil.ReadDir(file_path) if err != nil { return text.E("error reading directory: %v - %v", err.Error(), file_path) } count := 0 for _, fi := range files { p := fi.Name() if filepath.Ext(p) == ".mp3" { err := library.ImportMusic(p, m, force) if err != nil { if err.Error() == "in library" { log.Notice("already exists: %v", p) } else { log.Error("import error: %v - %v", err.Error(), p) } } else { count++ } } } log.Notice("imported %v tracks from: %v", count, file_path) } } else { err := library.ImportMusic(file_path, m, force) if err != nil { return err } } return nil } {{< /highlight >}}

Can anyone see what's wrong with this code? There's a number of things wrong and two I'd like to focus on are quite bad.

What was it given?

The first thing is the user-input handling. This function is entirely trusting that the user has provided a simple string representing the path on the filesystem to the track or directory meant to be imported. Any fuzzing tool, literally any, will probably crush this application, corrupt the database or even eat my tracks. Who knows?! Why? because I didn't guard the code fully... and more importantly... because there is exactly one user of this software and it's me and thus it seems I trust my self to do the right thing at all times.

Did it actually work?

An astute reader will notice that my fancy library.ImportMusic method is only returning an error object. Okay, cool, nothing wrong there... except that in the directory walk block of code, the error value isn't really an error (nothing went "wrong" in the import process) as the track was "in library" already. What makes this a faux pas is that the error object isn't intended to be the business logic, just notify the logic to handle the case.

Not a huge gripe, but it is a great example of...

technical debt.

In the library.ImportMusic example, we identified some odd use of the error handling and the lack of input sanitization as things that need to be addressed at some point. These are clear and obvious cases where instead of paying it forward and being thorough in my care while creating it, I chose to simply get it done right now because I wanted to try a new feature and was itching to spin a new set for fun and exercise.

Yes, that's correct, part of what I love about mixing is that I get so wrapped up in it that I'm dancing and bouncing and singing. I love it. Total fool of my self but it's just genuine fun and I'll never stop doing it.

Looking into what the library.ImportMusic func is doing, like just now as I'm rambling off-the-cuff about what I know is definitely flawed tech debt... lol the error message for the case being checked is not in fact "in library" at all anymore and is "in library: %v" where the value is the full path to the MP3.

HAHAHA and somehow I've never noticed this problem... and why? I'm a developer and have habits and assumptions that grew out of my years of writing and using code. Without realizing it, I'm constantly using software in ways that normal users would really use things. Yes, this means I can program VCRs. No I will not tell you what a VCR is, nor will I program yours. :P

Okay, so we've got two items of technical debt spotted in just the simple CLI handling of importing a track. There's like no complexity here and it's already showing real gangly warts.

Didn't you rewrite this a bunch of times already?

Yep. Don't look at me that way.

In truth, I've lost many features in the rewrites simply because I just wanted to get some things done. For example, in the second rewrite the GUI had this nifty thing for the track details - there were five or six buttons that caused the track to play but from: first 30s, next 30-60s, last 60-30s and the last 30s. Basically it allowed me to listen to the important parts of a track (in terms of transitioning from one to another) to learn if it's got vocals and if the songs in fact blend well. All of that fancy convenience was replaced with a simple "Play in VLC" option per-track's context menu and the same option but on the per-playlist's context menu (which of course plays the entire playlist). This adjustment to the workflow means I can now refine a setlist much easier and without having to ping-pong around in the GUI. Saves me overall about 1,000 clicks in the 20 minutes it takes me to prepare a setlist. Yes, OSDJS is a UX fail. All good though, I'm getting what I've paid for out of it.

stop, wait, how long? didn't you make this to automate things? LOL?

Yep. Stop looking at me sideways. LOL, makes me giggle.

Let's stop for a sidebar moment here... if you believe that every time someone builds software that "generates" or has "AI" and is now some kind of ultimate thing... boy do I have news for you!

HAHAHAHAHAHA

A lot of this type of stuff is garbage. Why? Because people like me who build it don't really know what we're doing and I'm not even sure that the people who are supposed to know what they're doing with AI even have a clue about it themselves. They just have more vocabulary than I do to talk about all the ways I'm farking this up! But it "works for me" so who cares right? Like, there is no AI or ML in the core project at all (spleeter luv ya), yet it sorta does the same thing and there's no real necessity to go beyond that with it.

In this case, let's look at the time I'd spend putting together a set list before this software was made. I would take my small box of CDs and randomly pick one track after the next based purely on what I imagined would be cool sounding. Having immediately discovered this was very stupid of me, because I don't have much sense of rhythmic pace and I'm probably tone def or at least partially so, I started to prepare my setlists. The first one took me about two to three days to work out. It sucked. The second one was a bit better, half as long to figure out but I used the decks and tried each transition before committing to the next track. It was around eight to ten total hours spent.

So now I have this software that spits out a randomly selected playlist, honouring the "DJ do's and don'ts" as best it can and enables me to breeze through the process with little to no effort.

Now my process looks like this:

Generate playlist (with or without seed depending on emotional states)
Read it over once for obvious "why do I still have this in here?" items
if no spark joy: GOTO 1 , else: GOTO 4
Replace the obvious bad tracks identified
load playlist in VLC, flip and skip around all the transitions, identifying replacement items
if no spark joy: GOTO 4 , else: rename the playlist and give it an RL spin

Makes my practice so much more convenient.

However, this of course brings me to the next focus for examining...

moar different technical debt.

Okay, so this playlist builder dialog. It gives you all these weird options doing curious things with magic nonsensical numbers and somehow produces modestly useful results. Results which I typically have go go through a round or two of pruning, swapping and tweaking before something playable is had. Don't get me wrong, sometimes the thing flukes out and produces a setlist that I didn't realize I had available to me. So that's nice.

The real necessity in all this is to make transitioning the mix as easy as possible in practice and over all my featureful testing and exploring, I discovered that the vocals are the biggest shortcoming in the system so far. The address to this arrived in the form of Deezer releasing their spleeter software. Machine learning based software to separate the various layers (stems) of any audio track. In the case of OSDJS I'm relying upon a simple split between the vocals and "everything else". I'm certain spleeter is capable of so much more.

Alright, so spleeter is this big, resource intensive process and requires considerable testing to get "just right", or perhaps I just don't know what the easy way would have been and thus it was a bit of a headscratcher for a bit. Once I got it tuned, I was able to use my same patterns of magic value creation and arrived at a new set variables for each track.

I had this whole system already, doing all the other things, how should I integrate this spleeter thing? OS calls out the shell? Write some bindings? Work out some sort of library interface?

Nope.

Waaaay too lazy for that! I had this perl script written as a proof of concept (PoC) and like the old adage goes, deploy first and ask questions later!

Given that integrating spleeter directly was out due to various factors and the excitement of what this new vox-certainty feature has the potential to do... it's hard to stop the drive. So, I added a new db table which has an implied relation via the track ID numbers with no foreign keys or other safeguards. This is somewhat intentional but certainly convenient. This meant that as long as the GUI rendering code didn't b0rk on missing values, nothing was actually broken and so it doesn't matter whether the original import or some other means are used to get the data in there.

Worked great. Okay, so now I've got this new table, the GUI is showing the extra columns but how do I get the vox-certainty data into the db? Well, when you're making a house of cards and cheating by using some glue, and when the glue isn't enough to cover the gaps? ... Add. More. Glue. This is the IDGAF at it's best, shinging bright.

It's hard to describe this modest sized pile of shame that I'm kind of tip-toeing around so let's just look at what it takes to import a single new track into the main library.

The first thing we need to do is to import the file "as normal", which in an ideal world is the only actual step but this isn't an ideal world.

$ time ./bin/osdjs import -v -c /media/psf/Home/Music/Incoming/DJ\ Le\ Roi\ -\ I\ Get\ Deep\ \(M.O.D.\ Remix\)\ \(feat.\ Roland\ Clark\).mp3
[DBG|osdjs/sys/log/log.go:65] set log level to: 2
[DBG|osdjs/sys/db/db.go:71] using db_path: /home/kck/.osdjs/library.db
[DBG|osdjs/music/playlist/playlist.go:65] init playlists
[DBG|osdjs/ux/gui/content/plsbuildmd.go:19] init playlists
[DBG|osdjs/ux/gui/window/window.go:69] gui.window Startup()
[DBG|osdjs/ux/ux.go:49] performing operation: import
[DBG|osdjs/music/library/library.go:37] ImportMusic(/media/psf/Home/Music/Incoming/DJ Le Roi - I Get Deep (M.O.D. Remix) (feat. Roland Clark).mp3,1,false)
[DBG|osdjs/music/library/library.go:74] Copying to: /home/kck/.osdjs/library/dj_le_roi/i_get_deep_remixes/02_-_i_get_deep_m_o_d_remix_feat_roland_clark.mp3
[DBG|osdjs/music/track/track.go:206] NewFromFile: /home/kck/.osdjs/library/dj_le_roi/i_get_deep_remixes/02_-_i_get_deep_m_o_d_remix_feat_roland_clark.mp3 (force=false)
[DBG|osdjs/music/track/samples/samples.go:136] CreateWaveformImage(/home/kck/.osdjs/cache/f23dcfce6dcc6541/0051700b4e5d225c/waveform.png)
[DBG|osdjs/music/track/track.go:482] track.Save(): 0
imported: I Get Deep (Remixes) - (02) I Get Deep (M.O.D. Remix) (feat. Roland Clark) (6m31s) DJ Le Roi [0|125.07|Am|♪]
[DBG|osdjs/ux/gui/window/window.go:74] gui.window Shutdown()

real	0m8.976s
user	0m7.540s
sys	0m1.652s

Okay, not bad, rougly nine seconds to import everything. The -v of course is the verbose flag, and, that's interesting... why is gui.window reporting a shutdown? Wasn't this a CLI command, not a GUI thing right?

That is correct though this is not the technical debt we're looking for.

So what did that import process really do? It populated the tracks table in a SQLite database with all sorts of metadata and whathaveyous. Alright, so what more needs to be done then? Well, technically nothing if you don't need to enable the vocal filtering features in the playlist builder. The GUI will load and happily select from the new tracks, but if the vox stuff is turned on, they're treated as "no vocals at all" because there is no value (emptyness parsed to zero).

Okay, so how do we get that stuff going? I'm glad you asked! My handy work in progress (WiP) scripting is just what we need.

So, I have a second GIT repo for this end of things, so we change directories to this PoC/WiP thing where we:

cd ../vox-detect/
$ cat README.md
# How-To Vox Detect for OSDJS

1. Import tracks as would normally
2. ./generate-library-list.sh > library.list
3. xargs -a library.list ./process.sh | tee library.csv
4. ./update-library.pl library.csv | sqlite ~/.osdjs/library.db

Yeah, ok so there's more than one glue script, in more than one language at this point. It's all good. Works for me.

So step one in the README file is to do what we did earlier. Step two creates a library.list file. What is that file?

$ cat generate-library-list.sh 
#!/bin/bash
find ~/.osdjs/library/ -iname "*.mp3" | sort -u -V

That file is my brain's way of remembering how to do step one, because I accidentally truncated my bash history once. That sucked.

Step three is to take the list of filenames in the library.list and run them through the process.sh script, which apparently outputs CSV content, which we also like watching scroll by as it works for some reason. Wait, was that a "hacker's progressbar"? I know of no such things. These are not the features you're looking for.

So what does this process.sh involve? Well, this script is 136 lines of shell scripting that:

uses mp3splt to create two new files per track, the first minute and the last minute
use spleeter to split all three track files (full,intro,extro) into vocals and instrumentals
call out to another script sox-bpm-energy.sh (which was the PoC for the current library.ImportMusic feature) to generate the energy slopes and other values using the vocal spleets as input and produces three values per track examined:
- average: total / count
- total: total value of all samples "added up"
- count: total number of samples "added up"
combine all of this into one line of CSV data leading with the file name

Finally step four is WHAT IN THE NAME OF QBASIC IS HAPPENING?! Easy there, it's all good. The astute reader would have noticed that the final script is written in Perl so it's just gotta be solid gold output that we can trust to not trash our precious library.db with erroneous SQL statements. Foolproof regexes all the way down.

Okay, this Perl script takes the CSV file generated from step three and rewrites the data into SQL statements which directly INSERT the data into the correct tables of the OSDJS database structure. This involves looking up the existing track IDs in the database and crafting the appropriately specific statements. I'm literally the only user of this "system" and I'm good with all of this as-is. Someday, it may become a necessity to roll all of this into one slick import process built into the actual codebase... but, I think today is Tuesday, not Someday, so I've got better things to do with my time. Works for me.

That is horrible, how is that "a good thing" at all?

LOL it's not! Of course it's not, but who cares! It's just a side-project and it's filling the need. There's lots of other things we could be doing like practicing writing posts or daydreaming while resting on a skyward gaze.

Let's step back and get an overview...

... of the technical debt present:

the QA process is not revealing the actual bugs present, like at all
the database needs to be modified "by hand" to get tracks fully imported
the codebase is spread in a clumpy fashion, few 10k-line files, lots of small boilerplate-ish things
the business logic relies upon Bad Math yet produces Good Results so it's hard to abandon what works for what's technically better
some features are CLI only, others are GUI only
there is bad UX and assumptions about user behaviour everywhere
dead code, which is pruned by the compiler but makes for bad reading
there's no documentation, anywhere
version numbers and other typical identifiers lie to the user
a lot of other things I'm forgetting
even more things I didn't bring up

So, again, why not clean all this up? Absolutely no need to. Works for me and I'm the only one paying for the development cycles... and I feel fine.

If there's any curious minds out there and want to see another post on the topic of my OSDJS project, for example, what's it take to cleanup some of this technical debt? could be an obvious starting point.

Side-note

Pageviews and arbitrary stats don't really tell creators anything about what you feel or think about their work. Artists, like all people, love recognition but I think the more valuable response is that of constructive criticism and open dialogue or debate. I love to be proven wrong, because then I can learn what is in fact right. Like what I'm saying? Let me know! Don't? Let me know too... on twitter.

Alright then, what is...

the real "Good Thing" from this project's "Bad Example"?

This posting. This not-so-mortem-postmortem. This is a message to anyone who cares to listen.

Your work does not define you. Your side-projects do no define you.

When they're half-done, you're not half-done.

You're all-good and it's the work that's half-done.

When we've got excessive idle time and an excessive number of "Things To Do" we all hit "decision paralysis", finally defaulting to our typical escapist retreats or falling pray to mental disorder and general disease (as-in "not at ease" or anything resembling "anxious" of mind). Stop that, take a breather, really, take ten breaths and count with me from zero to nine, counting on the exhale... zero... one... two... three... four... five... six... seven... eight... nine. In that time you've probably inadvertently had back-thoughts of two things: the thing you are supposed to be doing right now and the thing you really want to be doing right now... and they're probably not the same thing.

Take that insight and choose wisely. Always do the things that must be done, the things that put food on the table and keep the lights on. For everything else, follow wholeheartedly. Follow your heart whole-self-ed-ly.

I'm here, telling you, it's okay to not do things. To have a million ideas, a hundred started, a handful useful and at least one that others may find cool. So, share your not-so-mortem-postmortems. I'm on twitter and love reading about other people doing awesome things that I'd never think to do.

I'm also here, telling you, it's okay to do things. Have side-projects. Follow your interests. Study your self. Learn about what you like and don't like. Do what you wholeheartedly feel you should be doing right now. Whatever that is. Do it. Be you. Define your self by your self, not by others.

Every single person has a desire for joy within them. Every one needs some form of mental expression. When this stops, the mind stops and when the mind stops, we're no longer processing our self - we're processing whatever others have fed our mind for us.

You are what you eat.

Literally and metaphorically.

Take advantage of that.

Feed your mind.

Hack your mind.

Post-script

Again, I hope you enjoyed this as much as I enjoyed writing it.

Thanks for reading!