Use Yarn in Place of npm

by Ryan DavisonProgrammingPosted on No Comments

Condensed version of This Post

Use Yarn in place of npm: Workflows don’t change; Packages load faster; Consistent node_module structure.

yarn init = npm init
yarn install = npm install
yarn add [package] = npm install [package] --save
yarn add [package] --dev = npm install [package] --save-dev
yarn remove [package] = npm uninstall [package]

 

Longer Version of This Post

npm is currently king of the Node package managers. Yarn is an alternative package manager that tries to fix what could be problems for some npm users. Yarn provides faster load times, dependency consistency and shorter commands, all within the same workflow you are used to with npm.

 

Installation and Use

If you already use npm, install Yarn with npm install yarn -g . That’s it! You can now use the yarn commands just like you would with npm. If you feel silly installing npm’s replacement with npm you can download an installer instead. Use your existing package.json file or create a new one with yarn init . Run yarn add [package]  to install new package dependencies . Removing installed packages is as easy as yarn remove [package] . Install all of the dependencies of an existing project using yarn install,or even just yarn .

 

Deterministic Package Installs

Deterministic Package Installs is a fancy way of saying: The same module dependencies will be installed with the same structure on any machine using yarn. The structure of dependencies in the node_modules directory can be different from machine to machine when using npm. This can potentially cause a dependency to work on one machine but break on another.

Speed

Yarn installs packages faster than npm. Yarn starts by comparing a dependency against what’s already in the global yarn cache. If there’s no package cache, the package is downloaded and placed in cache. Once all dependencies are cached, yarn copies all necessary files only once to the project’s node_modules directory.

 

Downloaded and cached packages don’t need to be re-downloaded in the future. If you nuke your node_modules folder and run yarn install  again, your dependencies will be copied from the cache into your new node_modules directory very quickly. If you start a new project somewhere else on the same machine, only dependencies that have never been used elsewhere are downloaded. The rest are pulled from the cache and merged with the downloaded ones. This makes for a very fast load.

Conclusion

Do you really need to use Yarn? Of course not. Lot’s of people use npm for their projects with little problem. But on projects where dependencies have to be installed separately among several users, module consistency could become a problem. Yarn solves this and provides other great enhancements to npm. Yarn provides a similar use experience to npm. It provides all the same packages, is faster and has simpler commands. It can even tell you why a package is being used. There are few if any downsides and you can always go back to npm.

5 Ways to Comment Your JSON

by Ryan DavisonComPosts, ProgrammingPosted on No Comments

Comments aren’t part of the official JSON specification. According to an old (2012) Google Plus post by Douglas Crockford, he removed them to preserve interoperability.  But that same post suggests you can still use comments  so long as you remove them through minification before parsing.

There are a few other ways to handle JSON comments besides minification:

  1. You can add a new data element to your object. The element key would be named  “_comment_” or something similar. The value would be the actual comment. This method is slightly intriguing but feels kind of dirty. It looks like a hack. It is a hack! bulk is also added to the network payload. JSON is a lightweight data exchange. Adding comment elements takes away from this.
  2. Use Scripts that programmatically remove comments from your JSON before it’s parsed. Sindre Sorhus published a comment stripping module which does just that. This is similar to Crockord’s method of minification in that it removes the comments before parsing but you can inline it in your code rather than use it during a build process.
  3. You can forget comments in your JSON entirely. Put comments in the code where you make the data request in the first place. You should already know what kind of returned data is expected so comments would make sense here. You can stay in your code without the need to view a separate file. This makes your code easier to understand.
  4. Finally, if JSON is being used as a configuration file or some other static data store, you might even try commenting it in a separate file. Put a text README in the same directory the configuration file is stored. The README could contain a paragraph describing the data or you could copy the JSON into the README and use inline comments.

There are several ways to take care of the problem of commenting JSON files. All have their strengths and weaknesses. The best method depends on your particular situation and needs.

A First Crack at HTML5/JavaScript Game Development

by Ryan DavisonWeb DevelopmentPosted on No Comments

For a while now I’ve been wanting to start writing JavaScript/HTML5 based games for the web.  I’ve always been drawn to simple games like the old Asteroids and Galaga where you have a ship and a bunch of bad stuff trying to destroy you. They’re easy to learn, play and waste your time with and yes, they give you that easy, addictive sense that you are actually accomplishing something worthwhile when you break your high score by one point.

One of the coolest modern versions of these old arcade classics is actually kind of useful. Ztype  is a simple shooter game along the lines of Galaga but you are shooting words and you have to type them in correctly or your ship doesn’t shoot. It’s an addictive game with great graphics and sounds.

ZTYPE

But I needed something simpler to get me started and familiar with the game engine I had chosen to start with – Phaser. I found an amazing tutorial on how to re-build Asteroids over on zekechan.net. It is surprisingly straightforward, provides full code to check yours against, is easy to follow but goes in depth enough to take you through developing an entire game.

Asteroids screenshot

However, by the time I was finished building the game, I was a little bored with the idea of a ship trying to destroy asteroids so I switched it up a bit to include a political theme appropriate for the current presidential race. You can check it out at http://ryanrandom.com/ted .

Ted Cruz vs Donald Trump

EDIT 2020: I’ve long moved on from ryanrandom.com so the game above is no longer active.

Mapping The Dead

by Ryan DavisonGISPosted on One Comment

Skull and Crossbones

I’ve seen a lot of interesting mapping applications in the news over the last year. One that’s caught my attention is cemetery mapping. I had never really thought about just how ideal a cemetery is to be mapped. Each plot has a distinct spatial location. They have measurable attributes like occupant, location, depth and width. They are often laid out like a grid or a table but sometimes (especially on older properties) they are spread out seemingly without much thought to being easily located again.

Locating a plot is obviously the most important attribute for cemetery mapping. Caretakers have to be able to determine where a body is located so they can avoid accidentally digging it up when placing another body. Relatives of the deceased want to know where their family members are too so they and those in the future can find them again.

One of the first articles I cam across last year was about the cemeteries in the City of Mackinac Island Michigan. The city’s cemetery committee (I bet those meetings are fun) recognized that its current data holdings (hand-drawn paper maps, incomplete lists of cemetery residents and the memories of senior committee members who are increasingly ending up in the cemetery themselves) were not adequate. So they started mapping out plots using GPS and building a database of names.

The City of Mackinac Island Cemetery Committee hopes to have a completed digital mapping system by next June, which will help the city clerk’s office keep track of plots and burials more efficiently. The map is one of many updates the city is considering relating to its cemeteries and burial policies.

It didn’t surprise me to find that some cities are using GIS technology to keep track of cemeteries. What did surprise me was the number of software packages that have been created for mapping and managing them. A quick search for cemetery mapping software reveals several pages of apps, services and companies with interesting names like Memorial Business systems, CemMapper and The Crypt Keeper.

Yet with all of these software solutions, none of the cemeteries that I was interested in searching had detailed mapping of their plots. Only one even had a website. Although the mapping technology is there, this kind of project doesn’t seem like one many cemeteries are willing to undertake.

Why Gulp is Great

by Ryan DavisonProgrammingPosted on No Comments

In my last post I talked about why I started and then stopped using Grunt. Basically, Grunt seemed too slow and my workflow was being halted too often while I waited for it to build. There are several other task running/app building tools out there (Broccoli, Cake, Jake…) but I decided to try Gulp first since it has a large user base and there are plenty of plugins out there to keep me from having to think too much.

At first, Gulp didn’t seem quite as straightforward as Grunt. Grunt was easy to use. You just had to write (sometimes lengthy) configuration objects for the plugins you wanted to run and then fire off the tasks using the command window. Even someone like me could figure out how to add a source file and a destination location to a minification plugin and be reasonably sure I would get a minified file out of it.

It was also very easy to visualize what your Gruntfile was doing because every task plugin worked independently of the rest. You would configure ten different tasks and then register them all together in a row and expect them to run one after another until they all completed.

With Gulp, you don’t just configure plugins, you write JavaScript code to define your tasks and how you want them run. A Gulp task asks you to require the plugins you want to use, or write a custom task using plain old JavaScript, then call Gulp.src to provide a source file for the tasks to run on. Doing this opens a Node stream which keeps your source object in memory. If you want to run one of the task plugins you required at the top of your script, you simply pass the in-memory object to it by calling the .pipe() method. You can continue piping the object from one task to another until you’re finished. Finally, you call Gulp.dest and provide a destination location.

var gulp = require('gulp');
var plumber = require('gulp-plumber');
var addsrc = require('gulp-add-src');
var less = require('gulp-less');
var cssnano = require('gulp-cssnano');
var concatCss = require('gulp-concat-css');
var rename = require('gulp-rename');
var concat = require('gulp-concat');
var uglify = require('gulp-uglify');
var watch = require('gulp-watch');

gulp.task('less', function(){
    return gulp.src('./source/style/style.less')
        .pipe(plumber())
        .pipe(less())
        .pipe(cssnano())
        .pipe(addsrc.append(['./source/style/anotherStyleSheet.min.css', './source/style/stillAnother        StyleSheet.min.css']))
        .pipe(concatCss('concat.css'))
        .pipe(rename("style.min.css"))
        .pipe(gulp.dest('./destination/style/'));
});

gulp.task('js', function(){
    return gulp.src(['./source/scripts/javaScript.js'])
        .pipe(plumber())
        .pipe(uglify({
            mangle: false,
        }))
        .pipe(addsrc.prepend(['source/scripts/someJSLibrary.min.js', 
        'source/scripts/anotherJSFile.min.js','source/scripts/stillAnotherJSFile.min.js']))
        .pipe(concat("all.js"))
        .pipe(rename("finalFile.min.js"))
        .pipe(gulp.dest('./destination/scripts/'));
});

gulp.task('default', ['less', 'js'] , function() {
gulp.watch(['./source/style/style.less']);
gulp.watch(['./source/scripts/javaScript.js']);
});

The great thing about using Node streams is that you don’t have to keep opening and closing files for each task like in Grunt. This lack of i/o overhead makes running a series of tasks very fast. Even so, you really need to use the built-in watch task to take advantage of this speed. In my experience, running a default task with four or five tasks in it, from the command line, was almost as slow as in Grunt. With the watch task running, it only took milliseconds to rebuild what it needed to. But I’m new to Gulp so what do I know?

You can see in the code above that I used several plugins to manipulate the input file as it is piped down the stream. There are two that I found particularly helpful. The first is Gulp-Plumber which is basically a patch that keeps streams from being un-piped when an error is encountered. Supposedly, streams breaking on error will be fixed in version 4.0.

The second helpful plugin here is Gulp-Add-Src which does exactly what the title says. You can add additional source files to your stream so you can do neat things like concatenation. With these and other plugins I haven’t found anything with Gulp that would keep me from doing everything I could with Grunt.

The only thing I really don’t like about Gulp is the icon. It’s a cup with a straw in it and the word Gulp across its side. A cup by itself indicates an ability to gulp what is in it. But you don’t gulp through a straw, you sip or suck. Who wants their product to suck? And sip indicates a lack of passion. So what’s with the straw?

Gulp.js cup icon

Why Grunt is Gone From My Build Team Lineup

by Ryan DavisonProgrammingPosted on No Comments

I have to admit, I don’t always research every option when I’m looking for a solution to a problem. I’ll usually start out with a broad web search to see what others are using and if their solutions seem to fit my situation. Then I’ll take maybe the top two solutions and try to implement them. The first one that serves all of my requirements and is relatively easy to implement usually becomes my solution.This is exactly how I came to start utilizing Grunt as a build tool for a large web mapping application I develop.

When I first started using Grunt, the JavaScript API development team at ESRI was using it for their projects. Lots of other developers were using it too and I didn’t know any better than to follow. A few people were talking about Gulp too as an alternative to Grunt. So I took a brief look at Gulp, didn’t immediately understand it, then started putting together my Grunt configuration file and collecting all the plugins I needed.

What can I say – it worked great and I was happy that I wasn’t still using code minifiers and copying files by hand to production folders. When I started using Adobe Brackets as my default code editor, I was pleased to find it had a great Grunt plugin to integrate task running directly.

Things were great for a while but I was always bothered by how long Grunt took to run through all my tasks and complete my build. It would take 10+ seconds to finish and I would have to sit there waiting to check my latest edits. It can be really hard to develop a piece of code when you are constantly halting your flow.

However, I was lazy and didn’t want to have to learn another tool. What I had in place worked,  just not efficiently. But eventually I knew something had to change. Strangely, it wasn’t inefficiencies with Grunt that made me dump it, it was Brackets. My Brackets install was slowing down and freezing at inopportune times, like whenever I wanted to use it. I was also getting the Brackets “white screen of death” from time-to-time which required the Task Manager just to shut the program down. So now I was waiting 30 seconds for my editor to unfreeze so I could wait 10 seconds for my task runner to finish.

The upshot is, I revisited Atom and am now using it as my default editor. Fortunately, I wasn’t happy with Atom’s Grunt integration. I figured it was a great time to jump ship and try again with the second biggest player in the JavaScript task running world: Gulp.

In my next couple of posts I’ll talk more about why Gulp is great and why I shouldn’t have nuked Atom when I was first choosing a new editor.

Search Directory Trees with Python

by Ryan DavisonProgrammingPosted on No Comments

Here is a simple but powerful way to use Python to search and find all files with a specific extension within a given folder and all of its sub-folders using os.walk(). I use it a lot to find map documents with broken data sources, images that need to be organized or just to get a quick count of certain files in a directory.

import os

directorypath = raw_input("Enter a directory path: ")
extension = raw_input("enter an extension: .")

#Loop through all folders and subfolders in your target directory. 
for root, dirs, files in os.walk(directorypath):
    fileList = [os.path.join(root, f) for f in files if f.endswith(extension)]             
    for item in fileList:
        print item #or do something else with each file found.

 

When Django Met IIS

by Ryan DavisonProgrammingPosted on 2 Comments

The Problem:

The county I work for had a one page geography quiz with outdated questions and a poorly structured user interface. It was just an unstyled list of select dropdowns and a submit button. The answer page that was returned just listed each question with all the answer choices under them. One answer under each question was highlighted in yellow but it wasn’t made clear to the user whether this was the correct answer or just the answer they chose.

Old Mesa County quiz app.To compound things, the quiz was stuck in an enterprise CMS that gave it a really ugly url. I wanted to change that so it would look good and be really easy to understand. Thankfully I wasn’t restricted to any particular language or technology stack to build the new quiz.

The Solution:

A few years back I had played around with Django and thought it was a cool framework but I had never really applied it to any project. So I said to myself “why not?” and set out to build a new geography quiz app with Django. Over the next couple of days I put together the components of the new application:

  • A PostgreSQL database to store the questions and answers
  • Models to define the data in the database
  • Views to process and send the data and to route user answers to the answer page
  • Templates to render the data

Everything seemed to be going along smoothly although I wasn’t very happy with the way I wrote my views to handle the user submitted answers. The app basically builds a form on the fly (questions with radio button answers) then does a POST when the quiz taker clicks the submit button. The view then takes the POST data and turns it into a Python list. The rest of the view just slices and dices the list and uses offsets to pull out the matching questions and answers.

I know there are cleaner ways of doing this. Especially since the returned POST data is a querydict object which is basically just a Python dictionary. Manipulating key/value pairs seems neater. But the way I did it worked and I got it up and running fast. Maybe a project next time I’m bored will be to make the code cleaner and more maintainable.

Another Problem:

Remember earlier when I said that I asked myself “why not?” when considering using Django? Well, I was about to answer that question and the answer wasn’t pretty.

While I was developing my quiz app I used Django’s built-in server which is a lightweight “please don’t use in production” server. It worked great. Then I decided it was time to port the app to my production server and actually use it in the real world. At that point I remembered we use IIS on a Windows server.

IIS isn’t all bad, especially if you work in a place that uses .NET components heavily which I do. Unfortunately, Django was never really developed to run on IIS. Django was really designed to live in a Unix world and be served out with something like Apache. I remembered that from my prior experience with the platform – I just forgot.

Another Solution:

There are several tutorials on the web (most of them several years old) that discuss running Django under IIS but none of them were very straight-forward. The solution that seemed like the quickest route to a running app was to use Helicon’s Python Hosting Package (part of the Helicon Zoo repository available through the IIS Web Package Installer). The hosting package basically loads all of your dependencies and does all of the complicated work of getting IIS to run an antagonistic technology. You then load a Python Project module which builds everything for you including:

  • A virtual Python install specific to your app
  • A web config file with needed environment variables
  • Permissions and application pool settings

The only pain point I had with using Helicon’s solution was discovering it doesn’t work with Django 1.7. I had developed in 1.7 and then when I migrated into the Helicon environment everything broke. This really threw me for a loop for a while until I found a post suggesting using Django 1.6. This didn’t turn out to be a big deal as it didn’t affect my apps functionality. I just had to remove a couple of middleware classes from my settings file and I was good-to-go.

New Mesa County Quiz AppConclusion:

I love working with Django. If I wasn’t in a Windows environment I might be trying to use it throughout my office GIS site. But I can’t see trying to force uncommitted technologies into a relationship they don’t even seem to want. I guess Django is just going to be a hobby framework for me for now. Fortunately there are plenty of others out there just waiting to be learned and implemented.

Brackets – My New Favorite Code Editor

by Ryan DavisonWeb DevelopmentPosted on 2 Comments

Brackets IDE symbol.A couple of months ago I started searching for “the best” code editor for web development. I wanted to see what was out there and how it compared to what I have used and was currently using.

Since most of my co-workers live in the .NET world I have access to Visual Studio, which I actually like as an IDE. I’ve used it to do a good deal of development over the last couple of years. But I wanted to explore more of what was out there for code editors that might be more lightweight, fun and available wherever I might want to use it (work, home, on the road…).

For web and desktop work at home I’ve used Notepad, Notepad ++ and Aptana and have never been really happy.

Notepad ++ actually works really well but I hate the interface (it’s boring and ugly rolled into one). Besides, it would be nice to use something platform independent for portability. On the plus side (sorry), in Notepad ++ you can configure styles and keyboard mappings and there;s a lot you can do with the preferences to make things work the way you want them to.

That’s actually the story with the majority of editors and IDEs out there today. Most of them have customizable settings and functionality either built-in or available through plug-ins or extensions. Some of them are geared toward specific languages or uses but most of them seem to handle the most common languages.

No Magic Bullet

I’ve come to the conclusion that there is no “best” editor. There are only ones with fewer annoyances than others. Out of the editors I have been trying lately there are a few I have only used for a few seconds (like Atom) and some I’ve done some heavy lifting with (like Sublime Text). My favorite so far has been Brackets, the open source project from Adobe.

Brackets

I really like the look and feel of Brackets. It has a nice flat design. It doesn’t overwhelm you with controls and menus. But that led me to pause and question – where are all the controls and menus? It turns out, a lot of your customizations are done directly through json files or through extensions. That’s great because I love working in json.

Changing the keymap is not as straight-forward as most code editors and IDEs but it is intuitive and simple. You just have to override the default mappings in the keymap.json file. I set up my block and single line mappings because the defaults almost always annoy me.

I’ve been using Brackets both at work and at home. In my home setup I use the live preview feature all the time. It’s only available through Chrome which is not a problem for me. I usually like to debug my HTML, JavaScript and CSS in Firefox (Firebug) but the Chrome developer tools work just fine for most things. Live preview is great because the Chrome page refreshes automatically every time you save your HTML file. I have a different setup at work which doesn’t allow for the live preview to work but I might be changing that soon.

Brackets really shines with its extensions manager. This is where you can install/uninstall user created extensions or Brackets themes. You can also search Github for extension, download the zip file and drag the zip right into the extension manager. It only took a few minutes to search for and install a few extensions to make development easier. These included Grunt, indent guides, code folding and code beautification(formatting).

I have notices some of the extensions can slow Brackets way down so that’s something to watch as you’re loading new ones up. Now I’m just looking for a reason to create an extension of my own or some reason to hack Brackets itself.

Shapefile as a Multi User Editing Environment?

by Ryan DavisonGISPosted on One Comment

I had a ArcGIS user that I support come to me with a corrupted shapefile the other day. It had the old “number of shapes does not match number of table records” error. It turns out, he’s still using this shapefile as his layer’s main data source and he and several others regularly edit it! In this day and age?

I tried to convince him using a file geodatabase would be more stable for editing but he had been using shapefiles so long I don’t even think my comments registered. He just wanted a tool that could fix the shapefile.

I pointed him to the shapechk tool by Andrew Williamson. I’ve used the tool for years because <sarcasm>for some odd reason</sarcasm> I often run into people with corrupted shapefiles after people edit them over long periods of time. The shapefile works OK as a data exchange format but doesn’t always hold together under regular heavy use.

In the words of Pete Seeger, “when will we ever learn?”