Shit! I accidentally commited large files to my repo

So I realized that a git repo I've been using at work had some older commits
with unnecessary LARGE files.  The large files were things like executable binaries & application logs that weren't properly ignored in the .gitignore file.  Committing & leaving the unintentionally committed files in git repos can impact stability & performance if left unchecked.

Image from Ilmari Kontulainen's blog

Unintentionally committing unnecessary & large files is a very common
mistake that most new & veteran git users make & don't realize until a few commits down the road.  I published some scripts & git commands that will help you list & remove unintentionally committed files from the git repos after the fact.  These scripts will be helpful for when you don't realize that you polluted or if you inherit an already polluted repo.

Use this gist - python script to list files in the .git repo by setting a file size "greater than" threshold. The run the git command to actually remove the files from the git repo

The script is pretty much straight forward but hit me up if you need a hand


DNSSEC Your Domain or Else...

So last weekend I competed in TNW's Hack Battle in NYC which is very very rare for me since I usually serve as mentor/sponsor or judge at most of the hackathons that I attend.  I actually enjoy competing in hackathons & this time was no different. I enjoy being able to meet people with similar interests & sharing our creative ideas with the goal of just building shit to at the very least, flesh out these ideas.

My colleague affectionately A.K.A Martin "The Pullova" & I formed a team with no actual idea when we arrived but we quickly connected with some ppl that were sitting across from us at our table.  We pitched ideas back & forth before the sponsor talks/demos but after all of that I was inspired & an idea formed.  The Verisign sponsors pitched their Get DNS API project which basically has functionality to validate if domains are DNSSEC enabled.  For those of you who are asking WTF is DNSSEC bone up on it here

My idea basically consisted of a public shaming tool to build awareness around non-DNSSEC domains & engage the domain owners in securing their DNS in order to address privacy issues that are a result of non-secure DNS.  I believe in building awareness but also 100% believe in giving opportunities to resolve issues when discovered.  So the hack would shame ppl on social media but would also create a vehicle for the shamed to publicly make things right in full transparency. We would also provide some corrective recommendations for the offending domain based on the DNSViz tool also provided by Verisign.

We quickly found that the Get DNS API project already had a public shaming hack posted but all it did was shame the domains with no offer of redemption or recommendations on how to secure their DNS.  So we formed a team of 5 with some ppl that were sitting across the table.

Martin & I are primary living in devops roles these days our new teammates consisted of a very green developer, a business student & a pure graphical designer.   So the idea evolved into this hack:

  • Create a Chrome extension that validates the DNSSEC of visited domains
  • Get the Organization/Owner information for the domain via Whois
  • Post a shaming message to Twitter calling out their in-secure DNS
  • Send them a link to a todoist project that has some resources & actions they can perform to resolve the issue
  • Post some praise on social media addressing their willingness to fix the issue
We started working on the project & collaborated through the night building servers (on Linode props!!!), logo's, api's, landing pages, twitter accounts etc....  Everyone was passionately contributing & crushing the tasks at hand & the collective synergy was very truly inspiring.

At the end of it all we had most of the pieces for the hack prepared & functional but in true hackathon fashion integrating all the moving parts proved to be very difficult.  Our cloud server melted down because of overwritten dependency issue, the Chrome extension wasn't firing correctly on browse & last but not least we ran out of time!  Even with all this ruckus we were still able to band together & build a solid functional demo that clearly conveyed our concept & idea.

We proudly pitched our hack to the judges & shortly after I departed the event on a 2 hour drive home.  Upon arriving home I saw an msg from our teammate Sarah telling us that our hack won the Verisign sponsor prize for best use of their stuff!  It felt great but after reflecting on the win the biggest reward & gratification I got from the whole experience was actually collaborating with my teammates/peers. I was satisfied in working together to build out & demo the project no matter how fucked or incomplete we perceived it to be.

So to my teammates thank you for the experience & big ups to the TNWeb team for organizing a great weekend.  I had lots of fun competing in your event.  You should do it again next year!

Me & Windows 10

I've been using open source for most of my professional career & recently I've switched to Windows 10 to quell my curiosity.  I've been known to openly shit on closed proprietary software & those opinions haven't changed much.  My preference is Linux but these days I find my self hacking on OSX due to my current gig's work load & the nature of my current projects.

I just purchased a "new" used Dell laptop & decided to throw Windows 10 on it & give it a try.  I've been using it exclusively for about 2 weeks now and have to say that I like it so far.

It def has a much better feel regarding UX  than Windows 8 & I find myself not really bothered by annoying things I experienced in prev versions of the OS.

Having said this I've had to hack the shit out of Windows 10 in order to get it to a usable state where I can function in my normal day to day capacity.

Some of the things that I did to make using Windows 10 bearable were:

Overall Windows 10 is better than it's predecessors but really is still a non open sourced OS and that is still a HUGE problem for me.  Even if Microsoft gave it to me for free I still have issues with it being closed & proprietary.

 As for using it as a day to day OS it is usable but it still is Windows & not Linux which is where I'm most comfortable. So YMMV when using Windows 10 but it functions just fine for the moment.

Why No Posts for a Year

I am very disappointed in myself.  I just noticed that I haven't posted something in over a year.  I've had lots of ideas for posts over the last year that sat as drafts because I either didn't think they were relevant or not complete & I didn't know how to finish them.

During this period I've been reading other people's blogs & I realized that it doesn't matter.  Most of these blogs we more incomplete then my drafts but they were still set free into the world.  So my new goal is to write a new blog at least once a week.

Wish me luck

CoreOS Tutorial 01 - Installation

I'm writing these CoreOS tutorials to document & share my experience with the platform.  In my initial dealings with Docker & CoreOS I have found the CoreOS docs to be very minimal & unsatisfactory especially for new users.

Prep for the install

These instructions show how to perform CoreOS install using the .iso method.
  • Download the CoreOS .iso
  • Get familiar with the cloud-config.yaml example file & gather all the values for the < your xxxx> tags within the file ( your going to need them later)
  • Create a bootable CoreOS USB key or burn a CD using the .iso file
  • Create your own personal SSH Keys (you'll nee the contents of the file) 
  • Make sure you have an active internet connection
Now we're ready for some boot action

Install CoreOS

These directions show how to install CoreOS to a disk
  • Boot to CoreOS using the bootable USB Key or CD until you see the $ core@localhost~$ prompt
  • Download the cloud-config.yaml template I mentioned above:
         $ wget -O cloud-config.yaml
  • Get the Network Interface name:
        $ ifconfig - the nic name should be something like en... or ens...
  • Use the VIM text editor to modify the cloud-config.yaml file we just downloaded.
       $ vim cloud-config.yaml
  • Replace all the < your ... > tags with values pertinent to your installation, values such as Hostname, IP Address, Gateway, Public SSH Keys etc...
  • Save the file with your changes.
  • Now we'll install CoreOS using the core-install script:
        $ sudo coreos-install -d /dev/sda -C stable -c ~/cloud-config.yaml
  • After a successful installation reboot the server & remove the boot media
  • Reboot a second time for the new settings to take place

Assign Sudo Access to the cloud-config.yaml users

Unfortunately you have to manually add users to the sudoers file in order to assign root privileges.
  •  Login into the new server via SSH:
        $ ssh core@<your ip address>
  • $ sudo -i      to get root access
  • Create a new sudo rule & give the user(s) sudo access (replace the < ..> with your user's name:
        $ echo "< your user's name > ALL=(ALL) NOPASSWD: ALL" > /etc/sudoers.d/< your user's name >
  • Reboot the server
  • Login to the server via ssh using your new user & test your sudo access
CoreOS is a very cool tool but still lacking in documentation which is usually the case with cutting edge technologies.  I'll be writing a followup blog that will discuss CoreOS & using docker to create new application containers.

Props to Stevie Holdway's blog which served we well in writing this. It's also a great resource.

Stay tuned


Make it reign APIs... up in here

Pimp'n APIs at PhillyCodefest Drexel University
Now I've been evangelizing technology for most of my career & I can't remember a time that I felt as passionate about technology as I have been in the past few years.

The tech scene is on fire again and I 'm enjoying all of it.  Innovations in hardware & software are occurring faster than ever and their foot print is impacting our lives.  In the advent of all this innovation I'd like give a shout out to the RESTFUL APIs.  RESTFUL API's have truly redefined the way we access & interact with data.

Prior to REST APIs data was locked up behind our firewalls in some private database only to be shared & released to users on our private intranets using some outdated & usually home grown user un-friendly app.  That valuable data was sometimes let out to play, embedded in a hard to find & hard to maintain html file.  Organizations are sitting on data that they may not seem relevant to their specific use cases but in reality that same data, if made publicly, is extremely useful given the appropriate context.

Fast forward to present day... REST APIs are providing the modern day developer digital plumbing and access to that once imprisoned valuable data.  REST APIs are enabling developers to build the next evolution of apps which showcase the data like never before because after all data == information & information is the heart of most apps.

In closing I want to encourage developers to continue the innovations in building & using these awesome REST APIs because they're definitely changing the status quo in technology.  I also want to encourage developers to build apps that push the current  boundaries of REST APIs which should provoke evolution in future generations of APIs & data share concepts.

Please share your comments on this post


yep don't use it unless you ask ;-). Powered by Blogger.