Installing Spark 1.6.1 on a Mac with Scala 2.11

I have recently gone through the process of installing Spark in my mac for testing and development purposes. I also wanted to make sure I could use the installation not only with Scala, but also with PySpark through a Jupyter notebook.

If you are interested in doing the same, here are the steps I followed. First of all, here are the packages you will need:

  • Python 2.7 or higher
  • Java SE Development Kit
  • Scala and Scala Build Tool
  • Spark 1.6.1 (at the time of writing)
  • Jupyter Notebook

Python

You can chose the best python distribution that suits your needs. I find Anaconda to be fine for my purposes. You can obtain a graphical installer from https://www.continuum.io/downloads. I am using Python 2.7 at the time of writing.

Java SE Development Kit

You will need to download Oracle Java SE Development Kit 7 or 8 at Oracle JDK downloads page. In my case, at the time of writing I am using 1.7.0_80. You can check the version you have by opening a terminal and typing

java -version

You also have to make sure that the appropriate environment variable is set up. In your

~/.bashr_profile

  add the following lines:

export JAVA_HOME=$(/usr/libexec/java_home)

Scala and Scala Build Tool

In this case, I found it much easier to use Homebrew to install and manage the Scala language. I f you have never used Homebrew, I recommend that you take a look. To install it you have to type the following in your terminal:

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

Once you have Hombrew you can install Scala and the Scala Build Tool as follows:

> brew install scala
> brew install bst

You may want to create appropriate environments in your 

~/.bashr_profile

 :

export SCALA_HOME=/usr/local/bin/scala
export PATH=$PATH:$SCALA_HOME/bin

Spark 1.6.1

Obtain Spark from https://spark.apache.org/downloads.html

Note that for building Spark with Scala 2.11 you will need to download the Spark source code and build it appropriately.

Download_Spark

Once you have downloaded the tgz file, unzip it into an appropriate location (your home directory for example) and navigate to the unzipped folder (for example

~/spark-1.6.1

 )

To build Spark with Scala 2.11 you need to type the following commands:

> ./dev/change-version-to-2.11.sh
> build/sbt clean assembly

This may take a while, so sit tight! When finished, you can check that everything is working by launching either the Scala shell:

> ./bin/spark-shell

or the Python shell:

> ./bin/pyspark

Once again there are some environment variables that are recommended:

export SPARK_PATH=~/spark-1.6.1
export PYSPARK_DRIVER_PYTHON="jupyter" 
export PYSPARK_DRIVER_PYTHON_OPTS="notebook" 
alias sparknb='$SPARK_PATH/bin/pyspark --master local[2]'

The last line is an alias that will enable us to launch a Jupyter notebook with PySpark. Totally optional!

Jupyter Notebook

If all is working well you are ready to go. Source your

bash_profile

  and  launch a Jupyter notebook:

> sparknb

Et voilà!

Google Drive not synching on your Mac? Here is what to do

I am not a big used of Google Drive. It is a good service and it mostly does what one may need from a suite of productivity apps… but for some reason I only use it in very limited cases.

So, no surprise that I had not noticed that the synching between the cloud version of my documents and those in my mac had gone pear shaped. I tried logging out of Drive but that did not help. I attempted forcing the synch by making changed in both the cloud version and the Mac, but same result. Google Drive

I managed to sort it out in the end and here is what I did:

  1. Exit the Drive application
  2. Navigate to the Application Support folder and look for the Google folderYou may need to find the hidden Library folder
    • In Finder look for the Go menu and press Option + Cmd to reveal the hidden folder
    • Once there look for the “Application Support”
    • Alternatively you can press Cmd + Shift + G and go to “~/Library/Application Support/Google”
  3. Delete the Drive fokder
  4. Start the Drive application

Et voilà

Opening old Keynote/Pages files in new versions

Greetings readers! I hope you are all enjoying the break and getting ready for 2016.

This time I wanted to bring to your attention some information that you may find to be very useful. Particularly if, like me, you happen to have need some old slides, presentations or talks you have in Keynote but forgot (or rather did not need) to update to a newer version of the software. You may have thought that there would be some backward compatibility for this sort of thing, and you may be surprised that there is not an obvious click-and-update type solution. Nonetheless, not all is lost and you would not have to trash your presentations, unless of course they were not the slides you were looking for… This trick also works with Pages by the way.

You may find that when opening your old slide decks, Keynotes complains with:

This document can't be opened because it's too old. To open it, save it with Keynote '09 first.

Keynote Compatibility Issue

and Pages with:

This document can't be opened because it's too old. To open it, save it with Pages '09 first.

Of course, if you have both versions installed this should not be a problem, but why would you do that? So, if you cannot open the old file in the first place, here is what you need to do (please make sure that you have a backup copy of your file… you never know…):

  1. Open the Terminal and navigate to the directory where the old file is saved. So if your file is called
    my_presentation.keynote

    and it is saved in your Desktop just type 

    > cd Desktop
  2. Rename the file with a .zip extension:
    > mv my_presentation.keynote my_presentation.zip
  3. Unzip the file: 
    > unzip my_presentation.zip -d my_presentation
  4. Type the following command:
    gunzip --stdout index.apxl.gz | sed 's-:version="72007061400"-:version="92008102400"-g' > index.apxl

    and hit return. If you do not get any errors you are good to go.

  5. Remove the 
    index.apxl.gz
  6. Re-compress the folder and change the extension to the original one.

Try opening your file, it may still complain but at least you will be able to open it. Et voilà!

MacTex updates for El Capitan

El Capitan! Great! The new version of the OS X operating system. New features, new fonts, new problems… I knew that updating was going to bring some unexpected problems with my applications, but I wanted to update… And ditto, as soon as I tried to take a look under the hood for a couple of things I realised that a fresh installation of homebrewwas going to be needed.

More importantly, with my new book on data science (aka “Data Science and Analytics with Python”), LaTeX is probably one of the most used things in my computer. So, I wanted to check that things were fine and although I could compile (currently trying to finish Chapter 3 in case you are wondering) but there were some issues here and there, for example TeX Live thought I was using version 0 (yes zero!) and it could not find some files.

It turns out that El Capitan does not let us write to /usr and the 2015 TeX distribution creates symbolic links to /usr/texbin, is removed (if it was there from a previous OS version) and cannot be installed. If a GUI looks by default at that location it will sadly no longer find it. That is why the terminal was not affected! (Phew!)

The solution is to tell the broken applications to look at /Library/TeX/texbin, in /Library/TeX which is “owned” by MacTEX so is allowed by El Capitan. So to fix Tex Live do the following:

  •  Open TEX Live Utility  Preferences and click on the Choose. . .
  •  That opens a file chooser. Type Shift-Cmd-G , enter /Library/TeX  into the dialog box and then press Return .
  • Finally Double-Click  on texbin
  • Et voilà

 

For more info see this link.

El Capitan

No shuffle in new iOS 8.4 Music App

I was not too sure about the new Apple Music offering, but so far it seems quite alright! The music choices are generally good, and I hope that as I use the music app in iOS 8.4 more the choices get better.

Unfortunately I ended up using the app while not having mobile coverage and no WiFi either… so I reverted to “My Music” and since I was in the middle of a run, I wanted just to hit the shuffle button and hope for the best… However, I was surprised that there was no shuffle button to be seen… I ended up hitting the first song in the list and take it from there. It turns out that the shuffle option is set by default, you just have to seed it by starting playing any song. That seems good, except for the fact that it is not obvious at all.

You can select if you want the shuffle mode or not after starting playing any song and expanding the “Now Playing” bar:

iTunes Shuffle 1

 

And there you will be able to see the usual Shuffle icon:

iTunes Shuffle

 

Using curl to download a shortened URL – Dropbox, bit.ly

English: A download symbol.

I was in the middle of an introductory workshop for Data Science at General Assembly and I was talking about using command line instructions to facilitate the manipulation of files and folders. We covered some of the usual ones such as ls, mv, mkdir, cat, more, less, etc. I was then going to demonstrate how easy it was to download a file from the command line using curl and I had prepared a small file uploaded to Dropbox and shortened its URL with bit.ly.

“So far so good” – I thought – and then proceeded with the demonstration… Only to find out that the command I was using was indeed downloading a file, but it was the only downloading the wrapper html created by bit.ly for the re-directioning…  I should have known better than that! Of course all this happening while various pairs of gazing eyes were upon me… I tried again using a different flag and… nothing! and again… nothing… Pressure mounting, I decided to cut the embarrassment short and apologised. Got them to download the file in the less glamorous way by using the browser…

So, if you are ever in that predicament, here is the solution, use the -L flag with curl:

$ curl -L -o newname.ext http://your.shortened.url

The -L deals with the redirectioning of the shortened URL and make sure that you use the -o flag to assign a new name to your file.

E voilà!

Failed Battery

I have had this 17-in MacBook Pro for a few years… perhaps about 8 years? Probably a bit more? In any case, I have it more as a memento than anything else as I have a more modern one these days. I still keep it updated and all the rest of it so I was rather surprised to get it out and see that the battery has effectively bursted!!! I hope the rest of the machine still works though :(

Failed Battery

 

Apple Notes and Gmail Notes

dsc00455

I accidentally ended up creating some notes in the Gmail Notes inside my iDevice only to be completely confounded by the fact I could not see them in my desktop. I tried to find some resolution by looking at the instructions for the Apple notes, but got frustrated with the lack of information.

So, here it is how I solved my issue:

It seems that as an Apple Notes user, one can select to have the Notes saved “On my iPhone/iPad/Mac” or synced to any email account of one’s choice. If you chose the first option, then no issues there, but the “fun” part comes with the latter. In that case the application will send notes from the device via Gmail to the Gmail servers, or for that matter to the email account you designated under IMAP. This means that your notes are therefore treated as normal email and labelled as “Notes”. Not only that, they are automatically archived on arrival. The initial transfer is one-way only and this implies that the notes can’t be restored from Gmail to the device. In order to find your Notes in Gmail you have to search for the “Notes” label!

If you call up your note on your device, the application access it from Gmail and displays it. But if you deleted it, as many of us do, then the app gets confused as it does not know where they are… If they are deleted from the device removes the label in Gmail and thus they cannot be accessed by the device and they get zombiefied in Gmail! They will still be present in All Mail, but without label.

How to fix this… well it depends. If the Notes have been deleted from the Gmail account from the web interface they will still be there in the Trash for 30 days. You can “restore” then during that time and will be showing in the Notes App on the device.

If the Notes folder was deleted using the Mail App on the device, the notes will (probably) still be there under “All Mail” but without a label. You can search for them and re-apply the label!

My advice would be not to use the synching at all… it has caused more pains than it should be. Let me know if this helps.

Enable NTFS read and write in your Mac

CES 2013 - OWC Mac mini external storage - min...
CES 2013 – OWC Mac mini external storage – miniStack Max (Photo credit: the JoshMeister)

I was confronted with an old issue, that had not been an issue for a while: writing to an external hard drive that was formatted with Windows (NTFS) from my mac. I used to have NTFS-3G (together with MacFUSE) installed and that used to be fine. However, I guess something when a bit eerie with Mavericks as I was not able to get my old solution to work.

So, here is what I did (you will need superuser powers, so be prepared to type your password):

Open a Terminal (Terminal.app) and create a file called stab in the /etc folder. For instance you can type:

$ sudo nano /etc/fstab

You can now enter some information in your newly created file telling MacOS information about your device. If your external drive is called mydevice enter the following:

LABEL=mydevice none ntfs rw,auto,nobrowse

Use tabs between the fields listed above. Save your file and you are now ready to plug your device.

There is a small caveat: Once you do this, your hard drive is not going to appear in your Desktop. But do not disappear, you can still use the terminal to access the drives mounted by going to /Volumes folder as follows:

$ sudo ln -s /Volumes ~/Desktop/Volumes

et voilà!

Enhanced by Zemanta