Fast.ai v2 environment set-up on Google Cloud


#1

One of the AI Saturdays Singapore Mentors, How Khang, has kindly created a comprehensive step-by-step guide and script for setting up your fast.ai v2 environment on Google Cloud Platform. You can find the guide here: https://medium.com/@howkhang/ultimate-guide-to-setting-up-a-google-cloud-machine-for-fast-ai-version-2-f374208be43


#2

Can you please explain what the step 11 (that involves bash script) is doing exactly? … thanks!
curl https://raw.githubusercontent.com/howkhang/fastai-v2-setup/master/setup.sh | bash


#3

If by Step 11 you’re referring to

export PATH=~/anaconda3/bin:$PATH

It’s setting the Anaconda path in our BASH Profile.

So how BASH Works is, we have default variables and paths. If I want to run a command as simple as cd, cd would have a script that’d be defined already at some location which gets called from there, I don’t have to hardcode the system location every time.

Similarly, we don’t want to make our anaconda run every time only from it’s location.


#4

Oh was referring to the command curl https://raw.githubusercontent.com/howkhang/fastai-v2-setup/master/setup.sh2 | bash in How Khang guide
Just wondering what exactly that does…


#5

Also did install gcloud… but when i start the gcloud compute instance and try to run jupyter notebook with the static ip it says the page is not found…


#6

Curl downloads the file
The Pipe concatenates it with another command, bash after the pipe prompts the terminal that run this as a BASH Script.


#7

Can you share the details?


#8

I am at step 10 method 2 (Command line using Google Cloud SDK) of How Khang’s setup guide and encounter issue.
A PuTTY window pops up, saying: <Using username “admin”. Server refused our key>
Another PuTTY Fatal Error window also pops up saying: <Disconnected: No supported authentication methods available (server sent: publickey)>
Could someone advise if I am missing some steps?


#9

It’s likely that your file Windows paths/folder paths for PuTTY to see your public keys to access your GCP instance are either incorrect, missing, or the files themselves are missing. Check these settings in PuTTY. There are in: Connection -> SSH -> Auth. If that doesn’t resolve the issue, you might want to take a look at this Stack Exchange question. You are more likely to find the answer there.


#10

Melvin helped at the event yesterday and its working now. Thanks.

However, I am now trying to get some files copied from my local folder to the vm instance. How do i do that?


#11

If you want to upload small files, use GitHub. Upload them to GitHub and then download them to the instance.

Otherwise you’d use SCP


#12

Yeah, SCP gets the job done. Recently, I used magic wormhole: https://github.com/warner/magic-wormhole


#13

It might be a permissions issue with the .jupyter folder. Some people have come across that. Try:
sudo chown -R username:username .jupyter
in your VM instance on GCP. Where username is your Google username (basically your email minus the @gmail.com).

So if my username is jamsawamsa, it’ll be:
sudo chown -R jamsawamsa:jamsawamsa .jupyter


#14

Ah sorry didn’t see the you’ve already resolved it. Mind sharing how you fixed the issue?


#15

Hi, being new to virtual environments and Google Cloud and all, i would just like to enquire if we are charged for usage of GCloud if we don’t actually use it. Have not been using GCloud but apparently i have this list of transactions. Is this the norm? @jameslee


#16

Thanks Cedric for the information, I manage to connect to the VM instance by just running the command: “gcloud compute ssh <my_user_name>@<my_instance_name>” .
Before, I could not connect to the VM instance because I did “gcloud compute ssh <my_instance_name>”, without adding my user name.


#17

@JarylLim Yeah, think of it as renting a computer that google owns. So you have to pay sorta like a monthly rent and additional charges.

But please do remember to switch off your instance (which you did not). Each hour your instance is running will cost you money and you don’t want to be burning all your credits simply because you left it on.


#18

Did people have problem completing Step 11, using the below command provided in the tutorial, https://medium.com/@howkhang/ultimate-guide-to-setting-up-a-google-cloud-machine-for-fast-ai-version-2-f374208be43

I ran the below command as instructed on my terminal.

curl https://raw.githubusercontent.com/howkhang/fastai-v2-setup/master/setup.sh | bash

I received a long list of text displayed on the terminal but eventually the below text were displayed, where I suspect may have been the error:

All packages are up to date.
–2018-01-08 12:11:03-- http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/cuda-repo-ubuntu1604_9.0.176-1_amd64.deb
Resolving developer.download.nvidia.com (developer.download.nvidia.com)… 192.229.232.112, 2606:2800:247:2063:46e:21d:825:102e
Connecting to developer.download.nvidia.com (developer.download.nvidia.com)|192.229.232.112|:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 2838 (2.8K) [application/x-deb]
Saving to: ‘cuda-repo-ubuntu1604_9.0.176-1_amd64.deb’

**cuda-repo-ubuntu1604_9.0.176-1_amd64.d 100%[============================================================================>] 2.77K --.-KB/s in 0s **

2018-01-08 12:11:03 (645 MB/s) - ‘cuda-repo-ubuntu1604_9.0.176-1_amd64.deb’ saved [2838/2838]

Selecting previously unselected package cuda-repo-ubuntu1604.
(Reading database … 47451 files and directories currently installed.)
Preparing to unpack cuda-repo-ubuntu1604_9.0.176-1_amd64.deb …
Unpacking cuda-repo-ubuntu1604 (9.0.176-1) …
Setting up cuda-repo-ubuntu1604 (9.0.176-1) …

The public CUDA GPG key does not appear to be installed.
To install the key, run this command:
sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1604/x86_64/7fa2af80.pub

Should I proceed with the sudo command? Thanks


#19

I continue working to sort out the issues with why step 11 did not work. I realize afterwards, that I can try running line by line. I noticed that many programs were not installed. For example, ufw. Therefore, the commands were not able to complete. However, there was one command where I am stuck now. Below is the error messages that I received. What else can I try?

(fastai) [email protected]:~/fastai/courses/dl1$ sudo add-apt-repository ppa:graphics-drivers/ppa -y
gpg: keybox ‘/tmp/tmpg0ozcq20/pubring.gpg’ created
gpg: /tmp/tmpg0ozcq20/trustdb.gpg: trustdb created
gpg: key FCAE110B1118213C: public key “Launchpad PPA for Graphics Drivers Team” imported
gpg: Total number processed: 1
gpg: imported: 1
gpg: no valid OpenPGP data found.
Exception in thread Thread-1:
Traceback (most recent call last):
File “/usr/lib/python3.5/threading.py”, line 914, in _bootstrap_inner
self.run()
File “/usr/lib/python3.5/threading.py”, line 862, in run
self._target(*self._args, **self._kwargs)
File “/usr/lib/python3/dist-packages/softwareproperties/SoftwareProperties.py”, line 688, in addkey_func
func(**kwargs)
File “/usr/lib/python3/dist-packages/softwareproperties/ppa.py”, line 386, in add_key
return apsk.add_ppa_signing_key()
File “/usr/lib/python3/dist-packages/softwareproperties/ppa.py”, line 273, in add_ppa_signing_key
cleanup(tmp_keyring_dir)
File “/usr/lib/python3/dist-packages/softwareproperties/ppa.py”, line 234, in cleanup
shutil.rmtree(tmp_keyring_dir)
File “/usr/lib/python3.5/shutil.py”, line 480, in rmtree
_rmtree_safe_fd(fd, path, onerror)
File “/usr/lib/python3.5/shutil.py”, line 438, in _rmtree_safe_fd
onerror(os.unlink, fullname, sys.exc_info())
File “/usr/lib/python3.5/shutil.py”, line 436, in _rmtree_safe_fd
os.unlink(name, dir_fd=topfd)
FileNotFoundError: [Errno 2] No such file or directory: ‘S.gpg-agent.extra’


#20

Also, not sure how we ended up with /usr/local/cuda/include directory. We never created this directory in the sequence of commands that preceded this line.

sudo cp cuda/include/. /usr/local/cuda/include/